The site says privacy-first and also says "we cannot lose your data if we never collect it" but it makes a WHOLE lot of POST calls passing what appear to be encrypted payloads, and refuses to work offline-- so the user has no way to verify that the limited info you claim to be collecting is in fact what is being collected. Worse, if you simply visit and use the site, you never once see any mention of terms of use, and yet those published terms-- which you will only find if you actively scroll way down to the bottom of the SPA and click on a tiny link-- and claim to be binding merely by the use of the site, which could have easily happened without the user having any knowledge or notice whatsoever that they "agreed" to something (in other words, without actually agreeing to anything). The terms also do not say anything about your data collection, though if one looks hard enough one can find it mentioned in the privacy policy, well below the contradictory opening line that says "we cannot lose your data if we never collect it". Sorry, but meta data is still data, so "we never collect [your data]" is simply false.
So, maybe you did not intend it to be so, but to me the site comes off as being very sketchy and untrustworthy.
What is the advantage of this over the parser used by xsv? From the documentation, the only difference I can see is that xsv handles weird CSV better than this crate-- which in some situations is very important! So presumably this one must be faster? If so, how much faster? Or is there some other advantage to this?
Looks interesting and I gave it a whirl-- thank you. Your intro mentions filter + sort, but I couldn't find a way to do that in the web UI (maybe that's just my ineptitude).
Re your question whether it would be useful: hard to answer because I cannot tell right now whether it solves (or intends to solve) any specific problem better than plenty of other alternatives.
zsv was built because I needed a library to integrate with my application, and other CSV parsers had one or more of a variety of limitations (couldn't handle "real-world" CSV or malformed UTF8, were too slow, degraded when used on very large files, couldn't compile to web assembly, could not handle multi-row headers (seems like basically none of the other CSV parsers do this) etc-- more details are in the repo README). The closest solution to what I wanted was xsv, but was not designed as an API and I still needed a lot of flexibility that wasn't already built into it.
My first inclination was to use flex/bison but that approach yielded surprisingly slow performance; SIMD had just been shown to be useful in unprecedented performance gains for JSON parsing, so a friend and I took a page from that approach to create what afaik (though I could be wrong) is now the fastest CSV parser (and most customizable as well) that properly handles "real-world" CSV.
When I say "real-world CSV": if you've worked with CSV in the wild, you probably know what I mean, but feel free to check out the README for a more technical explanation.
With parser built, I found that some of the use cases I needed it for were generic, so I wrapped them up in a CLI. Most of the CLI commands are run-of-the-mill stuff: echo, select, count, sql, pretty, 2tsv, stack. Some of the commands are harder to find in other utilities: compare (cell-level comparison with customizable numerical tolerance-- useful when, for example, comparing CSV vs data from a deconstructed XLSX, where the latter may look the same but technically differ by < 0.000001), serialize/flatten, 2json (multiple different JSON schema output choices). A few are not directly CSV-related, but dovetail with others, such as 2db, which converts 2json output to sqlite3 with indexing options, allowing you to run e.g. `zsv 2json my.csv --unique-index mycolumn | zsv 2db -t mytable -o my.db`.
I've been using zsv for years now in commercial software running bare metal and also in the browser (see e.g. https://liquidaty.github.io/zsv/), so I finally got around to tagging v1.0.1 as the first production-ready release.
I'd love for you to try it out and would welcome any feedback, bug reports, or questions.
OK... this isn't useful to me because I now just only use mermaid and stopped using other diagraming tools, because mermaid can be embedded now in so many places (github, in-browser editing/rendering, shareable URL, python etc), can be easily saved as text and edited (whether manually or programmatically), there are no IP issues, etc. So more useful for me would be, whatever you need to use something other than mermaid for-- do what it takes to make mermaid fill that need, so that there is no need to use anything else.
Re the original analysis, my own opinion is that the outcome is only surprising when the critical detail, highlighting how the two are different, is omitted. It seems very unsurprising if it is rephrased to include that detail: "DuckDB, executed multi-threaded + parallelized, is 2.5x faster than wc, single-threaded, even though in doing so, DuckDB used 9.3x more CPU".
In fact, to me, the only thing that seems surprising about that is how poorly DuckDB does compared to WC-- 9x more CPU for only 2.5x more improvement.
But an interesting analysis regardless of the takeaways-- thank you
That is great, thank you. I'd love to continue the conversation-- maybe easier in a separate forum. Can I follow-up via the email address on your profile (gaven...)?
Glad to be helpful-- I'm in the business of data process automation, so I appreciate the opportunity to learn about new use cases. If you are willing to share what your end goal was in more detail (even as simple as an SQL query that you would now run want to run against your current schema), I'd be interested to see how an optimal process could be designed to easily generate that, and possibly suggest some tooling you could find useful. You may also want to try posting questions like this in forums such as the Seattle Data Guy's discord channel, and I'm suspect you will get lots of suggestions and advice.
This is misleading. First, as other comments have noted, it is comparing multi-threaded/parallelized vs single-threaded, and its total CPU time is much longer than wc's. Second, it suggests there is something special going on, when there is not. Just breaking the file into parts and running wc -l on it-- or even, running a CSV parser that is much more versatile than DuckDB's-- I'm pretty confident will perform significantly faster than this showing. Bets anyone?
I am always a proponent of starting with the end goal and then working backward. What are the end results you are aiming to achieve (or aiming to allow your audience to achieve)? Is marginal precision more important than the speed impact? The optimal database design will depend on that (i.e., on what you are optimizing for...).
It would also be very helpful, imho, to indicate keys and indexes, perhaps by modifying your schema diagram, or simply (and maybe better), just dump the actual SQL schema definition (i.e. the output from sqlite3's ".schema" command)
Haven't yet seen any of these beat https://github.com/liquidaty/zsv (of which I'm an author) when real-world constraints are applied (e.g. we no longer assume that line ends are always \n, or that there are no dbl-quote chars, embedded commas/newlines/dbl-quotes). And maybe under the artificial conditions as well.
If you think it is not good for producers and consumers, you are not disagreeing with me as much as with economic theory. Personally, I think there are many distortions in practice that can lead to exceptions, but that in the bigger picture of things, it nonetheless holds water.
---
Wouldn’t Consumers Be Better Off If Everybody Paid the Same Price?
---
In many cases, no. Different customer segments have different characteristics and different price points that they are willing to pay. If everything were priced at say the "average cost," people with lower price points could never afford it. Likewise, those with higher price points could hoard it. This is what is known as market segmentation. Economists have also identified market mechanisms whereby fixing static prices can lead to market inefficiencies from both the supply and demand sides.
Re car manufacturers using remote "unlock" pricing, I will admit to being horrified when I first heard about it.
Unfortunately for my emotions, there is a strong argument to support this new pricing approach-- which might be BS in specific cases but which, from a macro perspective, may nonetheless have merit-- in that fundamentally, all it is, is a mechanism for further granularity of the age-old thing called price discrimination, which is generally viewed as good for both producers and consumers (not that there cannot be exceptions to either that view, or the circumstances in which that view is valid...)
So, maybe you did not intend it to be so, but to me the site comes off as being very sketchy and untrustworthy.