No worries, it doesn't come off that aggressive ;)
Yes, I've definitely been involved in other projects where a single core developer can do a little too much "imposing" of their will. i.e. if they happen to have a personal preference one way or another on things, they not accept or even entertain certain changes.
I wouldn't expect most large, mature projects to be this way (otherwise they probably wouldn't have grown as large or attracted enough of a community!), but it's certainly a problem with some projects.
Unfortunately almost exclusively via http rest apis. It's not great, but it's the lowest common denominator between the vast "ingestion" service we've built (connectors to web apis, local application for local file upload, raw api endpoints, etc.).
We've started exploring the apache arrow format as a compressible binary format with a dedicated wire format just to cut down on parsing processing costs.
The biggest difference in these benchmarks comes down to how multiple threads are leveraged; (disclaimer: primary CSV.jl author here).
In CSV.jl, it was relatively straightforward to add multithreaded parsing support; we chunk up the file, find row starts, then spawn a threaded task to process each chunk. In data.table (fread), they have a constraint of R having a global String intern store; so it fundamentally restricts the multithreading capabilities by requiring strings to be interned sequentially. In pandas, there are similar nontrivialities jumping between the C++ source and python object world that adds a lot of complexity (and probably explains why no one has made the effort to do multithreaded parsing). It's interesting to note, however, that the apache arrow project (pyarrow package in Python), has a multithreaded csv parser integrated with the project. It provides similar performance to CSV.jl because it was built from the ground up to process chunks on multiple threads into "arrow tables". Unfortunately, it's tied very specifically to the arrow project, so pandas doesn't benefit from its work! It's one of my favorite features about Julia that CSV.jl automatically integrates with other "data table formats"; i.e. ODBC.jl, SQLite.jl, MySQL.jl, Arrow.jl, DataFrames.jl, etc. They can all leverage CSV.jl as "their csv parser" because it's seamlessly integrated via well-established "table" apis.
I agree; if I needed to parse CSVs in python and could utilize the arrow format, I would definitely use pyarrow.
I actually recently finished support for reading/writing the arrow format in Julia (https://github.com/JuliaData/Arrow.jl), and it's automatically integrated with the CSV.jl package; so you can do `Arrow.write("data.arrow", CSV.File("data.csv"))` and convert a csv file to arrow format directly. I'm very bullish on arrow as a standard binary data format for the future.
Take a deep look at Julia! Adding multithreaded parsing capabilities to CSV.jl was really a joy; basically just chunking up the file and spawning threaded tasks to process each chunk with the existing parsing code.
My favorite favorite thing about developing in Julia is the ability to write "high-level" type code and usually get decent performance, BUT THEN have the ability to fine tune for that extra boost of performance. The code inspection tools (see lowered, typed, LLVM, or native IR code levels) and super nice Task-based multithreading really make performance tuning a delight.
One advantage I've noticed at least (having been a contributor to the Julia language itself), is that it at least makes contributing to the language very approachable. Granted you see lower quality proposals from time to time, but in general, I even question whether I myself would have gotten involved or been brave enough to propose language features if I would have had to do such a rigorous, formal PEP writeup. In some respects, you're restricting the pool of people who actually contribute ideas to the language.
I also have appreciated the "meritocratic" approach of Julia language features; if someone can show that an idea/approach/algorithm is fundamentally faster, more flexible, etc. it's generally been accepted, regardless if the proposer is a first-time contributor or not. That also makes contributing feel approachable and very fair, as opposed to just trying to please a single core developer or playing some politics game.
No worries. The CSV.jl package has just been around for 5 years now, has some 700 issues opened/resolved, has built up a pretty extensive test suite, and is used in production by a number of companies, so I just wanted to hopefully clarify its level of maturity :D
I can definitely understand getting burnt in Julia early days (I was around back then as well), but since 1.0, the package ecosystem has matured quite a bit IMO; most of the really popular packages are very stable and work similarly to popular packages in other languages.
It can make huge amounts of difference in a production system; where I work, we process terabytes of csv data every day; saving minutes per file can add up to enormous differences in CPU cost/time for a production system running 24/7.
I agree that for a data scientist doing exploratory analysis locally on their computer, it doesn't make nearly as much a difference (also because they're usually not working on crazy large files).
The performance work in the CSV.jl package (that the article is about) was very much geared towards these kinds of production scenarios.
As the primary author of CSV.jl, I can help clarify on the posted issues:
* #720: ended up not being an issue at all, but a misconfigured environment
* #714: there was indeed a corner case when automatically detecting float values where the float started with '-' sign and only had a trailing decimal (e.g. '-123.'). Not super common, but indeed a bug
* #749, #734 are related to a new "beta" feature (CSV.Chunks) which allows iterating over large CSV files in batches. I've been trying to track down the issues 2 people have reported, but haven't been able to reproduce on the same large file. Once we iron out some of those issues, we'll mark the feature as "out of beta".
I agree that Julia packages in general are still evolving and you might run into issues, but at the same time, I strongly believe it's reached a similar maturity in most ways with other language package ecosystems. For example, I use a lot of Javascript/Python frameworks/packages and at least for me, I tend to run into corner case bugs/issues as often as I do for the most common/mature Julia packages.
Compared with, say, data.table in R, or pandas in python, one of the things I enjoy most about Julia packages is that they're almost exclusively written in pure Julia. Having had to dive into data.table/pandas source C/C++ + language binding glue code is a huge pain when trying to track down bugs, so I feel like my knowledge of Julia "goes further" in that if I run into a package bug, it's relatively much easier to track down what's going on and even submit a pull request to fix!
I made https://github.com/JuliaData/Strapping.jl, which is part of an ORM; i.e. it does the translating between Julia objects/vectors of objects and 2D tables. But it doesn't do some of the more "magical" things like SQL generation, automatic migrations, etc.
I think it'd be cool to do some of the SQL generation stuff, but no one has sat down to design something out and how it would work in Julia.
Yes, I've definitely been involved in other projects where a single core developer can do a little too much "imposing" of their will. i.e. if they happen to have a personal preference one way or another on things, they not accept or even entertain certain changes.
I wouldn't expect most large, mature projects to be this way (otherwise they probably wouldn't have grown as large or attracted enough of a community!), but it's certainly a problem with some projects.