I suspect the J package Jd is probably the most non-trivial public codebase. I don’t love the coding style (functions are long and scripted) and it doesn’t make use of newer lambda functions (“direct definitions”) which are easier to read. https://github.com/jsoftware/data_jd
Possibly so. We do simple IO primarily from parquet or csv data, and I am working on the Apache Arrow/Flight package currently. Our data sets typically fit in RAM but either if you have enough RAM or a file amenable to memory mapping, you shouldn’t have a problem. J has a memory mapping utility. Numerical data is straightforward. We do ETL, QC, and normalization in J. Avoid transposes where you can, but reshapes are trivial and done with metadata, so fast. Overall what you describe sounds like a fun project to try in J.
I cannot share specific projects. J is particularly good for numeric/algorithmic experimentation. It is interactive/REPL-based. Types and boilerplate are minimal. There is no compilation and minimal packages. This is the language I reach for when step 2 is not going to be “install these packages” or deploy to share code. It is most remarkable for data hacking, as it is trivial to manipulate structures and maintain performance. I use python, R, and clojure frequently, but the ability to move quickly in J is without parallel. Weaknesses include namespacing and deployment, although I have seen deployment of substantial codebases both on desktops and servers. Multi-threading and AX512 instructions in _your_ (not package) code, from the REPL are some of what you get with j904.
I did some work to compare Jd to data.tables and found that it was more performant in some instances such as on derived columns, and approximately equally performant on aggregations and queries. Jd is currently single-threaded, whereas multiple threads are important on some types of queries. I tried to further compare with Julia DB at the same time (maybe a year ago) and found that was incorrectly benchmarked by the authors and far slower than both; that might be different now. Jd is more equivalent to data.tables on disk; Clickhouse is far better at being a large-scale database.
Rules of thumb on memory usage:
Python/Pandas (not memory-mapped): "In Pandas, the rule of thumb is needing 5x-10x the memory for the size of your data.”
R (not memory-mapped): "A rough rule of thumb is that your RAM should be three times the size of your data set.”
Jd: "In general, performance will be good if available ram is more than 2 times the space required by the cols typically used in a query.”
Re CSV reading, Jd has a fast CSV reader whereas J itself does not. I have written an Arrow integration to enable J to get to that fast CSV reader and read Parquet.
The claim here is +80% reduction in time, +5x throughput relative the current standard.
Nice to see illumination of the benefit of high-level algorithmic design associated with the first-principals aspect of APL.
This is also an interesting claim presently, given there's an active conversation in the Jsoftware community currently on how to handle tree structures.
A bunch of questions: Is multi-threading happening at the level of user-defined functions? How are threads scheduled? What's the underlying method or library for enabling multi-threading (Cilk, openMP, some LWT library...)? To what extent has this level of granularity been tested against other levels of nested parallelism (e.g. SIMD or otherwise parallel operators)? Have you tested performance by OS, and if so, have you noted any necessary OS-level modifications related to thread management? Is this part of a broader roadmap for accelerator integration?
Without general theories, the only reasonable question is "Faster for What"?
What are generalizable theories of 'fast'? Fewer bits moving less with fewer instructions, and using the most suitable, specialized hardware available.
Even once you have a reasonably well-defined goal, if it's complex enough you may start hitting subsidiary questions, like numerical precision and stability.
Can you comment on experience (or contact me) regarding implementation efficiency? We have recently implemented task-based parallelism in the J language with openMP[0]. Improvements or critiques are appreciated. SIMD instructions there have been coded directly rather than via pragmas.
XLA is domain-specific compiler for linear algebra. Triton generates and compiles an intermediate representation for tiled computation. This IR allows more general functions and also claims higher performance.
While MATLAB and Numpy were influenced by the APL family of languages, they contain a small subset of the design available in the J/K/APL world. Informally most users of array languages would not consider them close family members, though perhaps they're cousins.
{{(-&2 +&$: -&1) ^: (1&<) M. y}} 155x
It'll run in your browser in 0.003 seconds.
https://jsoftware.github.io/j-playground/bin/html2/#code=%7B...