Yes, it's true that someone can put together a simple language like in a university course. The difficulties, as mentioned at the bottom of the post, are things like metaprogramming features or optimizing compilers.
The tail ends of a language implementation (parsing and code generation) are a fixed cost; the "middle end" can grow unbounded as more production-quality items are added.
std::execution is very interesting, but will be difficult to get started with, as cautioned by Sutter. This HPC Wire article demonstrates how to use standard C++ to benefit from asynchronously parallel computation on both CUDA and MPI:
Overlapping communication and computation has been a common technique for decades in high-performance computing to "hide latency", which leads to better scaling. Now standard C++ can be used to express parallel algorithms without tying to a specific scheduler.
The government is free to cancel its own contracts. The judge is saying that the government cannot prohibit other entities (like private contractors) from doing business with Anthropic.
The project relies on Rayon [1] for scheduling parallel tasks and Cranelift [2] to JIT the hot loops.
There are plenty of other interesting features like auto-FFI, bytecode caching (similar to Python's .pyc files), and "daemonize" mode (similar to mod_perl or FastCGI).
The one piece of advice I give new PhD students is to maintain a list of your references for a bibliography ahead of time. For every paper you read, copy the citation in BibTeX format and write a couple of sentences to remind yourself what the paper was about. Do this for every source, even if it doesn't seem important at the time.
> I think it would be the best to start interpreting the query and start compilation in another thread
This technique is known as a "tiered JIT". It's how production virtual machines operate for high-level languages like JavaScript.
There can be many tiers, like an interpreter, baseline compiler, optimizing compiler, etc. The runtime switches into the faster tier once it becomes ready.
I couldn't figure-out how Arthur's bin matched on symbol though, so I switched to a linear scan on the right table to record the last-seen index for each "by" element. While it worked, my hash table was messy because I relied on Python to handle a whole tuple as a key, which had some issues during initial testing.
The asof join I wrote for Empirical properly categorizes the keys before they are matched. That approach worked far better.
I first encountered q/kdb+ at a quant job in 2007. I learned so much from the array semantics about how to concisely represent time-series logic that I can't imagine ever using a scalar language for research.
Fun fact: the aj (asof join) function was my inspiration for pandas.merge_asof. I added the extra parameters (direction, tolerance, allow_exact_matches) because of the limitations I kept hitting in kdb.
The article points out that tools like TLA+ can prove that a system is correct, but can't demonstrate that a system is performant. The author asks for ways to assess latency et al., which is currently handled by simulation. While this has worked for one-off cases, OP requests more generalized tooling.
It's like the quote attributed to Don Knuth: "Beware of bugs in the above code; I have only proved it correct, not tried it."
I had been thinking about this idea for a long time, but I doubt I'll be able to get around to it. After speaking with a friend this evening, I decided to just jot it down for anyone interested.
Basically, use copy-and-patch compilation in a vector language to fuse loops and avoid temporaries. It can be employed for a baseline compiler that will use less memory than an interpreter and will have a much lower startup cost than an optimizing compiler.
The business model I've eyed is to license to service providers, like securities brokers and data vendors in the finance industry. That way, their customers can execute queries in the cloud.
For example, a crypto broker wants their traders to write their own models. A stock-exchange advisor wants their asset managers to compute their own transaction costs. Etc.
Basically, Empirical will be a value-add component for existing customers of these larger providers. Instead of shuffling market data over the Internet, the user will "send the query to the data."
Every firm I've shown Empirical to has had the same word-for-word response that most end users have had: "Does it handle streaming?" I put the sales calls on hold while I try to get this done.
I was pretty bad at explaining Empirical in the beginning. During alpha testing, I emphasized the technical merits (statically typed Dataframes!) rather than an application domain. Once I figured-out I needed to push time-series analysis, then the user feedback became more focused.
After starting the public beta, I heard the following sentence several dozen times: "This looks amazing! Does it handle streaming?" And thus began the quest I am currently on.
About the only advice I can give is to keep practicing and keep an open mind. The feedback will get less negative the more you hone your pitch.
Empirical is statically typed. I can run a backtest overnight knowing it won't crash because of a type error or misspelled column name.
I'm currently in the (long) process of adding streaming computation. Once that's ready, Empirical will be able to handle everything from gargantuan CSV files to real-time data using the same language primitives.
I talked about some of these goals when I launched on HN last year:
I did Startup School in 2018 and was one of the inaugural grant winners.
The program was extremely helpful for getting me through to the launch. My product is a moonshot sold to enterprises, so it took an enormous amount of work upfront to get something into users' hands.
The accountability of a cohort was critical for the lonely days of coding something that is both ambiguous and ambitious. Also, constantly being forced to explain my product really helped me hone my message and sales pitch.
I am still friends with some of the people I met back then. I seriously recommend this program to anyone who is starting out.
Right, it's same reason we still have ARM despite the success of x86. Or why some hardware vendors bundle FPGAs rather than rely on GPUs or custom ASICs. The instruction set is absolutely vital for a given domain's performance.
The tail ends of a language implementation (parsing and code generation) are a fixed cost; the "middle end" can grow unbounded as more production-quality items are added.
My language: https://www.empirical-soft.com