batterseapower

letztes Jahrdiscuss

Reinventing the programming paradigm has been Tim Sweeney's hobby horse since before 2006: https://groups.csail.mit.edu/cag/crg/papers/sweeney06games.p...

After Epic's wild success hes now got a bit of cash to splash on making his dream a reality :-)

batterseapower

vor 2 Jahrendiscuss

I tried it in OpenAI's O1. If I give it minimaxir's original prompt it writes the obvious loop, even if I include the postamble "Look for tricks that will make this function run as fast as possible in the common case".

However, if I then simply ask "What is the most probable result for this function to return?" it figures out the answer and a very good approximation of the probability (4.5e-5). From there it's easily able to rewrite the program to use the trick. So the creative step of spotting that this line of reasoning might be profitable seems missing for now, but 2025's models might solve this :-)

batterseapower

vor 2 Jahrendiscuss

Annual ridership is about 7m passengers (https://www.caltrain.com/media/34265) or 9m if we annualize the numbers in this article. At the same time, the Caltrain is losing more than $33m/year (https://www.caltrain.com/blog/2023/04/financial-future-caltr...) even though it recieved a huge $410m subsidy from the state government for the capital investment required to complete the electrification. So it's losing more than $5 per ride even ignoring capital costs. In fact, the farebox recovery ratio is only 25%, which is truly horrible even by public transport standards: https://www.caltrain.com/media/33996/download

The curmudgeon in my questions where it really make sense for the Caltrain to exist at all, given that it:

  1. It is extremely far from covering its own costs
  2. The subsidy provided by the state is regressive because it principally benefits the wealthy commuters along the line
  3. It imposes substantial negative externialities on the South Bay both in terms of noise & traffic delays at grade crossings
  4. The line and stations occupy very valuable real estate that could be sold and put to better uses

batterseapower

vor 2 Jahrendiscuss

I don't think that's naive - it could definitely help ameliorate the issues. To my knowledge noone has tried this approach.

The other idea I had when I was working on this stuff was to do JIT supercompliation. Clasically, supercompilers expand the whole graph at compile time, but it would be straightforward to delay the supercompilation process until the first time that the code is actually executed, which helps tame the problem of generating lots of specializations that may never actually be executed in practice.

(Choosing a generalization heuristic becomes more important in the JIT setting because you always have access to the full info about all parameters to the function you are specializing, and naively you would end up e.g. specializing sigmoid(x)=1/(1+exp(x)) for each value of x observed at runtime.)

batterseapower

vor 2 Jahrendiscuss

Supercompilation doesn't depend on having a typed language - you can more or less derive a supercompiler mechanically from any operational semantics, typed or untyped.

batterseapower

vor 2 Jahrendiscuss

My PhD was on supercompilation for Haskell (thanks for the cite in the repo :-))

Cool to see that people are still working on it, but I think that the main barrier to practical use of these techniques still remains unsolved. The problem is that it's just so easy for the supercomplier to go into some crazy exponential blowup of function unfolding, making the compilation step take impractically long.

Even if you avoid a literal exponential blowup you can easily end up generating tons of specializations that bloat your code cache but don't reveal any useful optimization opportunities/are infrequently used in practice. Similar performance problems also plague related techniques like trace-based JITs, even though the trace JIT happens at runtime and thus has access to strictly more information about the frequency with which a trace might be used.

You can try to use annotations like the @extract proposed in the article to control these problems, but it can be hard to predict in advance when this is going to occur.

One interesting research direction might be to use deep reinforcement learning to try to guide the generalization/termination decisions, where the reward is based on A) whether the unfolding leads to a tie-back later on, and B) to what extent the unfolding allowed deforestation/beta reduction to take place.

batterseapower

vor 2 Jahrendiscuss

The other recent improvement suggested for LoRA is DoRA: https://magazine.sebastianraschka.com/p/lora-and-dora-from-s.... It really does seem to strongly outperform LoRA - see also https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.htm...

comments