sarosh

But why does, as you explain "training goes brrr"?

Francis Bach, the author, makes a good faith effort to explain exactly why this material is beneficial (see https://francisbach.com/my-book-is-out/):

"Why yet another book on learning theory? ...the main reason is that I felt that the current trend in the mathematical analysis of machine learning was leading to overly complicated arguments and results that are often not relevant to practitioners. Therefore, my aim was to propose the simplest formulations that can be derived from first principles, trying to remain rigorous without overwhelming readers with more powerful results that require too much mathematical sophistication."

From my own reading and experience on the mathematical analysis approach of this "training goes brrr" approach, I thought the material in Chapter 12, Overparameterized Models, was interesting and coherent with 12.2.4 Linear Regression with Gaussian Projections being an especially elegant explanation. It would be interesting to hear if you had read/skimmed/purused this section and found it wanting etc.

sarosh

last yeardiscuss

This is the PDF of the following 2011 book focused on FFTs and fast arithmetic for both real numbers and finite fields. https://www.amazon.com/Matters-Computational-Ideas-Algorithm... The author is Jörg Arndt: born 1964 in Berlin, Germany. Study of theoretical physics at the University of Bayreuth, and the Technical University of Berlin, Diploma in 1995. PhD in Mathematics, supervised by Richard Brent, at the Australian National University, Canberra, in 2010.

sarosh

last yeardiscuss

Interesting that the underlying model, a LoRA fine-tune of Qwen2.5-Coder-32B, relies on synthetic data from Claude[1]:

  But we had a classic chicken-and-egg problem—we needed data to train the model, but we didn't have any real examples yet. So we started by having Claude generate about 50 synthetic examples that we added to our dataset. We then used that initial fine-tune to ship an early version of Zeta behind a feature flag and started collecting examples from our own team's usage.

  ...

  This approach let us quickly build up a solid dataset of around 400 high-quality examples, which improved the model a lot!

I checked the training set, but couldn't quickly identify which were 'Claude' produced[2]. Would be interesting to see them distinguished out.

[1]: https://zed.dev/blog/edit-prediction [2]: https://huggingface.co/datasets/zed-industries/zeta

sarosh

2 years agodiscuss

Defer to other experts, but (briefly) normalizing flows are a method for constructing complex distributions by transforming a probability density through a series of invertible transformations. Normalizing flows are trained using a plain log-likelihood function, and they are capable of exact density evaluation and efficient sampling. See:

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015. Link: https://bigdata.duke.edu/wp-content/uploads/2022/08/1505.057...

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In ICLR Workshop, 2015. Link: https://arxiv.org/pdf/1410.8516

And for your direct question, the following paper "Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods" appears upon a superficial glance to be relevant. Link: https://arxiv.org/pdf/2107.08001

sarosh

3 years agodiscuss

This actually seems like just a very clever market segmentation solution since the GPU was already limited to 8x PCIe lanes (its a laptop GPU see https://www.notebookcheck.net/NVIDIA-GeForce-RTX-4060-Laptop...). The 'addition' of the M.2 SSD makes it a unique offering. Limiting it to only one drive is another way to keep the thermal envelope down. Kudos to the Asus design and product development folks.

sarosh

3 years agodiscuss

The paper, Trigger-Level Event Reconstruction for Neutrino Telescopes Using Sparse Submanifold Convolutional Neural Networks, is here https://arxiv.org/abs/2303.08812.

The crucial advantage of SSCNN are that they "replaces traditional convolutions with sparse submanifold convolutions" which operate "only on the non-zero elements" and creates efficiencies.

sarosh

3 years agodiscuss

Looking at https://research.wmz.ninja/projects/phd/rulesets/default/eve... provides most of the key 'game loops', i.e.

# idea -> prelim -> major -> 2 figures -> submitted paper

interesting to see the hypothesis about reading more papers being borne out:

# increase the success rate as the player reads more papers probability: 0.60 + player.readPapers / 100 - itemCount('idea') / 20

Also interesting to see that passing the qualification exam provides the largest player.hope boost (+10)

Was fun to see the TooManyIdeas random event - now to actually get it to trigger.

sarosh

3 years agodiscuss

A (now 6 year old) discussion that might be helpful: Anders Hejlsberg on Modern Compiler Construction: https://www.youtube.com/watch?v=wSdV1M7n4gQ

Also https://github.com/salsa-rs/salsa

sarosh

5 years agodiscuss

An interesting take from Mike Pondsmith given his heavy involvement in the venture at the end of (at least in my opinion) well-written article: "[C]omparing the tabletop experience with its video-game incarnation, he noted that the latter doesn’t really compare to the former when it comes to self-expression. “You could be you in a tabletop game and bring all the stuff that you wanted to bring into it,” he said. “A tabletop game is limitless. A video game, by its very nature of how it’s designed, has some limits.” "

Submissions

Ask HN: Best Board Games of 2023?

comments