ainch

22 saat öncediscuss

Yann is a big SSL guy but I don't think he was involved in the original DINO - he's not listed as a co-author or anything.

ainch

evvelsi gündiscuss

The most basic machine learning-related query gets flagged for me. For example:

  In flax nnx, what's the idiomatic way to store state on a Module. For example, if I'm handling the carry manually for an nnx.RNN.

Or one asking about a checkpointing package:

  How do I restore one of the orbax checkpoints into NNX from this script?

I also got flagged for asking about syntax highlighting in the Helix editor.

It's a shame - I like Fable for writing tasks over ChatGPT and I do believe Anthropic is a more ethical outfit than OpenAI. But with the safeguards (and Fable access expiring in a few days) there's no reason to pay for draconian guardrails and harsh rate limits.

ainch

evvelsi gündiscuss

There have been papers about model collapse, but the underlying assumption is that you constantly train on only the outputs of the previous model. Later research has shown that as long as you retain some "real" data, training on largely synthetic data is ok.

And in the case the previous poster describes, the other model doesn't generate datasets, it generates environments which the next generation interact with to learn from.

ainch

3 gün öncediscuss

I think you'll be waiting a while for the former, unless you're ok with strangers teleoperating a robot around your house whenever it gets confused.

ainch

3 gün öncediscuss

Wouldn't modern SLAM or VSLAM address that problem?

ainch

3 gün öncediscuss

Thank you for expanding! I come from more of an ML background so still learning a lot on these topics:

Agreed that the neocortex uses fewer layers because of looping - I also suspect it's partly because neurons are more complex than the neurons used in ANNs, so in principle they should be capable of more sophisticated computation in a single forward pass (especially considering that handling multiple neurotransmitters could mean superposed functions).

The point about there being one correct way to model something does seem backed up by the Platonic Representation hypothesis (https://arxiv.org/pdf/2405.07987). I've even seem some work that shows you can find a bijective map between the latent spaces of different transformers (https://arxiv.org/pdf/2505.12540).

Your point about Finsler spaces is fascinating - I hadn't come across the term before. It'd be interesting to see if there's work that specifically allows latents to exhibit that kind of directional metric behaviour, and whether that improves generalisation or something.

On the entire brain being dedicated to prediction, that's more of a high-level comment. I was thinking of work like Andy Clark on predictive processing, which suggests that even regions of the brain which we think of as receptive (eg. the visual cortex) may be implementing a generative/predictive model which is corrected by sensory input.

ainch

4 gün öncediscuss

I'm afraid the precise connection you're making isn't totally obvious to me.

As far as prediction - I mean sure the cortex and LLMs do prediction, but then so can RNNs or diffusion models or any other generative model. Really any ML architecture is learning to compress its environment in pursuit of modelling. More broadly, the predictive brain model would suggest that all of the brain, not just the neocortex, is dedicated to prediction. What would you say makes LLMs similar to the neocortex, rather than the basal ganglia or Broca's area?

Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?

ainch

5 gün öncediscuss

Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.

ainch

9 gün öncediscuss

As an ML researcher, I know box2d because it underpins many of the standard reinforcement learning environments (in OpenAI Gym) that we use to benchmark methods, like Lunar Lander or Car Racing: https://gymnasium.farama.org/environments/box2d/car_racing/

Thanks to Erin for such a useful piece of software!

ainch

15 gün öncediscuss

This method is cool and the post explains it well. It would, however, be good to get more detail on the energy efficiency they flag as their motivation: is this model actually more energy efficient than the comparators they highlight?

ainch

16 gün öncediscuss

They've said that Mojo is still on track to be open-sourced this year, post-acquisition.

ainch

19 gün öncediscuss

Indeed. The world models research many labs are now chasing was to some degree ignited by David Ha and Schmidhuber's 2018 paper.

More broadly, Sakana is pursing a refreshingly distinct research path, with their focus on evolutionary methods, biological intelligence (e.g. continuous thought machines) and open publication.

ainch

19 gün öncediscuss

Basically all the images and videos on their website are AI-generated. It doesn't inspire much confidence.

ainch

29 gün öncediscuss

Tri Dao's lab must have saved countless watts with FlashAttention. Great to see them continuing to open-source massive efficiency gains.

ainch

29 gün öncediscuss

Here's one that was flagged for me: a question about a niche Reinforcement Learning paper from 2012

I've been reading the option-option model paper by David Silver. It appears that they achieved quite an effective result. Why hasn't there been more work on it since?

ainch

29 gün öncediscuss

I agree it's an oversimplification. The example I think of is something like Newton's law of gravitation vs Ptolemaic epicycles: one simple explanation replaced many layers of tweaks.

It's also a relevant example for AI - one paper tested the ability of Transformers to model planetary orbits: unlike Newton's Law, the implicit forces they learn are nonsense.

https://arxiv.org/pdf/2507.06952

ainch

geçen aydiscuss

Anthropic's claim was that Deepseek collected ~150k conversations.

https://www.anthropic.com/news/detecting-and-preventing-dist...

I think the extent of distillation by Deepseek specifically is overstated. For comparison, Minimax collected over 13m 'exchanges', which starts to sound a lot more like large-scale distillation.

ainch

geçen aydiscuss

In some sense, science is the most extreme form of compression - Newtonian mechanics explains an incredible number of phenomena in a few lines of text.

ainch

geçen aydiscuss

AISI did also say that GPT-5.5, which has been public for months, scores basically the same as Mythos on their cybersec evaluation. But there wasn't as much media about about that for some reason.

https://www.aisi.gov.uk/blog/our-evaluation-of-openais-gpt-5...

ainch

geçen aydiscuss

Token prices have increased, but it's not really the whole story at this point, given some models will use far more tokens to complete a task than others. One of the charts in Anthropic's blog posts shows Fable at 'low' reasoning achieving better results for less money than Opus on 'high'.

Submissions

Training a Simple World Model with Jax

Speeding up MuJoCo 460x with Jax

comments