tinygrad’s small set of operations and laziness made it easy to implement. Tho my overall sense is that neural network verification is currently more of a research interest than something practical.
I wonder if the authors have tried incorporating error feedback from Lean into their models.
Work from 2023 [1] showed general purpose models did better when they were able to incorporate error feedback, humans incorporate error feedback, but none of the SOTA models on minif2f seem to.
There’s something compelling about these, especially w.r.t. their ability to generalize. But what is the vision here? What might these be able to do in the future? Or even philosophically speaking, what do these teach us about the world? We know a 1D cellular automata is Turing equivalent, so, at least from one perspective, NCA/these aren’t terribly suprising.
If "we" all got together and reduced permitting costs, "our" homes would be much more valuable because a developer can build a highrise where the house is.
Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`?
Why not consider how far off the model is `abs(actual-expected)`? I wonder if there is an inflection point with that metric.
> ChatGPT specifically seems to absolutely adore the phrase [complex and multifaceted], using it at every opportunity to explain higher level concepts.
I'm curious why this is. Before the echo chamber, did the LLM lean some ontology of speech?
The title is an allusion to the experiment which samples the output of the 12th layer of LLaMA-7b. Surprisingly, there is a fair bit of structure on the principle components of those samples.
https://github.com/0xekez/tinyLIRPA
tinygrad’s small set of operations and laziness made it easy to implement. Tho my overall sense is that neural network verification is currently more of a research interest than something practical.