It's interesting that Claude also over-uses en-dashes. It's very willing to create compound-noun-phrases, especially in that compressed-summary-paragraph it often writes. The 0-days-vibes-vulns that started this thread looks a lot like that, but it could be Claude directly, or just Claude's style influencing people who spend too much time with it.
People are missing that Willison is among the very best people we have in the role of (for lack of a good name): early access to frontier models, evaluate them in real scenarios, no wishful thinking, hype, or doom, communicate the possibilities. Yes he could have fixed this himself but then he would have learned nothing about the AI, and we wouldn't have read a fascinating and important article.
"interpolate" has a technical meaning - in this meaning, LLMs almost never interpolate. It also has a very vague everyday meaning - in this meaning, LLMs do interpolate, but so do humans.
> Since these companies can’t improve their AI models without fresh data created by human beings
Totally wrong. Self-play dates back to Arthur Samuel in the 1950s and RL with verifiable rewards is a key part of training the most advanced models today.
I understand your point, but in response to GP (they should spend this money on houses for other poor people instead), the reduced reliance on other social welfare is totally legitimate to count.
Google Scholar provides imperfect citations - very often wrong article type (eg article versus conference paper), but up to and including missing authors, in my experience.
The best example of all is Prolog. It is always held up as the paradigmatic representative of logic programming, a rare language paradigm. But it doesn't need to be a language. It is really a collection of algorithms which should be a library in every language, together with a nice convention for expressing Prolog things in that language's syntax.
(My comment is slightly off-topic to the article but on-topic to the title.)
"Pelican on bicycle" is one special case, but the problem (and the interesting point) is that with LLMs, they are always generalising. If a lab focussed specially on pelicans on bicycles, they would as a by-product improve performance on, say, tigers on rollercoasters. This is new and counter-intuitive to most ML/AI people.
I would want to hear more detail about prompts, frameworks, thinking time, etc., but they don't matter too much. The main caveat would be that this is probably on the public test set, so could be in pretraining, and there could even be some ARC-focussed post-training - I think we don't know yet and might never know.
But for any reasonable setup, if no egregious cheating, that is an amazing score on ARC 2.
On HN it's very common to see a blog post along the lines of "I found this old piece of equipment with no brand name, I used some network traffic inspection to figure out what it does, I hacked around a bit, I got it working and turned it into a self-ringing doorbell with wifi" (or whatever). All of that is anecdotal, N=1, "I did what worked for me, I hope it's interesting to you". And those posts are highly prized and rightly so.