My main takeaway from LeCun's thesis isn't that you can't build LLMs to do useful things better than the best human, it's that these systems don't learn arbitrary skills efficiently, like humans do. And the question is, why not? 8% on ARC-AGI-3 is amazing for a machine considering how far we've come since digital computers were first built. But it is pretty poor if you're claiming something is well on its way to exhibiting human-like intelligence.
Mythos can do some amazing things (I'm assuming, I've never seen it). A young child can learn to control its body without reading any books on dynamical systems and kinematics. Mythos cannot learn to control a humanoid robot after sucking in every piece of data Anthropic can get their hands on.
2024 is a good cut-off. I use these additional heuristics. Bad cover art. Not from a real publisher or not from an author that hasn't published before 2024.
I hope that real books that just happen to slip through the cracks, that are also genuinely good books will eventually surface as easily findable somehow. Maybe that's wishful thinking, I don't know.
I know one thing. It's grim out there for publishing and real humans who have something good to say but don't have a popular voice.
I didn't mean nobody lives there. I meant if you plucked random people from all over the country and told them to relocate to NYC, odds are they would need a massive income increase to survive. And even then, it would be touch-and-go for most.
I'm from Harlem, and other parts of NYC. I moved out to pursue a career in an industry that is relatively non-existent in the NYC metro area. If I moved back now I could probably afford it on my current salary. But there are no jobs there for what I do. And if there was a bigger tech industry in NYC the costs would likely be even higher.
Religion used to be the biggest supporter of science. Then science surpassed religion in civilization relevancy. Now conservative politics is stepping up to the fight.
> how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who?
I don't.
My company doesn't host any code on GitHub, we have our own Git servers.
Regarding the privacy issues with OpenAI, Anthropic or anyone else, we engineers are just using the tools we've been authorized to use by the company. If there's a security issue, that would need to be worked out between the company and the model providers.
Would I use external model providers through an API for personal projects? For everything I'm doing now, probably. Could I see a day where I'm working on something too sensitive to be willing to give any data to these companies? Possibly.
I don't even look at benchmarks anymore. I just try different models as they're released on our large, proprietary, systems software codebases in real, shipping products or projects that will ship eventually. It's pretty clear which models help me do my job better or faster. I'm fortunate enough to have the token budget to use basically as much as I need, for now.
No need for benchmarks, evals, marketing, system cards or anything like that. I read the web for tips, practices and release announcements. My colleagues and I share our experiences with each other but beyond that, everything else is just noise.
In the broadest sense, I don't think we're there yet. I asked an SoC vendor to provide their chip documentation in Markdown. They refused. So, I went ahead and tried to do myself with AI.
I tried various AI tools and the results ranged from absolute garbage to something-but-not-something-but-not-quite.
I went ahead and did a section of a huge PDF by hand, just to see if what I was asking for was even feasible. After more than several hours of painstaking work spread across multiple days, I got several chapters to look identical to the source PDF in some Markdown renderers. I had to use some HTML for the more complex tables. I converted some diagrams to Markdown and some to images linked to from the Markdown.
Of course this can't go on forever. Especially not on LLMs. But are we really close to the limits of what these LLMs can do? I'm not sure we are.
The difference between GPT-5/Opus 4 and GPT-5.5/Opus 4.8 is striking. For software development anyway, there's no comparison. And all this has happened in a year.
My assumption is there will be another 2-3 years of improvements ahead of us on LLMs alone. Through hardware upgrades, larger training runs, better data quality, better algorithms, etc.
Of course, by then these models will be quite expensive. Will my company pay for it? I don't know. I'm sure some people will though.
Not sure why this got downvoted. The amount of book slop on Amazon these days is staggering. It has completely changed my experience of shopping for books on Amazon and it's for the worse.
The very understandable desire to not have to rely on huge, centralized companies or powers for tokens has clouded people's judgment on how well these local models actually perform. They've improving, which is great, but for real work I use the best models available right now because they're so much better than local models.
Current: Embedded software, robotics.