Based on a cursory read of the situation, it seems similar (at least on its face) to the Waymo vs Uber situation. In that case, Uber payed a Waymo an equity stake and signed an agreement about which technology they would/wouldn't use. The key person involved also was sentenced to 18 months in prison (pardoned after 6 months).
I mean, OpenAI delayed the public release of GPT-2 back in 2019 because it seemed capable of authoring interesting blog posts (that also happened to be untrue). It was a pretty big deal the first time Transformer models were capable of generating that kind of output--no one found it weird. We've just grown to take it for granted that large Transformer models are this capable.
The same cycle is happening now for a harder frontier. And proofs represent a pretty good benchmark for model capabilities, so a new model proving a result that a previous model didn't is generally notable in the same way that a model scoring higher on a benchmark is.
I'm sure we'll take it for granted in the not-too-distant future.
I think the author largely agrees with you re: type systems and LLMs. He's pretty explicit that Haskell should be very well positioned to be a power language for LLM-assisted programming, but that the Haskell ecosystem presents the bottlenecks that make it harder.
I don't personally use Haskell for anything, but I use Lean and occasionally some other languages with expressive type systems, and like you I've found it to be a pretty great experience for working with LLMs. But I've also experienced what the author is talking about, with languages that sit at different points on the type system spectrum, regarding a languages ecosystem/infra layer becoming a bottleneck. I don't think it's ultimately about the type system but the broader ergonomics of the language/ecosystem.
So I think his criticism is less than expressive type systems are a pre-LLM concept, and more that Haskell has an individually bad "agentic coding story".
I've been a power user of LLMs for software development for a while now, and I've found two things to be true:
- The benefits of more "extreme" type systems are more accessible and valuable than ever. I have a fairly involved project built on Lean that I hope to open source this month, and it's been a joy to work in even for uses outside of mathematics.
- Readability, build time, infra complexity, and everything that affects your speed after finishing your implementation--these things now matter more than ever.
It's sort of a dual ergonomics problem, in some sense. And given that, the author's lament makes complete sense to me, especially:
"An AI-enabled Haskell ecosystem would ask different questions. How do we make Haskell easier for agents to use well? How do we get more high-quality Haskell examples into model training data? How can we scale reviews? How do we make library docs full of copy-pastable, realistic examples, not just beautiful types? How do we make project bootstrap fast? How do we make error messages more agent-friendly? How do we reduce cold build times? How do we make common industrial patterns obvious to a model that is trying to help?"
Lots of researchers have done just this! There's a really rich history of research + lots of contemporary work on different encoding/representation strategies. This might be interesting to you: https://sbert.net/
What makes the DeepSeek-OCR and related results exciting to some researchers is less about the fact that you could devise a tokenization scheme that has fewer tokens, and more about how well it works.
Nah, optical compression is a thing. You see it in a lot of different areas in ML. In this case, the "trick" has been known for a while, and belongs to a whole world of compression research. But I think where you're maybe getting mixed up is in where that 60% gain is coming from.
It's not a 60% percent reduction in cost for 100% of the same output. If you have a model and input text A, and you fix the seed etc. and run Text A through the model as text tokens and as compressed image tokens, you will not get identical outputs. You're specifically reducing the number of tensors needed to represent your input, which saves you on raw compute, but also by definition gives you less room to represent the information in your input. It's lossy, in other words.
Put another way, if you're using a model like Fable because you need the absolute frontier of capability and cheaper models cannot solve your tasks, then there is a very real chance that a compression strategy like this drops Fable's accuracy such that it's no longer suitable for your task. Which defeats the point of you paying for the most expensive model in the first place.
So, it's cool research. Might be useful for some people. Probably isn't something that has incredible utility in real use cases.
This has been a (noble) goal of lots of different projects in the community for a long time. Federated learning projects like Flower have been chipping away at it for a long time. There are many many hurdles to be cleared before anything in this area is super feasible as an alternative, but I applaud everyone who works on it.
But yes, in general, merging refers to techniques that directly blend the weights of different models mathematically. It had a big moment of popularity ~2 years ago, with many so-called "Frankenmodels" popping up on leaderboards.
I tend to think of merging as belonging to the same general umbrella as things like "abliteration", or other techniques that surgically modify the weights of a model without a traditional training/tuning loop. Maxime Labonne is a great person to follow if you're interested in this general area.
I don't understand this line of criticism exactly. By putting new information in the context window, you are materially changing the activations at your point of sampling, which is literally "customizing with mere markdown files."
Taken to the extreme, the attitude that there is some special incantation that will unlock all capabilities is silly, and a lot of the "prompt engineering" discourse is similarly kind of dumb, but in-context learning is clearly a real thing.
Eh, Watson was a classic open domain QA system originally, no deep learning or much of what we think of in an "AI platform" today. It was one of a bunch of such systems that were built in that early 2000s period. They all failed because the approach fundamentally didn't work very well.
My experience has been that this is not unique to tech, and is common in all large enough industries. I think it's just the natural emergence of reward hacking i.e. if you're an executive at Pepsi and your job is largely to increase the stock price, and you know that you can do something to change the way your numbers are presented such that Wall St will like it, you'll likely do it.
I do think tech certainly has its own flavor though, particularly because of how differently it is treated by investors.
If I'm remembering right, it was weirder than that, as Llama's originally release strategy was sort of bizarre.
You did have to apply for access, but if you met their criteria (basically if you were the right profile of researcher or in government), you got direct access to the model weights, not just an API for a hosted model. So access was restricted, but the full weights were shared.
I believe that the model was leaked by multiple people, some of which didn't work at Meta but had been granted access to the weights.
I've worked on two open source infrastructure projects that raised money now, and am friends with people involved in many more. I'd put a couple of asterisks next to the claims in this article:
- VCs definitely cared about our Stars, especially in early stages, but not as our primary metric. I suppose Stars might be the primary metric if they're truly off the charts, but usually they're just one of many social proof signals an investor might look at.
- Investors, especially at the earliest stages, are quite a varied bunch. Some were diligent about looking at who was leaving Stars on the repo (i.e. are these accounts fake/do they belong to potential future customers). Some less so. This is true for basically every metric (see: startups that grossly misreport ARR)
- Fake GitHub stars were a thing way before 2022. I'd have to look in more detail at the methodology here, but I'd question any analysis that finds that paying for GitHub Stars (or any social following kind of metric) is a strictly post-2022 thing. Any metric that can be construed as social proof will immediately have its own grifter economy. Investors know this and (mostly) do their diligence.
Finally, showing numbers is hard for an early stage open source startup. At later stages, you should be able to show an actual business with typical metrics, but at the seed stage you often just have a repo and a website. Your goal is just to get a lot of people using your software. You can add telemetry to track that, but that's a thorny decision. GitHub Stars aren't a terrible proxy for popularity, provided that you audit the quality of the following. A project with a lot of organic stars and forks is, at the very least, a project that a lot of people are familiar with.
I'm not saying that GitHub Stars aren't wildly overvalued or gamed, but contextualized properly, they're a reasonable metric to consider, particularly at earlier stages. Most investors aren't just throwing millions at random repositories with 20k Stars from obviously spam accounts.
I work with/am friends with many junior-ish developers who are in the same place as you (got into programming in their late 20s around the 2020 hiring cycle). I'm very sorry for the stress you're dealing with.
I don't know if this describes your situation, but I know many people who are dealing with positions where they have no technical mentorship, no real engineering culture to grow in, and a lot of deadlines and work pressure. Coupled with this, they often don't have a large social group within programming/tech, because they've only been in it for a few years and have been heads down grinding to get a good job the whole time. They're experiencing a weird mixture of isolation, directionless-ness, and intense pressure. The work is joyless for them, and they don't see a future.
If I can offer any advice, be selfish for a bit. Outsource as much as you want to LLMs, but use whatever time savings you get out of this to spend time on programming-related things you enjoy. Maybe work the tickets you find mildly interesting without LLMs, even if they aren't mission critical. Find something interesting to tinker with. Learn a niche language. Or slack off in a discord group/make friends in programming circles that aren't strictly about career advancement and networking.
I think it's basically impossible to get better past a certain level if you can't enjoy programming, LLM-assisted or otherwise. There's such a focus on "up-skilling" and grinding through study materials in the culture right now, and that's all well and good if you're trying to pass an interview in 6 weeks, but all of that stuff is pretty useless when you're burned out and overwhelmed.
I don't really understand this line of criticism, in this context.
What would "generalizing" the information in this article mean? I think the author does a good job of contextualizing most of the techniques under the general umbrella of in-context learning. What would it mean to generalize further beyond that?