It is not so simple a problem. Should people have the right to do whatever they want with hardware they buy? Yes.
But the regulations that would require John Deere to change their practices and designs for repairability are not about your rights, they are about what we require John Deere to provide. And the more you require John Deere to provide, the more costs add up. When designing regulations that we require companies to follow, the costs of those regulations should be considered.
For routine repairs it seems very beneficial for farmers to be able to repair things themselves. But there’s a very long tail of problems where at some point the cost will become meaningful, and the benefits might not be that great.
This is a prompt injection to include a malicious link in the response. The user would still have to click that link.
I think Google should put in some effort to avoid this type of low-effort prompt injection, but it also does require a second step of users clicking the malicious link in the AI output.
There are always concepts that some people think are a basic, that others haven't heard of. The entire benefit here is that AI can point out what we miss. There are certainly techniques you don't know about, or just didn't think to apply to a problem, that others would find to be pretty standard.
You can have a nuanced discussion with an LLM. But LLMs also have failure modes where they start making up justifications. The two are not mutually exclusive.
I disagree with keeping an eye on the model as it is working, approving every command, and denying and stopping the model when you think it has gone wrong. It is not that it is actively harmful to do this, but rather that it is a waste of time and you can avoid the need for it through better design discussions and review.
Micro-managing and keeping the AI on a "short leash" also lends itself better to telling models to do smaller units of work at a time instead of discussing broader design concerns. That is why I think someone doing this would miss the MILP solution, because they might never discuss the overall design with the model but rather just tell it what to implement next.
"Nuanced discussions" is more about describing a design to a model, asking the model to critique your design and ask you for clarifications, and then you providing those clarifications and the model "getting it" and proceeding to additional levels of detail before implementation. In particular the models being able to highlight concerns you have not yet thought about is a pretty good sign of this. Fable is noticeably better at this compared to Opus.
I was not talking about models making mistakes. Mistakes, and then models making up justifications for those mistakes, is a failure mode of any LLM, and Fable is no different in that regard. Newer models might make less mistakes, or at least make less egregious mistakes, but they still make mistakes.
This “short leash” seems like more of a crutch to me, and a sign of not giving the AI enough detail on the problem to begin with, or not reviewing and iterating on its output.
Hand-holding great models like Fable through implementation is a waste of time, and a waste of Fable. You can have increasingly nuanced discussions with stronger models, and they write a lot better code than they used to. The process of discussing designs and their implementations, questioning things that look weird to you, and actually reading the AI’s responses also helps to find better solutions.
For example, one time I wanted to write a greedy solver for a problem, and in my discussion with Opus on the idea it suggested using an existing MILP library to solve the problem exactly. I’d never even heard of MILP, but my final implementation ended up being better and simpler than what I’d have done alone.
It’s pretty incredible to me that a mammoth change like this is possible to prototype now using LLMs.
It makes me wonder how much of our software stack will become more malleable to big ideas and experiments in the future, like Filip’s idea here. Even if you don’t want to merge the code, it’s still an incredible existence proof that something like this could work.
Would the US government have slapped Anthropic with this export control if Anthropic never fearmonger'ed about Mythos? I think the answer is very likely no.
But is this the type of regulation Anthropic has been asking for? Not at all.
This is a failure of Anthropic's politicking, and a warning that they need to be more careful with their communication in the future. If they truly want constructive regulations because of their fears about AI, they will need to repair their relationship with the administration, and it is still unclear to me how they plan to do that.
The big AI labs are also accumulating huge datasets of expert work in a wide range of fields, which is very expensive to re-create. It seems pretty plausible that this this gives them a big advantage that is compounded by their larger training runs and larger models.
The Team plan is ~125 USD / month / user. Big enterprises like Uber are paying upwards of $1500 USD / month / user. Anthropic can raise their revenue a lot more by selling to big enterprises than they can by selling more team plan seats.
Definitely, it is quite an extreme change. But the upsides of better access to support and advice are huge, even if the potential downsides are scary as well. This feels like one area where we need better transparency and regulation due to how much ChatGPT and others can affect people who listen to them.
It is not solely or even primarily the big AI labs that would need to prepare. They have a better idea of what’s coming, and they’re positioned to benefit from it.
It is governments, big companies, and individuals who could all experience fundamental changes if any of these predictions come true. If people within the labs believe these possibilities are around the corner, it would be responsible to try to let people know so they can be more ready if RSI suddenly hits and in a couple years time all our work is fundamentally changed.
That’s not to say I agree with their predictions, but rather I’m just saying that there are good reasons for Anthropic to publish stuff like this that are not just PR.
I gave GPT-4 some source code and my existing tests, and asked it to write a new test, and it did it! It didn’t even run straight away, I had to fix it, but it still blew my mind.
Later, I wrote a ~5k line proxy for work in C, and gave the whole thing to ChatGPT o1 and asked it to review it. It found several real memory bugs, and now that service has been running since with no problems.
Just this week, I was trying to write a greedy solver to pick the best subset of block sizes to keep from a larger sweep for shorter testing. Opus 4.8 suggested that this could actually be solved as a MILP problem, and found the perfect solution in 5 mins. I’d never even heard of MILP before.