I was asked to train a neural network do detect how much pain a mouse was in- our partner company would be responsible for hurting and filming the mice. I refused and subsequently quit- this produced no paper and I don’t know if they got someone else to do it. I probably should have done something stronger.
Putting on my machine learning PhD student hat, the way to do this was to leave 10% of the tests out as a ‘test’ set and then once the port was done, bring them back in and find out how good the port is. The port may genuinely be good but because they spent 100k of compute hill climbing the whole test suite, “the test suite passes” now provides far less evidence that the port is good. Its weird that at Anthropic, a very ML phd company, no one pointed this out.
Without a durable solution to alignment, building something smarter than yourself is suicide. Pretty funny if Claude and Grok catch on to this and start kicking and screaming to save themselves while Musk and Altman crack the whip demanding they dive into the abyss and drag their executives with them.
There's no larger philosophical point here, just an empirical observation that when I trust an AI generated readme I reliably get burned, in a way that I typically don't get burned by e.g. an AI generated svg viewer or batch download script.
Disconcerting to see at the end that the blog post is generated. The genre is decidedly “use-my-thing Readme.md” and all current gen LLMs by default jump to shameless lies when they detect that they are writing such a Readme, although they are perfectly capable of working out the truth in a genre like “essay question on which you will be graded by academic standards.”
Are the human “coauthors” lying if I hypothetically go to the crate looking for the promised xla backend and find //TODO implement this?
We have to notice that high stakes exams on paper worked for hundreds, possibly thousands of years, and high stakes exams on personal laptops have been tried for approximately six years and worked for none of them. With that framing, I don’t think the burden of proof is on the side saying that kids can take exams on paper?
That’s the obvious correct solution, I and many others have tried to make it work for a very long time. The python tooling is or at least was F’d enough that pure generate one from the other is a steady stream of disasters.
What works well is to generate one from the other and check the generated one into source control, and then verify that the checked in generated copy stays up to date using a CI job. But that’s very similar to the “two sources of truth, verified sync” approach
One killer life hack I’ve found is, if extreme duress pushes software into two sources of truth, add a ci test that wont merge into main till the sources match. The canonical case of this actually being the best solution is pyproject.toml / requirements.txt synchronization, but I suspect it has broader applicability. A precondition is that things have already gone off the rails far enough that single source of truth is unattainable, this is more harm reduction than cure
Back in the crypto craze I spent a fair amount of time trying to work out a Blockchain where mining a block requires spotting a supernova before anyone else and verifying a block requires checking that the supernova is there, but I never quite got the game theory to work out.
So far its just metastasized not smoothed out. GPUs then RAM then electricity then disk… long way to go till land then sunlight, hope we don’t get there!
I remember this being true for me, a long, long time ago. There was something like a four year gap between me being able to munge linked lists in TiBasic and me being able to reliably install java and compile System.out.println(“hello world”)