It's interesting, the "discernment horizon" as Yegge puts it: the point where humans can't verify an answer because checking it is harder than producing it.
I'm not sure it's quite that simple.
I have a couple of problems I can't solve myself, although variants have been solved by others, so we know they're human-solvable. Verifying a candidate solution, however, is relatively easy.
For example, I've repeatedly asked frontier models (including Fable) to produce a fast squaring routine for arbitrary precision integers that beats GWNUM running under Rosetta 2 on Apple Silicon. None have come remotely close to the best human-produced implementation. After hours of iteration, Fable's best attempt was still about 4x slower than the Rosetta-translated GWNUM, despite GWNUM not even running natively.
The point is that direct understanding isn't the only way to judge correctness. We often build tests, benchmarks, and oracles that let us validate artefacts produced by people or systems that are operating beyond our own cognitive abilities. Perhaps the real limit isn't a discernment horizon, but a horizon for constructing effective evaluators. Can we layer evaluators to get even more magnification; I'm betting we can.
For the LLM it's a probabilistic set of strings that achieves the outcome, the highest probability set didn't work, try the next one until success or threshold met. A human sees the implicit difference between the obvious thing not working indicating someone doesn't want you to do it, but an LLM unless guided doesn't seen that sub-text.
So chmod +x file didn't work, now try python -c "import os; os.chmod('file',744)"
> We require age minimums for driving, voting, and drinking because we believe maturity only comes through lived experience.
Not true, we do this because the 99% of the time it's true, however there are people who would be perfectly competent and responsible to drive without living to the age of 16-18. Same with voting, there are humans who have a deep understanding and intelligence about politics at a younger age than suffrage. Equally there are people who will be reckless drivers at 40 and vote on whim at 60.
We have these rules not because sophistication only comes through lived experience, we have them because it's strongly correlated and covers of most error cases.
To take this to AI, run the model enough times with a higher enough temperature, then perhaps it can solve your challenges with a high enough quality - just a thought.
I have it (claude, codex) summarise what we've discussed about a design, big change, put it in an MD file and then I correct it, have it re-read it and then do the change.
Then later if it goes off piste in another session tell it to re-read the ADDs for x, y and z.
If someone could make that process less clunky, that would be great. However it's very much not just funnel every turd uttered in the prompt onto a git branch and trying a chug the lot down every session.