this all sounds very plausible and convincing until this part:
> But that's what we get with humans, right? No!
> Humans can explain their reasoning.
can you explain your reasoning as to why a language model would never be able to match a human's ability to explain its reasoning?
i've met a lot of humans that are quite bad at this, and i will likely never know for sure why they wrote the unit tests that they wrote unless i rewrite those tests to myself.
but if you have them explain their reasoning enough, and if that reasoning is plausible enough, and if the relationship between that reasoning and what they did is strong enough, consistently enough — you start to trust them.
you don't trust gpt4 to write code for you. which makes sense. but that doesn't mean as much as you think it means, i think.