This is the biggest lesson I got from LMMs. I have a 1 million LOC vibe coded project that I can only imagine would fit in a few hundred thousand lines. But it's still holding up, I expected some kind of development collapse long before this point.
I don't think the bug is anything special, just another confusion the model can make from it's own context. Even if the harness correctly identifies user messages, the model still has the power to make this mistake.
DeepSeek would print all it's mental gymnastics to censor itself in the reasoning phase directly to the user, before shutting down the conversation. Apparantly such an odd move is a thing in China.
General Intelligence has already been demonstrated to be possible by the human brain, so I don't really get how physicality is an argument against AGI. Who is to say biological computers won't be built?
LLMs can actually make up for their negative contributions. They could go through all the references of all papers and verify them, assuming someone would also look into what gets flagged for that final seal of disapproval.
But this would be more powerfull with an open knowledge base where all papers and citation verifications were registered, so that all the effort put into verification could be reused, and errors propagated through the citation chain.
I was curious what kind of company needed to develop all these different kinds of advanced tech. Turns out it's just so they can be sneaky and trade shares quickly. Kind of sad IMO.