Okay, so it doesn't work, you know it doesn't work, it's just that you accept the specific ways in which it doesn't work.
I've lost track of all the myriad stupid ways in which these ancient systems are hugely ineffectual without even being outright faulty.
Like airline tickets where your name is printed as "LASTFIRSTMR" in all caps and no spaces because their systems are ancient beyond belief.
Similarly, my bank statements are security-critical, because anyone with a copy of my credit card details can pull money out of my account without my express authorization. But...
... because they're stored in terrible ancient mainframe databases, the text fields all have tiny maximum lengths. Hence they're all abbreviations. Attacker-controlled abbreviations without any authenticity assurance of any kind!
I have no idea who actually transfers money out of my accounts! There are no URLs, no metadata, nothing to actually confirm the identity of the other party. Every field in a transaction record is 100% attacker-controlled and unverified by my bank.
If you look at it from the perspective of someone used to modern web security, then you realise that banking is a raging tyre in comparison. Banks literally just accept a certain rate of criminal activity and "price that in", reversing transactions when asked -- which itself can also be a criminal activity. They just shrug their shoulders.
"What can we do about this?" -- says the people that have tried nothing and are all out of ideas.
Rewrite it. The whole thing.
Use an actual database, something made in the last three decades instead of half a century ago.
Use cryptography. No, not crypto coins! I just mean a bog-standard algorithms like public-private key signing so that it is possible to confirm the source of transactions.
Etc.
I would much rather have something generated with the assistance of a modern LLM than what we have now, which is security holes big enough to drive a panamax container ship through.
I don't bother to use open-weight models, but for all of the "security" or "security adjacent" work I have tried recently, GPT 5.5 x-high has been the first model that produced useful output.
Even very old models could spot the most glaring issues, but it's a different story if you scan a source repository where humans can't find security vulnerabilities even after hours of reading through the code. Feed something like that to, say, Gemini Pro 3.1 and you'll get a bunch of false positives back, nit-picking, or variants of "this could be insecure if the code around it changes in unreasonable ways in the future".
Feed the same thing into GPT 5.5 x-high and then tens of minutes later it'll find half a dozen unauthenticated remote code execution vulnerabilities, arbitrary file read/write vulnerabilities, or similar.
Until it got nerfed, Mythos was similarly a huge step up for a lot of people working on code security.
Is this bug fixed with 5.6? If not, it probably doesn’t matter which version Codex users are getting because the overall result is dramatically worse than stated by Open AI advertising: https://github.com/openai/codex/issues/30364
> while you could ask your LLM to do that you aren't going to run your bank on the result.
Why not?
I feel like we're entering a new era of prejudice against not a category of humans, but against non-human intelligences.
The design patterns for distributed and fault-tolerant systems are well-known and established in the industry. Both humans and AIs are familiar with them!
So if you sketch a design for the AI to follow, establish the rules in AGENTS.md, have a robust test suite, use a frontier model dialed up to eleven, etc... why not rely on the LLM output?
At the end of the day, humans are not without fault either.
I've been wading through some legacy "pre-AI" code recently and it has more bugs than a rainforest! Static fields used incorrectly, causing data races. Floating point types used for money amounts. JavaScript and SQL injection up the wazoo. Wildly unsafe password handling. So on, and so forth. This is the norm for most human-written software, not the exception.
As a proof-of-concept, I tried an AI rewrite of one such legacy app[1], and it is not bug free, but it notably has fewer bugs than the original. Different bugs, sure, and I'll have to iron them out after a round or two of UAT, but I'm honestly more confident with what I got from the chatbot than the code inherited from humans.
[1] Deals with money, but admittedly at a much lower level of risk and consequence than a banking app running on a mainframe.
If this really is widespread and degrading performance in 40% of the cases, then if OpenAI simultaneously fixes this bug and releases GPT 5.6 within a day or two, then the sudden boost in capability is going to blow people's hair back.
Something I'd like to see is for someone to finally come to the realisation that the right thing to do is to make the front-end web templating language truly polyglot and support SQL natively, without an ORM wrapper.
For example, the ASP.NET Razor syntax allows HTML and C# code to be interspersed surprisingly freely:
<ul>
@foreach (var user in Model.Users)
{
<li>@user.Name</li>
}
</ul>
Just picture the same kind of thing, but with SQL expressions freely interspersed with the programming language.
Just like how Cargo, NuGet, NPM, etc... can import packages and/or how you can cross-reference projects in build systems, web apps should be able to reference a database schema project directly, importing the SQL definitions without any explicit "mapping". If the SQL changes, the type changes, and the build system picks that up automatically without any additional manual steps.
.NET with EF Core is almost there, and I've seen some half-hearted attempts in various languages over the years, but it's like the industry has an allergy to the concept.
I bet that since this bug has made headlines, there are some panicked engineers at OpenAI desperately trying to figure out how to fix it without undoing their “magic optimisation”.
> I find it interesting how far we've come so far from the mindset of "You can do that."
Outside of Silicon Valley, most organisations strictly only buy commercial-off-the-shelf (CotS) software. If they do get something developed bespoke, it'll be an isolated thing with very defined boundaries.
This gets absurd after a point, where'd I've seen organisations was millions of dollars to avoid what amounts to an afternoon of scripting... and that was before LLMs!
If you even suggest custom code "an executable" (gasp!) to some people, they'll nearly faint from the stress of even thinking about it.
I've always found this kind of self-imposed constraint a bit odd, but it is surprisingly pervasive.
There can be selection pressure for females to mate with multiple males.
Releasing sterile males only works for species that mate only once or at most twice, and rapidly falls off in effectiveness for species where the females mate many times.
Look, even if in this one instance Trump isn’t simply asking get his beak wet, that is the less believable scenario.
When “not cartoonishly corrupt” becomes hard to believe due to firmly established character and endless precedent, then it almost doesn’t matter if everything is on the level in this one matter.
The trust has been lost, undermining the ability to govern.
The fact that he’s still in power with firm support from everyone else that matters is all you need to know about how rotten to the core things are in your country.
> but there are a lot of tasks where it's cheaper and more efficient to use other models.
Sure… but which ones? How can you know ahead of time?
I just did a “simple” upgrade project where both me and the AI kept tripping over dead code, subtle typos, and difficult-to-trace live versus dead code.
Many times I used “Medium” thinking I got bitten, but not every time, and I couldn’t predict when.
The irony is that now, in the LLM age, it wouldn't be that hard to feed a few thousand browser tabs and/or bookmarks into a vector database or whatever and then if some fuzzy memory tickles your brain about something you saw a year ago, you can just query it.
It's the same logic as keeping RAW digital photos. Lightroom is already gaining "find the keepers" AI features. Sooner or later it'll be possible to feed a bunch of burst shots into an AI that just weren't worth the trouble to manually sort through. The AI can do the drudge-work of digging up any gems in the rough...
Your thoughts are just some ions sloshing around a lump of meat.
That meat follows an ill-defined pattern encoded in fewer bits than the source code of PyTorch and its pretraining phase used a tiny fraction of the available data.
You’re a poor imitation of an LLM.
I mean… you’re fluent in, what, at most five or six languages? Can program in maybe another dozen if we’re being generous about your capabilities?
Pfft… who would trust anything to meat brains!? They’re famously prone to hallucinations!
What do you even mean by "works", specifically?
> it might have bugs, you know what they are.
Okay, so it doesn't work, you know it doesn't work, it's just that you accept the specific ways in which it doesn't work.
I've lost track of all the myriad stupid ways in which these ancient systems are hugely ineffectual without even being outright faulty.
Like airline tickets where your name is printed as "LASTFIRSTMR" in all caps and no spaces because their systems are ancient beyond belief.
Similarly, my bank statements are security-critical, because anyone with a copy of my credit card details can pull money out of my account without my express authorization. But...
... because they're stored in terrible ancient mainframe databases, the text fields all have tiny maximum lengths. Hence they're all abbreviations. Attacker-controlled abbreviations without any authenticity assurance of any kind!
I have no idea who actually transfers money out of my accounts! There are no URLs, no metadata, nothing to actually confirm the identity of the other party. Every field in a transaction record is 100% attacker-controlled and unverified by my bank.
If you look at it from the perspective of someone used to modern web security, then you realise that banking is a raging tyre in comparison. Banks literally just accept a certain rate of criminal activity and "price that in", reversing transactions when asked -- which itself can also be a criminal activity. They just shrug their shoulders.
"What can we do about this?" -- says the people that have tried nothing and are all out of ideas.
Rewrite it. The whole thing.
Use an actual database, something made in the last three decades instead of half a century ago.
Use cryptography. No, not crypto coins! I just mean a bog-standard algorithms like public-private key signing so that it is possible to confirm the source of transactions.
Etc.
I would much rather have something generated with the assistance of a modern LLM than what we have now, which is security holes big enough to drive a panamax container ship through.