Its funny to see how researchers bypass Githubs praised guardrails with a simple word like "Additionally". It just proves that any attempt to build hard security boundaries inside an llm context window is bound to fail. The model is naturally built to follow instructions, so if you mix system rules and user input together, the newer or more persistent instruction will always win
I think they arent even trying to build an AI detector. This is more of a social signal like "dont send us an automatically generated flood of changes"
And it just keeps looping like that until the context window bursts. In practice the model is great at writing new code, but when you feed it its own six month old spaghetti code with a floating bug in the state machine it just starts hallucinating and silently breaking neighboring features
AI accidentally found one of the most expensive resources in the industry: the free time of people who maintain open source in the evenings after their day job
That's just the basics. To craft a prompt for a complex architectural task, you need to know the solution at least on an abstraction level. If you don't have the right system design in your head, no llm is gonna conjure it out of thin air
Funny how almost every wave of automation starts the same way: "we're gonna cut headcount," but ends with "we're just shifting roles"
AI is pretty good at scaling existing knowledge, but if the actual knowledge is just in the head of an engineer who can hear that a press is acting up, the model doesn't really have much to go on
Good code is absent code
LLMs by nature work like autocomplete on steroids. They're always trying to write more than necessary to please the prompt. Seniority now is measured by the ability to break down a task so that the agent doesn't even think to drag in unnecessary abstractions