“ There have been many cases where I've started a piece of work by laying down very rigid abstractions and a few examples of using them, and I explicitly prompt to not only exclusively use the specific abstraction API but also copy the way I've used it. And the (frontier) LLM does neither, it just steams ahead re-implementing things from scratch from bottom up basic structures”
Yup, this is the sad state of affairs that we’re currently in, and the only way to avoid this is to specifically instruct the model on how exactly to implement things.
From my point of view, I think this is fine, to be able to use these LLMs as fast implementation engines. The challenge is to make it surface these types of implementation decisions before it goes off doing it the wrong way.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.
And then people wonder why there’s so little faith in the EU and why there is a perception of them being disconnected from the real interests of the people.
Ensuring our remote employees’ machines are secure is a serious problem for us, and it’s absolutely impossible to require employees to be diligent. We require attestation upon connection to our corporate VPN that checks for basic things such as latest security patches, certain tools installed, etc.
Giving an app full scope to all repos in an org does not automatically imply that it would leak information from private repo A in comments on public repo B. That’s the issue being discussed here.
Like I said earlier, I can see both points of view, and I think the answer is more granular scoped permissions (eg on a per-workflow basis). Right now the permissions are crude.
That doesn’t sound like the “it hammers until it’s done”-type of intent.
Just last night Fable decided to get into a rabbit hole of debugging a database driver issue by packet sniffing the network traffic instead of just adding debug statements to the code. Definitely needed steering, and I don’t know many people whose first intuition would be to use pcap when they have a segfault.
"How is this a Github vulnerability? The researchers are the ones that grant the agent access to private repos and then ask it to answer questions in public repos.. of course this allows extracting private information?"
I think the assumption is that the permissions are scoped to the repository you're currently asking questions on, rather than your private repositories as well.
I see the exact same discussion as we’re having right now there; people stating that local models aren’t as good as the state of the art, but good enough for certain tasks.
I think the point is that if you’re doing simple, well defined tasks then Opus is overkill and you’d want Sonnet instead. Meaning, GLM5.2 is Sonnet-quality, not Opus-quality.
But Deepseek has a very different way of operating their business as the underdog. They also publish their models as open weight, which Anthropic also doesn’t do.