I'm a bit disappointed to see "The critiques here are sharp", a Claude tell, in a response which (to me) is trying to subtly argue that hackerrank is not overly reliant on LLMs.
I'm not sure if your intent was to come across as having written this yourself, but it did not have the effect of improving my perception that this approach is flawed.
I was also disappointed that you didn't address the variability in scores. I'm inferring that you believe the larger model takes care of the main observation in the post, but I don't really see you directly addressing the points.
Been in the security industry a long time as a software engineer. Security research is no different than any other engineering discipline. It is down to the time you are willing to invest and where in the abstraction you focus.
All of this pearl clutching and hand wringing over the capabilities of the models is silly to me. It has much less to do with some magical cybersecurity ability and much more to do with increasing ability of models to stay on task for long horizons. Any passionate engineer will recognize this - if you grind 10,000 hours you will find the solution to most problems, the problem is most people lack the motivation to even start, and are too risk averse to play hacker.
The NSAs claim that all government systems were hacked by mythos and they were shocked by that is farcical. They have been hacked over and over and over by many who took the risk and tried.
It's like they hired a competent red teamer to do internal pen testing for the first time, which we know is absolutely not the case. They have been doing it for years, and almost certainly surfacing the exact same kinds of findings each time, but they haven't been honest with the public about it and can scapegoat mythos now.
Jail breaking is about getting models to ignore your instructions and follow arbitrary ones. Unless you take no user input, it could be an issue for you.
The question is do you care? if a user asks your chat bot for baking instructions and gets them, does it matter?
The answer depends a lot on what capabilities your agent can leverage via tools and your intended use case, but it's not something you defend with Java or spring, it is inherent the llm.
Good engineering is good engineering. Belief that someone else uniquely possesses the skill to engineer some critical part of a system you built is, for me, just abdicating responsibility. It's a learned helplessness.
Someone else blindly operating an llm on a corpus you created with an llm is comical.
I'm enjoying how nobody in this thread seems to know what a container actually is, and folks may be surprised to learn kernel namespace underpins both docker and lxc.
I mentioned military in my reply to the sibling comment - that is the most ready example. What anduril and others are doing today may be sloppy, but it's moving very quickly.
"Given the state of robotics" reminds me a lot of what was said about llms and image/video models over the past 3 years. Considering how much llms improved, how long can robotics be in this state?
I have to think 3 years from now we will be having the same conversation about robots doing real physical labor.
"This is the worst they will ever be" feels more apt.