I'm the OP and wanted to clarify. We do not give Claude Code access to production API keys. The intent was staging databases and I've updated the post to make it more clear.
We assume that every API key we put into Claude Code can be leaked and therefore make sure there's nothing sensitive behind them.
I'm the OP and this is the exact point of the post. Whoever handed you that project did not have Claude code running it end to end in a production-like environment.
Whether you like the vibe coded app or not, you would have had fewer errors if Claude had been testing it end to end while vibe coding.
As written elsewhere, we dont give access to prod! The DBs are staging and our assumption is that every key we give Claude will be leaked. I'll update the post to clarify
I'm the OP and to clarify we dont give access to prod DBs. The point is you need to give the LLM the ability to test end to end, and that can be done with staging data.
Is it possible to run a Kubernetes cluster inside one? (E.g. via KIND.)
If so, we'd very much like to test this. We make extensive use of Claude Code web but it can't effectively test our product inside the sandbox without running a K8s cluster
HolmesGPT maintainer here: our benchmarks [1] tell a very different story, as does anecdotal evidence from our customers- including Fortune 500 using SRE agents in incredibly complex production environments.
We're actually struggling a bit with benchmark saturation right now. Opus does much better in the real world than Sonnet but it's hard to create sophisticated enough benchmarks to show that in the lab. When we run benchmarks with a small number of iterations Sonnet even wins sometimes.
Haiku is called often, but not always the way you think. E.g. every time you write something CC invokes Haiku multiple times to generate the 'delightful 1-2 word phrase used to indicate progress to the user' (Doing Stuff, Wizarding, etc)
Not necessarily true. Subagents allow for parallelization but they can decrease accuracy dramatically if you're not careful because there are often dependencies between tasks and swapping context windows with a summary is extremely lossy.
For the longest time, Claude Code itself didnt really use subagents much by default, other than supporting them as a feature eager users could configure. (Source is reverse engineering we did on Claude code using the fantastic CC tracing tool Simon Willison wrote about once. This is also no longer true on latest versions that have e.g. an Explore subagent that is actively used.)
> - how can I reliably call tools with the right schema?
This is typically done by enabling strict mode for tool calling which is a hermetic solution. Makes llm unable to generate tokens that would violate the schema. (I.e. LLM samples tokens only from the subset of tokens that lead to valid schema generation.)
Or just use the Claude Code SDK that does this all for you! (You can also use various provider-specific features for 2 like automatic compaction on OpenAI responses endpoint.)
For example, the agent in the post will demonstrate 'early stopping' where it finishes before the task is really done. You'd think you can solve this with reasoning models, but it doesn't actually work on SOTA models.
To fix 'early stopping' you need extra features in the agent harness. Claude Code does this with TODOs that are injected back into every prompt to remind the LLM what tasks remain open. (If you're curious somewhere in the public repo for HolmesGPT we have benchamrks with all the experiments we ran to solve this - from hypothesis tracking to other exotic approaches - but TODOs always performed best.)
Still, good article. Agents really are just tools in a loop. It's not rocket science.
Yeah, I'm familiar with them. Once they're widespread this will be a whole different story. (Here too, if you control the fleet of servers then this is easy. If you're deploying software as a vendor you're stuck with older kernel versions.)