It depends on what you're measuring. I agree that model resourcefulness is useful, but if you're trying to simulate real user sessions, then Claude looking at upstream Git and fetching the answer directly is somewhat worthless.
In my case, I'm trying to measure how coding agents perform under realistic scenarios when implementing tasks, as a proxy for how agents perform when used by actual users for those same tasks, so it's important to ensure the agents are behaving realistically instead of "cheating" and looking up answers.
Happy to share resources! I've been pretty deep in the space :)
as a tip - models will always find a way to cheat, you will probably need to impose some restrictions on what they do / are able to access in the sandbox environment
- Mines tasks from your merged PRs/commits
- Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc)
- Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
The most salient point here is the societal acceptance of consuming slop - somehow we've gotten to a point where the majority of people are ok with mediocre art. I feel that this is a trend that AI has only amplified. The commodification of attention has gradually led us to a point where we're optimizing for engagement instead of for intrinsic value of the content itself.
Personally, I will continue seeking out high-quality music/art/movies/books that speak to me, and most of my friends do the same. There will always be a demand for human-created art, regardless of any plagiarism or replication by labs.
Agree - all of this is based on vibes (I also use TDD based on vibes FWIW). The only way to settle "does TDD / caveman / [insert random skill here] help" is to replay real PRs from your repo and measure quality
> Seems like the progressive disclosure approach is the best for context efficiency; I wound up with a somewhat tight generic AGENTS.md, and the .cursor/rules individual files with glob matching for file names. Cursor honored those well.
This is also generally where I've landed - keep the AGENTS.md super light, and link out to docs as needed. Same idea with skills as well. Basically, preserve the context window at all costs.
The part I'm curious about is, when we're making the sorts of behavior changes you're describing on shared repos, how do we actually measure and quantify impact? It's one thing to tell the team that the agent should perform better, and it's another to say that you made the agent 5% better across a variety of tasks for every dev in the repo.
This has been bothering me for a while - the entire dev community is running on vibes when talking about AI. We're operating in an old paradigm, thinking that smart and logical additions to AGENTS.md result in good agent behavior, when in fact agents behavior is such a black box, that measurement is necessary.
> Even when all the rigging is controlled. (Which implies we need multiple experiments to compare against.)
Yes, agree that low n makes overclaiming a real risk with this sort of optimization loop. Low n results can be useful directionally but can't claim superiority without expanding the dataset. If I were running this for a shared repo with real consequences / value to improving AGENTS.md, instead of just as an experiment, I would expand n by a few factors for training / holdout, depending on expected variation on the tasks.
I'm also noticing similar patterns with needing to update AGENTS.md / skills per model release. E.g with Opus 4.6 -> 4.7, it became much more instruction adherent, so instructions written for the prior model generation might cause unexpected behavior in the new generation. I'm also convinced that an optimal AGENTS.md for Codex is not the same file as an optimized CLAUDE.md for Claude - the model personalities and behaviors are so different that we probably need to tune the instructions differently as well.
Yeah, I've found that to be more effective. Going with the example "Always clarify intent before acting" > "Never act without getting intent first", seemingly because telling the agent NOT to do something sometimes primes it to do that exact thing
My advice, from doing this myself and reading best practices, would be:
- Keep it concise, use progressive disclosure / nested AGENTS.md for information expansion
- Give agent the high level repo structure if necessary
- Have a "why" section to align the agent, high level, what your code is doing
- Keep behavior instructions positive where possible, eg Always clarify intent before acting
AGENTS.md is extremely important - it's probably the highest leverage thing you can give your agent. It's injected into every turn, and the agents are trained to follow instructions. If anything, I think people are under-investing into AGENTS.md and going purely based on vibes.
For example, if I write a bad AGENTS.md for a repo with 100 engineers actively working in it, then every agent for every engineer gets worse, without anyone really noticing.
I think we should move towards data-based tuning of AGENTS.md, testing out changes, gathering data, and then making a decision on whether or not to ship it.
I've been building a tool to do this - build a dataset based on tasks from your repo, then A/B test the agent with whatever change you're making to determine the impact prior to actually shipping it. If you want to check it out - stet.sh
Not the OP, but I've been thinking about this problem a lot - as devs we're overly reliant on vibes for evaluating coding agents. This is already a problem, and especially so if you're working in an engineering organization where a bad edit to AGENTS.md can cause silent regressions for everyone in the codebase.
To solve this, I've built an agent-native tool to run evaluations based on merged PRs in your codebase. Basically you can ask Claude to evaluate whether the skill made things better/worse on real tasks, and to then iteratively improve it
Stalking your profile (sorry..) I see you're pretty deep in the eval space, so I'm super curious what your approach has been to being rigorous for things like skill changes?
Claude does appear to work for longer, and use more tokens, when at higher reasoning modes. It just doesn't seem like this increased token usage leads to better actual outcomes
Agree, it's impossible to tell if someone else's workflow works with your codebase without actually trying it, which takes time/tokens. I've been thinking about how to make running quick, directional evals easier / more efficient to give more confidence in using / developing skills. Basically, how do we go from vibes to data?
I'm actually currently working on benchmarking the opus 4.7 reasoning curve against real-world tasks, and have found that reasoning effort does not seem to monotonically improve results (at least on the slice I'm looking at). I've been puzzling about this but perhaps the fact that claude code has adaptive thinking explains some of it - even at medium reasoning effort, it can use more thinking tokens when needed to solve a complex problem.
Snapshot of the results (sorry for busted format, ask your llm for dataviz. cant seem to format a good table in the comments)
(custom avg is a set of rubrics used for llm-as-a-judge, graded out of 4)
Practically, the results indicate that medium has better outcomes, or at least the same outcomes, considering variance, as higher reasoning efforts, at a much lower cost/time.
yep, ran a controlled experiment on 28 tasks comparing old opus 4.6 vs new opus 4.6 vs 4.7, and found that 4.7 is comparable in cost to old 4.6, and ~20% more expensive then new 4.6 (because new 4.6 is thinking less)