AI Coding: A Sober Review(ubicloud.com)
ubicloud.com
AI Coding: A Sober Review
https://www.ubicloud.com/blog/ai-coding-a-sober-review
15 comments
This matches my experience as well. When I'm working on a codebase that I started and know well, it feels like magic to chat with an AI and watch patches appear on the screen to accept/deny. I only accept about 50% of the AI patches before tweaks because it's my project and I care about keeping on the track I laid out.
When I'm vibe coding something from scratch I don't have the mental model, I don't always review everything closely, and eventually it becomes an "AI project" that I'm just making requests against to hopefully achieve my goal.
When I'm vibe coding something from scratch I don't have the mental model, I don't always review everything closely, and eventually it becomes an "AI project" that I'm just making requests against to hopefully achieve my goal.
And when you lose your mental model it’s harder to prompt the LLM for good code.
This space is filled with personal anecdotes and studies from providers. It's hard to get objective perspectives from independent labs.
Hi! Author of the blog post here.
I completely agree, getting an objective measure for the developer experience from these various tools is not easy. On one hand, you have a series of benchmarks from LLM providers. While reflecting some degree of fitness to specific tasks, they often fail to translate to real-world usage. On the other hand, you have the tool providers with different features and product claims, and user anecdotes for very different use-cases.
The attempt with this post was to summarize my experience across some of these tools and highlight some specific features which worked better for me vs others. Given how quickly things are changing in this space, the primary conclusion is that using a tool day-to-day, discovering its strengths and deficiencies and working to eliminate the ones with high hit-rate is best at this point.
I completely agree, getting an objective measure for the developer experience from these various tools is not easy. On one hand, you have a series of benchmarks from LLM providers. While reflecting some degree of fitness to specific tasks, they often fail to translate to real-world usage. On the other hand, you have the tool providers with different features and product claims, and user anecdotes for very different use-cases.
The attempt with this post was to summarize my experience across some of these tools and highlight some specific features which worked better for me vs others. Given how quickly things are changing in this space, the primary conclusion is that using a tool day-to-day, discovering its strengths and deficiencies and working to eliminate the ones with high hit-rate is best at this point.
(Disclaimer: Ozgun from Ubicloud)
I agree with you. I feel the challenge is that using AI coding tools is still an art, and not a science. That's why we see many qualitative studies that sometimes conflict with each other.
In this case, we found the following interesting. That's why we nudged Shikhar to blog about his experience and put a disclaimer at the top.
* Our codebase is in Ruby and follows a design pattern uncommon industry * We don't have a horse in this game * I haven't seen an evaluation that evaluates coding tools in (a) coding, (b) testing, and (c) debugging dimension
I agree with you. I feel the challenge is that using AI coding tools is still an art, and not a science. That's why we see many qualitative studies that sometimes conflict with each other.
In this case, we found the following interesting. That's why we nudged Shikhar to blog about his experience and put a disclaimer at the top.
* Our codebase is in Ruby and follows a design pattern uncommon industry * We don't have a horse in this game * I haven't seen an evaluation that evaluates coding tools in (a) coding, (b) testing, and (c) debugging dimension
It's hard to go beyond anecdotes because it's impossible to measure outcomes objectively.
Is it? Tests turn green seems pretty objective, as does time/tokens to test green, code delta size, patch performance, etc. Not sure why people have such a hard time with agent evals.
Just remember to keep a holdout test set for validation.
Just remember to keep a holdout test set for validation.
> Is it?
Yes. You are "testing" a non-deterministic black box, and we usually know nothing about the code base, the prompts, the tasks etc.
Which is further complicated by whatever wrapper you're using (cursor/amp/windsurf/opencode/whatever).
Which is further complicated by the "oops we nerfed the model, but it was bug trust us".
> Tests turn green seems pretty objective, as does time/tokens to test green, code delta size, patch performance, etc. Not sure why people have such a hard time with agent evals.
What is the distribution between running the same test on the same model with the same prompt, also with distribution by time?
I've already had several instances when the same model with the same prompt on the same code would produce completely different results.
Yes. You are "testing" a non-deterministic black box, and we usually know nothing about the code base, the prompts, the tasks etc.
Which is further complicated by whatever wrapper you're using (cursor/amp/windsurf/opencode/whatever).
Which is further complicated by the "oops we nerfed the model, but it was bug trust us".
> Tests turn green seems pretty objective, as does time/tokens to test green, code delta size, patch performance, etc. Not sure why people have such a hard time with agent evals.
What is the distribution between running the same test on the same model with the same prompt, also with distribution by time?
I've already had several instances when the same model with the same prompt on the same code would produce completely different results.
You can construct or curate code bases (parametric construction is cheaper and gives you 100% knowledge).
You are testing a series of traces from starting prompt -> agent stops or creates a PR. Your signal is %pass + time to green + code metrics as I said.
You can control for the model and drift by doing bootstraps on individual repo evals to get a distribution, any model nerf will show using statistical tests.
Capturing a distribution is the whole point. I run my agent evals 20x on a given problem for this exact reason. This way you can tune prompts and not only do you get your average improvement in pass/time to green, but you can see the shape of the distribution and optionally tune for things like maximum error magnitude that point statistics won't show you.
If you want to talk about how to eval in more depth, share your specific case and I'll help you set it up.
You are testing a series of traces from starting prompt -> agent stops or creates a PR. Your signal is %pass + time to green + code metrics as I said.
You can control for the model and drift by doing bootstraps on individual repo evals to get a distribution, any model nerf will show using statistical tests.
Capturing a distribution is the whole point. I run my agent evals 20x on a given problem for this exact reason. This way you can tune prompts and not only do you get your average improvement in pass/time to green, but you can see the shape of the distribution and optionally tune for things like maximum error magnitude that point statistics won't show you.
If you want to talk about how to eval in more depth, share your specific case and I'll help you set it up.
You have either too much time, or too much money, or both to curate code bases, to run 20x agent evals on those curated databases and spend time micro-optimising your agents... for those curated codebases. The moment you step outside of those curated codebases and run the agents against non-curated codebases?
Well, no one knows. They may or may not work because the actual codebase may be similar to, or may be completely different from the curated one.
And how do I know that it may not work? Well, let's turn to our friends at Anthropic: https://www.anthropic.com/engineering/a-postmortem-of-three-...
--- start quote ---
When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution. We use "top-p sampling" to avoid nonsensical outputs—only considering words whose cumulative probability reaches a threshold (typically 0.99 or 0.999). On TPUs, our models run across multiple chips, with probability calculations happening in different locations. To sort these probabilities, we need to coordinate data between chips, which is complex
--- end quote ---
So it's a probabilistic next word (which is quite likely to be different for a non-curated codebase), and there's top sampling, and then the complex sorting of probabilities, and on top of that are all the changes and bugs and limits and input/output transforms that Anthropic introduces.
> share your specific case and I'll help you set it up.
I have several side projects in Elixir. And at work we're developing a product that runs across 14 different (but similar, but different enough) platforms using the company's proprietary services.
It's especially funny to see the claims of "oh just one more fine-tuning, bro, and everything will be gazillion times better" when I have already used and found issues with every "diligently researched" "guaranteed eval'ed" hype tool under the sun. This is just one of the results: https://x.com/dmitriid/status/1967306828418818217
Yours are unlikely to be any different.
Well, no one knows. They may or may not work because the actual codebase may be similar to, or may be completely different from the curated one.
And how do I know that it may not work? Well, let's turn to our friends at Anthropic: https://www.anthropic.com/engineering/a-postmortem-of-three-...
--- start quote ---
When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution. We use "top-p sampling" to avoid nonsensical outputs—only considering words whose cumulative probability reaches a threshold (typically 0.99 or 0.999). On TPUs, our models run across multiple chips, with probability calculations happening in different locations. To sort these probabilities, we need to coordinate data between chips, which is complex
--- end quote ---
So it's a probabilistic next word (which is quite likely to be different for a non-curated codebase), and there's top sampling, and then the complex sorting of probabilities, and on top of that are all the changes and bugs and limits and input/output transforms that Anthropic introduces.
> share your specific case and I'll help you set it up.
I have several side projects in Elixir. And at work we're developing a product that runs across 14 different (but similar, but different enough) platforms using the company's proprietary services.
It's especially funny to see the claims of "oh just one more fine-tuning, bro, and everything will be gazillion times better" when I have already used and found issues with every "diligently researched" "guaranteed eval'ed" hype tool under the sun. This is just one of the results: https://x.com/dmitriid/status/1967306828418818217
Yours are unlikely to be any different.
A vibe article on vibe coding.
This reads like an advert for Continue.dev
Good: I can prototype things very quickly thanks to these tools
Bad: After couple of vibe coding iterations, I don't have a mental model of the project.
Good: When I open my past projects where I have very good mental models, I can come up with a nice prompt and build anything quickly again.
Bad: After couple of iterations I become lazy, and eventually my mental models break.
There's definitely a use for these tools. But be careful, job of engineers are not only coding but also training their memory to build solutions and bridge real world problem with software solution. If you lose this skill of thinking, you will be obsolete quickly