In general, yes. I believe there's a difference between writing a performance review based on your memory/notes of how someone was doing 6 months ago vs getting into the weeds of a sample of PRs/specs and reviewing those from a performance viewpoint.
Author here. This is valid feedback, thank you. This post was originally an internal post. Internally, I have credibility as "the guy that does his homework". That doesn't carry over to the broader internet!
I think you're right that the data presented doesn't prove or disprove the thesis that "customer issues remain broadly stable". I think it lends credibility to the idea that "AI didn't lead to an obvious and immediate slopfest of bugs".
As for the bumps, we've noticed them annually and we haven't yet found a solid reason why. There’s no correlation with customer growth, PRs shipped, or features shipped. It's likely a combination of various seasonal effects in our own hiring and customer onboarding. What isn't in this article: we've more than doubled our customer base in this time period, our customers are more mature and raise bugs more readily, we've got ~60% more engineers, and our product surface area has grown a lot too. I'm yet to land on a satisfying metric that normalizes bug rates in a changing environment.
Author here! That opt out issue is definitely not intentional, I just gave it a quick try and it worked for me in a few different browsers. Feel free to ping me on colin AT ashbyhq.com with any additional details and I'll see if we can get a reproduction.
We only hire engineering managers that are deeply technical. This gives us more ability to work closely with engineers and make sure they're using new tools in the ways we want. It's not perfect. Sometimes, it goes wrong. But, that's why we continuously add tests and other tools to help us spot mistakes.
Author here. 100% agree with your sentiment. When we do performance reviews of engineers we take the time to comb over at least some of the PRs the engineer has written. We flag any patterns around code quality (regardless of whether it came from AI or not). This is on top of any other ongoing feedback we're giving.
E.g. in a recent review I wrote I picked up on an engineer not splitting up React components aggressively enough and over-relying on deeply nested ternaries. In that same review I also commended the engineer for being particularly excellent at explaining complex bugs to our support people and called out one example of a nasty race condition.