I thought arenas were one of the simplest and most easy to deal with and conceptualize memory management strategies around. Arguably easy, even easier to understand and just as easy to manage as GC. Did they do something special?
I'm saying Claude didn't introduce any more bugs than the human maintainers of rsync had in the past, not that it introduced no bugs at all. That "arbitrary amount" is the historical amount of bugs. That's why I'm confused. You're completely missing my point and talking past me. Yes, Claude may have introduced a bug. That doesn't change that it doesn't really matter.
Additionally, I quoted Tridge in response to a comment about an increase in changes to rsync, not in response to the person pointing at one bug Claude introduced. If you actually looked at the thread, you'd see that. I didn't deny the Claude introduced bug at all.
Whether commits decrease the sev/10c depends on if there are a lot of small commits increasing the demoninator. In reality, we have the opposite: the post-Claude releases have way fewer commits than the pre-Claude ones.
Thus, if anything their sev/10c is inflated. If I changed it to lines of code changed, the relative bug ratios would be much smaller, and the conclusion wouldn't change. In fact, the conclusion would look "better" for Claude; if I was using "mental gymnastics" to come to this conclusion, I would have already used a metric other than adjusting per commits!
What different metric would you suggest that would change the conclusion?
Showing "humility", as you so moralistically and condescendingly put it, would require being wrong first.
The statistical methodology I used is mine. As is the interpretation. Completely. To the degree that I misunderstood statistics (and it is under debate even in the thread you link, and the people accusing me of misunderstanding statistics there are universally misrepresenting my point, which is to point out a total absence of evidence for any difference, not to prove the null hypothesis) that's on me
Okay, so let me get this straight. Because I used AI to, among other things, write the prose of the original draft of this article, all of the days of effort researching and carefully thinking through the metrics I would use, and the methodologies to analyze them, and rewriting the entire analysis multiple times from scratch based on specifically asking people in my life who are qualified in statistics what I should do and trying to come up with the fairest analysis I could with the little data available doesn't matter at all? The post just have some sort of essential AI nature that makes it low effort just because one aspect of it didn't have sufficient effort put into it for you personally?
And once it's originally posted, it doesn't matter the great extent I go to address metrics and methodological critiques in order to ensure that the data is as robust and helpful as possible. And the effort in writing and refining my prose and the organization of the report in response to people's complaints and criticisms because I do value their time. And when people told me the AI prose was bad, I spent two hours to to make sure that it was something people would want to read, that doesn't matter at all? It's only the original intention that matters. So you just have this arbitrary cutoff point for what counts towards my intentions in the post and my character. No allowance for learning or adaptation, and the fact that I'm clearly committed to putting a lot of effort into making this something that is useful and pleasant to read for people, I just didn't do it for the first draft originally, doesn't matter, only the original version matters?
And more than that, you're not going to actually deal with the substance of the issue, the actual calculations and methodology and conclusions that I came to, instead, the only semi-substantive critique you're going to make of the post is to tone police me and dance around the real issues, as if you're afraid of ever touching them?
The best argument you could make that my bias actually influenced my conclusions would be to point into the methodology and metrics where I did that. I made it all extremely open and transparent and auditable both by describing it in extreme detail in the post and by providing all of my source code and the ability to build the database it runs on from scratch. If there was an actual flaw or bias that my intentions going into this created your biggest possible Smackdown, your best weapon in your arsenal would be to actually point that out. But instead, again, you're just tone policing me. but a polemical style in the presentation of an objective statistical analysis does not in the least undercut its accuracy. Have you considered that my polemic became so fiery, in fact, precisely because I ran the tests and found how non-existent the evidence was for this outrage and that's what made me angry? No, you didn't because you saw some words that hurt your feelings and now you won't listen to facts.
If they're the statistical methods and metrics hold up, or they don't. Also, if you don't want to read my opinion on things, then just grab the GitHub repo and run the end-to-end replication and look at the output data yourself.
> Why was v3.4.1 the most buggy, right before the Claude commits? Why did "nobody notice"? It's way to strange to just say welp, it must be human error.
Why wouldn't it be except question begging priors assuming it couldn't be?
> Why does v3.4.2 have 0 bugs, or 0 bug score. And why was such an outlier (no other commit seemingly has this??) allowed to mix into aggregate statistics and bring all the "is Claude buggy?" scores down.
My original metrics which didn't filter out feature requests and questions had it at four bugs and prior to that it was even higher and it didn't make much of a difference to the overall analysis (fell well within the IQR, the lower end of it too). Also, removing one outlier just because it looks kind of funny to you, especially when we only have two Claude releases at all, would be worse in my opinion and more arbitrary.
"Like many developers of open source packages I’ve been hit by a flood of security reports lately in my role as the rsync maintainer. Many of those reports are AI generated (not all though, there are some notable ones with very careful and high quality manual analysis).
As this flood started to get more intense I realised I needed to raise the defences on rsync a lot — we needed much more thorough test suites, code coverage analysis, CI testing on a lot more platforms, deliberate and thorough scanning for possible security issues (so I find at least some of them before other people!) and the addition of a whole lot of defence-in-depth hardening techniques. This is all a huge amount of work. "
> This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecdotes with zero empirical evidence to back it up.
Let's go back to remedial classes on this one.
"I have found that [tool] has made me more effective" is what we call lived experience. It is an "i" statement communicating something about the person’s life. It does not require evidence by default, and you are a crazy person if you call bullshit without good reason, because many "I" statements are epistemically justified in ways that can't be empirically demonstrated or require tacit knowledge.
"[tool] has been buggier since [change]" is a falsifiable claim; you need to actually provide evidence for believing it, and what I'm showing is literally that there isn't any.
> This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecdotes with zero empirical evidence to back it up. It's way, way too late to claim hypocrisy here. As I stated under the original submission about this topic, irrational anti-AI behavior is usually just an equal and opposite reaction to irrational pro-AI behavior.
I'm talking about the double standard on the anti-AI side about what evidence should count, not some vague industry-wide epistemic standard, whatever that means. I'm aware LinkedIn Lunatics and Steve Yegge are also being crazy. And it seems to me that even your response here is engaging in a bit of a double standard, or something akin to it, in that you think the irrational anti-AI behavior should be given a pass — and the conclusions perhaps even taken seriously — just because pro-AI people did it too.
> And that doesn't help. If anything, editing the AI output to make it read less like blatant slop just comes off as deceptive, like you're trying to hide the fact that the analysis was AI generated.
Okay, so, if I don't spend the time to write everything myself, that's bad because it's AI slop. If I do rewrite everything myself, then it's evidence of deceptiveness... despite being asked by multiple people to do that, and being extremely explicit about my methods and process and the commit history being (as you've shown), very public.
Also, the AI-generatedness of the text doesn't mean the analysis is AI generated, in terms of what was actually done. That's a category error.
> Looking at the commits, you were adding more AI generated text less than 2 hours ago[0] before quickly editing out one of the most blatantly sloppy sentences I've ever read[1].
The second commit literally says that that was my prose it was fucking with by adding slop. It's just that me adding my prose, and it adding slop to it, were in the same previous commit. Additionally, my process is often giving it exactly what I want to say, more or less, and having it HTML-format it and insert the templated numbers and UI widgets around that text.
But again, even if I'm spending the time to read through and edit everything it's writing to de-slop it, then I'm clearly also reading it through enough to make sure the analysis makes sense, and is accurate; how is that not enough "effort" for you, if effort is supposed to be a proxy for verification?
> Even if we ignore the bias clearly on display there, the premise alone is enough to dismiss the entire thing as heavily biased and chasing a pre-determined conclusion - of course someone who is so dependent and trustful of AI that they decide such an analysis on the bugginess of AI code should itself be written by AI is going to steer the conclusion towards "actually AI code is good and you luddites are overreacting".
That's not ignoring the bias, that's literally restating that you think the bias is there. But if you really think that my bias meaningfully "steered the results," then show me how that happened. Tell me how you would've proven the Claude releases were meaningfully worse, or unusual, at all, or how the methods I chose biased the data against that result, or literally anything except shifting the goalposts and using accusations of "bias" as a get-out-of-jail-free-card.
> The entire concept is so tone-deaf that failing to notice it or predict the criticism before publishing is enough to prove the bias.
And you're so committed to your preconceived notions that anything made with AI must be bad, wrong, or not worth your time, that you'll spend your entire time begging the question ("it's made with AI, therefore it's wrong") and shifting the goalposts instead of engaging meaningfully.
Also, I certainly predicted the criticism (in general, anyway, to the fact that it was made with AI; not the prose being AI) but I made it this way anyway, because if someone is so AI-blinded that they can't read and evaluate the actual metrics, methodology, and provide meaningful criticism to it, and instead can only see that it was made with AI, and they're so it doesn't matter.
Nothing you have said makes the analysis wrong. At this point, you're essentially just resorting to ad homenem and begging the question.
Edited that claim, and made several clarifications elsewhere. The whole point of this analysis is that outrage is unjustified on the basis of two totally statistically unremarkable releases that no one would have remarked on pre-AI (my further proof of this is that there was a pre-AI remarkably broken release, and no one did comment!) and zero positive evidence outside cherry-picked anecdotes for any negative impact. We should wait for outrage and version pinning and cancelation until there is evidence, no? I'm just trying to say that these specific releases are unremarkable, and there's no evidence at all of harm currently; I'm not trying to build any kind of predictive model for future Claude releases to say anything grander than "these specific releases are fine, what are we freaking out about?", not some claim about what Claude-exposed releases will look like or trend like in the future or in general.
> This analysis showed that there is indeed an absence of evidence, but it concludes there is evidence of absence.
I tried pretty hard to avoid saying that, can you point me at how to rephrase? The point I'm trying to make is just that there is absolutely no evidence at all for what people are saying with such absolutism and claimed objectivity (that Claude made rsync worse), and thus it doesn't justify the outrage.
> Under-sampling, and concluding with p > 0.05
How would I avoid under-sampling here? And if you're going to say it's because I only have 2 data points, well, the side making the positive claim — that Claude made rsync worse — only had two as well, and unremarkable ones at that, as I've tried very hard to show.