Computer Architects Can't Find the Average(dgsq.net)
dgsq.net
Computer Architects Can't Find the Average
https://dgsq.net/2025-04-27-averages/
32 comments
Averages are the mind-killer. Don't show me the average, show me the distribution. Any time that you present an average over a data set that isn't approximately a normal distribution, you're committing statistical malpractice.
> Don't show me the average, show me the distribution.
Put another way, don't give me a measure of center (mean) without a corresponding measure of spread (variance/standard deviation).
Providing a measure of center for something that is approximately normal doesn't obviate the need for a measure of spread--you could have two distributions with the same mean but wildly different spread of results due to variance.
Put another way, don't give me a measure of center (mean) without a corresponding measure of spread (variance/standard deviation).
Providing a measure of center for something that is approximately normal doesn't obviate the need for a measure of spread--you could have two distributions with the same mean but wildly different spread of results due to variance.
I think the GP wants to see the actual distribution graph.
Yes, single scalar values tell you nothing useful if you have no a priori knowledge of the shape of your distribution. Mean and variance are meaningless figures if your distribution is multimodal, for example. If you truly have to compress the distribution into a few numbers, then the best thing to do is to represent it as a series of quantile values. In cases where the distribution is unknown, evenly spaced quantile values are a good start.
This is the underlying flaw of Ground News and both-sides-ism shifting the Overton Window right.
Figuring out which of two things is bigger is hard. The article lightly touches on the two main reasons:
- Sometimes the definition is complicated. E.g., I'd accept a computer 10x slower at most tasks if a particular small subset were sped up even 2x. No symmetric mean (geometric, arithmetic, harmonic, or otherwise) will appropriately capture that.
- Nobody agrees on the definition. Even for very regular workloads, parsing protobufs is very different from old-school ML is very different from reversing the endianness of an unaligned data fragment. Venturing into even more diverse workloads muddies the waters further.
The article basically gives up and says that the geometric mean is the least bad option because it has some nice properties (ones I don't think anyone cares about for this purpose...) and because it's what everyone expects.
That latter argument is at least based in something sound, but I think the real point here is that attempting to come up with a single definitive "speed" is foolhardy in the first place. The only point is to compare two architectures, but without a particular context in mind no such linear ordering exists, and that's before we get to the difficulties in defining it appropriately even when given a context.
- Sometimes the definition is complicated. E.g., I'd accept a computer 10x slower at most tasks if a particular small subset were sped up even 2x. No symmetric mean (geometric, arithmetic, harmonic, or otherwise) will appropriately capture that.
- Nobody agrees on the definition. Even for very regular workloads, parsing protobufs is very different from old-school ML is very different from reversing the endianness of an unaligned data fragment. Venturing into even more diverse workloads muddies the waters further.
The article basically gives up and says that the geometric mean is the least bad option because it has some nice properties (ones I don't think anyone cares about for this purpose...) and because it's what everyone expects.
That latter argument is at least based in something sound, but I think the real point here is that attempting to come up with a single definitive "speed" is foolhardy in the first place. The only point is to compare two architectures, but without a particular context in mind no such linear ordering exists, and that's before we get to the difficulties in defining it appropriately even when given a context.
Isn't this why benchmarking is used? You select some common payload and then use that as a realworld simulation of how your architecture performs with it.
Of course this comes with its own can of worms, like overfitting and such, but I could imagine a benchmarking solution that gives you a more granular look at which specific tasks an architecture performs well.
Of course this comes with its own can of worms, like overfitting and such, but I could imagine a benchmarking solution that gives you a more granular look at which specific tasks an architecture performs well.
Or even better just measure your actual workload.
Yeah
For the specific purpose of what the article presents, I feel something simpler like 10%/50%/90% times can be better (two systems that have times as 0.1/1/10 or 0.9/1/1.1 will have "the same average" but where one might be adequate the other one might not)
For the specific purpose of what the article presents, I feel something simpler like 10%/50%/90% times can be better (two systems that have times as 0.1/1/10 or 0.9/1/1.1 will have "the same average" but where one might be adequate the other one might not)
Metrics can be useful when they are thoughtfully chosen for the context in which they are used. The more abstract or synthetic a benchmark is, the less context there is to inform such a choice. Said another way, metrics characterizing synthetic benchmarks inherit/expose one of the key problems of synthetic benchmarks - a lack of context.
In practical settings, the way to characterize performance will be more clear from context, often giving one (-ish) clear metric for each dimension you care about. For example, if an endpoint has a hard timeout of 100 ms, then it's pretty interesting to look at the percentage of real world requests that are faster than 100 ms. If the same solution is also used in a setting where throughput is more important than latency, then an additional metric is probably needed for that use case. Multiple metrics are needed for multiple use cases to capture trade-offs being made between those use cases.
In practical settings, the way to characterize performance will be more clear from context, often giving one (-ish) clear metric for each dimension you care about. For example, if an endpoint has a hard timeout of 100 ms, then it's pretty interesting to look at the percentage of real world requests that are faster than 100 ms. If the same solution is also used in a setting where throughput is more important than latency, then an additional metric is probably needed for that use case. Multiple metrics are needed for multiple use cases to capture trade-offs being made between those use cases.
This kind of debate isn't that unusual, is it? To figure out what calculation to use, one has to made tradeoffs about desirable properties such as resilience to outliers, being bias-free, and/or having the best predictive performance in least-squared error. Bringing some grizzled statisticians into the conversation will probably clear a lot up. They are familiar with the pros and cons of various measures of central tendency.
In the era of multivariate models, machine learning, and AI, some of the classic wisdom from good-old linear modeling gets overlooked.
In the era of multivariate models, machine learning, and AI, some of the classic wisdom from good-old linear modeling gets overlooked.
Ironically, AI research canonically uses arithmetic mean for averaging basically everything. Whatever suite of experiments you are running, somebody will sooner or later ask you to average them.
Sometimes, you can't find the average because it's undefined: it can happen with a Cauchy and a few other statistical distributions: the wikipedia page has a nice plot of how the first 2 moments don't converge https://en.wikipedia.org/wiki/Cauchy_distribution#History
When in doubt, don't use the mean: prefer more robust estimates, as even with degenerate statistical distributions, there are still some "good numbers to report" like the mode or the median.
And if you don't know statistics, just use a plot!
When in doubt, don't use the mean: prefer more robust estimates, as even with degenerate statistical distributions, there are still some "good numbers to report" like the mode or the median.
And if you don't know statistics, just use a plot!
Indeed, the best averaging method depends on the underlying probability distribution from which data is drawn. Arithmetic is best for normal whereas geometric is better suited for lognormal distribution, and as the above comment suggests average is meaningless for most power law distributions where exponent is less than 2.
However, When all else fails, define your own Von Neumann entropy. Figure out how often you compile GCC, FFT, or video compression, then compute probabilities (ratios) and multiply by logarithm of speedups for each use case. Sum them up and report it as machine/architecture entropy and you'll win every argument about it.
However, When all else fails, define your own Von Neumann entropy. Figure out how often you compile GCC, FFT, or video compression, then compute probabilities (ratios) and multiply by logarithm of speedups for each use case. Sum them up and report it as machine/architecture entropy and you'll win every argument about it.
I agree with your point, but it is funny to think about true consumer workloads: I... mostly JIT and run Javascript, layout algorithms, and whatever compositing cannot be offloaded to the GPU.
Yep, I always end up using a bunch of graphs because picking one number feels off every time. Honestly, stats can get slippery real fast if you don't watch it.
I disagree with the article's claim that the geometric mean lacks physical meaning.
Say you have two benchmarks with different units, frames/second and instructions/second. You can't take the arithmetic mean of these unless you divide by some baseline first (or a conversion factor).
But the geometric mean has well-defined units of (frames * instructions)^.5/second. And the reason you can divide by another geometric mean is because these units are always the same.
Having coherent units isn't exactly the same as "physical meaning", but it's a prerequisite at the least.
Say you have two benchmarks with different units, frames/second and instructions/second. You can't take the arithmetic mean of these unless you divide by some baseline first (or a conversion factor).
But the geometric mean has well-defined units of (frames * instructions)^.5/second. And the reason you can divide by another geometric mean is because these units are always the same.
Having coherent units isn't exactly the same as "physical meaning", but it's a prerequisite at the least.
Geometric average is just estimating the mean of logarithmic units. Units of measurement are arbitrary, and justification for using logarithmic units for statistical purposes can be easily made with distributional considerations. E.g. durations are bounded to be positive, so there will be at least some skew in the distribution (although this can be negligible in practice).
Given only speedups any formula to combine them is meaningless, because the real question involves a concrete workload: would the performance of what I'm going to do with my computer improve enough to justify the cost of improving it?
At the very least, weights for each benchmark task are needed; complications of the cost model (beyond a binary old system/new system comparison) are also likely.
Measuring performance improvement by speedup of throughput is also often naive, since there are other dimensions (like power and latency) and complex requirements.
At the very least, weights for each benchmark task are needed; complications of the cost model (beyond a binary old system/new system comparison) are also likely.
Measuring performance improvement by speedup of throughput is also often naive, since there are other dimensions (like power and latency) and complex requirements.
Physical interpretation of geometric mean… you have a collection of multiplicative speedups, and some of them apply to any given workload. I think the geometric mean falls out of the resulting statistical distribution in a natural way but I’m sleep-deprived at the moment and can’t write out the math. Like, in this distribution geomean is the median speedup.
Think of the case with two values, 2.00 and 2.42. Decompose it into three speedups… 2x, 1.1x, and 1.1x. The speedup 2x happens with probability 1. Each of the 1.1x speedups happen with probability 0.5. The geomean is 1 2x speedup and 1 1.1x speedup, giving 2.2x.
There are many such decompositions, that one is not unique. Exercise for the reader to show which conditions give you the geometric mean and explain why that is reasonable… I’m terribly sleep-deprived at the moment and this is where I stop.
There is of course no correct choice of mean here, just a bunch of different choices with different interpretations justifying them.
Think of the case with two values, 2.00 and 2.42. Decompose it into three speedups… 2x, 1.1x, and 1.1x. The speedup 2x happens with probability 1. Each of the 1.1x speedups happen with probability 0.5. The geomean is 1 2x speedup and 1 1.1x speedup, giving 2.2x.
There are many such decompositions, that one is not unique. Exercise for the reader to show which conditions give you the geometric mean and explain why that is reasonable… I’m terribly sleep-deprived at the moment and this is where I stop.
There is of course no correct choice of mean here, just a bunch of different choices with different interpretations justifying them.
Your favorite stock doubled in 6 months. What was it growing on average in a day?
An “average” (whether a mean/ median etc) is a very lossy compression algorithm.
You’re attempting to describe a whole series of numbers with just one (or two) numbers.
Trying to come up with a good general purpose way to reduce/compress/aggregate data via a lossy algorithm is intractable.
While that all might sound obvious, it can be very hard to internalise this.
(And that’s before getting into the motivated reasoning that biased actors [aka normal people] will use to preference one lossy algorithm over another)
You’re attempting to describe a whole series of numbers with just one (or two) numbers.
Trying to come up with a good general purpose way to reduce/compress/aggregate data via a lossy algorithm is intractable.
While that all might sound obvious, it can be very hard to internalise this.
(And that’s before getting into the motivated reasoning that biased actors [aka normal people] will use to preference one lossy algorithm over another)
You can use an increasing number of statistical moments.
https://en.wikipedia.org/wiki/Moment_(mathematics)
The arithmetic mean is one of them, which would be an argument in favor of it.
https://en.wikipedia.org/wiki/Moment_(mathematics)
The arithmetic mean is one of them, which would be an argument in favor of it.
I don't think it is an argument in favour of it.
arith-mean = E[x] , the first moment of x geo-mean = exp(E[log x]) , so log geo-mean = first moment of log x
They are both equivalent in amount of information preserved, but arithmetic preserves additive structure whereas geo preserves multiplicative structure
arith-mean = E[x] , the first moment of x geo-mean = exp(E[log x]) , so log geo-mean = first moment of log x
They are both equivalent in amount of information preserved, but arithmetic preserves additive structure whereas geo preserves multiplicative structure
I didn't want to imply that there is a loss of information. Yes, it's one level up in the hyperoperation chain, I suppose, but what I meant is that it's not a typical way of doing statistics, especially the higher moments.
This is very relevant for compiler optimization and video codecs as well - both involve testing changes on lots of small benchmarks. Almost any change will hurt some benchmarks and help others. Maybe one benchmark improves a lot while causing a small slowdown on many others. The overall improvement for any one change can be minuscule, less than one percent. And yet if you keep doing these small improvements it adds up, hopefully to the point of improving every benchmark over time.
I'm using median because it is more robust against outliers.
But you often want a number that’s more correlated with the average. Imagine a distribution that has 80% very low values and 20% insanely high values. The median won’t capture changes in that 20% region at all. So you could see numbers not changing when in fact they may have very dramatic changes
The linked paper has some tables that explain why this matters: How not to Lie with Statistics: The Correct Way to Summarize Benchmark Results https://dl.acm.org/doi/pdf/10.1145/5666.5673
it seems pretty similar to voting problems in political elections. No matter the algorithm you choose, someone will complain.
arithmetic mean of percentages = terrorism