Everything is correlated (2014–23)(gwern.net)
gwern.net
Everything is correlated (2014–23)
https://gwern.net/everything
129 comments
You're spot on that significant ≠ meaningful effect. But I'd push back slightly on the example. A very low p-value doesn't always imply a meaningful effect, but it's not independent of effect size either. A p-value comes from a test statistic that's basically:
(effect size) / (noise / sqrt(n))
Note that bigger test statistic means smaller p-value.
So very low p-values usually come from bigger effects or from very large sample sizes (n). That's why you can technically get p<0.001 with a microscopic effect, but only if you have astronomical sample sizes. In most empirical studies, though, p<0.001 does suggest the effect is going to be large because there are practical limits on the sample size.
(effect size) / (noise / sqrt(n))
Note that bigger test statistic means smaller p-value.
So very low p-values usually come from bigger effects or from very large sample sizes (n). That's why you can technically get p<0.001 with a microscopic effect, but only if you have astronomical sample sizes. In most empirical studies, though, p<0.001 does suggest the effect is going to be large because there are practical limits on the sample size.
The challenge is that datasets are just much bigger now. These tools grew up in a world where n=2000 was considered pretty solid. I do a lot of work with social science types, and that's still a decent sized survey.
I'm regularly working with datasets in the hundreds of thousands to millions, and that's small fry compared with what's out there.
The use of regression, for me at least, is not getting that p-gotcha for a paper, but as a posh pivot table that accounts for all the variables at once.
I'm regularly working with datasets in the hundreds of thousands to millions, and that's small fry compared with what's out there.
The use of regression, for me at least, is not getting that p-gotcha for a paper, but as a posh pivot table that accounts for all the variables at once.
There’s a common misconception that high throughput methods = large n.
For example, I’ve encountered the belief that just by recording something at ultra high temporal resolution gives you “millions of datapoints”. This then has all sorts of effects on the breakdown of statistics and hypothesis testing (seemingly).
In reality, the replicability of the entire setup, the day it was performed, the person doing it, etc. means the n for the day is probably closer to 1. So to ensure replicability you’d have to at least do it on separate days, with separately prepared samples. Otherwise, how can you eliminate the chance that your ultra finicky sample just happened to vibe with that day’s temperature and humidity?
But they don’t teach you in statistics what exactly “n” means, probably because a hundred years ago it was much more literal in nature. 100 samples is because I counted 100 mice, 100 peas, or 100 surveys.
For example, I’ve encountered the belief that just by recording something at ultra high temporal resolution gives you “millions of datapoints”. This then has all sorts of effects on the breakdown of statistics and hypothesis testing (seemingly).
In reality, the replicability of the entire setup, the day it was performed, the person doing it, etc. means the n for the day is probably closer to 1. So to ensure replicability you’d have to at least do it on separate days, with separately prepared samples. Otherwise, how can you eliminate the chance that your ultra finicky sample just happened to vibe with that day’s temperature and humidity?
But they don’t teach you in statistics what exactly “n” means, probably because a hundred years ago it was much more literal in nature. 100 samples is because I counted 100 mice, 100 peas, or 100 surveys.
I learned about experiment design in statistics, so I wouldn’t blame statisticians for this.
There’s a lot of folks out there though who learned the mechanics of linear regression in a bootcamp or something without gaining an appreciation for the underlying theories, and those folks are looking for low p-value and as long as they get it it’s good enough.
I saw this link yesterday and could barely believe it, but I guess these folks really live among us.
https://stats.stackexchange.com/questions/185507/what-happen...
There’s a lot of folks out there though who learned the mechanics of linear regression in a bootcamp or something without gaining an appreciation for the underlying theories, and those folks are looking for low p-value and as long as they get it it’s good enough.
I saw this link yesterday and could barely believe it, but I guess these folks really live among us.
https://stats.stackexchange.com/questions/185507/what-happen...
This is an interesting point. I've been trying to think about something similar recently but don't have much of an idea how to proceed. I'm gathering periodic time series data and am wondering how to factor in the frequency of my sampling for the statistical tests. I'm not sure how to assess the difference between 50Hz and 100Hz on the outcome, given that my periods are significantly longer. Would you have an idea of how to proceed? The person I'm working with currently just bins everything in hour long buckets and uses the mean for comparison between time series but this seems flawed to me.
I don't know if you'll be reading this, but my first intuition would be to determine my effective sampling rate, and determine if samples are comparable at all in the first place.
For example, if your phenomenon is observable at 50 Hz, maybe even 10 Hz, then any higher temporal resolution does not give you new information, because any two adjacent datapoints in the time-series are extremely correlated. Going the other way, at a very low sampling frequency you'd just get the mean, which might not reveal anything of interest.
If you bin 100 Hz data at 50 Hz, are they the same? Is the Fourier spectrum the same? If you have samples of different resolution you must choose the lowest common denominator for a fair statistical comparison. Otherwise, a recording between a potato and an advanced instrument would always be "statistically different", which doesn't make sense.
If you don't find "anything", the old adage goes "the absence of evidence is not the evidence of absence", so statistics don't really fail here. You can only conclude that your method is not sensitive enough.
For example, if your phenomenon is observable at 50 Hz, maybe even 10 Hz, then any higher temporal resolution does not give you new information, because any two adjacent datapoints in the time-series are extremely correlated. Going the other way, at a very low sampling frequency you'd just get the mean, which might not reveal anything of interest.
If you bin 100 Hz data at 50 Hz, are they the same? Is the Fourier spectrum the same? If you have samples of different resolution you must choose the lowest common denominator for a fair statistical comparison. Otherwise, a recording between a potato and an advanced instrument would always be "statistically different", which doesn't make sense.
If you don't find "anything", the old adage goes "the absence of evidence is not the evidence of absence", so statistics don't really fail here. You can only conclude that your method is not sensitive enough.
Depending on the nature of the study, there's lots of scientific disciplines where it's trivial to get populations in the millions. I got to see a fresh new student's poster where they had a p-value in the range of 10^-146 because every cell in their experiment was counted as it's own sample.
https://pmc.ncbi.nlm.nih.gov/articles/PMC3444174/
> Using Effect Size—or Why the P Value Is Not Enough
> Statistical significance is the least interesting thing about the results. You should describe the results in terms of measures of magnitude –not just, does a treatment affect people, but how much does it affect them.
– Gene V. Glass
> Using Effect Size—or Why the P Value Is Not Enough
> Statistical significance is the least interesting thing about the results. You should describe the results in terms of measures of magnitude –not just, does a treatment affect people, but how much does it affect them.
– Gene V. Glass
Agreed. However, I think you're being overly charitable in calling it a "pet peeve", it's more like a pathological misunderstanding of stats that leads to a lot of bad outcomes especially in popular wellness media.
As an example, read just about any health or nutrition research article referenced in popular media and there's very often a pretty weak effect size even though they've achieved "statistical significance." People then end up making big changes to their lifestyles and habits based on research that really does not justify those changes.
As an example, read just about any health or nutrition research article referenced in popular media and there's very often a pretty weak effect size even though they've achieved "statistical significance." People then end up making big changes to their lifestyles and habits based on research that really does not justify those changes.
^
And if we increase N enough we will be able to find these 'good measurements' and 'statistically significant differences' everywhere.
Worse still if we did not agree in advance what hypotheses we were testing, and go looking back through historical data to find 'statistically significant' correlations.
And if we increase N enough we will be able to find these 'good measurements' and 'statistically significant differences' everywhere.
Worse still if we did not agree in advance what hypotheses we were testing, and go looking back through historical data to find 'statistically significant' correlations.
Which means that statistical significance is really a measure of whether N is big enough
This has been known ever since the beginning of frequentist hypothesis testing. Fisher warned us not to place too much emphasis on the p-value he asked us to calculate, specifically because it is mainly a measure of sample size, not clinical significance.
Yes the whole thing has been a bit of a tragedy IMO. A minor tragedy all things considered, but still one nonetheless.
One interesting thing to keep in mind is that Ronald Fisher did most of his work before the publication of Kolmogorov's probability axioms (1933). There's a real sense in which the statistics used in social sciences diverged from mathematics before the rise of modern statistics.
So there's a lot of tradition going back to the 19th century that's misguided, wrong, or maybe just not best practice.
One interesting thing to keep in mind is that Ronald Fisher did most of his work before the publication of Kolmogorov's probability axioms (1933). There's a real sense in which the statistics used in social sciences diverged from mathematics before the rise of modern statistics.
So there's a lot of tradition going back to the 19th century that's misguided, wrong, or maybe just not best practice.
It's not, that would be quite the misunderstanding of statistical power.
N being big means that small real effects can plausibly be detected as being statistically significant.
It doesn't mean that a larger proportion of measurements are falsely identified as being statistically significant. That will still occur at a 5% frequency or whatever your alpha value is, unless your null is misspecified.
N being big means that small real effects can plausibly be detected as being statistically significant.
It doesn't mean that a larger proportion of measurements are falsely identified as being statistically significant. That will still occur at a 5% frequency or whatever your alpha value is, unless your null is misspecified.
It's standard to set the null hypothesis to be a measure zero set (e.g. mu = 0 or mu1 = mu2). So the probability of the null hypothesis is 0 and the only question remaining is whether your measurement is good enough to detect that.
But even though you know the measurement can't be exactly 0.000 (with infinitely many decimal places) a priori, you don't know if your measurement is any good a priori or whether you're measuring the right thing.
But even though you know the measurement can't be exactly 0.000 (with infinitely many decimal places) a priori, you don't know if your measurement is any good a priori or whether you're measuring the right thing.
The probability is only zero a.s., it's not zero. That's a very big difference. And hypothesis tests aren't estimating the probability of the null being true, they're estimating the probability of rejecting the null if the null was true.
It's less of a big difference than it might seem, because it takes infinitely long to specify a real number to infinite precision. If you think about something like trying to tell if you hit the exact center of the bullseye, you eventually get down to the quantum mechanical scale and you find that the idea of an atom being in the exact center isn't even that well defined.
In a finite or countable number of trials you won't see a measure zero event.
> they're estimating the probability of rejecting the null if the null was true.
Right, but the null hypothesis is usually false and so it's a weird thing to measure. It's a proxy for the real thing you want, which is the probability of your hypothesis being true given the data. These are just some of the reasons why many statisticians consider the tradition of null hypothesis testing to be a mistake.
In a finite or countable number of trials you won't see a measure zero event.
> they're estimating the probability of rejecting the null if the null was true.
Right, but the null hypothesis is usually false and so it's a weird thing to measure. It's a proxy for the real thing you want, which is the probability of your hypothesis being true given the data. These are just some of the reasons why many statisticians consider the tradition of null hypothesis testing to be a mistake.
I really like this video [1] from 3blue1brown, where he proposes to think about significance as a way to update the probability. One positive test (or in this analog a study) updates the probability by X % and thus you nearly always need more tests (or studies) for a 'meaningful' judgment.
[1] https://www.youtube.com/watch?v=lG4VkPoG3ko
[1] https://www.youtube.com/watch?v=lG4VkPoG3ko
To add nuance, it is not that bad. Given reasonable levels of statistical power, experiments cannot show meaningless effect sizes with statistical significance. Of course, some people design experiments at power levels way beyond what's useful, and this is perhaps even more true when it comes to things where big data is available (like website analytics), but I would argue the problem is the unreasonable power level, rather than a problem with statistical significance itself.
When wielded correctly, statistical significance is a useful guide both to what's a real signal worth further investigation, and it filters out meaningless effect sizes.
A bigger problem even when statistical significance is used right is publication bias. If, out of 100 experiments, we only get to see the 7 that were significant, we already have a false:true ratio of 5:2 in the results we see – even though all are presented as true.
When wielded correctly, statistical significance is a useful guide both to what's a real signal worth further investigation, and it filters out meaningless effect sizes.
A bigger problem even when statistical significance is used right is publication bias. If, out of 100 experiments, we only get to see the 7 that were significant, we already have a false:true ratio of 5:2 in the results we see – even though all are presented as true.
> Significance testing only tells you the probability that the measured difference is a "good measurement". With a certain degree of confidence, you can say "the difference exists as measured".
Significance does not tell you this. The p-value can be arbitrarily close to 0 while the probability of the null hypothesis being true is simultaneously arbitrarily close to one
Significance does not tell you this. The p-value can be arbitrarily close to 0 while the probability of the null hypothesis being true is simultaneously arbitrarily close to one
Right. The meaning of p-value is, in a world where there is no effect, what is the probability of getting the result you got purely by random chance? It doesn’t directly tell you anything about whether this is such a world or not.
This is sort of the basis of econometrics, as well as a driving thought behind causal inference.
Econometrics cares not only about statistical significance but also usefulness/economic usefulness.
Causal inference builds on base statistics and ML, but its strength lies in how it uses design and assumptions to isolate causality. Tools like sensitivity analysis, robustness checks, and falsification tests help assess whether the causal story holds up. My one beef is that these tools still lean heavily on the assumption that the underlying theoretical model is correctly specified. In other words, causal inference helps stress-test assumptions, but it doesn’t always provide a clear way to judge whether one theoretical framework is more valid than another!
Econometrics cares not only about statistical significance but also usefulness/economic usefulness.
Causal inference builds on base statistics and ML, but its strength lies in how it uses design and assumptions to isolate causality. Tools like sensitivity analysis, robustness checks, and falsification tests help assess whether the causal story holds up. My one beef is that these tools still lean heavily on the assumption that the underlying theoretical model is correctly specified. In other words, causal inference helps stress-test assumptions, but it doesn’t always provide a clear way to judge whether one theoretical framework is more valid than another!
I’d say rather that “statistically significance” is a measure of surprise. It’s saying “If this default (the null hypothesis) is true, how surprised would I be to make these observations?”
Maybe you can think of it as saying "should I be surprised" but certainly not "how surprised should I be". The magnitude of the p-value is a function of sample size. It is not an odds ratio for updating your beliefs.
For reference, this is the https://en.wikipedia.org/wiki/Effect_size
For all the shit that HN gives to MBAs, one thing they instill into you during the Managerial Stats class is Stag Sig not the same as Managerial Sig.
Really classic "rationalist" style writing: a soup of correct observations about statistical phenomena with chunks of weird political bullshit thrown in here and there. For example: "On a more contemporary note, these theoretical & empirical considerations also throw doubt on concerns about ‘algorithmic bias’ or inferences drawing on ‘protected classes’: not drawing on them may not be desirable, possible, or even meaningful."
This is such a bizarre sentence. The way its tossed in, not explained in any way, not supported by references, etc. Like I guess the implication being made is something like "because there is a hidden latent variable that determines criminality and we can never escape from correlations with it, its ok to use "is_black" in our black box model which decides if someone is going to get parole? Ridiculous. Does this really "throw doubt" on whether we should care about this?
The concerns about how models work are deeper than the statistical challenges of creating or interpreting them. For one thing, all the degrees of freedom we include in our model selection process allow us to construct models which do anything that we want. If we see a parole model which includes "likes_hiphop" as an explanatory variable we ought to ask ourselves who decided that should be there and whether there was an agenda at play beyond "producing the best model possible."
These concerns about everything being correlated actually warrant much more careful understanding about the political ramifications of how and what we choose to model and based on which variables, because they tell us that in almost any non-trivial case a model is at least partly necessarily a political object almost certainly consciously or subconsciously decorated with some conception of how the world is or ought to be explained.
This is such a bizarre sentence. The way its tossed in, not explained in any way, not supported by references, etc. Like I guess the implication being made is something like "because there is a hidden latent variable that determines criminality and we can never escape from correlations with it, its ok to use "is_black" in our black box model which decides if someone is going to get parole? Ridiculous. Does this really "throw doubt" on whether we should care about this?
The concerns about how models work are deeper than the statistical challenges of creating or interpreting them. For one thing, all the degrees of freedom we include in our model selection process allow us to construct models which do anything that we want. If we see a parole model which includes "likes_hiphop" as an explanatory variable we ought to ask ourselves who decided that should be there and whether there was an agenda at play beyond "producing the best model possible."
These concerns about everything being correlated actually warrant much more careful understanding about the political ramifications of how and what we choose to model and based on which variables, because they tell us that in almost any non-trivial case a model is at least partly necessarily a political object almost certainly consciously or subconsciously decorated with some conception of how the world is or ought to be explained.
> This is such a bizarre sentence. The way its tossed in, not explained in any way,
It reads naturally in context and is explained by the foregoing text. For example, the phrase "these theoretical & empirical considerations" refers to theoretical and empirical considerations described above. The basic idea is that, because everything correlates with everything else, you can't just look at correlations and infer that they're more than incidental. The political implications are not at all "weird", and follow naturally. The author observes that social scientists build complex models and observe huge amounts of variables, which allows them to find correlations that support their hypothesis; but these correlations, exactly because they can be found everywhere, are not anywhere near as solid evidence as they are presented as being.
> Like I guess the implication being made is something like "because there is a hidden latent variable that determines criminality and we can never escape from correlations with it, its ok to use "is_black" in our black box model which decides if someone is going to get parole?
No, not at all. The implication is that we cannot conclude that the black box model actually has an "is_black" variable, even if it is observed to have disparate impact on black people.
It reads naturally in context and is explained by the foregoing text. For example, the phrase "these theoretical & empirical considerations" refers to theoretical and empirical considerations described above. The basic idea is that, because everything correlates with everything else, you can't just look at correlations and infer that they're more than incidental. The political implications are not at all "weird", and follow naturally. The author observes that social scientists build complex models and observe huge amounts of variables, which allows them to find correlations that support their hypothesis; but these correlations, exactly because they can be found everywhere, are not anywhere near as solid evidence as they are presented as being.
> Like I guess the implication being made is something like "because there is a hidden latent variable that determines criminality and we can never escape from correlations with it, its ok to use "is_black" in our black box model which decides if someone is going to get parole?
No, not at all. The implication is that we cannot conclude that the black box model actually has an "is_black" variable, even if it is observed to have disparate impact on black people.
Sorry, but I don't think that is a reasonable read. The phrase "not drawing on them may not be desirable, possible, or even meaningful" is a political statement except perhaps for "possible," which is just a flat statement that its hard to separate causal variables from non-causal ones.
Nothing in the statistical observation that variables tend to be correlated suggests we should somehow reject the moral perspective that that its desirable for a model to be based on causal rather than merely correlated variables, even if finding such variables is difficult or even, impossible to do perfectly. And its certainly also _meaningful_ to do so, even if there are statistical challenges. A model based on "socioeconomic status" has a totally different social meaning than one based on race, even if we cannot fully disentangle the two statistically. He is mixing up statistical and social, moral and even philosophical questions in a way which is, in my opinion, misleading.
Nothing in the statistical observation that variables tend to be correlated suggests we should somehow reject the moral perspective that that its desirable for a model to be based on causal rather than merely correlated variables, even if finding such variables is difficult or even, impossible to do perfectly. And its certainly also _meaningful_ to do so, even if there are statistical challenges. A model based on "socioeconomic status" has a totally different social meaning than one based on race, even if we cannot fully disentangle the two statistically. He is mixing up statistical and social, moral and even philosophical questions in a way which is, in my opinion, misleading.
Or maybe your own announced bias against “rationalists” is affecting your reading of this. I agree with GPs interpretation.
> Nothing in the statistical observation that variables tend to be correlated suggests we should somehow reject the moral perspective that that its desirable for a model to be based on causal rather than merely correlated variables, even if finding such variables is difficult or even, impossible to do perfectly.
Perfect is the enemy of good. That it would be desirable to construct a model based on causal variables is self-evident, but we don't have those, and if a correlative model can demonstrably improve people's material conditions, even if conditioned on variables you find "distasteful", what is your argument that such a model shouldn't be used?
Perfect is the enemy of good. That it would be desirable to construct a model based on causal variables is self-evident, but we don't have those, and if a correlative model can demonstrably improve people's material conditions, even if conditioned on variables you find "distasteful", what is your argument that such a model shouldn't be used?
It really depends on a lot of things, frankly. For one thing, we, as a society, aren't optimizing for short term material conditions exclusively. The abstract dignity of not letting arbitrary variables determine important aspects of our lives might outweigh certain material benefits.
I think abstract principles deserve extreme skepticism. Pursuing material improvements will yield conditions that improve dignity, but pursuing dignity will not necessarily yield material improvements. Dignity is a luxury for those who don't have to desperately scrounge to live. There's a long and sad history of ignoring people's immediate material conditions to pursue some utopian vision of what "ought" to be. Unless there's some argument that sacrificing immediate material improvements will (not may) yield large scale material improvements, we're right back at the perfect being the enemy of the good.
>its desirable for a model to be based on causal rather than merely correlated variables
Ironically, your "likes_hiphop" example would appear to be an unusually clean case of a variable that is likely to exert causal influence.
What do you think the causal effect of listening to lyrics like "Prolly leave my fuckin' show in a cop car" might be, on an impressionable teenage boy say?
From one of the most-streamed hip-hop songs of all time:
https://genius.com/Post-malone-rockstar-lyrics
https://newsroom.spotify.com/2024-05-20/best-hip-hop-songs-1...
>A model based on "socioeconomic status" has a totally different social meaning than one based on race, even if we cannot fully disentangle the two statistically.
I see no evidence Gwern disagrees with this claim. He just seems to be arguing the "cannot fully disentangle the two statistically" part.
Ironically, your "likes_hiphop" example would appear to be an unusually clean case of a variable that is likely to exert causal influence.
What do you think the causal effect of listening to lyrics like "Prolly leave my fuckin' show in a cop car" might be, on an impressionable teenage boy say?
From one of the most-streamed hip-hop songs of all time:
https://genius.com/Post-malone-rockstar-lyrics
https://newsroom.spotify.com/2024-05-20/best-hip-hop-songs-1...
>A model based on "socioeconomic status" has a totally different social meaning than one based on race, even if we cannot fully disentangle the two statistically.
I see no evidence Gwern disagrees with this claim. He just seems to be arguing the "cannot fully disentangle the two statistically" part.
Doesn't it seem like the opposite to you? Since this is one of the most streamed songs of all time AND most kids have not "[left their] fuckin' show in a cop car" it seems that the causal power of media is small, which is pretty consistent with the literature on the subject.
The vast, vast majority of people understand the difference between media and real life. I mean I wouldn't go so far as to suggest that Post Malone is "good," either "morally" or aesthetically, but I don't think there is a strong case for lyrics, tv, or video games having a strong effect on violent behavior. But if it were the case it would be good to identify it accurately. There is plenty of violent "rock" music too, after all. The Columbine shooters weren't listening to hip hop.
The vast, vast majority of people understand the difference between media and real life. I mean I wouldn't go so far as to suggest that Post Malone is "good," either "morally" or aesthetically, but I don't think there is a strong case for lyrics, tv, or video games having a strong effect on violent behavior. But if it were the case it would be good to identify it accurately. There is plenty of violent "rock" music too, after all. The Columbine shooters weren't listening to hip hop.
>most kids have not "[left their] fuckin' show in a cop car"
Many people commit crimes. I'll bet criminals are more likely to listen to hip hop than the population at large is.
>it seems that the causal power of media is small
If the causal power of media is small, why are you concerned with Gwern's article? Even if he made claims that are blatantly racist, it wouldn't matter much, since the causal power of media is small.
>The vast, vast majority of people understand the difference between media and real life.
Suppose 99% understand that, and 1% don't. That can still be a big relative increase in the rate of crimes which do serious harm.
If you read the message of the song lyrics I linked, the clear implication (very common with this sort of music) is that criminal behavior will make lots of women want to have sex with you. This can easily be a self-fulfilling prophecy. Women listen to the lyrics and think to themselves "criminals sound cool and rebellious; criminal behavior is kinda hot -- all the other women are going for criminals; perhaps I will as well". Men who are trying to become attractive to women listen to the lyrics, and engage in crime alongside the other things they are doing which make them more attractive. Thus the prophecy becomes self-fulfilling, to society's detriment.
Anyways, as an exercise, ask ChatGPT to generate a list of top gangster rap artists. Then pick a few at random and ask if they've run into trouble with the law. There's a much higher rate of lawbreakers in this group than the population at large.
Many people commit crimes. I'll bet criminals are more likely to listen to hip hop than the population at large is.
>it seems that the causal power of media is small
If the causal power of media is small, why are you concerned with Gwern's article? Even if he made claims that are blatantly racist, it wouldn't matter much, since the causal power of media is small.
>The vast, vast majority of people understand the difference between media and real life.
Suppose 99% understand that, and 1% don't. That can still be a big relative increase in the rate of crimes which do serious harm.
If you read the message of the song lyrics I linked, the clear implication (very common with this sort of music) is that criminal behavior will make lots of women want to have sex with you. This can easily be a self-fulfilling prophecy. Women listen to the lyrics and think to themselves "criminals sound cool and rebellious; criminal behavior is kinda hot -- all the other women are going for criminals; perhaps I will as well". Men who are trying to become attractive to women listen to the lyrics, and engage in crime alongside the other things they are doing which make them more attractive. Thus the prophecy becomes self-fulfilling, to society's detriment.
Anyways, as an exercise, ask ChatGPT to generate a list of top gangster rap artists. Then pick a few at random and ask if they've run into trouble with the law. There's a much higher rate of lawbreakers in this group than the population at large.
Yes, much like an AI, we can arrange a series of tokens in any order we want to create the appearance of an argument. All I'm saying is that given that many, many people listen to hip hop (which is, incidentally, a much more expansive genre than Post Malone) and very, very, few people commit violent crimes, it is clear that hip hop is probably not the proximal cause of violent crime. The vast, vast, majority of people who listen to hip hop never commit a crime. Furthermore, to the extent that social science research means anything, correlations between media and criminality have been difficult to definitively find, for example: https://www.ucanmadison.org/wp-content/uploads/2024/12/2017_....
I should note that if Gwern's observations about correlations are true, then a negative result should be taken seriously, since positive correlations should be easy to find. Absence of strong correlations should reasonably be taken as a sign that a definitive connection is hard to come by. Of course, any good research in this field will attempt to control for confounds and if you ask me personally, I'm not optimistic about that prospect. But to the extent that this research says anything at all, the case isn't strong.
I'm not even saying you are per se wrong - it does seem reasonable that media that glorifies lawlessness might increase lawlessness. But if it does, it clearly only does so in a small population which also share a lot of other factors (like poverty, for example). Given that most humans enjoy hip hop without negative consequences, focusing on it as a potential intervention seems off base. A ban on hip-hop would be very unlikely to reduce crime, but a decrease in poverty would probably do so (accepting that we can't really figure out how to do that). A focus on hip hop is extremely flaccid.
I should note that if Gwern's observations about correlations are true, then a negative result should be taken seriously, since positive correlations should be easy to find. Absence of strong correlations should reasonably be taken as a sign that a definitive connection is hard to come by. Of course, any good research in this field will attempt to control for confounds and if you ask me personally, I'm not optimistic about that prospect. But to the extent that this research says anything at all, the case isn't strong.
I'm not even saying you are per se wrong - it does seem reasonable that media that glorifies lawlessness might increase lawlessness. But if it does, it clearly only does so in a small population which also share a lot of other factors (like poverty, for example). Given that most humans enjoy hip hop without negative consequences, focusing on it as a potential intervention seems off base. A ban on hip-hop would be very unlikely to reduce crime, but a decrease in poverty would probably do so (accepting that we can't really figure out how to do that). A focus on hip hop is extremely flaccid.
>All I'm saying is that given that many, many people listen to hip hop (which is, incidentally, a much more expansive genre than Post Malone) and very, very, few people commit violent crimes, it is clear that hip hop is probably not the proximal cause of violent crime.
This "crime is rare" argument is a fully general way to argue that almost nothing causes crime. Because crime is rare, by your logic, we know e.g. that gun ownership and poverty cannot be proximal causes, since gun ownership and poverty are both common.
Ultimately this line of reasoning is simply innumerate. Rare events can have common causes. For example, even if house fires are rare, they can still be caused by e.g. careless cigarette smokers, even if careless cigarette smokers are common.
>focusing on it as a potential intervention seems off base.
I never said it should be focused on as an intervention or it should be banned. All I said was likes_hiphop is "a variable that is likely to exert causal influence".
>A focus on hip hop is extremely flaccid.
Tell that to the guy who introduced "likes_hiphop" as a topic of discussion :-)
This "crime is rare" argument is a fully general way to argue that almost nothing causes crime. Because crime is rare, by your logic, we know e.g. that gun ownership and poverty cannot be proximal causes, since gun ownership and poverty are both common.
Ultimately this line of reasoning is simply innumerate. Rare events can have common causes. For example, even if house fires are rare, they can still be caused by e.g. careless cigarette smokers, even if careless cigarette smokers are common.
>focusing on it as a potential intervention seems off base.
I never said it should be focused on as an intervention or it should be banned. All I said was likes_hiphop is "a variable that is likely to exert causal influence".
>A focus on hip hop is extremely flaccid.
Tell that to the guy who introduced "likes_hiphop" as a topic of discussion :-)
"Rationalists" do seem to have a fetish for ranking people and groups of people. Oddly enough, they frequently use poorly performed studies and under-powered data to reach their conclusions about genetics and IQ especially.
> For example: "On a more contemporary note, these theoretical & empirical considerations also throw doubt on concerns about ‘algorithmic bias’ or inferences drawing on ‘protected classes’: not drawing on them may not be desirable, possible, or even meaningful."
As much as I do think that good, parsimonious social science modeling _requires_ theoretical commitments, the test is whether TFA would say the same thing about political cause du jour - say, `is_white` in hiring in an organization that does outreach to minority communities.
As much as I do think that good, parsimonious social science modeling _requires_ theoretical commitments, the test is whether TFA would say the same thing about political cause du jour - say, `is_white` in hiring in an organization that does outreach to minority communities.
Yes this is gwern to a "T". Overwhelm with a r/iamverysmart screed whilst insidiously inserting baseless speculation and opinion as fact as if the references provided cover those too. Weirdly the scaling/AI community loves him.
The article missed the chance to include the quote from that standard compendium of information and wisdom, The Hitchhiker's Guide to the Galaxy:
> Since every piece of matter in the Universe is in some way affected by every other piece of matter in the Universe, it is in theory possible to extrapolate the whole of creation — every sun, every planet, their orbits, their composition and their economic and social history from, say, one small piece of fairy cake.
> Since every piece of matter in the Universe is in some way affected by every other piece of matter in the Universe, it is in theory possible to extrapolate the whole of creation — every sun, every planet, their orbits, their composition and their economic and social history from, say, one small piece of fairy cake.
Wouldnt you need the T_zero configuration of the universe for this to work?
Given different T_zero configs of matter and energies T_current would be different. and there are many pathways that could lead to same physical configuration (position + energies etc) with different (Universe minus cake) configurations.
Also we are assuming there is no non-deterministic processed happening at all.
Given different T_zero configs of matter and energies T_current would be different. and there are many pathways that could lead to same physical configuration (position + energies etc) with different (Universe minus cake) configurations.
Also we are assuming there is no non-deterministic processed happening at all.
I am assuming integrating over all possible configurations would be a component of The Total Perspective Vortex.
After all, Feynman showed this is in principle possible, even with local nondeterminism.
(this being a text medium with a high probability of another commenter misunderstanding my intent, I must end this with a note that I am, of course, BSing :)
After all, Feynman showed this is in principle possible, even with local nondeterminism.
(this being a text medium with a high probability of another commenter misunderstanding my intent, I must end this with a note that I am, of course, BSing :)
> Wouldnt you need the T_zero configuration of the universe for this to work?
Why? We learn about the past by looking at the present all the time. We also learn about the future by looking at the present.
> Also we are assuming there is no non-deterministic processed happening at all.
Depends on the kind of non-determinism. If there's randomness, you 'just' deal with probability distributions instead. Since you have measurement error anyway, you need to do that anyway.
There are other forms of non-determinism, of course.
Why? We learn about the past by looking at the present all the time. We also learn about the future by looking at the present.
> Also we are assuming there is no non-deterministic processed happening at all.
Depends on the kind of non-determinism. If there's randomness, you 'just' deal with probability distributions instead. Since you have measurement error anyway, you need to do that anyway.
There are other forms of non-determinism, of course.
> Why? We learn about the past by looking at the present all the time. We also learn about the future by looking at the present.
The original comment's was in a differnt spirit or at least how I interpreted it. It was more implying by looking at a very small slice of reality you should in theory be able to re-construct the whole universe because every particle and space quantum is being influenced (to tiniest degrees) by every other particle in the universe, which will not work if you dont know all the rules, no determinism and T_zero.
The original comment's was in a differnt spirit or at least how I interpreted it. It was more implying by looking at a very small slice of reality you should in theory be able to re-construct the whole universe because every particle and space quantum is being influenced (to tiniest degrees) by every other particle in the universe, which will not work if you dont know all the rules, no determinism and T_zero.
The rules are presumably also gleanable from looking at the small speck of dust.
I don't know why you want to know T_zero. I assume T means time here, not temperature or so?
If you have eg randomness as your non-determinism, you can still build probability distributions of the rest of the universe and the rest of time. (And honestly, even if you have determinism in the laws, you always have measurement errors. Even in classical mechanics.)
I don't know why you want to know T_zero. I assume T means time here, not temperature or so?
If you have eg randomness as your non-determinism, you can still build probability distributions of the rest of the universe and the rest of time. (And honestly, even if you have determinism in the laws, you always have measurement errors. Even in classical mechanics.)
> We learn about the past by looking at the present all the time. We also learn about the future by looking at the present.
We infer about the past, based a bit on some material evidence we can subjectively partially get some acquaintance with. Through thick cultural biases. And the actual material suggestions should not come to far from our already integrated internal narrative, without what we will ignore it or actively fight it.
Future is pure fantasm, only bound by our imagination and what we take for unchallengeable fundamentals of what the world allows according to our inner model of it.
At least, that's one possible interpretation of the thoughts when an attention focus on present.
We infer about the past, based a bit on some material evidence we can subjectively partially get some acquaintance with. Through thick cultural biases. And the actual material suggestions should not come to far from our already integrated internal narrative, without what we will ignore it or actively fight it.
Future is pure fantasm, only bound by our imagination and what we take for unchallengeable fundamentals of what the world allows according to our inner model of it.
At least, that's one possible interpretation of the thoughts when an attention focus on present.
Why are you going so soft?
Having observed the sun go up and down reliably over my lifetime so far, I infer that it will keep doing so far quite a while, and that it has done so before I was born, too. Not much culture about it.
Every scientific measurement has elements of this.
And to be honest, we only learn about the present through a thick layer of interpretation and inference, too. The past and future aren't that special in that regard.
It becomes even more fun, when you add the finite propagation time of signals into the mix: here on earth we can never learn about the moon in the present, only how it was about 1s ago.
Having observed the sun go up and down reliably over my lifetime so far, I infer that it will keep doing so far quite a while, and that it has done so before I was born, too. Not much culture about it.
Every scientific measurement has elements of this.
And to be honest, we only learn about the present through a thick layer of interpretation and inference, too. The past and future aren't that special in that regard.
It becomes even more fun, when you add the finite propagation time of signals into the mix: here on earth we can never learn about the moon in the present, only how it was about 1s ago.
The real problem is you need a real-number-valued universe for this to work, where the measurer needs access to the full real values [1]. In our universe, which has a Planck size and Planck time and related limits, the statement is simply untrue. Even if you knew every last detail about a piece of fairy cake, whatever "every last detail" may actually be, and even if the universe is for some reason deterministic, you still could not derive the entire rest of the universe from it correctly. Some sort of perfect intelligence with access to massive amounts of computation may be able to derive a great deal more than you realize, especially about the environment in the vicinity of the cake, but it couldn't derive the entire universe.
[1]: Arguments are ongoing about whether the universe has "real" numbers (in the mathematical sense) or not. However it is undeniable the Planck constants still provide a practical barrier to any hypothetical real valued numbers in the universe that make them in practice inaccessible.
[1]: Arguments are ongoing about whether the universe has "real" numbers (in the mathematical sense) or not. However it is undeniable the Planck constants still provide a practical barrier to any hypothetical real valued numbers in the universe that make them in practice inaccessible.
In Buddhism we have dependent origination : https://en.wikipedia.org/wiki/Prat%C4%ABtyasamutp%C4%81da
Also the concept of implicate order, proposed by the theoretical physicist David Bohm.
> Bohm employed the hologram as a means of characterising implicate order, noting that each region of a photographic plate in which a hologram is observable contains within it the whole three-dimensional image, which can be viewed from a range of perspectives.
> That is, each region contains a whole and undivided image.
> "There is the germ of a new notion of order here. This order is not to be understood solely in terms of a regular arrangement of objects (e.g., in rows) or as a regular arrangement of events (e.g., in a series). Rather, a total order is contained, in some implicit sense, in each region of space and time."
> "Now, the word 'implicit' is based on the verb 'to implicate'. This means 'to fold inward' ... so we may be led to explore the notion that in some sense each region contains a total structure 'enfolded' within it."
> Bohm employed the hologram as a means of characterising implicate order, noting that each region of a photographic plate in which a hologram is observable contains within it the whole three-dimensional image, which can be viewed from a range of perspectives.
> That is, each region contains a whole and undivided image.
> "There is the germ of a new notion of order here. This order is not to be understood solely in terms of a regular arrangement of objects (e.g., in rows) or as a regular arrangement of events (e.g., in a series). Rather, a total order is contained, in some implicit sense, in each region of space and time."
> "Now, the word 'implicit' is based on the verb 'to implicate'. This means 'to fold inward' ... so we may be led to explore the notion that in some sense each region contains a total structure 'enfolded' within it."
Particles do not suffer from predestination, do they ?
People didn't always use statistics to discover truths about the world.
This, once developed, just happened to be a useful method. But given the abuse using those methods, and the proliferation of stupidity disguised as intelligence, it's always fitting to question it, and this time with this correlation noise observation.
Logic, fundamental knowledge about domains, you need that first. Just counting things without understanding them in at least one or two other ways, is a tempting invitation for misleading conclusions.
This, once developed, just happened to be a useful method. But given the abuse using those methods, and the proliferation of stupidity disguised as intelligence, it's always fitting to question it, and this time with this correlation noise observation.
Logic, fundamental knowledge about domains, you need that first. Just counting things without understanding them in at least one or two other ways, is a tempting invitation for misleading conclusions.
> People didn't always use statistics to discover truths about the world.
And they were much, much worse off for it. Logic does not let you learn anything new. All logic allows you to do is restate what you already know. Fundamental knowledge comes from experience or experiments, which need to be interpreted through a statistical lens because observations are never perfect.
Before statistics, our alternatives for understanding the world was (a) rich people sitting down and thinking deeply about how things could be, (b) charismatic people standing up and giving sermons on how they would like things to be, or (c) clever people guessing things right every now and then.
With statistics, we have to a large degree mechanised the process of learning how the world works, and anyone sensible can participate, and they can know with reasonable certainty whether they are right or wrong. It was impossible to prove a philosopher or a clergyman wrong!
That said, I think I agree with your overall point. One of the strengths of statistical reasoning is what's sometimes called intercomparison, the fact that we can draw conclusions from differences between processes without understanding anything about those processes. This is also a weakness because it makes it easy to accidentally or intentionally manipulate results.
And they were much, much worse off for it. Logic does not let you learn anything new. All logic allows you to do is restate what you already know. Fundamental knowledge comes from experience or experiments, which need to be interpreted through a statistical lens because observations are never perfect.
Before statistics, our alternatives for understanding the world was (a) rich people sitting down and thinking deeply about how things could be, (b) charismatic people standing up and giving sermons on how they would like things to be, or (c) clever people guessing things right every now and then.
With statistics, we have to a large degree mechanised the process of learning how the world works, and anyone sensible can participate, and they can know with reasonable certainty whether they are right or wrong. It was impossible to prove a philosopher or a clergyman wrong!
That said, I think I agree with your overall point. One of the strengths of statistical reasoning is what's sometimes called intercomparison, the fact that we can draw conclusions from differences between processes without understanding anything about those processes. This is also a weakness because it makes it easy to accidentally or intentionally manipulate results.
Discovering that two different-seeming statements reduce to the same truth is new knowledge.
There is a quote from George Lucas where he talks about how when new things come into a society people have a tend to over do it.
https://www.youtube.com/watch?v=VEIrQUXm_hY
https://www.youtube.com/watch?v=VEIrQUXm_hY
Quoted repeatedly by Mr. Plinkett in his new drop yesterday.
https://www.youtube.com/watch?v=0xeMak4RqJA
https://www.youtube.com/watch?v=0xeMak4RqJA
Nice, yeah. With many movies one has to ask: What's the point? Especially all Disney Star Wars..
Not commenting on the topic at hand, but my goodness, what a beautiful blog. That drop cap, the inline comments on the right hand side that appear on larger screens, the progress bar, chef's kiss. This is how a love project looks like.
You may be interested in gwern's dropcap article:
https://gwern.net/dropcap
https://gwern.net/dropcap
This is such a massive article. I wish I had the ability to grind out treatises like that. Looking at other content on the guy's website, he must be like a machine.
IIRC Gwern lives extremely frugally somewhere remote and is thus able to spend a lot of time on private research.
IIRC people funded moving gwern to the bay not too long ago.
That and early bitcoin adoption. There’s a short bio somewhere on the site.
lots of time, many iterations, affinity for the hard questions, some expertise in research (and Haskell). oh, and also it helps if someone is funding your little endeavor :)
I wish I would be even able to read things like that.
gwern is goated
Arguments like this have been around for decades. I think it's important to keep in mind — critical even.
At the same time, as I've been forced to wrestle with it more in my work, I've increasingly felt that it's sort of empty and unhelpful. "Crud" does happen in patterns, like a kind of statistical cosmic background radiation — it's not meaningless. Sometimes it's important to understand it, and treating it as such gets no one anywhere. Sometimes the associations are difficult to explain easily when you try to pick it apart, and other times I think they're key to understanding uncontrolled confounds that should be controlled for.
As much as this background association is present too, it's not always there. Sometimes things do have zero association.
Also, trying to come up with a "meaningful" effect size that's not zero is pretty arbitrary and subjective.
There's probably more productive ways of framing the phenomenon.
At the same time, as I've been forced to wrestle with it more in my work, I've increasingly felt that it's sort of empty and unhelpful. "Crud" does happen in patterns, like a kind of statistical cosmic background radiation — it's not meaningless. Sometimes it's important to understand it, and treating it as such gets no one anywhere. Sometimes the associations are difficult to explain easily when you try to pick it apart, and other times I think they're key to understanding uncontrolled confounds that should be controlled for.
As much as this background association is present too, it's not always there. Sometimes things do have zero association.
Also, trying to come up with a "meaningful" effect size that's not zero is pretty arbitrary and subjective.
There's probably more productive ways of framing the phenomenon.
Correlated. Others?
Everything Is Correlated - https://news.ycombinator.com/item?id=19797844 - May 2019 (53 comments)
Everything Is Correlated - https://news.ycombinator.com/item?id=19797844 - May 2019 (53 comments)
Correlated, you mean?
That's so good I had to nick it for the parent comment. Thanks!
(It said "Related" before, of course: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....)
(It said "Related" before, of course: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....)
Those would be all articles posted in HN :)
This is why experimental science is different from observational studies.
Statistical analyses provide a reason to believe one hypothesis over another, but any scientist will extend that with an experimental approach.
Most of the examples given in this blog post refer to medical, sociological or behavioral studies, where properly controlled experiments are hard to perform, and as such are frequently under-powered to reveal true cause-effect associations.
Statistical analyses provide a reason to believe one hypothesis over another, but any scientist will extend that with an experimental approach.
Most of the examples given in this blog post refer to medical, sociological or behavioral studies, where properly controlled experiments are hard to perform, and as such are frequently under-powered to reveal true cause-effect associations.
This was my take as well. At least microeconomics has moved away from large-scale observational studies and has moved into experimental and quasi-experimental studies.
While the methods alone cannot fix it all ("You can’t fix by analysis what you bungled by design" [1] after all), it gets somewhat closer to unbiased results.
[1]: https://www.degruyterbrill.com/document/doi/10.4159/97806740...
While the methods alone cannot fix it all ("You can’t fix by analysis what you bungled by design" [1] after all), it gets somewhat closer to unbiased results.
[1]: https://www.degruyterbrill.com/document/doi/10.4159/97806740...
[deleted]
„This renders the meaning of significance-testing unclear; it is calculating precisely the odds of the data under scenarios known a priori to be false.“
I cannot see the problem in that. To get to meaningful results we often calculate with simplyfied models - which are known to be false in a strict sense. We use Newtons laws - we analyze electric networks based on simplifications - a bank-year used to be 360 days! Works well.
What did i miss?
I cannot see the problem in that. To get to meaningful results we often calculate with simplyfied models - which are known to be false in a strict sense. We use Newtons laws - we analyze electric networks based on simplifications - a bank-year used to be 360 days! Works well.
What did i miss?
The problem is basically that you can always buy a significant result with money (large enough N always leads to ”significant” result). That’s a serious issue if you see research as pursuit of truth.
Reporting effect size mitigates this problem. If observed effect size is too small, its statistical significance isn't viewed as meaningful.
Sure (and of course). But did you see the effect size histogram in the OP?
Are you referring to the first figure, from Smith, et al, 2007? If so, I couldn't evaluate whether gwern's claim makes sense without reading that paper to get an idea of, e.g., sample size and how they control for false positives. I don't think it's self-evident from that figure alone.
One rule of thumb for interpreting (presumably Pearson) correlation coefficients is given in [0] and states that correlations with magnitude 0.3 or less are negligible, in which case most of the bins in that histogram correspond to cases that aren't considered meaningful.
[0]: https://pmc.ncbi.nlm.nih.gov/articles/PMC3576830/table/T1/
One rule of thumb for interpreting (presumably Pearson) correlation coefficients is given in [0] and states that correlations with magnitude 0.3 or less are negligible, in which case most of the bins in that histogram correspond to cases that aren't considered meaningful.
[0]: https://pmc.ncbi.nlm.nih.gov/articles/PMC3576830/table/T1/
I’m not arguing that there’s something fundamentally wrong with mathematics or the scientific method. I’m arguing that the social norms around how we do science in practice have some serious flaws. Gwern points out one of them. One that IMHO is quite interesting.
EDIT: I also get the feeling that you think it’s okay to do an incorrect hypothesis test (c > 0), as long as you also look at the effect size. I don’t think it is. You need to test the c > 0.3 hypothesis to get a mathematically sound hypothesis test. How many papers do that?
EDIT: I also get the feeling that you think it’s okay to do an incorrect hypothesis test (c > 0), as long as you also look at the effect size. I don’t think it is. You need to test the c > 0.3 hypothesis to get a mathematically sound hypothesis test. How many papers do that?
My opinion of Gwern's piece is that some of the arguments he makes don't require correlations. For example, A/B tests of differences in means using a zero difference null hypothesis will reject the null, given enough data.
In that A/B testing scenario, I think if someone wants to test whether the difference is zero, that's fine, but if the effect size is small, they shouldn't claim that there's any meaningful difference. I believe the pharma literature calls this scenario equivalence testing.
Assuming a positive difference in means is desirable, I think testing for a null hypothesis of a change of at least some positive value (e.g., +5% of control) is a better idea. I believe the pharma literature calls this scenario superiority testing.
I believe superiority testing is preferable to equivalence testing, and in professional settings, I have made this case to managers. I have not succeeded in persuading them, and thus do the equivalence testing they request.
I don't think the idea of a zero null hypothesis is necessarily mathematically unsound. In cases like the difference in means, a zero null hypothesis is well-posed. However, I agree with you that there are better practices, like a null hypothesis incorporating a nonzero effect.
I don't entirely agree with the arguments Gwern puts forth in the Implications section because some of them seem at odds with one another. Betting on sparsity would imply neglecting some of the correlations he's arguing are so essential to capture. The bit about algorithmic bias strikes me as a bizarre proposition to include with little supporting evidence, especially when there are empirical examples of algorithmic bias.
What I find lacking about Gwern's piece is that it's a bit like lighting a match to widespread statistical practice, and then walking away. Yes, I think null hypothesis statistical testing is widely overused, and that statistical significance alone is not a good determinant of what constitutes a "discovery". I agree that modeling is hard, and that "everything is correlated" is, to an extent, true because the correlations are not literally or exactly zero. But if you're going to take the strong stance that null hypothesis statistical testing is meaningless, I believe you need to provide some kind of concrete alternative. I don't think Gwern's piece explicitly advocates an alternative, and it only hints the alternative might be causal inference. Asking people who may not have much statistics training to leap from frequentist concepts taught in high school to causal inference would be a big ask. If Gwern isn't asking that, then I'd want to know what a suggested alternative would be. Notably, Gwern does not mention testing for nonzero positive effects (e.g., in the vein of the "c > 0.3" case above). If there isn't an alternative, I'm not sure what the argument is. Don't use statistics, perhaps? It's tough to say.
In that A/B testing scenario, I think if someone wants to test whether the difference is zero, that's fine, but if the effect size is small, they shouldn't claim that there's any meaningful difference. I believe the pharma literature calls this scenario equivalence testing.
Assuming a positive difference in means is desirable, I think testing for a null hypothesis of a change of at least some positive value (e.g., +5% of control) is a better idea. I believe the pharma literature calls this scenario superiority testing.
I believe superiority testing is preferable to equivalence testing, and in professional settings, I have made this case to managers. I have not succeeded in persuading them, and thus do the equivalence testing they request.
I don't think the idea of a zero null hypothesis is necessarily mathematically unsound. In cases like the difference in means, a zero null hypothesis is well-posed. However, I agree with you that there are better practices, like a null hypothesis incorporating a nonzero effect.
I don't entirely agree with the arguments Gwern puts forth in the Implications section because some of them seem at odds with one another. Betting on sparsity would imply neglecting some of the correlations he's arguing are so essential to capture. The bit about algorithmic bias strikes me as a bizarre proposition to include with little supporting evidence, especially when there are empirical examples of algorithmic bias.
What I find lacking about Gwern's piece is that it's a bit like lighting a match to widespread statistical practice, and then walking away. Yes, I think null hypothesis statistical testing is widely overused, and that statistical significance alone is not a good determinant of what constitutes a "discovery". I agree that modeling is hard, and that "everything is correlated" is, to an extent, true because the correlations are not literally or exactly zero. But if you're going to take the strong stance that null hypothesis statistical testing is meaningless, I believe you need to provide some kind of concrete alternative. I don't think Gwern's piece explicitly advocates an alternative, and it only hints the alternative might be causal inference. Asking people who may not have much statistics training to leap from frequentist concepts taught in high school to causal inference would be a big ask. If Gwern isn't asking that, then I'd want to know what a suggested alternative would be. Notably, Gwern does not mention testing for nonzero positive effects (e.g., in the vein of the "c > 0.3" case above). If there isn't an alternative, I'm not sure what the argument is. Don't use statistics, perhaps? It's tough to say.
Thanks for the extensive answer.
> I don't think the idea of a zero null hypothesis is necessarily mathematically unsound. In cases like the difference in means, a zero null hypothesis is well-posed. However, I agree with you that there are better practices, like a null hypothesis incorporating a nonzero effect.
I don’t think a zero null hypothesis is mathematically unsound of course. But I think it is unsound to do one and then look at the effect size as a known quantity. It’s not a known quantity, it’s a point estimate with a lot of uncertainty. The real underlying correlation may well be a lot lower than the point estimate.
And of course it’s hard to get people in charge interested in better hypothesis testing. That testing will result in fewer conclusions being drawn / fewer papers being published. It’s just another symptom of the core issue: it’s quite convenient to be able to buy the conclusions you want with money.
> I don't think the idea of a zero null hypothesis is necessarily mathematically unsound. In cases like the difference in means, a zero null hypothesis is well-posed. However, I agree with you that there are better practices, like a null hypothesis incorporating a nonzero effect.
I don’t think a zero null hypothesis is mathematically unsound of course. But I think it is unsound to do one and then look at the effect size as a known quantity. It’s not a known quantity, it’s a point estimate with a lot of uncertainty. The real underlying correlation may well be a lot lower than the point estimate.
And of course it’s hard to get people in charge interested in better hypothesis testing. That testing will result in fewer conclusions being drawn / fewer papers being published. It’s just another symptom of the core issue: it’s quite convenient to be able to buy the conclusions you want with money.
Back when I wrote a loan repayment calculator, there were 47 common different ways to 'day count' (used in calculating payments for incomplete repayment periods, e.g in monthly payments, what is the 1st-13th of aug 2025 as a fraction of aug 2025?).
There is a known maximum error introduced by those simplifications. Put the other way around, Einstein is a refinement of Newton. Special relativity converges towards Newtonian motion for low speeds.
You didn't really miss anything. The article is incomplete, and wrongly suggests that something like "false" even exists in statistics. But really something is only false "with a x% probability of it actually being true nonetheless". Meaning that you have to "statistic harder" if you want to get x down. Usually the best way to do that is to increase the number of tries/samples N. What the article gets completely wrong is that for sufficiently large N, you don't have to care anymore, and might as well use false/true as absolutes, because you pass the threshold of "will happen once within the lifetime of a bazillion universes" or something.
Problem is, of course, that lots and lots of statistics are done with a low N. Social sciences, medicine, and economy are necessarily always in the very-low-N range, and therefore always have problematic statistics. And try to "statistic harder" without being able to increase N, thereby just massaging their numbers enough to get a desired conclusion proved. Or just increase N a little, claiming to have escaped the low-N-problem.
You didn't really miss anything. The article is incomplete, and wrongly suggests that something like "false" even exists in statistics. But really something is only false "with a x% probability of it actually being true nonetheless". Meaning that you have to "statistic harder" if you want to get x down. Usually the best way to do that is to increase the number of tries/samples N. What the article gets completely wrong is that for sufficiently large N, you don't have to care anymore, and might as well use false/true as absolutes, because you pass the threshold of "will happen once within the lifetime of a bazillion universes" or something.
Problem is, of course, that lots and lots of statistics are done with a low N. Social sciences, medicine, and economy are necessarily always in the very-low-N range, and therefore always have problematic statistics. And try to "statistic harder" without being able to increase N, thereby just massaging their numbers enough to get a desired conclusion proved. Or just increase N a little, claiming to have escaped the low-N-problem.
A frequentist interpretation of inference assumes parameters have fixed, but unknown values. In this paradigm, it is sensible to speak of the statement "this parameter's value is zero" as either true or false.
I do not think it is accurate to portray the author as someone who does not understand asymptotic statistics.
I do not think it is accurate to portray the author as someone who does not understand asymptotic statistics.
> it is sensible to speak of the statement "this parameter's value is zero" as either true or false.
Nope. The correct way is rather something like "the measurements/polls/statistics x ± ε are consistent with this parameter's true value to be zero", where x is your measured value and ε is some measurement error, accuracy or statistical deviation. x will never really be zero, but zero can be within an interval [x - ε; x + ε].
Nope. The correct way is rather something like "the measurements/polls/statistics x ± ε are consistent with this parameter's true value to be zero", where x is your measured value and ε is some measurement error, accuracy or statistical deviation. x will never really be zero, but zero can be within an interval [x - ε; x + ε].
As you yourself point out, a consistent estimator of a parameter converges to that parameter's value in the infinite sample limit. That limit is zero or it's not.
It's a quantitative problem. How big is the error introduced by the simplification?
Any two linearly increasing or decreasing datasets will be necessarily highly correlated. But that doesn't mean anything unless you also have a plausible explanation why it is so.
Also correlation is often taken to mean linear correlation. For example, you can have two nonlinearly varying dataset to be perfectly (rank) correlated while their linear correlation can be close to zero.
People often attach undue meaning to correlation.
Also correlation is often taken to mean linear correlation. For example, you can have two nonlinearly varying dataset to be perfectly (rank) correlated while their linear correlation can be close to zero.
People often attach undue meaning to correlation.
I mouse over the first graphic in dark mode and get flashbanged with a white diagram. Hover text obscures the axis text. I click the diagram hoping to read the text. In the expanded picture the alt text now obscures the axis instead. Close the picture and read on to the starting section. A dropped capital. It looks like an 'R' with some flourish. 'Rnowing'. Ok, it must be a 'k' based on context, but appears to resemble a lower case cursive 'k' and not a capital 'K'. Dropping the capital letter to lowercase in a drop cap. A minuscule pun one character in. Off to a good start. I continue reading to the second diagram. Fearful from my previous encounter I rest my cursor on what looks like a safe zone: 'click to expand'. The section immediately unfurls and pushes the diagram off the page. 'Everything is correlated' I tell myself. If there's a hover event there must be an 'on-not-hover' event too, right? I move my cursor to an empty zone. Nothing happens. I click in the empty space to remind the browser my cursor is not hovering. Nothing. 'Only some things are correlated' I think, closing the section manually. Eventually I see a hovernote 4 and think to myself 'it's nice these notes won't interrupt the flow of reading' as I click on it. The page shifts to show text already on the screen and I lose my place briefly. The word 'error' is now partially obscured by the footnote bounding box. I chuckle to myself. Maybe there is an option for reader mode I think as I start scrolling up. A gear reveals itself in the top corner. As I scroll down the options I see a 'Auto Reader Mode'. Exactly what I need. I move my mouse over to the text. This was a poor choice. The menu retracts and now that my mouse is partially over a hover note it scrolls itself fully into view. 'Tibshirani 2014' is revealed.
A really interesting essay. I'm genuinely surprised the author didn't delve more into philosophy here. I think the biggest issue this essay illustrates is the inductive-deductive jump that modeling (especially mathematical modeling makes). That is, if the axioms we are using are correct, then the output of the model should reflect reality. This is very obviously not always the case, and because of the black swan problem (in a very real sense) we can never really know if it does.
Any time we wade into solipsism, however, I think it's important to remember that statistical analysis is a tool, not an arbiter of Truth. We are trying to exist in the mess of a world we live in, and we're going to be using every possible advanced tool we have in our arsenal to do that, and the standard model of science is probably the best tool we have. At the same time, we should often return to that solipsism an remember that where we can improve the model, we should.
Any time we wade into solipsism, however, I think it's important to remember that statistical analysis is a tool, not an arbiter of Truth. We are trying to exist in the mess of a world we live in, and we're going to be using every possible advanced tool we have in our arsenal to do that, and the standard model of science is probably the best tool we have. At the same time, we should often return to that solipsism an remember that where we can improve the model, we should.
> For example, while looking at biometric samples with up to thousands of observations, Karl Pearson declared that a result departing by more than 3 standard deviations is “definitely significant.”
Wait. Sir Arthur Conan Doyle lived at basically the exact same time as this Karl Pearson.
Is that why the Sherlock Holmes stories had handwriting analysis so frequently? Was there just pop science going around at the time that like, let's find correlations between anything and anything, and we can see that a criminal mastermind like Moriarty would certainly cross their T's this way and not that way?
Wait. Sir Arthur Conan Doyle lived at basically the exact same time as this Karl Pearson.
Is that why the Sherlock Holmes stories had handwriting analysis so frequently? Was there just pop science going around at the time that like, let's find correlations between anything and anything, and we can see that a criminal mastermind like Moriarty would certainly cross their T's this way and not that way?
Also, I'm convinced that the reason humans intuitively struggle to figure out causality is because the vast majority of causes and effects are self-reinforcing cycles and go both ways. There was little evolutionary pressure for us to understand the concept of causality because it doesn't play a strong role in natural selection.
For example, eat a lot and you will gain weight, gain weight and you will feel more hungry and will likely eat more.
Or exercise more and it becomes easier to exercise.
Earning money becomes easier as you have more money.
Public speaking becomes easier as you do it more and the more you do it, the easier it becomes.
Etc...
For example, eat a lot and you will gain weight, gain weight and you will feel more hungry and will likely eat more.
Or exercise more and it becomes easier to exercise.
Earning money becomes easier as you have more money.
Public speaking becomes easier as you do it more and the more you do it, the easier it becomes.
Etc...
> Or exercise more and it becomes easier to exercise.
Only if you don't injure yourself while exercising.
Only if you don't injure yourself while exercising.
A lot of things can happen to break a self-perpetuating cycle. But it's usually some extreme event. The cycle keeps optimizing in a particular direction and this eventually leads to an extreme situation which becomes unsustainable. The natural equilibrium of a self-reinforcing cycle is not static but drifting towards some extreme unstable state. There is usually a point where it breaks down.
But I suspect that being able to figure out causation doesn't matter much from a survival or reproduction perspective because cause and effect are just labels.
Reality in a self-perpetuating cycle is probably like Condition A is 70% responsible and Condition B is 30% responsible for a problem but they feedback and exacerbate each other... You could argue that Condition A is the cause and Condition B is the effect because B < A but that's not quite right IMO. Also, it's not quite right to say that because A happened first, that A is the cause of a severe problem... The problem would never have gotten so bad to such extent without feedback from B.
But I suspect that being able to figure out causation doesn't matter much from a survival or reproduction perspective because cause and effect are just labels.
Reality in a self-perpetuating cycle is probably like Condition A is 70% responsible and Condition B is 30% responsible for a problem but they feedback and exacerbate each other... You could argue that Condition A is the cause and Condition B is the effect because B < A but that's not quite right IMO. Also, it's not quite right to say that because A happened first, that A is the cause of a severe problem... The problem would never have gotten so bad to such extent without feedback from B.
Or, exercise more and you will eat more. Drat.
I was recently looking at a large timeseries dataset.
I noticed when doing a scatter plot of two variables and noticed that there were several "lines" of dots.
This generally implies that subsets of the two variables may have correlations or there is a third variable to be added.
I did some additional research and it is possible for two variables with large N to show correlation for short bursts even if both variables are random.
I mention for two reasons:
1. I was just doing the above and saw the OP article today
2. Despite taking multiple college level stats classes, I don't remember this ever being mentioned.
I noticed when doing a scatter plot of two variables and noticed that there were several "lines" of dots.
This generally implies that subsets of the two variables may have correlations or there is a third variable to be added.
I did some additional research and it is possible for two variables with large N to show correlation for short bursts even if both variables are random.
I mention for two reasons:
1. I was just doing the above and saw the OP article today
2. Despite taking multiple college level stats classes, I don't remember this ever being mentioned.
Did they quote https://www.tylervigen.com/spurious-correlations ?
It's just as I suspected - there are NO coincidences.
Looks like an impressive thorough piece of investigation. Well done.
That said, holistic supposition can certainly be traced back as far as writting dawns. Here the focus on more modern/contemporary era is legitimate to keep the focus delimited on a more specific concern, but is a bit obfuscating this fact. Maybe it's already acknowledged in the document, I read it all yet.
That said, holistic supposition can certainly be traced back as far as writting dawns. Here the focus on more modern/contemporary era is legitimate to keep the focus delimited on a more specific concern, but is a bit obfuscating this fact. Maybe it's already acknowledged in the document, I read it all yet.
Statistical correlations anre important to establish but there are the easiest and least useful part of the research. Creating theories as to “why” and “how” these correlations exist are what advances our knowledge.
I read lot of papers that painstakingly show a correlation in the data, but then their theory about the correlation is a complete non sequitur.
I read lot of papers that painstakingly show a correlation in the data, but then their theory about the correlation is a complete non sequitur.
If two things e.g. both change over time, they will be correlated. I think it can be good to keep this article in mind
> If two things e.g. both change over time, they will be correlated.
No?
You can have two independent random walks. Eg flip a coin, gain a dollar or lose a dollar. Do that to times in parallel. Your two account balances will change over time, but they won't be correlated.
No?
You can have two independent random walks. Eg flip a coin, gain a dollar or lose a dollar. Do that to times in parallel. Your two account balances will change over time, but they won't be correlated.
I don't disagree with the title, but I'm left wondering what they want us to do about it beyond hinting at causal inference. I'd also be curious what the author thinks of minimum effect sizes (re: Implication 1) and noninferiority testing (re: Implication 2).
Does the author account for time autocorrelation? Pearson’s correlation assumes independence between observations, yet time series data often violate this assumption. Without adjusting for autocorrelation, correlations may be artificially inflated.
there is but a single unfolding, and everything is part of it
Doesn't everything means all things that exist in universe and since they exist in same universe they are correlated?
Causality from relativity prevents this from being generally true. There could be things in this universe sufficiently far away that we cannot see because whatever interaction we could observe from it has not reached here yet.
It is actually inflation that precludes this from being true (along with relativity, of course). If there wasn't a period where spacetime inflated extremely rapidly, given two points, no matter how far apart, their causal pasts would eventually intersect, if going sufficiently back in the past. As such, they could correlate through a common cause C, which lies in both of their causal pasts.
I wonder if this tendency to correlate truly holds for everything? Intuitively it more or less demonstrates that nature tends to favor zero-sum games. Maybe analyzing correlations within the domain of theoretical physics would highlight true non-correlations in some particular approaches? (pun only slightly intended)
> Intuitively it more or less demonstrates that nature tends to favor zero-sum games.
Please explain.
Please explain.
For every action, there is an opposite and equal reaction. For example, there is a correlation between the acceleration and deceleration of colliding objects: inertia is transferred, not created or destroyed.
Similarly, for every chemical and nuclear reaction, when something is gained, something else is lost. For example, when two ions bond covalently by sharing electrons, a new molecule is gained, but the two ions are no longer what they previously were. So there is a correlation between gain of reaction products and loss of reactants.
But perhaps such analogies cannot be found everywhere in theoretical physics. Perhaps such a non-correlation would be a sign of a novel discovery, or a sign that a theory is physically invalid. It could be a signal of something for sure.
Similarly, for every chemical and nuclear reaction, when something is gained, something else is lost. For example, when two ions bond covalently by sharing electrons, a new molecule is gained, but the two ions are no longer what they previously were. So there is a correlation between gain of reaction products and loss of reactants.
But perhaps such analogies cannot be found everywhere in theoretical physics. Perhaps such a non-correlation would be a sign of a novel discovery, or a sign that a theory is physically invalid. It could be a signal of something for sure.
How do I reconcile this with "entropy invariably increases" which is a contradiction to your hypothesis that "nature tends to favor zero-sum games"?
How do I reconcile "for every chemical and nuclear reaction, when something is gained, something else is lost" with catalysts increasing rate but not being consumed themselves?
In fact you can show there are an uncountably infinite number of broken symmetries in nature, so it is mathematically possible to concoct a parallel number of cases where nature does not have some "zero sum game" by Noether's Theorem.
Your statement is just cherry picking a few and then (uncountably infinitely) overgeneralizing.
How do I reconcile "for every chemical and nuclear reaction, when something is gained, something else is lost" with catalysts increasing rate but not being consumed themselves?
In fact you can show there are an uncountably infinite number of broken symmetries in nature, so it is mathematically possible to concoct a parallel number of cases where nature does not have some "zero sum game" by Noether's Theorem.
Your statement is just cherry picking a few and then (uncountably infinitely) overgeneralizing.
You are picking apart the zero-sum part, I would also pick at the 'game' part. None of the examples are 'games'. They are just a few examples of conservation laws.
No decision making, no min-maxing actors etc.
Btw, when you have a single optimising actor, then moving along the efficiency barrier is also a set of trade-offs, which can be made constant-sum, if you set up your conversion factors just right; even if the optimisation itself is otherwise variable sum. (As a silly illustration: to produce more guns, you need to produce less butter.) But that observation doesn't really prove anything.
No decision making, no min-maxing actors etc.
Btw, when you have a single optimising actor, then moving along the efficiency barrier is also a set of trade-offs, which can be made constant-sum, if you set up your conversion factors just right; even if the optimisation itself is otherwise variable sum. (As a silly illustration: to produce more guns, you need to produce less butter.) But that observation doesn't really prove anything.
Entropy is a measurement of the system itself and doesn’t describe the dynamics within that system.
Catalysts increase reaction rate just as a train runs faster on a track. Is a railway a catalyst?
Are symmetries broken in nature or just models of nature? Or are you referring to accepted theories in theoretical physics, which was the entire point here?
Catalysts increase reaction rate just as a train runs faster on a track. Is a railway a catalyst?
Are symmetries broken in nature or just models of nature? Or are you referring to accepted theories in theoretical physics, which was the entire point here?
the rest of the page has amazing design, but there's just something about the graphs switching from dark to light that flashbangs my eyes really badly - i think it's the sudden light!
That is unintentional and a bug in the dark-mode.
For dark-mode, we rely on https://invertornot.com/ to dynamically decide whether to fade or negate/invert. (Background: https://gwern.net/invertornot ) The service uses a small NN and is not always correct, as in these cases. Sorry.
I have filed them as errors with InvertOrNot, and will manually set them to invert.
For dark-mode, we rely on https://invertornot.com/ to dynamically decide whether to fade or negate/invert. (Background: https://gwern.net/invertornot ) The service uses a small NN and is not always correct, as in these cases. Sorry.
I have filed them as errors with InvertOrNot, and will manually set them to invert.
Ah lovely! Not a critique as enjoyed the rest of the article, just wanted to comment in case it helped resolve it!
And that, kids, is why A/B testing is nonsense.
There is no zero in the real world
People interpret "statistically significant" to mean "notable"/"meaningful". I detected a difference, and statistics say that it matters. That's the wrong way to think about things.
Significance testing only tells you the probability that the measured difference is a "good measurement". With a certain degree of confidence, you can say "the difference exists as measured".
Whether the measured difference is significant in the sense of "meaningful" is a value judgement that we / stakeholders should impose on top of that, usually based on the magnitude of the measured difference, not the statistical significance.
It sounds obvious, but this is one of the most common fallacies I observe in industry and a lot of science.
For example: "This intervention causes an uplift in [metric] with p<0.001. High statistical significance! The uplift: 0.000001%." Meaningful? Probably not.