Interesting read and worth thinking about the question the way it's framed, although I don't know there's actually an answer. I agree that how much something matters is how much things would change if it were changed, but I think that depends heavily on the scenario, in terms of the questions asked, the design, and so forth. I also think problems in contemporary academics go far beyond statistics, and won't be addressed with statistics, but rather funding and sociopolitical changes.
Take the conclusion that "issues like measurement error or distributions, which are equally common, are often not important." I strongly disagree with this. In a classic randomized controlled trial, maybe yes. But even then there are serious problems potentially. Just to offer a few examples:
Measurement is key in this age of unreproducability. It's not uncommon for claims to be made based on results involving one particular measure, when the hypothesis would apply to many measures of the same thing in the sample. When an author claims X causes Y, and there are multiple measures of Y, but results are only reported for one, problems are there. Modeling the joint effect on Y, rather than the measures of Y, is key.
Overfitting goes hand in hand with distributional misassumptions, because to the extent that an observed distribution deviates from the assumed one, overfitting models can capitalize on excess information missed by the base distribution. A classic example of this is assuming a normal distribution when fitting a linear regression but fitting to a very nonnormal distribution: in many cases an interaction term will add significantly even in the absence of a real interaction, because it captures more of the interestingness of the data, information-theoretically speaking.
Measurement error also becomes key in modeling things like mediation effects, or in trying to control for covariates. Residual confounds are increasingly being recognized, which is all about measurement error. This is maybe related to overfitting, but many claimed effects can be attributed to measurement error, especially differential measurement error between variables. This is often more of a problem in observational studies, but it can easily apply to experimental designs as well, when there's some ambiguity about how an effect is acting, or if it's actually acting through the mechanisms being hypothesized.
Take the conclusion that "issues like measurement error or distributions, which are equally common, are often not important." I strongly disagree with this. In a classic randomized controlled trial, maybe yes. But even then there are serious problems potentially. Just to offer a few examples:
Measurement is key in this age of unreproducability. It's not uncommon for claims to be made based on results involving one particular measure, when the hypothesis would apply to many measures of the same thing in the sample. When an author claims X causes Y, and there are multiple measures of Y, but results are only reported for one, problems are there. Modeling the joint effect on Y, rather than the measures of Y, is key.
Overfitting goes hand in hand with distributional misassumptions, because to the extent that an observed distribution deviates from the assumed one, overfitting models can capitalize on excess information missed by the base distribution. A classic example of this is assuming a normal distribution when fitting a linear regression but fitting to a very nonnormal distribution: in many cases an interaction term will add significantly even in the absence of a real interaction, because it captures more of the interestingness of the data, information-theoretically speaking.
Measurement error also becomes key in modeling things like mediation effects, or in trying to control for covariates. Residual confounds are increasingly being recognized, which is all about measurement error. This is maybe related to overfitting, but many claimed effects can be attributed to measurement error, especially differential measurement error between variables. This is often more of a problem in observational studies, but it can easily apply to experimental designs as well, when there's some ambiguity about how an effect is acting, or if it's actually acting through the mechanisms being hypothesized.