This is a complex topic, but it's a bit simpler when outcomes are bounded, such as a binary outcome that either occurs or does not occur. In that case, the impact of any one factor is bounded.
In the scenario you're describing, this other factor drowns out any influence the treatment has on the outcome. You'll struggle to get a statistically significant result (low power) and the confidence interval on the treatment effect will include 0. This too can be a valuable finding: sometimes the answer is that the treatment is not particularly effective.
The author is using the technical definitions of confounders and covariates without sufficient explanation, and the technical definitions do not match the normal English definitions.
In English, a confounder is any factor that distorts an observation. (My dictionary defines it as throwing into confusion or disarray.)
In causal inference, a confounder is a factor that is correlated with both treatment and outcome. If the treatment is randomly assigned, by construction it is independent of all other factors. This, there can be no confounders.
Your example is about observed occurrences of imbalance, but the technical definition is about probabilities. Observed imbalances can still skew inference, but that causes high variance (or low precision). It doesn't cause bias (or affect accuracy).
Adjusting for observed imbalances can reduce variance, but in some circumstances can actually cause bias.
In the hard sciences, it's often possible to isolate the phenomenon of interest away from any other influencing factors, e.g. in a laboratory. But many phenomena, like social interactions, or even agriculture, are difficult to isolate in this way. Randomization provides another way of "zooming in" on the treatment of interest.
In the example you gave, a test is going to have very low power because of the important factor with huge variance. If that factor is observed, you can create pairs of units with that factor identical within the pair, then randomly assign treatment to one unit in each pair.
As someone who works with causal inference most days, I expected to find much fault with this article. I was pleasantly surprised to find how rigorous the article is, despite some other comments here. For more information on the role of randomization in causal inference (experimental and observational), I recommend the books by Paul Rosenbaum, especially, "Observation and Experiment".
The textbook, "Bayesian Data Analysis" by Gelman et al, has a good discussion on this in Chapter 8. Here are some relevant bits:
"A naive student of Bayesian inference might claim that because all inference is conditional on the observed data, it makes no difference how those data were collected. This misplaced appeal to the likelihood principle would assert that given (1) a fixed model (including the prior distribution) for the underlying data and (2) fixed observed values of the data, Bayesian inference is determined regardless of the design for the collection of the data. Under this view there would be no formal role for randomization in either sample surveys or experiments."
"The notion that the method of data collection is irrelevant to Bayesian analysis can be dispelled by the simplest of examples. Suppose for instance that we, the authors, give you, the reader, a collection of the outcomes of ten rolls of a die and all are 6's. Certainly your attitude toward the nature of the die after analyzing these data would be different if we told you (i) these were the only rolls we performed, versus (ii) we rolled the die 60 times but decided to report only the 6's, versus (iii) we decided in advance that we were going to report honestly that ten 6's appeared but would conceal how many rolls it took, and we had to wait 500 rolls to attain that result."
I played around with different tap&hold durations (how long you have to hold before it registers as a hold rather than a tap), but I just couldn't find a duration that matched my rhythm. I kept getting holds when I wanted taps (for short durations) or vice versa for longer durations. My current config uses tap&hold on two keys (home and end, which I don't use often anyway). Holding leads to "hyper" and "meh" which are two extra modifier keys (like control or alt/option/meta). I'm an emacs user so extra modifiers means I can basically store as many macros as I want!
I have a Kinesis Advantage 2 and the thumb clusters are my favorite part. I think I probably spend too much time optimizing the layout of those keys though. Like, Escape is really awkwardly placed in the default layout so I moved it to the right thumb cluster, but I was actually just thinking about moving it to the left cluster. Talk about micro-optimizations!
There are some great technical solutions offered, but the real one is to publish your data and your code, and be as transparent as you can about what you did.
Agreed. But as Cox and Hinkley discuss, the likelihood principle is sometimes at odds with the repeated sampling principle, so in any particular application, you need to identify if there is a conflict, and if so, which principle is more important. In my domain (simple A/B tests), you can claw the repeated sampling principle from my cold, dead hands.
There is less of a conflict than many would have you believe. In many situations, both approaches yield the same answer. There are some edge cases. For example, in A/B testing, is early peeking bad? From a frequentist perspective the answer is "yes, either use a sequential method, or don't early peek at all". From a Bayesian perspective the answer is "early peeking is fine".
It boils down to what properties you want your analysis to have. Cox and Hinkley's "Theoretical Statistics" has a great discussion (section 2.4). Basically, you might want your analysis to have a certain kind of internal consistency. But you might also want your analysis to be replicable either by yourself or by another researcher. Those both seem like pretty important things! But there are edge cases (like the early peeking example) where you can't have it both ways. So you have to pick which one you want, and use the corresponding methods.
Gelman is one of the few self-proclaimed Bayesians who doesn't seem to outright hate frequentist approaches. They're complementary approaches. Bayesian methods are great for combining different sources of information. Frequentist methods are great for validating that a method is working well. (For example, Gelman often recommends running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do.)
Frequentism is mostly about how to evaluate a methodology. It's pretty agnostic about what that methodology is. Bayesian methods are about combining different sources of information. In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer.
People say you might as well always use Bayesian methods then. But no matter what, you should always try to validate or poke holes in your model, and Frequentist techniques are great for that. So it's best to be familiar with both!
Or maybe businesses see more of a need for causal reasoning, with or without a controlled, randomized experiment. If you're making a decision, you need to have some model of how your decision affects outcomes you care about. But that model needs to be causal--predictive models can be useful starting points but may also lead you astray due to spurious correlations.
Economists are trained to approach these kinds of problems methodically, so it makes total sense to me that there is high demand.
Really interesting. I've always wished there was a formula to be better at conversations. I've never been good at it! I've tried to find books but the ones I've found come off as manipulative, not as a cure for social awkwardness! Doorknobs are a great mental model!
At the risk of a no-true-Scotsman argument, I want to point out the difference between Frequentism as practiced by statisticians and non-statisticians. Non-statisticians will simply say that a result was stat sig, without specifying even the corresponding p-value, let alone an effect size point estimate or confidence interval.
I always report confidence intervals front and center, and bring in point estimates and p values as supporting characters. And of course discussing to what extent the study design supports causal conclusions.
Regardless of the pros and cons of Bayesian methods, here is what I believe is needed:
- Pre-register all studies, declaring sample sizes and power analysis.
- Report results regardless of outcome. Eliminate the "we only publish stat sig results" baloney.
- Report confidence/credible intervals, adjusting for multiple comparisons as appropriate. Plot the posterior distribution of the effect size if appropriate.
- Publish all data and code.
- Provide funding for duplicating important studies.
In the scenario you're describing, this other factor drowns out any influence the treatment has on the outcome. You'll struggle to get a statistically significant result (low power) and the confidence interval on the treatment effect will include 0. This too can be a valuable finding: sometimes the answer is that the treatment is not particularly effective.