This is the exact comparison we make in the paper.
We have subsampled ImageNet experiments as well; see Figure 3 in the full paper: https://arxiv.org/abs/2010.08127
Note that we also have ImageNet experiments, with entirely real data (non-synthetic). See Section 4, and in particular Figure 3 of the full paper: https://arxiv.org/abs/2010.08127
To clarify some other comments on this post: In all settings, we compare "Real World" and "Ideal World" for the same underlying distribution. Eg, we never compare CIFAR-10 and CIFAR-5m, we only compare "Real World CIFAR-5m" vs "Ideal World CIFAR-5m".
Oh yes, absolutely. We only make claims for "natural distributions and models" (^).
It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme.
But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings".
---
(^) Whatever "natural distributions" means... (which I think is a good research direction in itself).
Hi, we define the interpolation threshold in Section 2 of the full paper (https://arxiv.org/abs/1912.02292)
as the point when the "Effective Model Complexity" = # of train samples.
Where the "Effective Model Complexity (EMC)" of a model + training procedure (w.r.t an input distribution) is the maximum number of samples from the distribution that the model+training can fit to ~0 train error.
Our experiments are consistent with the hypothesis that the double-descent peak occurs when EMC = n; that is, when the model+training is just barely able to fit the train set.