This is a fascinating discussion, to which I have little to add, except this. Quoting the article (including the footnote):
> [I]f you carefully craft random data so that it does not contain a Dunning-Kruger effect, you will still find the effect. The reason turns out to be embarrassingly simple: the Dunning-Kruger effect has nothing to do with human psychology[1].
> [1]: The Dunning-Kruger effect tells us nothing about the people it purports to measure. But it does tell us about the psychology of social scientists, who apparently struggle with statistics.
It seems to me that despite rudely criticizing a broad swath of academics for their lack of statistical prowess, the author here is himself guilty of a cardinal statistical sin: accepting the null hypothesis.
The fact that data resemble a random simulation in which no effect exists does not disprove the existence of such an effect. In traditional statistical language, we might say such an effect is not statistically significant, but that is different from saying that the effect is absolutely and completely the result of a statistical artifact.
I'm probably nitpicking your language, but L1 regularization is precisely that: regularization. (See https://en.wikipedia.org/wiki/Regularization_(mathematics)#R....) In your typical linear regression setting, it does not replace the squared error loss but rather augments it. In regularized linear regression, for example, your loss function becomes a weighted sum of the usual squared error loss (aiming to minimize residuals/maximize model fit) and the norm of the vector of estimated coefficients (aiming to minimize model complexity).
Releasing any data or statistic based on sensitive data--even once--bears a privacy risk. The primary purpose of differential privacy is to quantify that risk, both for a single release of data and over many releases of data.
As for the number of analyses you can run, that depends on what you mean. You're right that differential privacy won't allow you to set up a database of _confidential data_ that can be arbitrarily queried infinitely many times with any meaningful privacy guarantee, but this is in no way unique to differential privacy.
What you can do with differential privacy is release noisy statistics once and let researchers use those statistics for arbitrarily many analyses. This is what the 2020 US Census is doing, for example.
One of the interesting insights in differential privacy is that to provide privacy protections that can't be reverse-engineered, the process has to be random rather than deterministic. The sort of algorithm that OP describes is really neat, but in addition to what dp_throw says, deterministic algorithms like this that choose how to anonymize things based on private data can reveal information about that private data in the very way that they format the final data. (This may be less relevant in the case at hand, but consider a setting where it would be sensitive to know if someone is in the database at all, e.g., a medical study.)
> [I]f you carefully craft random data so that it does not contain a Dunning-Kruger effect, you will still find the effect. The reason turns out to be embarrassingly simple: the Dunning-Kruger effect has nothing to do with human psychology[1].
> [1]: The Dunning-Kruger effect tells us nothing about the people it purports to measure. But it does tell us about the psychology of social scientists, who apparently struggle with statistics.
It seems to me that despite rudely criticizing a broad swath of academics for their lack of statistical prowess, the author here is himself guilty of a cardinal statistical sin: accepting the null hypothesis.
The fact that data resemble a random simulation in which no effect exists does not disprove the existence of such an effect. In traditional statistical language, we might say such an effect is not statistically significant, but that is different from saying that the effect is absolutely and completely the result of a statistical artifact.
The nuance of statistics is never-ending.