It's always refreshing to meet a fellow R hacker on HN!
May I ask you why you chose to use the normal distribution in your example or any distribution at all, for that matter? What I was replying to was
">they only test null hypothesis that are true."
Which means that the null hypothesis is always true no matter what data you collect trying to reject it. It does not depend on the null distribution (normal in your example), the value of the test statistic (the mean of the sample in your example), or the threshold (crit in your example). In fact, the null distribution in this case is not a distribution at all since there's no randomness in the null hypothesis. We know for a fact that it is always true (in the hypothetical situation we are considering).
It's more like
> rep(FALSE, simulations) # is the null hypothesis false? nope
or, if you insist on using the normal distribution,
In fact, in your example, since you are essentially running 1000 hypothesis tests on different samples, multiple hypothesis correction would solve the "problem" with p-value. This is how I would do it.
The only comment to the point so far has gotten downvoted. How telling... Bashing p-values without really understanding what they are about is quite trending these days.
If a null hypothesis is invariably true, it's impossible to reject it. Which means the scientists will not be able to find any statistic or data to support any of their bad, original hypotheses. Not 5%, not 0.005%, nor whatever.
p-values are not flawed. They are a useful tool for a certain category of jobs: namely to check how likely your sample is, given a certain hypothesis.
The argument in the original post is a bit of a straw man fallacy.
"I want to know the probability that the null is true given that an observed effect is significant. We can call this probability "p(null | significant effect)"
OK, hypothesis testing can't answer this type of questions.
Then "However, what NHST actually tells me is the probability that I will get a significant effect if the null is true. We can call this probability "p(significant effect | null)"."
Not quite correct. It's "p(still NOT a significant effect whatever it means | null)".
May I ask you why you chose to use the normal distribution in your example or any distribution at all, for that matter? What I was replying to was
">they only test null hypothesis that are true."
Which means that the null hypothesis is always true no matter what data you collect trying to reject it. It does not depend on the null distribution (normal in your example), the value of the test statistic (the mean of the sample in your example), or the threshold (crit in your example). In fact, the null distribution in this case is not a distribution at all since there's no randomness in the null hypothesis. We know for a fact that it is always true (in the hypothetical situation we are considering).
It's more like
or, if you insist on using the normal distribution,
In fact, in your example, since you are essentially running 1000 hypothesis tests on different samples, multiple hypothesis correction would solve the "problem" with p-value. This is how I would do it.
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS Based on 1000 bootstrap replicates
CALL : boot.ci(boot.out = boot.out, type = "basic")
Intervals : Level Basic 95% ( 0.9774, 0.9780 ) Calculations and Intervals on Original Scale
P.S. p-values are great when used appropriately.