I stand corrected on that point then. Thanks for the info!
So it looks like they cherry-picked the 6 subjects who did (by chance) worse on the memory task than the other tasks. Then regression to the mean happened, and voila! Some positive results.
it sounds like you are assuming they collected more data than they reported, is there evidence of that? They preregistered to collect 60, but they couldve simply been unable to collect that many (eg COVID).
If it was preregistered perhaps the journal were obligated to publish the results no matter the outcome. Some journals do this ostensibly to combat publication bias.
I didn’t read the whole thing: are the authors actually claiming the human data are informative? If so, that is wrong. But if they are just saying “hey, these are the data!” I wouldn’t be so harsh.
Edit: Just read the methods. Data collection was 2016 to 2020, so COVID probably wasn’t the issue. They were selected for poor memory yet normal cognition in other domains, so it is possible they couldn’t recruit the number of subjects they proposed who met this criteria and also were willing and able to complete the treatment.
I’m now quite confident the memory result is regression to the mean, and therefore nothing to do with the treatment.
I also read the portion of the discussion related to cognition. They are claiming to have found something meaningful (in the human data) when they definitely haven’t.
I skimmed the article for the pertinent table. Three observations:
1. There is no control group. Since people get better with cognitive tests with practice, we don’t know whether these improvements were due to the treatment or just the normal practice effect. (Though if I were a betting person, my money is on the latter.)
2. Cognitive tests are not noiseless. The same person may vary by a few points between repeated tests. The modest improvements on most tests found here might not be above this noise floor.
3. The most impressive result (memory) is almost certainly regression to the mean. That is, the group did unusually poorly on this task in the first session by chance, then they performed more typically in the second, which looks falsely like a big improvement.
Basically, I think this paper provides no meaningful evidence of cognitive improvement in humans. I can’t speak about the mice stuff, that might be fine.
This is so not true on many levels. Science is not about being right every time. Positive experimental findings can turn out to be wrong even if the study was excellent. This is called a type 1 error, and indeed they are guaranteed to happen with 5% probability assuming every study is conducted correctly and people don’t cheat.
Also, I don’t think you understand that there is a difference between “replicate” and “reproduce” as those terms are typically used in social and biological science. Reproduction refers to the methods and is simply conducting the experiment again, regardless of the outcome. An experiment should be reproducible, of course—if it isn’t, the paper was poorly written (the experiment might still be ok though).
Replication refers to the results: do we get the same answers as the original experiment? Bad science is one way an experiment might not replicate but other reasons are 1. Just a regular type 1 error in the original study, 2. A type 2 error in the new one, 3. an overlooked methodological difference or mistake somewhere. No. 3 doesn’t mean bad science, it could be just normal scientists making normal mistakes, as they sometimes do.
Also, you don’t “hypothesize” an experiment. You come up with a hypothesis then test it with an experiment. (Sometimes scientists HARK but that’s a different discussion …)
In this case, it sounds like there truly was some cheating going on of the worst kind (data fabrication).
I'm assuming you mean to imply that I'm being overdramatic by suspecting something nefarious is going on when it isn't. That might be true, yes, thanks for your input.
> Which means Comcast has your current address. It is not hard to look up, in their own database, a name when given an address from a rental listing website.
Yes, this is exactly what I imagined, which is why I wrote, "I suppose Comcast could deduce their customers’ behavior by scraping rental sites," but perhaps it wasn't totally clear.
> Their marketing department likely saw it as a "customer retention" activity. By offering to make the transfer of your Comcast account to your new location easy and effortless (they hope) their belief would be that they might retain you as a customer.
So it looks like they cherry-picked the 6 subjects who did (by chance) worse on the memory task than the other tasks. Then regression to the mean happened, and voila! Some positive results.