Reliable novelty: New should not trump true(journals.plos.org)
journals.plos.org
Reliable novelty: New should not trump true
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000117
40 comments
I'm not a scientist, but in software development some people used to think that planning and designing everything up front was a good way to go about things. Turns out that taking all the big decisions before getting your hands dirty and really learning about the problem domain isn't really a great idea.
There's a long and unpleasant track record in science of people doing forming a hypothesis, doing trials, mining and changing methods and hypotheses and p-hacking for anything significant, and publishing a misleading paper that completely fails to mention that the original hypothesis was abandoned halfway through.
That may sound pretty agile. It is! You're absolutely right that this is an excellent match! The issue is that while agile may work pretty well for software dev, it perhaps doesn't work so well for science. It's given rise to a number of abuses.
It might also be noted that researchers doing experimental design are often quite familiar with their field, having gotten their hands dirty in a problem domain repeatedly.
That may sound pretty agile. It is! You're absolutely right that this is an excellent match! The issue is that while agile may work pretty well for software dev, it perhaps doesn't work so well for science. It's given rise to a number of abuses.
It might also be noted that researchers doing experimental design are often quite familiar with their field, having gotten their hands dirty in a problem domain repeatedly.
There's the XKCD about significant P-values: https://www.xkcd.com/882/
To prevent that from happening, all the experiments that failed need to be reported too, and the link between them has to be obvious.
To prevent that from happening, all the experiments that failed need to be reported too, and the link between them has to be obvious.
Exactly this kind of registration is mandatory for clinical trials under the Declaration of Helsinki and in US and EU law, but it's still poorly enforced.
If this issue matters to you, I would encourage you to support the AllTrials campaign; they have done stellar work in exposing publication bias.
http://www.alltrials.net/
If this issue matters to you, I would encourage you to support the AllTrials campaign; they have done stellar work in exposing publication bias.
http://www.alltrials.net/
To put on a different hat for this: when I went to do my Master's thesis, I actually did not have a hypothesis nor did I have expected results. Maybe it was unique for us, but we actually did not know what to expect for the outcome of the research, that was the whole point of it.
We had procedures so that when we came up with a result, it could be validated and we made sure we didn't do anything that could have tainted the research.
We had procedures so that when we came up with a result, it could be validated and we made sure we didn't do anything that could have tainted the research.
I believe this practice should be inculcated in most research projects. However, if I put my adversarial hat on - I see one challenge. Scientists could still perform their experiments beforehand with the hypothesis, procedures and results. During submissions, they will _phase_ these two-stage information and give an impression of not performing posthoc rationalization. However, in reality they would. Such a system could be business as usual for the scientists unless you control the power for scientists to perform their experiments which seems impractical.
The idea is extremely great but I believe this could be difficult to implement at a publication level.
The idea is extremely great but I believe this could be difficult to implement at a publication level.
For short experiments you could game the system, but for an experiment that requires two years of work, it’s unlikely that you’d hold your results for two years to maintain the illusion.
Also you could tie the release of funds to publication of expected methods.
Also you could tie the release of funds to publication of expected methods.
[deleted]
Global. Warming. Exponential. Curve.
This is a risk, yes. I don’t necessarily think it’s a big problem.
First off, you’re describing premeditated, wilfully fraudulent behaviour. I suspect a lot of the problems are either less deliberate (you react to new information by making changes to your hypothesis or protocol without stopping to think what that does to your experiment’s reliability), crimes of opportunity (“this is so close, there’s no harm in fudging it a little bit, right?”) or a combination of both. It’s ok to enact rules that fight pervasive small issues while doing nothing about rare malicious behaviour!
Even the malicious behaviour would become much harder though. First off, the strategy you described only works for setups where the data collection stage is relatively short. If it takes a long time, then the fraudster’s timelines start looking suspicious. Second, that type of behaviour could really harm you if caught.
First off, you’re describing premeditated, wilfully fraudulent behaviour. I suspect a lot of the problems are either less deliberate (you react to new information by making changes to your hypothesis or protocol without stopping to think what that does to your experiment’s reliability), crimes of opportunity (“this is so close, there’s no harm in fudging it a little bit, right?”) or a combination of both. It’s ok to enact rules that fight pervasive small issues while doing nothing about rare malicious behaviour!
Even the malicious behaviour would become much harder though. First off, the strategy you described only works for setups where the data collection stage is relatively short. If it takes a long time, then the fraudster’s timelines start looking suspicious. Second, that type of behaviour could really harm you if caught.
It would be difficult in practice to maintain the secrecy needed to pull this off.
While in many cases scientists currently keep what they're working on "mostly secret" until it's published, in fact lots of their friends and colleagues have a pretty good idea of what they're doing and would be in a position to ask hard questions if they tried to take your approach.
While in many cases scientists currently keep what they're working on "mostly secret" until it's published, in fact lots of their friends and colleagues have a pretty good idea of what they're doing and would be in a position to ask hard questions if they tried to take your approach.
Wait, there are groups that don’t do this already?
“We thank reviewer 3 for the insightful suggestion...” (ok now copy paste the results that we cut for submission)
Ha ha only serious. Otherwise you get “helpful suggestions” like “run another few phase III trials first”
“We thank reviewer 3 for the insightful suggestion...” (ok now copy paste the results that we cut for submission)
Ha ha only serious. Otherwise you get “helpful suggestions” like “run another few phase III trials first”
That would make it an explicit cheat. Right now, not publishing negative results, tweaking the procedure description a little and things like that lie somewhere in a gray zone and can be socially acceptable in some lab to some extent. However, if this would be explicitly defined, breaking this would be an explicit straightforward lie, which would make it harder to make socially acceptable in some limited circles, people would have to go to greater length to hide it and this kind of thing would not happen that often.
Doing that would require researchers to overtly and explicitly lie, which would act as at least some disincentive.
If caught, the dishonesty would be truly obvious and harmful to a reputation
This would lead to a long string of embarrassing cancelled experiments being visible in the literature, and I can only guess that's a main reason it's not done. It gets worse if the experiment was to test a controversial hypothesis (or even, bad publicity around the hypothesis being tested could shut down the experiment, or in some cases, compromise the results).
Hypotheses could be sealed upon filing. The existence of sealed hypotheses could/should be public, to prevent the system from being gamed by filing dozens of hypotheses at once for the same experiment and then selectively unsealing the one that matched their results. If the experimenter had too many sealed hypotheses to their name, that would be seen as cause for concern. But only a handful of sealed hypotheses would be unlikely to raise any eyebrows. Furthermore keeping one's sealed:unsealed ratio low would be an incentive to publish negative results. This system would also protect the experimenter from political pressure if they were testing a controversial hypothesis.
This has been discussed before:
https://dynamicecology.wordpress.com/2013/10/21/two-stage-pe...
http://neurochambers.blogspot.com/2013/04/scientific-publish...
Personally as a researcher, I think the idea is definitely a good one.
https://dynamicecology.wordpress.com/2013/10/21/two-stage-pe...
http://neurochambers.blogspot.com/2013/04/scientific-publish...
Personally as a researcher, I think the idea is definitely a good one.
In my experience quality vs prestige is a U-curve.
At one extreme you have the really crappy journals that publish nonsense, at the other the really "prestigious" journals that are more like tabloids reporting exciting sounding conclusions with totally inadequate methods sections.
In the middle are the topic-specific journals that you need to read to guess what they did to get the "exciting" results.
"Science" in the late 1990s and early 2000s is the absolute worst example of this.
At one extreme you have the really crappy journals that publish nonsense, at the other the really "prestigious" journals that are more like tabloids reporting exciting sounding conclusions with totally inadequate methods sections.
In the middle are the topic-specific journals that you need to read to guess what they did to get the "exciting" results.
"Science" in the late 1990s and early 2000s is the absolute worst example of this.
At least in my field there is the running joke of "it was published in Nature, but it still might be correct."
Well yeah, obviously: high risk, high reward. Without the benefit of hindsight (and subsequent tests), a paper that proposes something truly transformative will look a lot like one that is just wrong (albeit, wrong in a creative/novel way). If you want to publish the former, you're going to have to take a lot of risks and publish a lot of duds. Don't we complain all the time that NSF doesn't fund high-risk science?
That isn't what is going on. The stuff that gets published in the "tabloids" (Nature, Science, PNAS, etc) is sloppier.
I don't know your field, but that hasn't really been my experience at all. Just about as much sloppiness all the way across the board.
My only systematic complaint here is that short format means much of the substance has to go in an inconvenient supplement, but that's hardly unique to Nature and Science.
My only systematic complaint here is that short format means much of the substance has to go in an inconvenient supplement, but that's hardly unique to Nature and Science.
The supplements have vastly improved the situation. There is no denying that.
"Nature", outside the biology area, tends to be an example of this.
And honestly even though we all agree about this in my field (condensed matter) we still try to get our work published there because it's what helps people get ahead in academic life.
It's such a perverse vicious cycle that can't seem to be broken...
It's such a perverse vicious cycle that can't seem to be broken...
Why do you make an exception for biology?
They might be familiar with a subfield where there's a strong culture of accountability through reproduction of experimental results. Certain parts of molecular biology at least used to be that way.
most bio papers in nature are irreproducible, except by a few top competitive labs, and even then, most don't bother unless they don't believe the results.
Nature inside biology too!
Whenever I look at a biology paper in Nature I frequently see figures like this:
https://www.nature.com/articles/s41586-019-1112-8/figures/5
They’re utterly incomprehensible to my lowly engineers eye and seem deliberately designed to look as complex as possible.
What do these kinds of images and plots tell us of any value?
https://www.nature.com/articles/s41586-019-1112-8/figures/5
They’re utterly incomprehensible to my lowly engineers eye and seem deliberately designed to look as complex as possible.
What do these kinds of images and plots tell us of any value?
As someone who did computational biophysics, my opinion is that MD is meaningless and those plots don't tell you anything of value. Most likely, the senior author is friends with a MD expert and wanted an excuse to put the friend on the paper.
This type of visualisation appears pretty common in such papers so I think perhaps the authors believe that they now need to include them to get published, a cargo cult of sorts.
> "What do these kinds of images and plots tell us of any value?"
It tells you "p38γ is essential for cell cycle progression and liver tumorigenesis". You have to be blind not to see that.
It tells you "p38γ is essential for cell cycle progression and liver tumorigenesis". You have to be blind not to see that.
[deleted]
“Should” is an amazingly dangerous word (in the usage often embraced in the US, at least).
I find that asking why something “should” be XYZ is much more enlightening, both for the writer and the reader, in terms of exposing structural problems that make it so.
“People seldom do what they know to be right. They do what is convenient, then repent” —attributed to bob dylan
(Corollary: make the “right” thing the least inconvenient thing to do, and the odds of people doing it will rise appreciably.)
I find that asking why something “should” be XYZ is much more enlightening, both for the writer and the reader, in terms of exposing structural problems that make it so.
“People seldom do what they know to be right. They do what is convenient, then repent” —attributed to bob dylan
(Corollary: make the “right” thing the least inconvenient thing to do, and the odds of people doing it will rise appreciably.)
Sounds like a dangerous title since the word true is subjective and requires variable amounts of support, even no evidence, to qualify its use.
What?
it's better to think of the top journals as idea forges, designed to maximize the flow of thought between the top labs, rather than the authoritative location for technically correct research.
The other solutions presented were also good though. Making the data cleaner for other to analyze is always a good thing, especially in the "machine learning" age.