A sharp-eyed scientist became biology’s image detective(newyorker.com)
newyorker.com
A sharp-eyed scientist became biology’s image detective
https://www.newyorker.com/science/elements/how-a-sharp-eyed-scientist-became-biologys-image-detective
26 comments
Thanks for letting me/us know about the Patreon! I'm now a happy contributor.
Elisabeth Bik is worth following on Twitter: https://twitter.com/MicrobiomDigest
She posts images from widely-cited scientific papers and then her followers race to find the duplications in the images. These challenges range from easy to very hard.
Her work is pure positive externality: nobody has incentive to pay for it. Journals barely care about fraud (as long as it's not caught by someone else), reviewers don't have much incentive to check carefully for it (and might even be in on the fraud), and the authors engaging in fraud obviously prefer that journals not look too closely.
There are many more interested parties than journals, reviewers, and authors. Universities have incentives to protect their reputations from fraudulent scientists; funding agencies have incentives to protect their funds from being misused, etc. Certainly they don't have resources to exhaustively check the outputs of their scientists, however, and it seems difficult or impossible to quantify the return on investment of such proactive research integrity efforts.
This seems like an area where one of the large science-focused philanthropic organizations -- the Gates Foundation, Allen Institute, Chan Zuckerberg, HHMI, etc. -- would be well positioned to have a huge impact on the quality of scientific output and also reap a large return in terms of positive press for their efforts. Any of these organizations could easily fund an entire team of Elisabeth Biks, not to mention an AI team on the side to augment and partially automate this work.
There are many more interested parties than journals, reviewers, and authors. Universities have incentives to protect their reputations from fraudulent scientists; funding agencies have incentives to protect their funds from being misused, etc. Certainly they don't have resources to exhaustively check the outputs of their scientists, however, and it seems difficult or impossible to quantify the return on investment of such proactive research integrity efforts.
This seems like an area where one of the large science-focused philanthropic organizations -- the Gates Foundation, Allen Institute, Chan Zuckerberg, HHMI, etc. -- would be well positioned to have a huge impact on the quality of scientific output and also reap a large return in terms of positive press for their efforts. Any of these organizations could easily fund an entire team of Elisabeth Biks, not to mention an AI team on the side to augment and partially automate this work.
The problem is really that there are strong incentives to produce this stuff, usually from the universities or hospitals themselves. These aren’t prestigious places, and it’s not in their interest to spotlight how many garbage papers they produce.
She advocates to „Limit spread of [scientific] misinformation on social media“ [1] yet uses social media to attack published and peer reviewed science (the gold standard used by fact checkers).
[1] https://mobile.twitter.com/MicrobiomDigest/status/1410653746...
[1] https://mobile.twitter.com/MicrobiomDigest/status/1410653746...
She calls attention to instances of fraud that have made it through peer review and publication. Are you saying that is somehow comparable to spreading "misinformation" on social media?
Published research is not perfect; tons of crap gets through peer review. Every scientist knows this. Peer reviewed science may be more reliable than other forms of information, but treating it as settled and unquestionable is foolish and, in fact, unscientific.
Published research is not perfect; tons of crap gets through peer review. Every scientist knows this. Peer reviewed science may be more reliable than other forms of information, but treating it as settled and unquestionable is foolish and, in fact, unscientific.
My problem is her call for censorship of misinformation.
Currently institutional judgment on misinformation relies on fact checkers, that in best case turn to published research, which is not perfect as you pointed out.
Depending on the zeitgeist in the future she calls for, she may be censored herself.
Currently institutional judgment on misinformation relies on fact checkers, that in best case turn to published research, which is not perfect as you pointed out.
Depending on the zeitgeist in the future she calls for, she may be censored herself.
An expansion of that point in your original message might have been better received. The way it was phrased read like you yourself held the opinion that she was using social media to spread lies.
That tweet you linked was talking about her ideas about how to improve scientific integrity. I would hardly characterize it as a call to censorship.
As a "gold-standard" it's still fallible. Peer-reviewed studies can and do fail replication.
Elisabeth Bik has done some amazing work catching these kinds of errors. I occasionally follow Retraction Watch which cites her efforts regularly, [0] about 500 times or so. However, as she points out, she doesn't get compensated for this work:
> We need this to be a career that people can make money and use their talents in. [1]
Journals, IMO, should adopt something like a Knuth reward check [2] as an actual system. They, after all, are the ones profiting the most from our present system of scientific publications. I'm not sure what's fair; maybe 100 USD for catching a major post-publication error?
It's currently a system where most of the incentives operate in opposition to self-correction.
[0] https://retractionwatch.com/
[1] https://retractionwatch.com/2019/05/07/meet-elisabeth-bik-wh...
[2] https://en.wikipedia.org/wiki/Knuth_reward_check
> We need this to be a career that people can make money and use their talents in. [1]
Journals, IMO, should adopt something like a Knuth reward check [2] as an actual system. They, after all, are the ones profiting the most from our present system of scientific publications. I'm not sure what's fair; maybe 100 USD for catching a major post-publication error?
It's currently a system where most of the incentives operate in opposition to self-correction.
[0] https://retractionwatch.com/
[1] https://retractionwatch.com/2019/05/07/meet-elisabeth-bik-wh...
[2] https://en.wikipedia.org/wiki/Knuth_reward_check
This seems like an interesting use for crypto-style staking. Rather than simply paying Elsevier or Springer to publish, which is a transaction that increasingly makes no sense, instead take those fees and put them in an interest-generating escrow account. If the paper is retracted within some window of time (say, 3 years), the money goes to the group that found the problems. If not, it gets returned plus interest to the authors or their organization.
This would create an economic incentive to hunt scientific fraud, and possibly even replication studies, while also over time putting the money that's currently just being gouged by legacy publishers back into science.
This would create an economic incentive to hunt scientific fraud, and possibly even replication studies, while also over time putting the money that's currently just being gouged by legacy publishers back into science.
I like this because it'd be easy to relate to. It is a formal version of:
"Look, I bet you that when [scenario details] happens then [result] will happen. And here is money on the line to prove it."
"Look, I bet you that when [scenario details] happens then [result] will happen. And here is money on the line to prove it."
Before someone asks about why can't this work be automated, here's the FAQ where she answers this question.
https://scienceintegritydigest.com/frequently-asked-question...
https://scienceintegritydigest.com/frequently-asked-question...
She doesn't say it can't be automated; in fact she basically says the opposite, only that it hasn't been automated yet:
Q: Someone should develop software to find these things. Software is going to be much better than the human eye.
A: I agree, and several groups are working hard on that. I have shared a dataset of 400 papers with image problems and 800 matched controls in which I did not find any problems with several teams that are developing such software. I have not heard back from any of them, so I assume this is much harder than most people assume. But this is only a matter of time.
Q: Someone should develop software to find these things. Software is going to be much better than the human eye.
A: I agree, and several groups are working hard on that. I have shared a dataset of 400 papers with image problems and 800 matched controls in which I did not find any problems with several teams that are developing such software. I have not heard back from any of them, so I assume this is much harder than most people assume. But this is only a matter of time.
Yeah, it's great that she is doing this.
But I am still left shaking my head. How did we get to this point? Science is the peak achievement of our civilisation, the pandemic is a clear testament to that. The kind of scientific fraud she picks out is incredibly disappointing, whether it is careless or mercenary or (usually) some combination of both.
But I am still left shaking my head. How did we get to this point? Science is the peak achievement of our civilisation, the pandemic is a clear testament to that. The kind of scientific fraud she picks out is incredibly disappointing, whether it is careless or mercenary or (usually) some combination of both.
How did we get to this point?
We've always been here. There have been tales of fraud peppered throughout history, after all. Pirates and thieves and literal snake oil salesmen.
We've always been here. There have been tales of fraud peppered throughout history, after all. Pirates and thieves and literal snake oil salesmen.
Science is the peak achievement of our civilisation
Maybe this kind of uncritical scientism played a part?
the pandemic is a clear testament to that.
It's not even clear yet that the practice of science did not cause the pandemic. Science is a tool and when people start treating it like an end, such problems will arise.
Maybe this kind of uncritical scientism played a part?
the pandemic is a clear testament to that.
It's not even clear yet that the practice of science did not cause the pandemic. Science is a tool and when people start treating it like an end, such problems will arise.
Scientism is an extreme view that has long been considered untenable. My statement about the value of science has nothing to do with scientism. I did not say it is the only thing of value, nor do I think that is true.
For those who may appreciate it:
https://outline.com/pKdazw
https://outline.com/pKdazw
This kind of faking scientific results should be a crime. Maybe then we will clean the academic circles from that frauds. She is a real hero we need.
This kind of faking scientific results should be a crime.
In many cases, it already is -- generally falling under misuse of government research funds. However it is usually only pursued as such in extreme cases. For example: https://en.wikipedia.org/wiki/Dong-Pyou_Han
In many cases, it already is -- generally falling under misuse of government research funds. However it is usually only pursued as such in extreme cases. For example: https://en.wikipedia.org/wiki/Dong-Pyou_Han
makes me sad to read that there are so many bad actors out there in the biology scientific community. hope she continues what she does - what a great service to the community and she seems to be bootstrapping everything herself.
Non-paywalled link:
https://web.archive.org/web/20210701031117/https://www.newyo...
https://web.archive.org/web/20210701031117/https://www.newyo...
Her work is pure positive externality: nobody has incentive to pay for it. Journals barely care about fraud (as long as it's not caught by someone else), reviewers don't have much incentive to check carefully for it (and might even be in on the fraud), and the authors engaging in fraud obviously prefer that journals not look too closely.
That's why I am a patron: https://www.patreon.com/elisabethbik
She maintains a blog if you want to read more about what she does, e.g.: https://scienceintegritydigest.com/2020/12/31/2020-a-year-in...