The Absurdity of Peer Review: What the Pandemic Revealed(elemental.medium.com)
elemental.medium.com
The Absurdity of Peer Review: What the Pandemic Revealed
https://elemental.medium.com/the-absurdity-of-peer-review-1d58e5d9e661
29 comments
One of the worst hydroxychloroquine studies - the now-retracted one in the Lancet claiming that patients receiving the drug were dying at a higher rate, which appears to have used completely fake data and where there was no possible way they could've got the data they purported to have used - was in fact peer reviewed. Peer review really doesn't do as much as people seem to think it does.
So then what? Even if peer review does not solve the problem of bad research completely, it certainly mitigates it better than not doing anything.
How would you solve this problem?
How would you solve this problem?
By attributing less weight to the imprimatur of peer review.
It's a sign that something might well hold up, but it's not definitive. I think people know this about their own field, and then immediately forget it when dealing with another field.
It's a sign that something might well hold up, but it's not definitive. I think people know this about their own field, and then immediately forget it when dealing with another field.
> Even if peer review does not solve the problem of bad research completely, it certainly mitigates it better than not doing anything.
This isn't self-evident. In fact, it's conceivable that peer review could actually lower the quality of published research by delaying paradigm shifting work from seeing the light of day.
This isn't self-evident. In fact, it's conceivable that peer review could actually lower the quality of published research by delaying paradigm shifting work from seeing the light of day.
How? If someone is going to rely upon the conclusions of a paper, simply read that paper and evaluate the claims. i.e. stop the sole use of the titles of papers and their cliffs notes to support a point.
I think the problem people are getting is that "peer reviewed" has become a badge of expertise or authority in itself, inflating people's trust in it and weakening skepticism.
The problem is that "peers" of a like mind are easily found, when churning papers is rewarded. Quantity beats Quality in today's world.
Peer review can’t catch fake data. Peer review can look at if an experiment is repeatable and repeating the experiment can show fake data.
Peer review definitely could and should have caught the fake data in this case - there were too many implausible things about this paper that one could verify with a google search and see cannot be right.
In fact, it took people about one day from publication to start pointing them out with proof - and these people were (supposedly) less qualified than the peer reviewers.
It is true that, with properly faked data, a reviewer cannot be expected to catch it. Nor can p-hacking generally be discovered from an incomplete description.
But quite a bit of faked and even p-hacked data is less-than-competently done in a way that a reviewer can and should be able to catch;
And observational data or meta-study are much easier to verify, since no experiment was done by the researcher either.
In fact, it took people about one day from publication to start pointing them out with proof - and these people were (supposedly) less qualified than the peer reviewers.
It is true that, with properly faked data, a reviewer cannot be expected to catch it. Nor can p-hacking generally be discovered from an incomplete description.
But quite a bit of faked and even p-hacked data is less-than-competently done in a way that a reviewer can and should be able to catch;
And observational data or meta-study are much easier to verify, since no experiment was done by the researcher either.
It is ridiculous to think covid-19 could have originated in a lab’
Have there been any peer-reviewed studies that unambiguously admit the possibility that Covid-19 may have originated in a lab? Although I think it's something that needs serious consideration, the alleged lack of Medical Reliable Sources (basically medical journal articles) is being used aggressively by a small group of editors on Wikipedia to enforce the line that a lab leak is "conspiracy theory".
Have there been any peer-reviewed studies that unambiguously admit the possibility that Covid-19 may have originated in a lab? Although I think it's something that needs serious consideration, the alleged lack of Medical Reliable Sources (basically medical journal articles) is being used aggressively by a small group of editors on Wikipedia to enforce the line that a lab leak is "conspiracy theory".
There's the https://drasticresearch.org/ group, which follows the lab theory. They unambiguously admit the possibility.
> only one of Einstein’s 300 or so published papers was ever peer-reviewed, which so disgusted him that he never submitted a paper to that journal again
I never thought of this, and it is one of those shifts in habits and uses that go unremarked (until, of course, someone points them out like this piece did).
Perhaps due to the industrialization of research after the middle of the XX century?
Now that we don't really need to publish in journals, since anything can be made available on the internet, the filter of credibility can also be changed - for instance to something like being cited approvingly by other reputable researchers.
I never thought of this, and it is one of those shifts in habits and uses that go unremarked (until, of course, someone points them out like this piece did).
Perhaps due to the industrialization of research after the middle of the XX century?
Now that we don't really need to publish in journals, since anything can be made available on the internet, the filter of credibility can also be changed - for instance to something like being cited approvingly by other reputable researchers.
Even if peer review doesn't actually signal paper quality, it's still useful for science journalists to note whether a paper is in its final form or still a work in progress. Referring to unpublished papers as "pre-prints" or "working papers" could be a less biased way to do so.
Also, it's hard to tell whether peer review actually improves paper quality by comparing published / unpublished papers in a world with peer review, where everyone is writing with the knowledge they'll be intensely scrutinized. Without some sort of detailed review process -- even one that's potentially deeply flawed -- researchers would have fewer incentives to be careful.
Also, it's hard to tell whether peer review actually improves paper quality by comparing published / unpublished papers in a world with peer review, where everyone is writing with the knowledge they'll be intensely scrutinized. Without some sort of detailed review process -- even one that's potentially deeply flawed -- researchers would have fewer incentives to be careful.
Ok, so what's the alternative to peer review then? We'll "just know" when that paper/blog entry/tweet/4chan post is written by the next Einstein and we can take it for a fact?
There really isn't any good alternative, having one or more qualified peers go over the results is a lot more trustworthy.
Of course it requires that there isn't any hidden agenda or that a specific result of the review is dictated.
Of course it requires that there isn't any hidden agenda or that a specific result of the review is dictated.
"Fact" in science needs to be based as much as possible on reproducibility by anyone and as little as possible on the support of those with institutional authority.
Alternatives to peer review are seen everywhere these days: open review and discussion and even the open source software model.
A scientist does some work, writes it up in rough form, puts data files online, and tells some friends/colleagues about it. After some feedback and corrections, the scientist puts the writeup online and notifies discussion forums. Then, anyone interested can look it over and provide feedback of all sorts: good, bad, and ugly.
The scientist then incrementally upgrades the paper, including edits and rewording and donated graphics, etc., along the lines of open source software and pull requests. I'm not claiming that many eyes make all bugs shallow, but if someone points out that your algorithm omits the first line of each data table, or your evidence of warming only measured temperature during the day while comparing to temperatures measured day and night, it's true or not regardless of the credentials of the person making the claim.
The paper doesn't have to be fossilized but can be versioned, because "paper" doesn't mean paper anymore, and your institutional worth (as opposed to scientific worth) can be based on citations of all versions rather than the prestige of the journal.
You might fear that authors will be overwhelmed by mostly low-quality feedback, but most papers are read by so few people that this isn't much of a risk, in my opinion. The biggest risk is deliberate lies, but we already have a reproducibility crisis, and only reproducibility--not the filters of authority--can ultimately solve it.
Alternatives to peer review are seen everywhere these days: open review and discussion and even the open source software model.
A scientist does some work, writes it up in rough form, puts data files online, and tells some friends/colleagues about it. After some feedback and corrections, the scientist puts the writeup online and notifies discussion forums. Then, anyone interested can look it over and provide feedback of all sorts: good, bad, and ugly.
The scientist then incrementally upgrades the paper, including edits and rewording and donated graphics, etc., along the lines of open source software and pull requests. I'm not claiming that many eyes make all bugs shallow, but if someone points out that your algorithm omits the first line of each data table, or your evidence of warming only measured temperature during the day while comparing to temperatures measured day and night, it's true or not regardless of the credentials of the person making the claim.
The paper doesn't have to be fossilized but can be versioned, because "paper" doesn't mean paper anymore, and your institutional worth (as opposed to scientific worth) can be based on citations of all versions rather than the prestige of the journal.
You might fear that authors will be overwhelmed by mostly low-quality feedback, but most papers are read by so few people that this isn't much of a risk, in my opinion. The biggest risk is deliberate lies, but we already have a reproducibility crisis, and only reproducibility--not the filters of authority--can ultimately solve it.
A group of us are working on a solution. Here's a reference to a prior comment thread: https://news.ycombinator.com/item?id=27297013
The solution is to make peer-review better, rather than remove it entirely. Imagine that you can choose a set of people who you trust on any particular topic, and rank them with how much you trust them. Then you, and those people, can all rank any paper (or in general any URL) you read, and each of you will have a feed of incoming papers (or URLs) on any topic computed by a function that you control, in your personal reader app, which crawls your network of trusted people, and the people they trust, and so on, and aggregates all rankings and reviews. After about 7 degrees of hops across that network, you'll cover most of the world.
The key new idea here is that peer review will no longer try to be a single, objective, determination of truth. Each person can have their own slightly different feed, computed by slightly different criteria, using slightly different sets of trust metrics of people over the community, all competing for other people's trust, by writing good reviews, and good papers, in order to gain reputation and influence within the distributed network.
This won't be a single app. Rather, it'll be a distributed use of the WWW. Anyone can host their ratings at any web URL, on any website, and subscribe to the ratings from anyone else, at any other URL, at any other website. We can make this all realtime using the web extensions defined by https://braid.org, so that you can have a continuously-updating feed.
In effect, we'll be building a distributed, subjective hacker news (or equivalently, a distributed subjective reddit). We don't need to fight over objectivity anymore. Instead, we'll all be entitled to our own view of the web, and be able to share our views with others. And we'll find that we are best off ourselves when we import the best views from the best people we know, and that way we will see the world through each other's eyes, and even be able to — with the flip of a dial — see the world through our enemies' eyes.
Please reach out if you are interested in building anything similar! I am [email protected], and am coordinating a group to make this happen.
The solution is to make peer-review better, rather than remove it entirely. Imagine that you can choose a set of people who you trust on any particular topic, and rank them with how much you trust them. Then you, and those people, can all rank any paper (or in general any URL) you read, and each of you will have a feed of incoming papers (or URLs) on any topic computed by a function that you control, in your personal reader app, which crawls your network of trusted people, and the people they trust, and so on, and aggregates all rankings and reviews. After about 7 degrees of hops across that network, you'll cover most of the world.
The key new idea here is that peer review will no longer try to be a single, objective, determination of truth. Each person can have their own slightly different feed, computed by slightly different criteria, using slightly different sets of trust metrics of people over the community, all competing for other people's trust, by writing good reviews, and good papers, in order to gain reputation and influence within the distributed network.
This won't be a single app. Rather, it'll be a distributed use of the WWW. Anyone can host their ratings at any web URL, on any website, and subscribe to the ratings from anyone else, at any other URL, at any other website. We can make this all realtime using the web extensions defined by https://braid.org, so that you can have a continuously-updating feed.
In effect, we'll be building a distributed, subjective hacker news (or equivalently, a distributed subjective reddit). We don't need to fight over objectivity anymore. Instead, we'll all be entitled to our own view of the web, and be able to share our views with others. And we'll find that we are best off ourselves when we import the best views from the best people we know, and that way we will see the world through each other's eyes, and even be able to — with the flip of a dial — see the world through our enemies' eyes.
Please reach out if you are interested in building anything similar! I am [email protected], and am coordinating a group to make this happen.
[deleted]
As the article points out, the previous Einstein’s papers were not peer reviewed.
Peer review is not what the pubic and journalists think it is.
An example:
The Physical Review journals publish experimental high energy physics papers from large collaborations without review, because there is no point to it. Anyone competent to review these papers is already on the list of 400 authors.
Science is strongly specialized. Results are largely disseminated by preprint. You know if a preprint makes sense if you’re working in the field. You probably know better than whomever the editor can get to agree to review the thing.
By the time something gets published, a year has gone by and research is already building on work reported in the preprint.
Peer review is not what the pubic and journalists think it is.
An example:
The Physical Review journals publish experimental high energy physics papers from large collaborations without review, because there is no point to it. Anyone competent to review these papers is already on the list of 400 authors.
Science is strongly specialized. Results are largely disseminated by preprint. You know if a preprint makes sense if you’re working in the field. You probably know better than whomever the editor can get to agree to review the thing.
By the time something gets published, a year has gone by and research is already building on work reported in the preprint.
Peer review is necessary.
Peer review is totally failing to do its job.
There is nothing in the current structure of peer review which would suggest that it could be expected to do its job.
Discuss.
This is a timely article about an important issue in the sciences.
The author begins by discussing the importance given in popular media to whether articles are peer reviewed or not, using somewhat absurd examples of foundational historical papers to stress the point.
The author then makes a logical jump, and claims that "Science worships peer review". However neither the journal Science nor the discipline of the same name are capable of worshipping anything. The author might mean to say that "Scientists worship peer review"? But for such a claim the author provides little evidence, and further it would contradict the later statement that "simple inertia, and ..." are the reasons "we still have peer review".
Subsequently the author tries to demonstrate that peer review is useless by stressing the multitude of problematic papers that did make it through peer review. This, of course, is a standard logical fallacy and again in direct contradiction with the latter statement that "we do not know whether the science we produce is better with or without it" (emphasis mine).
As I mentioned above, it is clear that the author raises an important point and at its root the message it tries to convey is certainly valid. However the article itself adds little to what scientists (and I would wager the informed public) already know. Besides the issues mentions above, I think a further weakness is that it claims to speak for all of science but I might imagine that in a field like mathematics the act of peer review can be more meaningful.
Altogether, then, I am of the opinion that the above issues need to be corrected in a substantial revision before I can accept the article for publication in your journal.
Sincerely,
The author begins by discussing the importance given in popular media to whether articles are peer reviewed or not, using somewhat absurd examples of foundational historical papers to stress the point.
The author then makes a logical jump, and claims that "Science worships peer review". However neither the journal Science nor the discipline of the same name are capable of worshipping anything. The author might mean to say that "Scientists worship peer review"? But for such a claim the author provides little evidence, and further it would contradict the later statement that "simple inertia, and ..." are the reasons "we still have peer review".
Subsequently the author tries to demonstrate that peer review is useless by stressing the multitude of problematic papers that did make it through peer review. This, of course, is a standard logical fallacy and again in direct contradiction with the latter statement that "we do not know whether the science we produce is better with or without it" (emphasis mine).
As I mentioned above, it is clear that the author raises an important point and at its root the message it tries to convey is certainly valid. However the article itself adds little to what scientists (and I would wager the informed public) already know. Besides the issues mentions above, I think a further weakness is that it claims to speak for all of science but I might imagine that in a field like mathematics the act of peer review can be more meaningful.
Altogether, then, I am of the opinion that the above issues need to be corrected in a substantial revision before I can accept the article for publication in your journal.
Sincerely,
Good joke. :)
While we are at it, let's also change the road sign "left lane must turn left", because a lane is not capable of turning left.
Proper peer review has people at or above your level in the research area of the paper = they understand it and can know the nature of what you are telling in the paper. Sadly projealosy can rear it's head and some weasel sees your 'Nobel grade' paper and wants to delay and knock off kilter and get his similar paper ahead of you. Admittedly this is rare. In addition, the readers or referees of the paper could be very busy and drag their ass. This has become such a problem that the idea of preprint server has come into being where yourpaper is made public right away - establishing your priority as all stuff preprint served is scanned by your field and a number of people will give it intensive study for their being able to stay abreast of the field and let ideas spread. Often journals have a substabtial fee for this.
The whole journal game needs to be burned at the stake and replaced by electronic.
Their is another weasel, their are thousand of fake journals who copy and paraphrase paper and release them online via a nice sounding fake journal. It is a plague, however, those in the field know these rats - sadly newspapers and other media do not and are often tricked.
I did my Ph.D in a relatively niche area of physics. We had conferences, attendeed by a handful of people from all over the world, and presented and discussed our research. None of this was "peer reviewed" in the sense that it had undergone a third party review like you'd have for paper or conference submission. But the whole point of the conferences was peer review. This is what I expect mostly happened in the days of Einstein and Watson. Just because there was not an editor assigning three reviewers, doesn't mean their work wasn't scrutinized by their peers.
Now extrapolating this outside of niche science and into research that is underpinning policy decisions (or tenure and promotion decisions), of course one would expect it to undergo some kind of review. And if something had not, and was just a preprint being circulated, it's a pretty important consideration that its not yet part of established canon.
Certainly I agree there are lots of flaws with peer review, that a system that originally evolved to build out the body of accepted knowledge in a field had now been formalized and used for other purposes, and that it can be gamed. But it don't find it credible to say that we just don't need it, and that we shouldn't distinguish between new results and better scrutinized ones.
Now extrapolating this outside of niche science and into research that is underpinning policy decisions (or tenure and promotion decisions), of course one would expect it to undergo some kind of review. And if something had not, and was just a preprint being circulated, it's a pretty important consideration that its not yet part of established canon.
Certainly I agree there are lots of flaws with peer review, that a system that originally evolved to build out the body of accepted knowledge in a field had now been formalized and used for other purposes, and that it can be gamed. But it don't find it credible to say that we just don't need it, and that we shouldn't distinguish between new results and better scrutinized ones.
I think Humphries is using "peer review" as a short-hand for "external peer review".
Compare Humphries' statement: "None of the classic papers on DNA, nor Einstein’s four in his miracle year, were peer reviewed.
with this quote from Wikipedia at https://en.wikipedia.org/wiki/Scholarly_peer_review (emphasis mine):
> Peer review became a touchstone of the scientific method, but until the end of the 19th century was often performed directly by an editor-in-chief or editorial committee. Editors of scientific journals at that time made publication decisions without seeking outside input, i.e. an external panel of reviewers, giving established authors latitude in their journalistic discretion. For example, Albert Einstein's four revolutionary Annus Mirabilis papers in the 1905 issue of Annalen der Physik were peer-reviewed by the journal's editor-in-chief, Max Planck, and its co-editor, Wilhelm Wien, both future Nobel prize winners and together experts on the topics of these papers.
The latter quote supports your expectation that Einstein's work was indeed scrutinized by his peers.
I'll use that distinction between internal and external peer review to analyze this quote by Humphries: "And the media have dealt with this explosion by consistently pointing out when research has not yet been peer reviewed. Presumably they do this to warn the reader that the research lacks the safeguards that peer review brings."
This means the media is reporting that a given preprint has had neither internal peer review by an editor, nor external peer review.
Compare Humphries' statement: "None of the classic papers on DNA, nor Einstein’s four in his miracle year, were peer reviewed.
with this quote from Wikipedia at https://en.wikipedia.org/wiki/Scholarly_peer_review (emphasis mine):
> Peer review became a touchstone of the scientific method, but until the end of the 19th century was often performed directly by an editor-in-chief or editorial committee. Editors of scientific journals at that time made publication decisions without seeking outside input, i.e. an external panel of reviewers, giving established authors latitude in their journalistic discretion. For example, Albert Einstein's four revolutionary Annus Mirabilis papers in the 1905 issue of Annalen der Physik were peer-reviewed by the journal's editor-in-chief, Max Planck, and its co-editor, Wilhelm Wien, both future Nobel prize winners and together experts on the topics of these papers.
The latter quote supports your expectation that Einstein's work was indeed scrutinized by his peers.
I'll use that distinction between internal and external peer review to analyze this quote by Humphries: "And the media have dealt with this explosion by consistently pointing out when research has not yet been peer reviewed. Presumably they do this to warn the reader that the research lacks the safeguards that peer review brings."
This means the media is reporting that a given preprint has had neither internal peer review by an editor, nor external peer review.
I mean, seems pretty obvious, on the face of it, that enshrining a practice like peer review would lead to disingenuous gate-keeping, normalcy bias, intellectual sloppiness, politicisation, cliques, groupthink, obscurantism and nerdification.
I don't agree with everything in this article, but just to give my 2 cents as somebody who has been on both sides of the equation:
Before I ever did any reviewing, I always felt like some conferences really are like a lottery on whether or not you get accepted. You can submit the same paper and get nothing but strong accepts or strong rejects maybe even with the same reviewers.
When reviewing papers, I think I understood more about why. When reviewing you can see other reviewers' comments when you're done with your review, as well as their level of expertise. With some really really niche areas, some people simply aren't qualified to be reviewing, specify their expertise in an area as higher than it is, and give a really half-assed review. Part of that is also because we need to review a whole bunch of papers and can't dedicate hours to each one.
The only time I've found this process to work properly is with really good papers or really shit papers. There, everybody tends to agree. For average papers, it's not that you have a mixture of weak accepts and weak rejects and it comes down to the wire, but rather it's a mixture of strong accepts and strong rejects, with comments that praise/roast a paper that you're not even sure is the same one you read, and you doubt your own understanding of the topic. There is usually very little structure and objectivity built into the system, so mediocre science presented really well is much more likely to do well. That's why it's so random.
Before I ever did any reviewing, I always felt like some conferences really are like a lottery on whether or not you get accepted. You can submit the same paper and get nothing but strong accepts or strong rejects maybe even with the same reviewers.
When reviewing papers, I think I understood more about why. When reviewing you can see other reviewers' comments when you're done with your review, as well as their level of expertise. With some really really niche areas, some people simply aren't qualified to be reviewing, specify their expertise in an area as higher than it is, and give a really half-assed review. Part of that is also because we need to review a whole bunch of papers and can't dedicate hours to each one.
The only time I've found this process to work properly is with really good papers or really shit papers. There, everybody tends to agree. For average papers, it's not that you have a mixture of weak accepts and weak rejects and it comes down to the wire, but rather it's a mixture of strong accepts and strong rejects, with comments that praise/roast a paper that you're not even sure is the same one you read, and you doubt your own understanding of the topic. There is usually very little structure and objectivity built into the system, so mediocre science presented really well is much more likely to do well. That's why it's so random.
‘It is ridiculous to think covid-19 could have originated in a lab’
‘Covid-19 spreads through touching surfaces’
‘Hydroxychloroquine cures covid-19’
‘Hydroxychloroquine does not cure covid-19’
‘Any opinions that go against this popular scientific stance are dangerous and should be blocked’