We shouldn’t take peer review as the ‘gold standard’(washingtonpost.com)
washingtonpost.com
We shouldn’t take peer review as the ‘gold standard’
https://www.washingtonpost.com/outlook/why-we-shouldnt-take-peer-review-as-the-gold-standard/2019/08/01/fd90749a-b229-11e9-8949-5f36ff92706e_story.html
149 comments
You are totally right about peer review being misunderstood as replication. That's why preprints have gained so much traction lately. Things have become so complex that sometimes peer review is only good enough to filter results that are obviously wrong.
But replicating a proof or a complex experiment is way beyond what reviewers can accomplish.
Another problem with peer review, especially in experimental fields, are malicious reviewers. Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.
But replicating a proof or a complex experiment is way beyond what reviewers can accomplish.
Another problem with peer review, especially in experimental fields, are malicious reviewers. Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.
Big PIs typically gain control of a particular subfield by publishing something first, then getting lots of papers to review and blocking said papers and/or stealing good ideas from them.
That's a bold claim, especially the "typically" part. How can you back it up?
That's a bold claim, especially the "typically" part. How can you back it up?
I know it's a bold claim. But PIs that dominate a field employ predatory tactics that are very similar to those used by big corporations. It's just economics.
I have witnessed this in some groups I have been a member of. Very unethical behaviors. As I said above blocking papers in peer review, or even rejecting them, and simultaneously sharing key bits from said papers with postdocs is a routine practice.
Knowing journal editors and reaching publication agreements before papers are even written is also very common, and hardly surprising to someone that has been in the field for some time.
I have witnessed this in some groups I have been a member of. Very unethical behaviors. As I said above blocking papers in peer review, or even rejecting them, and simultaneously sharing key bits from said papers with postdocs is a routine practice.
Knowing journal editors and reaching publication agreements before papers are even written is also very common, and hardly surprising to someone that has been in the field for some time.
I have absolutely seen this in my field. The top-of-its-field journal editor actually requested our paper for publication and one of the reviewers poured vinegar all over it, creating requirements literally no paper of this kind had ever been held to. We actually sent it to another journal, and got the verbatim same review back. They had sent it to the same top reviewer. Amazingly, said reviewer comes out with a pre-print of a similar paper that addresses some of his own concerns, about 9 months later. Amazing how that happened, hmmm?
The one consolation is that at least we know who's reviewing, who's an asshole, and that this space is as hot as we thought it was.
The one consolation is that at least we know who's reviewing, who's an asshole, and that this space is as hot as we thought it was.
That sucks. I guess the solution is to publish preprints and move quickly. Then ignore scammy reviewers and publish elsewhere.
Proofs aren't very hard to review, especially because math doesn't have the proprietary problems of wet science so drafts aren't hidden. Such is the benefit of working in a field no one can profit from :-)
Math doesn't have a replication crisis; false papers are rare.
Proofs are actually incredibly hard to review. Problems will be found in what had been accepted to be a "good" proof years to decades later. There's a whole movement to move proofs over to something/anything more verifiable (e.g. representing all proofs in Coq--but even then you're relying on the Coq proof assistant to have zero bugs).
Furthermore, the standard for mathematical proof has also changed over time, most significantly in the early 20th century. This led to a number of existing results needing to be re-proved (or thrown out! Some were incorrect!).
Exactly what qualifies as a proof is a FASCINATING debate. Mathematics is created by consensus, just like all other knowledge.
Furthermore, the standard for mathematical proof has also changed over time, most significantly in the early 20th century. This led to a number of existing results needing to be re-proved (or thrown out! Some were incorrect!).
Exactly what qualifies as a proof is a FASCINATING debate. Mathematics is created by consensus, just like all other knowledge.
Proving a math problem is mentally challenging, but there are other interesting definitions of hard. In medicine, for instance. None of it requires particularly fancy logic. But can you cite any instances in math here you had to invest, say, $10-15M and collect data from every available human patient at multiple hospitals over 2-3 years in order to replicate?
What are some examples of proofs that were originally accepted and then later shown to be incorrect?
Here are a few of them:
https://mathoverflow.net/questions/35468/widely-accepted-mat...
https://mathoverflow.net/questions/35468/widely-accepted-mat...
Is there any proof peer review has increased scientific productivity? It is relatively new, and I don't see any reason prima facie why it would work better than an editorial teaming willing to delegate when necessary. Anecdotally, I see lots of academics complain about very important results being unable to make it through peer review. Ralph Merkel's early groundbreaking cryptography work was suppressed for years by unthinking peer review. From Merklel:
>My first paper (and, in fact, the first paper) on public key cryptography was submitted in 1974 and initially rejected by an unknown "cryptography expert" because it was "...not in the main stream of present cryptography thinking...."
>My first paper (and, in fact, the first paper) on public key cryptography was submitted in 1974 and initially rejected by an unknown "cryptography expert" because it was "...not in the main stream of present cryptography thinking...."
Is there any proof peer review has
increased scientific productivity?
It is relatively new
Peer review is a relatively new procedure in the same way that California is a relatively new state :-p
[1] https://en.wikipedia.org/wiki/Philosophical_Transactions_of_...
"Peer review" nowadays refers almost exclusively to prepublication review by people other than the editor. This was still relatively rare at the beginning of the 20th century.
If you name a famous publication prior to 1900, it almost certainly was published without peer review.
If you name a famous publication prior to 1900, it almost certainly was published without peer review.
However, some famous papers like Watson & Crick's DNA model were published without peer review.
You can publish today in PNAS without review if you are a member of the national academy of sciences.
Not exactly any more. It's true that it used to be that a member could just contribute their own manuscripts which would be published as is without review. But after some unfortunate cases of questionable stuff getting published, even the contributed track has to be peer reviewed these days. However members still have leverage in that they can choose the reviewers, which obviously could be abused.
The concept is certainly old, but it hasn't been a systematic part of the "universal scientific process" for more than a few decades.
Most revolutionary scientific discoveries were made without it.
Most revolutionary scientific discoveries were made without it.
Isn’t peer review at most journals done by “an editorial team delegating as necessary”?
I married into the field, didn’t work in it, but that’s what I’ve observed from the side.
I married into the field, didn’t work in it, but that’s what I’ve observed from the side.
The ACM and Usenix peer review work I did involved rotating teams of volunteers and very little guidance from the publication/venue.
Well yes, but the review power (both nominally and in practice) does lie with the publication/venue editor[ial team], it's just that they have decided to delegate almost all that power to rotating teams of volunteers, because they can, and because they want to for obvious practical and financial reasons.
A system with an "editorial teaming willing to delegate when necessary" simply becomes the current system as the editorial team is willing and able to delegate, and it's neccessary pretty much all the time because of resource constraints - because being an editor generally also is an unpaid volunteer position done in addition to your normal work in academia.
A system with an "editorial teaming willing to delegate when necessary" simply becomes the current system as the editorial team is willing and able to delegate, and it's neccessary pretty much all the time because of resource constraints - because being an editor generally also is an unpaid volunteer position done in addition to your normal work in academia.
You are thinking of journal review, which hijacked peer review until arxiv and the internet in general rescued it.
Yup exactly correct. I've also been a reviewer from time to time on various ACM and IEEE papers and there are clearly authors who are trying to get reviewed by their 'friends' rather than by objective reviewers. And there is also reviewer biases, I reviewed a paper where one of the other reviewers was extremely harsh, and that reviewer turned out to have a stake in the outcome. Now a good journal is supposed to curate these sorts of conflicts of interest but my experience is that isn't always the case.
To be cynical, sorting through conflicts of interests would take more effort and Elsevier wouldn’t find that profitable. More realistically running an unbiased journal will always be difficult unless human nature changes significantly. Nevertheless it’s still worthwhile to do.
Same as you, I’ve had reviewers reject research based on a reviewer’s stake in the outcome, and also had good reviews with constructive feedback. As a reviewer I’ve also helped "tear apart" an article that had good results but was poorly structured to help them clean up their article.
Same as you, I’ve had reviewers reject research based on a reviewer’s stake in the outcome, and also had good reviews with constructive feedback. As a reviewer I’ve also helped "tear apart" an article that had good results but was poorly structured to help them clean up their article.
Getting a harsh review stings but the (normally) impartial feedback can be invaluable. I've come to believe that only by learning how to pass your articles by peer review do you learn how to write a good article.
Even in medical research, legacy publishers do precious little to respect even the very modest standards of the field for the disclosure of conflict of interest.
https://bmjopen.bmj.com/content/bmjopen/9/7/e029796.full.pdf
What exactly was bogus about sokal squared? They made stuff up, they got it peer reviewed and they got it published. Yeah the journals weren't great but none of them were publication mills and not everything published gets published in Nature.
I wouldn't consider nature to be necessarily so reputable either. If you want an almost unassailable example of peer review, check this out:
https://en.m.wikipedia.org/wiki/Organic_Syntheses
https://en.m.wikipedia.org/wiki/Organic_Syntheses
In my field we say "it was published in Nature, but it might still be correct".
The underlying problem with Nature is that they pick the flashy stuff AND present it in a very very condensed matter -- removing all the important technical details -- to make it look exciting for a wider audience. For the few hundred people actually in the subfield the best course of action is often to ignore the PR piece in nature and go hunt for the technical paper with all the details that is published in parallel in a "lesser" journal. That is the one the tells what the actual progress was and what you want to consider for your own work.
The underlying problem with Nature is that they pick the flashy stuff AND present it in a very very condensed matter -- removing all the important technical details -- to make it look exciting for a wider audience. For the few hundred people actually in the subfield the best course of action is often to ignore the PR piece in nature and go hunt for the technical paper with all the details that is published in parallel in a "lesser" journal. That is the one the tells what the actual progress was and what you want to consider for your own work.
Indeed. Nature ranks very high in the retraction index.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3690355/figure/...
> The underlying problem with Nature is that they pick the flashy stuff AND present it in a very very condensed
I don't object to the condensed part so much. When I was a physicist, people had a similarly dim view of Nature. But Physical Review Letters also published four-page papers. And PRL was definitely a bigger prize than the long form PR(xyz) papers.
Now if you happened to want to drill in and do work based on someone else's, then you would no doubt prefer the long-form. But if you are keeping up with the field, then the 4-pager is much preferred.
You should be able to lay down what the actual science is in a small space. And if it's a theory paper, that space is enough to have a few equations that the reader could then puzzle out justifications for herself.
I don't object to the condensed part so much. When I was a physicist, people had a similarly dim view of Nature. But Physical Review Letters also published four-page papers. And PRL was definitely a bigger prize than the long form PR(xyz) papers.
Now if you happened to want to drill in and do work based on someone else's, then you would no doubt prefer the long-form. But if you are keeping up with the field, then the 4-pager is much preferred.
You should be able to lay down what the actual science is in a small space. And if it's a theory paper, that space is enough to have a few equations that the reader could then puzzle out justifications for herself.
"it was published in Nature, but it might still be correct"
We said that in biochemistry as well. Also science. There are some huge stinkers there. When arsenic life came out we were like, of course, it's horsedung, it's in science!!
We said that in biochemistry as well. Also science. There are some huge stinkers there. When arsenic life came out we were like, of course, it's horsedung, it's in science!!
There's also Andrew Kelman's renaming of PPNAS, the Prestigious Proceedings of the National Academy of Science, where Prestige is used instead of Truth.
Interesting. I'm not in academia so I guess that's showing here.
There were a couple issues with the Sokal Squared hoax. The biggest one was that they relied a lot on their own definition of "obviously absurd" rather than using some form of empirical or unbiased "made stuff up" generator.
In the process, they wrote some papers that, especially if you weren't assuming the person made up the data underlying them, were...middling compelling.
I read one of the papers - the dog park one - and my conclusion was much the same as one of the peer reviewers. That it was an interesting concept, presented slightly too stridently (a common thing with papers written by say...students), and interesting enough as a frame to jump off from.
In the process, they wrote some papers that, especially if you weren't assuming the person made up the data underlying them, were...middling compelling.
I read one of the papers - the dog park one - and my conclusion was much the same as one of the peer reviewers. That it was an interesting concept, presented slightly too stridently (a common thing with papers written by say...students), and interesting enough as a frame to jump off from.
I must compliment the creativity of defending someone from having published nonsense by claiming that it wasn't sufficiently-scientifically-generated nonsense.
If someone not in the field can write a "middling compelling" paper in a distant field while trying to be absurd, then that still means something. If some jerk (basically) can just scribble a "publishable paper" off in some semi-major discipline while literally doing the opposite of trying, then clearly that semi-major discipline has very little value to it. We're not creating these large infrastructures with journals and university libraries and peer review and the Imprimatur of Scientific Credibility and credentialed university graduate programs and millions of dollars in grants for things any idiot can scribble down. We've got the Internet at large for that already.
If it's by design or somehow correct that peer review can't catch these things, then my opinion of the situation is even worse than if they were merely a demonstration of things slipping through a net, and peer review on any level is apparently utterly worthless.
Your defense may be superficially interesting, but if taken seriously, only deepens the problem.
If someone not in the field can write a "middling compelling" paper in a distant field while trying to be absurd, then that still means something. If some jerk (basically) can just scribble a "publishable paper" off in some semi-major discipline while literally doing the opposite of trying, then clearly that semi-major discipline has very little value to it. We're not creating these large infrastructures with journals and university libraries and peer review and the Imprimatur of Scientific Credibility and credentialed university graduate programs and millions of dollars in grants for things any idiot can scribble down. We've got the Internet at large for that already.
If it's by design or somehow correct that peer review can't catch these things, then my opinion of the situation is even worse than if they were merely a demonstration of things slipping through a net, and peer review on any level is apparently utterly worthless.
Your defense may be superficially interesting, but if taken seriously, only deepens the problem.
They weren’t outside the field though. They did a deep dive, learned the language, and relied on a definition of “absurd” rooted in their own ideology rather than reason.
In doing so, they ended up mimicking the field too well.
In doing so, they ended up mimicking the field too well.
Again... even that is still pretty serious condemnation. If you can "deep dive" that many fields to the point that you're fooling the gatekeepers, a "deep dive" must not actually be all that "deep".
I mean, we can try to puff up Sokal as some kind of massive genius who got to publication-quality skill levels in multiple fields in less than a year, but... in that case, why shouldn't I accept his take on the results, since he's apparently that much of a genius?
There just isn't a way around it. If it's that easy to get to "publishable", it isn't worth being an academic discipline.
Compare someone in one of those fields trying to publish a particle physics paper by just imitating what they think are the important characteristics.
I mean, we can try to puff up Sokal as some kind of massive genius who got to publication-quality skill levels in multiple fields in less than a year, but... in that case, why shouldn't I accept his take on the results, since he's apparently that much of a genius?
There just isn't a way around it. If it's that easy to get to "publishable", it isn't worth being an academic discipline.
Compare someone in one of those fields trying to publish a particle physics paper by just imitating what they think are the important characteristics.
Note we’re not talking about Sokal, but a separate hoax that decided to call itself that and which was distinctly less impressive. They started, at best, one field adjacent from what they targeted.
I’m not at all surprised they managed, after devoted effort to do so, in having middling success getting papers into some okay journals.
I’m not at all surprised they managed, after devoted effort to do so, in having middling success getting papers into some okay journals.
Couldn't agree with you more. I too have done peer review, and the most disappointing thing in the world is when you point out something that is inaccurate and the response from the publisher is, "if we made it accurate there is too much nuance for the layperson to understand, so we'll keep it the way it is even though it's wrong because they'll get the gist of it".
Wait, when does this happen. In my experience, the sort of document undergoing peer review is rarely the sort to be read by laypeople to begin with.
There's sometimes peer review on popular science books. Maybe he reviewed a few chapters of "Netflix: What happens when you press play?" (https://www.amazon.com/Netflix-What-Happens-When-Press-ebook...) or something.
Ha that's funny. I literally gave a talk with that title. But no, I didn't review that book. But yes, it was similar popular science type stuff.
Technical books tend to get peer reviewed.
As a machine learning / probabilistic programming friend put it, (journal) peer review is boosting with three weak learners.
Better than nothing, but very far from ironclad. Only replication really verifies scientific findings. Everything else is just window dressing.
(I say this as a regular reviewer; for whatever reason, this particular week I’m reviewing for both Science and Nature.)
Better than nothing, but very far from ironclad. Only replication really verifies scientific findings. Everything else is just window dressing.
(I say this as a regular reviewer; for whatever reason, this particular week I’m reviewing for both Science and Nature.)
Test results are very easy to fake. The "gold standard" is whether outside groups can run the same tests and reproduce the same results. Peer review doesn't mean much more than "this paper seems interesting and is well-written and properly formatted."
Self consistent is another major check. A lot of math and similar errors are caught in peer review.
Peer review isn't (and never was meant to be) a golden seal declaring a result to be unchangeable truth. It is a process that makes sure that at least one competent colleague has read it and caught all the embarrassing Monday morning mistakes that might have been in there. This is done before wider publication to avoid that the 100 people in your field all stumble over the same stupid little mistake and can spent their time thinking about the actual content.
Think of it as the type checking by the compiler or a code review by somebody on your team, NOT a papal announcement of unchangeable dogma.
Think of it as the type checking by the compiler or a code review by somebody on your team, NOT a papal announcement of unchangeable dogma.
I disagree. The gold standard is if outside groups subject your findings to new experiments and reproduce the same results.
I agree with you.
"I can run your code/experiment/etc. in your population and get your results" isn't the gold standard either. "We approached it from a different way, and with a different population, and we get a consistent answer" is both harder and more compelling.
"I can run your code/experiment/etc. in your population and get your results" isn't the gold standard either. "We approached it from a different way, and with a different population, and we get a consistent answer" is both harder and more compelling.
Well, there's replicating the experiment and then there's challenging the conclusion. Those are two separate ideas, and are actually fairly independent.
Depends on the journal, for example
https://en.m.wikipedia.org/wiki/Organic_Syntheses
https://en.m.wikipedia.org/wiki/Organic_Syntheses
I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed here.
Peer review:
1. dramatically reduces the pace of progress
2. exacerbates publication bias
3. creates a false sense of accomplishment
4. creates a false meritocracy
5. creates many perverse incentives
(3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications).
The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else.
What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc.
Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons.
Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value.
Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).
Peer review:
1. dramatically reduces the pace of progress
2. exacerbates publication bias
3. creates a false sense of accomplishment
4. creates a false meritocracy
5. creates many perverse incentives
(3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications).
The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else.
What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc.
Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons.
Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value.
Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).
Those are all very real problems (and as you point out, they impact grant review as well). On the other hand, fully transparent peer review doesn't necessary address most of them. Reviewers who have to sign their reviews may be reluctant to anger colleagues (or try to curry favor with them).
As you presumably know a many journals have experimented with open peer review, but editors still need to police the reviews to look for bias. It solves some problems but creates others.
As you presumably know a many journals have experimented with open peer review, but editors still need to police the reviews to look for bias. It solves some problems but creates others.
transparent review isn't a perfect solution. But the problem you suggest of reluctance to anger colleagues or currying favor are worse under the current system than they would be under a transparent one.
Just because other journals failed in the past doesn't mean we shouldn't try again in the future! Maybe there were some mistakes we could learn from? Also the internet is becoming a more familiar tool used more often by more people every day, maybe it just wasn't the right time previously.
Just because other journals failed in the past doesn't mean we shouldn't try again in the future! Maybe there were some mistakes we could learn from? Also the internet is becoming a more familiar tool used more often by more people every day, maybe it just wasn't the right time previously.
Double-blind peer review fixes "reluctance to anger colleagues or currying favor" and is widely used by journals in my field.
Maybe.
> In the late 2000s, widespread debate and controversy ensued after Budden and colleagues (2008a) found that a switch to double-blind review in the journal Behavioural Ecol- ogy led to a small but notable 7.9% increase in the 2proportion of articles with female first authors
https://www.eswnonline.org/wp-content/uploads/gravity_forms/...
Others agree: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5629234/
However:
> Findings from studies of journals that have actually adopted the practice are non-conclusive. For example, Budden and colleagues show that the introduction of double-blind review by Behavioral Ecology was followed by an increase in papers with female first authors (Budden et al., 2008). However, a second paper reports that female first authorship also increased in comparable journals that do not use double blind review (Webb et al., 2008). A study by Madden and colleagues shows that the adoption of double-blind reviewing in the SIGMOD conference led to no measureable change in the proportion of accepted papers coming from “prolific” and less well-established authors (Madden and DeWitt, 2006), but a later study contests this conclusion (Tung, 2006).
https://www.frontiersin.org/articles/10.3389/fnins.2015.0016...
> In the late 2000s, widespread debate and controversy ensued after Budden and colleagues (2008a) found that a switch to double-blind review in the journal Behavioural Ecol- ogy led to a small but notable 7.9% increase in the 2proportion of articles with female first authors
https://www.eswnonline.org/wp-content/uploads/gravity_forms/...
Others agree: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5629234/
However:
> Findings from studies of journals that have actually adopted the practice are non-conclusive. For example, Budden and colleagues show that the introduction of double-blind review by Behavioral Ecology was followed by an increase in papers with female first authors (Budden et al., 2008). However, a second paper reports that female first authorship also increased in comparable journals that do not use double blind review (Webb et al., 2008). A study by Madden and colleagues shows that the adoption of double-blind reviewing in the SIGMOD conference led to no measureable change in the proportion of accepted papers coming from “prolific” and less well-established authors (Madden and DeWitt, 2006), but a later study contests this conclusion (Tung, 2006).
https://www.frontiersin.org/articles/10.3389/fnins.2015.0016...
Simple solution: have a panel of mixed open and anonymous reviewers, and an editorial arbiter.
I totally agree with the transparency thing. I would love to see something like open review (https://openreview.net/forum?id=ryQu7f-RZ) become more common, and even have something like this replace the traditional peer review system.
We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.
We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.
I've badly soured on open peer review, having had some experiences with it. I think you'll have very differential levels of review, and very differential values of how free people feel to go after authors.
I'd much prefer double-blind peer review.
I'd much prefer double-blind peer review.
What would be the alternative to peer review? White papers on “pop” topics? Opinion pieces by think tanks?
The ideal of peer review is pretty much the gold standard of scientific work. By that, I mean the idea that your work should undergo rigorous scrutiny by your peers before it gets accepted for publications. As any other human system, though, the process is still subject to politics and biases.
My hope is that a truly open review system might one day democratize science and ameliorate the issues with the current review system. For example, it would be cool to put all the revisions, reviews and authors’ comments in the open as an online appendix. That will show how the paper changes as it moves trough the process, the issues that the reviewers brought up, and how the authors responded to them. It will also help show if there is systemic bias against certain researchers or topics.
The ideal of peer review is pretty much the gold standard of scientific work. By that, I mean the idea that your work should undergo rigorous scrutiny by your peers before it gets accepted for publications. As any other human system, though, the process is still subject to politics and biases.
My hope is that a truly open review system might one day democratize science and ameliorate the issues with the current review system. For example, it would be cool to put all the revisions, reviews and authors’ comments in the open as an online appendix. That will show how the paper changes as it moves trough the process, the issues that the reviewers brought up, and how the authors responded to them. It will also help show if there is systemic bias against certain researchers or topics.
I think that publication should not be gated by peer review. Release early, release often, publish experimental setup and what you are looking for prior to the actual experiment, etc.
We have arxiv now; the whole process could open up even more, slightly closer to research in the open, like open-source development in the open. This is to make peer review easier.
Then what is currently "publication" becomes what it should be, in the sense of adding value: curation, filtering the most interesting, least dubious results.
We have arxiv now; the whole process could open up even more, slightly closer to research in the open, like open-source development in the open. This is to make peer review easier.
Then what is currently "publication" becomes what it should be, in the sense of adding value: curation, filtering the most interesting, least dubious results.
You're not the only one to comment this, but from what I know per review is a long iterative process, wher reviewers raise objections and submitted respond with edits.
So, what you're suggesting seems a lot like shipping code before code review, and seems to have many of the same problems. What do you do in the long period before critical issues have been addressed? What do you do if the submitter just says "fuck you, my code works, I have more important things to do"?
So, what you're suggesting seems a lot like shipping code before code review, and seems to have many of the same problems. What do you do in the long period before critical issues have been addressed? What do you do if the submitter just says "fuck you, my code works, I have more important things to do"?
This. I have definitely dug my heels in on some peer reviewed papers where I'm rather glad I was "gatekeeping".
arxiv is actually less open than most traditional venues.
Anyone can submit to most conferences and journals. Anyone at all! No PhD or reputation needed and it’s blind. All that matters is the quality of the research.
To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity.
It’s somewhat a step backwards in some ways.
Anyone can submit to most conferences and journals. Anyone at all! No PhD or reputation needed and it’s blind. All that matters is the quality of the research.
To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity.
It’s somewhat a step backwards in some ways.
It is my impression that, if the quality of work is good, it is easy to get arXiv endorsements. I believe that intent is not to form a clique nor gatekeep, but rather to keep arXiv from being buried under a giant pile of internet spam and crackpots.
https://arxiv.org/help/endorsement
https://arxiv.org/help/endorsement
This goes both ways. It’s obviously an unfortunate barrier to entry but it generally means the content on the arxiv is reputable and meets a certain bar (and submission is not in any way anonymous because members are verified which can discourage low quality submissions). Compare the arxiv to vixra. It seems unfair to Cornell university library to have them moderate submissions for scientific quality and those people who currently use it would not want to have to filter through the new papers in their topic of interest to determine which papers were worth reading and which would be a largely wrong waste of time (I think here I’m perhaps being a bit unfair and for many fields there would likely be no time wasters and in others it would be easy if annoying to filter out the p=np or Riemann hypothesis papers)
I don’t think that letting anyone submit is a good solution.
I don’t think that letting anyone submit is a good solution.
> I don’t think that letting anyone submit is a good solution.
I think their process is totally reasonable and I’d probably do the same... but I do think it’s a fact that it’s now less accessible and more based on reputation and credentials and we should acknowledge that.
I think their process is totally reasonable and I’d probably do the same... but I do think it’s a fact that it’s now less accessible and more based on reputation and credentials and we should acknowledge that.
I agree
I wonder which is more likely though. An unpublished and non-PhD author being published in a journal after submission, or that same author being allowed to publish on arXiv.
My guess would be on arXiv.
If you really have a decent breakthrough, I'm pretty sure your best bet is to directly contact some relevant people in the field.
If you really have a decent breakthrough, I'm pretty sure your best bet is to directly contact some relevant people in the field.
Which is more likely: you get a paper, yet unreviewed for formatting and style by any expert in the field, accepted into a conference or journal, or you get someone with field expertise to review for format and style, and they let you post to arxiv?
I see. I just meant that we have a place where publication is technically easy. No year-long waits. Everything is immediately available online. No physical paper involved.
I suppose that changing the technoligcal base to fully electronic, immediate-mode publishing should help.
Building a review workflow, or several, around the fully electronic and open archive of all published results should be doable.
Storing online everything by default, including all the negative results, proofs of the null hypothesis, full datasets and code, etc, should not be overly expensive, but would help reproducibility and further research a lot.
I suppose that changing the technoligcal base to fully electronic, immediate-mode publishing should help.
Building a review workflow, or several, around the fully electronic and open archive of all published results should be doable.
Storing online everything by default, including all the negative results, proofs of the null hypothesis, full datasets and code, etc, should not be overly expensive, but would help reproducibility and further research a lot.
I think those are called blogs. And they haven't really made much of an impact.
> To submit to arxiv you need to be approved by someone as a legitimate researcher, or beg for reviews from people you’ve never met without any anonymity.
I did not have to do something like this.
I did not have to do something like this.
I did not have to do so formally, either. Being a co-author on an arXiv paper or two counts, I believe, as endorsement.
Whenever the problem has come up for a colleague, it has been straightforward to endorse.
Whenever the problem has come up for a colleague, it has been straightforward to endorse.
Well that’s good for you isn’t it! What about everyone else? Not everyone is automatically approved because of things like their email address. That’s my point - they focus on reputation and credentials rather than the blind value of the work. It’s less accessible.
The confusion comes from the 'journal' aka topic on arXiv, which can have different settings for when an author can submit. Some are more stringent than others. It is not an arXiv wide issue.
Nonsense. Only the editorial board can approve you to publish in the journal or comference. Approximately anyone on Arxiv can approve to you to publish on Arxiv.
Yes all results should get published. Peer review should only affect ranking/discoverability.
[deleted]
The article doesn't propose we do away with peer review, but rather that lay audiences stop pretending that peer review is a "gold standard" by which the quality of research can be judged. It's "gold standards" the author proposes to eliminate.
I get that. Sorry my reply wasn’t very clear. The article gives examples of how politicians and policymakers have “weaponized” peer review to silence critics. The author proposes moving away from peer review (and the gold standard it sometimes represents) as a way to solve this issue.
In my reply, I was just pointing out that peer review still plays an important role in the development of knowledge and I was wondering if alternative systems that could replace it would actually be feasible. That lead me somewhere off topic.
By the way, the “weaponization” of peer review is nothing new. You can see it used in the past to discredit research linking smoking or lead to health issues. You can also see of the possible risks or benefits (depending on your point of view) of rejecting peer reviewed research in the anti-vax movement.
In my reply, I was just pointing out that peer review still plays an important role in the development of knowledge and I was wondering if alternative systems that could replace it would actually be feasible. That lead me somewhere off topic.
By the way, the “weaponization” of peer review is nothing new. You can see it used in the past to discredit research linking smoking or lead to health issues. You can also see of the possible risks or benefits (depending on your point of view) of rejecting peer reviewed research in the anti-vax movement.
The author does not propose moving away from peer review and, in the conclusion, says it's a necessity for science.
Some journals do this. It is an interesting approach.
I am however ambivalent, or even perhaps against allowing public comments directly into a paper. We've all experienced, no doubt, the a naive but loud internet com-mentor spreading FUD on comment sections and social media on topics they simply know nothing about, and do so prolifically. It is nice to think that good arguments will float to the top, but often it is the loudest and least nuanced argument, right or wrong, which gets taken up.
I am however ambivalent, or even perhaps against allowing public comments directly into a paper. We've all experienced, no doubt, the a naive but loud internet com-mentor spreading FUD on comment sections and social media on topics they simply know nothing about, and do so prolifically. It is nice to think that good arguments will float to the top, but often it is the loudest and least nuanced argument, right or wrong, which gets taken up.
I'm an editor of a journal that has one of (the?) highest IFs in its field, among other things. Lots of experience with scientific publishing.
I've had lengthy discussions about this with my colleagues, and I don't think the problem is with peer review per se, it's with how peer review, and by extension, scientific literature, is interpreted and utilized. It's not just with the lay community, it's among scientists themselves, as well as scientific consumers as in engineering, healthcare, etc.
People now treat science, and scientific publications, as if it's a farm or factory. The produced commodity is the peer-reviewed paper. Once research, via a paper, is peer-reviewed, it's blessed as the truth, and if it's not, it's worthless. The goal of many institutions is now not to produce sound research per se, but to receive money (largely due to indirect cost income) to produce the commodity of peer-reviewed paper. Once a paper is accepted as in press, it can be cited as if it were truth, which provides a building block for something else. The truth per se doesn't matter, only the citability of it as peer-reviewed.
I realize this sounds a bit conspiratorial but I do think this is basically how it works at this point in time, at least in certain fields. Peer-review is overvalued, as are single papers, and contributions of particular researchers in many cases. That doesn't mean they don't have any value, just that their real value is probably much less than we assume.
I agree that peer review can be improved a lot, and will likely always have an important role in some form, but I don't think fundamental problems in the field will go away unless there's a change in perspective on the real scientific process as a whole, to something more nuanced and gradual.
I've had lengthy discussions about this with my colleagues, and I don't think the problem is with peer review per se, it's with how peer review, and by extension, scientific literature, is interpreted and utilized. It's not just with the lay community, it's among scientists themselves, as well as scientific consumers as in engineering, healthcare, etc.
People now treat science, and scientific publications, as if it's a farm or factory. The produced commodity is the peer-reviewed paper. Once research, via a paper, is peer-reviewed, it's blessed as the truth, and if it's not, it's worthless. The goal of many institutions is now not to produce sound research per se, but to receive money (largely due to indirect cost income) to produce the commodity of peer-reviewed paper. Once a paper is accepted as in press, it can be cited as if it were truth, which provides a building block for something else. The truth per se doesn't matter, only the citability of it as peer-reviewed.
I realize this sounds a bit conspiratorial but I do think this is basically how it works at this point in time, at least in certain fields. Peer-review is overvalued, as are single papers, and contributions of particular researchers in many cases. That doesn't mean they don't have any value, just that their real value is probably much less than we assume.
I agree that peer review can be improved a lot, and will likely always have an important role in some form, but I don't think fundamental problems in the field will go away unless there's a change in perspective on the real scientific process as a whole, to something more nuanced and gradual.
Personally, I view any study in the less scientific fields (i.e. those that routinely accept p < 0.05 as "proven") as suspect, unless it's preregistered, has full data available for the public to check, and has p < 0.01. (I'm neither aware of any such study, nor have I looked for it... I'm just saying that that should be the minimal acceptable standard for e.g. psychology and medicine.)
I would love to see an open review system like that. Comments out in the open, along with the experiments and experimental data which supports or does not support the work.
Science is all about experimentation, yet we currently measure papers by number of citations rather than by experimental support.
Science is all about experimentation, yet we currently measure papers by number of citations rather than by experimental support.
> What would be the alternative to peer review?
Replication. If science has a gold standard, it is this.
Replication. If science has a gold standard, it is this.
The gold standard of science is a different experiment whose findings agree with the findings of the first.
Replication is much less valuable, scientifically and professionally, than non-scientists think it is. Simply repeating a published investigation runs a much higher risk of repeating any experimental errors or faulty assumptions that might have harmed the first one.
The whole point of science is to find knowledge that persists beyond one particular perspective; knowledge that is independently verifiable. Rote replication is not the best way to find this type of knowledge.
As for peer review, its purpose is simply to sharpen the communication of a completed study. Even if every study was replicated, the papers would still benefit from peer review.
Replication is much less valuable, scientifically and professionally, than non-scientists think it is. Simply repeating a published investigation runs a much higher risk of repeating any experimental errors or faulty assumptions that might have harmed the first one.
The whole point of science is to find knowledge that persists beyond one particular perspective; knowledge that is independently verifiable. Rote replication is not the best way to find this type of knowledge.
As for peer review, its purpose is simply to sharpen the communication of a completed study. Even if every study was replicated, the papers would still benefit from peer review.
It would be interesting to see if replication is really the glue that holds our greatest scientific achievements together. I suspect not. In my view a higher standard is progress towards powerful unifying theories, even if getting there is a messy business.
I'm thinking of something like relativity or quantum mechanics. Suppose a study in those fields fails replication. The whole thing still holds together, to the point where controversies at the part per billion level make it to the front page of the newspaper. Perhaps even most studies, taken in isolation, would be found to have problems when subjected to the strictest criteria for replication. Choosing replication as a silver bullet would be an unnecessary distraction.
Now, what about fields where there is no unifying theory on the horizon? If replication is all we've got, then sure. I can certainly see the point, especially if the results affect personal decisions (diet, medications, etc.) or public policy.
I suspect that "gold standards" can hurt science as much as help. Telling people that science is bunk because of the "replication crisis" contradicts the fact that messy science has produced results of astounding accuracy and predictive power. Learning from success should be at least as important as installing safeguards against failure.
I'm thinking of something like relativity or quantum mechanics. Suppose a study in those fields fails replication. The whole thing still holds together, to the point where controversies at the part per billion level make it to the front page of the newspaper. Perhaps even most studies, taken in isolation, would be found to have problems when subjected to the strictest criteria for replication. Choosing replication as a silver bullet would be an unnecessary distraction.
Now, what about fields where there is no unifying theory on the horizon? If replication is all we've got, then sure. I can certainly see the point, especially if the results affect personal decisions (diet, medications, etc.) or public policy.
I suspect that "gold standards" can hurt science as much as help. Telling people that science is bunk because of the "replication crisis" contradicts the fact that messy science has produced results of astounding accuracy and predictive power. Learning from success should be at least as important as installing safeguards against failure.
Replication is slow and can be extremely expensive. It's a good thing, but it doesn't serve the same function peer review does.
> Replication is slow and can be extremely expensive
Gold standards are identified as such because they’re the best. Not everything needs to meet the gold standard to be debatable. Peer review is adequate for further research, but perhaps not policy initiatives. An unreviewed paper is enough to start simple inquiries. Et cetera
Gold standards are identified as such because they’re the best. Not everything needs to meet the gold standard to be debatable. Peer review is adequate for further research, but perhaps not policy initiatives. An unreviewed paper is enough to start simple inquiries. Et cetera
Replication is a more powerful statement about the validity of research findings than peer review. Lots of valid research findings will not be replicated before they influence other researchers and the public. The point of this piece is that lay audiences shouldn't expect a simple "gold standard" by which they can distinguish the good research from the bad; understanding research requires critical thinking and access to domain expertise.
And there are also ways in which replication is orthogonal to peer review. Replication can't by itself tell you whether a piece of research makes a significant contribution to the field, or whether it is itself derivative, or poorly presented.
And there are also ways in which replication is orthogonal to peer review. Replication can't by itself tell you whether a piece of research makes a significant contribution to the field, or whether it is itself derivative, or poorly presented.
It does not have to be slow in all domains. In my field there are many modeling papers. Journals could require that all data and code be open access, or at least define a submission type where this is the case. You could even automate the process of running submitted containers / packages.
All that would prove is that the the code that implements a buggy version of a model gives consistently wrong results. Actual replication requires that somebody else implements the code independently. And that both implementations are checked over a reasonably wide range of parameters over which the model should be valid.
Actual replication is much more effort and consequently slower than just a "docker run".
Actual replication is much more effort and consequently slower than just a "docker run".
While I agree that this should be mandatory in many cases, it doesn't prove too much. I've experienced many cases where the open code worked fine with the test data provided, but failed completely when I tried it with my own real world data.
E-life is already doing peer review right. Now you need get others on board
The idea that in scientific publication, more is better, makes little sense to me as a practicing scientist. The scientific literature is already filled with papers with conclusions that are not well supported by the data, and that is after 50-80% of submissions are rejected.
To argue that publication should have fewer restrictions, one needs to show that the modest number of papers that are improperly rejected by every journal (not just the first submission) equals or outnumbers the overwhelming number of submissions that should not be published because they reflect a misunderstanding of their field or a misinterpretation of the data. For many (most) scientists, the problem is not that important results are unpublished, the problem is that it is almost impossible to keep up with the current gated literature, particularly when that literature is full of mistakes. While it is certainly true that some reviewers are biased and some ill-informed, most of the time papers are rejected because the authors did not communicate well (for papers that should be published) or because the authors did not understand that their data did not support their conclusion (papers that should be rejected).
The scientific literature needs a better signal to noise ratio, not more noise.
To argue that publication should have fewer restrictions, one needs to show that the modest number of papers that are improperly rejected by every journal (not just the first submission) equals or outnumbers the overwhelming number of submissions that should not be published because they reflect a misunderstanding of their field or a misinterpretation of the data. For many (most) scientists, the problem is not that important results are unpublished, the problem is that it is almost impossible to keep up with the current gated literature, particularly when that literature is full of mistakes. While it is certainly true that some reviewers are biased and some ill-informed, most of the time papers are rejected because the authors did not communicate well (for papers that should be published) or because the authors did not understand that their data did not support their conclusion (papers that should be rejected).
The scientific literature needs a better signal to noise ratio, not more noise.
The rejection of a submission does cancel a work, it merely shifts elsewhere. The question is whether peer review gives the right incentives; many suspect that publication venues with an ostensibly stricter peer review actually encourage the production of more noise, not less. See https://www.frontiersin.org/articles/10.3389/fnhum.2013.0029...
> A 2015 study of 1,000 medical journal submissions found that of the papers that were eventually published, the 14 that became the most frequently cited were initially rejected. Groundbreaking studies by Sir Frank MacFarlane Burnet, Rosalind Yalow, Baruch Blumberg and others were rejected by peer reviewers, yet later led to Nobel Prizes.
Yup, if you are actually doing good work it is likely to be so different to what is going on that people have a hard time evaluating it.
Yup, if you are actually doing good work it is likely to be so different to what is going on that people have a hard time evaluating it.
I can't read the article, but being rejected doesn't have to mean it's bad work.
My mom has helped many PhD students take their degree over the years, and in the process helped with their papers. Several have been rejected, some multiple times. Frequently it was down to presentation issues (the student had not explained the certain things clearly enough or similar), or there was disagreement about how strong conclusions one could draw from the results.
For example, recently a paper was rejected because the experimenters had forgotten or overlooked an important detail which meant they couldn't control for a certain variable. One of the reviewers picked this up, and rejected the paper because the findings could possibly be explained by this uncontrolled variable. So they had to resubmit with a weaker conclusion (this is when my mom got involved).
In this case the review process was harsh but fair. The experimenters had goofed, and the reviewer caught it.
Of course when doing ground breaking experiments, I guess the process might not be optimal. But the majority of scientists are not doing that.
My mom has helped many PhD students take their degree over the years, and in the process helped with their papers. Several have been rejected, some multiple times. Frequently it was down to presentation issues (the student had not explained the certain things clearly enough or similar), or there was disagreement about how strong conclusions one could draw from the results.
For example, recently a paper was rejected because the experimenters had forgotten or overlooked an important detail which meant they couldn't control for a certain variable. One of the reviewers picked this up, and rejected the paper because the findings could possibly be explained by this uncontrolled variable. So they had to resubmit with a weaker conclusion (this is when my mom got involved).
In this case the review process was harsh but fair. The experimenters had goofed, and the reviewer caught it.
Of course when doing ground breaking experiments, I guess the process might not be optimal. But the majority of scientists are not doing that.
The question is why where they rejected? My wife is a researcher and does a fair amount of peer review and sometimes she has to reject papers, not because their necessarily wrong, but because they're so badly written its hard to tell if they're right or wrong. Other times the research and paper is fine, but the conclusions are overstated, for example claiming that A is better than B, when all they've actually proven is that A is not worse than B. Other times it's great paper in every way, but covering a topic that really isn't relevant for that particular journal.
Basically there are lots of reasons to get rejected that have nothing to do with the quality or validity of the research.
Basically there are lots of reasons to get rejected that have nothing to do with the quality or validity of the research.
I recently rejected a paper as "This is fine, and really interesting, but no one who reads this journal is going to get it".
One of the problems with peer review is that behind the veil of anonymity people can get away with such presumptuous nonsense. You are part of the problem.
An extremely technical paper on centrality metrics in a clinical journal is not the right audience.
I suggested there were ways to approach writing it for that audience, but they'd need to extensively rework the paper. The reason I know that? I've done the same thing, for the same journal.
I suggested there were ways to approach writing it for that audience, but they'd need to extensively rework the paper. The reason I know that? I've done the same thing, for the same journal.
Surely making sure the paper gets published in the place it will have the greatest impact is good thing.
Also, the review wasn’t anonymous.
So you have to be a generation ahead of your time for your work to be truly enlightening. Rejection is therefore to be expected if your work is really good rather than something churned out. Science progresses one funeral at a time.
A simpler and more elegant solution is hard for the old guard to take in. You are sweeping away years of knowledge that they have built up about their earth-centred universe to propose an all new sun-centred solar system.
If you are in the position of having written some work that is ahead of its time then how do you know that? Rejection isn't good enough. The reasons why are important. If you have not presented your work neatly or used language that others don't like then that won't help. If you haven't talked about some of the smaller points that the learned people think are important about the topic then you could be done for. We need something more than 'rejection', failed for presentation reasons could be helpful so we can know if the jury is out of some 'rejected' ideas.
Nothing ever gets rejected for being twenty years ahead of its time. It would be helpful if this was an option.
A simpler and more elegant solution is hard for the old guard to take in. You are sweeping away years of knowledge that they have built up about their earth-centred universe to propose an all new sun-centred solar system.
If you are in the position of having written some work that is ahead of its time then how do you know that? Rejection isn't good enough. The reasons why are important. If you have not presented your work neatly or used language that others don't like then that won't help. If you haven't talked about some of the smaller points that the learned people think are important about the topic then you could be done for. We need something more than 'rejection', failed for presentation reasons could be helpful so we can know if the jury is out of some 'rejected' ideas.
Nothing ever gets rejected for being twenty years ahead of its time. It would be helpful if this was an option.
If you are actually doing ground-breaking work. There's plenty of scope for doing good work that is boring and mainstream and still high quality enough to be published.
Indeed, my work was of the latter type. It was rejected by Physical Review Letters but accepted by another journal.
One feature of peer review that's overlooked, is that there are different journals with different standards, and you can choose a journal that fits what you're trying to accomplish.
One feature of peer review that's overlooked, is that there are different journals with different standards, and you can choose a journal that fits what you're trying to accomplish.
A curious thing that’s happened in some parts of mathematics is that formal peer review, needing coordination of journals, editors, and reviewers, can happen after many people have already read the paper (effectively informally peer reviewing it but with fewer comments), and even after people have cited the paper.
This peer review then seems to not serve much purpose.
An example is the recent first finite prime gap paper where the result was quickly strengthened several times by other authors long before the paper made it into any journal.
Another example here: https://mobile.twitter.com/adrianprw/status/1156534906618597...
This peer review then seems to not serve much purpose.
An example is the recent first finite prime gap paper where the result was quickly strengthened several times by other authors long before the paper made it into any journal.
Another example here: https://mobile.twitter.com/adrianprw/status/1156534906618597...
"Studies have shown that journal editors prefer reviewers of the same gender, that women are underrepresented in the peer review process, and that reviewers tend to be influenced by demographic factors like the author’s gender or institutional affiliation. "
Most venues are double blind, so reviewers wouldn't know about the latter things. Having fewer women in the reviewer pool is unfortunate, but it's hard to see how it would effect the mean review quality in a direct way assuming that men and women are equally good at reviewing (maybe in an indirect way it could harm reviews by increasing the load per-reviewer).
Most venues are double blind, so reviewers wouldn't know about the latter things. Having fewer women in the reviewer pool is unfortunate, but it's hard to see how it would effect the mean review quality in a direct way assuming that men and women are equally good at reviewing (maybe in an indirect way it could harm reviews by increasing the load per-reviewer).
Not sure if this is the case, but I'd like to play devils advocate. While most venues are double blind, it is still possible there is a subconscious gender bias. Many studies have shown that gender can be readily detected from anonymized authors based style alone.
https://www.aclweb.org/anthology/W11-0310
http://cs224d.stanford.edu/reports/BartleAric.pdf
https://www.sciencedirect.com/science/article/pii/S187705091...
https://www.aclweb.org/anthology/W11-0310
http://cs224d.stanford.edu/reports/BartleAric.pdf
https://www.sciencedirect.com/science/article/pii/S187705091...
Whether venues are double blind is extremely field dependent. In my field no journals are to my knowledge. However, many people use an initial rather than a first name, so it's difficult to assign gender unless you already know the person.
Depends on the field. In my field, I know pretty much the twenty groups who have the expertise to review my work and every paper of mine I could identify with high certainity from the referee's comment which group they belonged to.
For everyone complaining about peer review, another alternative to open review is paid reviewers who focus on a particular topic. For example, I'm an editor at a journal where statistical reviewers are paid and evaluated, while the remaining (content) reviewers are not. This ensures that we have a rigorous review of the statistical methods, at least. I do think payment increases the reviewers' responsibility and conscientiousness, and the overall quality of the subsequent articles. (nb This probably has been tested somewhere.)
BUT - paid reviews are not feasible for many journals, especially open access/low-cost journals where margins may be thin. (Conversely, Elsevier should be able to pay their reviewers in gold bullion, rather than taking advantage of the scientific community's altruism, but that's another conversation).
BUT - paid reviews are not feasible for many journals, especially open access/low-cost journals where margins may be thin. (Conversely, Elsevier should be able to pay their reviewers in gold bullion, rather than taking advantage of the scientific community's altruism, but that's another conversation).
I think the peer review system is ripe for, as SV people call it, "disruption".
Even something like an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period would go a long way compared to the current process. In my lab at a prestigious Canadian university, the papers' authors names were visible as well as the reviewers names because any back and forth post review would happen through email. I think journals and universities need to buy into a system that standardizes this process so at least these basic requirements of impartiality can be met.
Even something like an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period would go a long way compared to the current process. In my lab at a prestigious Canadian university, the papers' authors names were visible as well as the reviewers names because any back and forth post review would happen through email. I think journals and universities need to buy into a system that standardizes this process so at least these basic requirements of impartiality can be met.
While I'd apreciate a little fresh air in the process as well...
> an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period
Except for that it's not an app (that would be kind of pointless, why would I want that on a phone instead of a webapp), that's pretty much the standard in my field. Anonymization is done through the authors (if the conference/journal wants it to be) and ensured by the conference submission system. Formatting is standardized anyway.
The problem is the "randomly selected experts" part, for many (most?) subfields suitable reviewers are a scarce resource, to the point that in quite a few double-blind processes I've seen one or both parties could recognize each other's work or review.
I've always found openreview [1] a nice way to at least make the process public, especially when anonymity is an illusion anyway.
https://openreview.net/
> an app that you submit papers to, which then anonymizes them, standardizes formatting and presents them to randomly selected experts for review in a certain time period
Except for that it's not an app (that would be kind of pointless, why would I want that on a phone instead of a webapp), that's pretty much the standard in my field. Anonymization is done through the authors (if the conference/journal wants it to be) and ensured by the conference submission system. Formatting is standardized anyway.
The problem is the "randomly selected experts" part, for many (most?) subfields suitable reviewers are a scarce resource, to the point that in quite a few double-blind processes I've seen one or both parties could recognize each other's work or review.
I've always found openreview [1] a nice way to at least make the process public, especially when anonymity is an illusion anyway.
https://openreview.net/
By app I meant software not a phone app.
Secondly, I think the problem of having few experts in the field is a legitimate problem but I don't think the status quo is a good solution either. I think it erodes trust in the scientific community which is very hard to build back up.
Secondly, I think the problem of having few experts in the field is a legitimate problem but I don't think the status quo is a good solution either. I think it erodes trust in the scientific community which is very hard to build back up.
I think we need an overall shift in incentives in academia before that will change. Popular conferences see an all time high in submissions, though I do not have data on that I'm sure Journals face similar problems these days. I feel like that's not an aspect that will get better in the short term.
On the app/phone app thing: Ah, fair enough, sorry about that.
On the app/phone app thing: Ah, fair enough, sorry about that.
I'm going to propose something that is unconventional today, but I believe will be standard practice at somepoint.
Overtime we have shown that many mistakes are made in labs and we know that many experiments are not replicable.
I think its time that we ask researchers using public funds to film their entire experiment from beginning to end. Every pipette, every prep etc.
Peer review should also have a video audit. Then a replication in another lab. The cost of ensuring results are what they say they are would be offset by less wasted research dollars on false rabbit holes.
Overtime we have shown that many mistakes are made in labs and we know that many experiments are not replicable.
I think its time that we ask researchers using public funds to film their entire experiment from beginning to end. Every pipette, every prep etc.
Peer review should also have a video audit. Then a replication in another lab. The cost of ensuring results are what they say they are would be offset by less wasted research dollars on false rabbit holes.
I do actually agree that we should apply more funding to replicating results. I disagree that a video log is an efficient way to ensure replicatability. In many disciplines this is not useful at all (math, high energy physics). And in most others, like your biology example, I think it adds an undue burden without much gain.
What evidence do you have that suggests that the cost of doing a video audit (an expert carefully watching hundreds of hours of experimentation) and a replication study for every submitted article would be cheaper than the occasional paper that leads to a second experiment that doesn't pan out?
Are you trying to re-invent https://en.wikipedia.org/wiki/Reproducibility_Project ?
No. Reproducibility requires double the investment and may also have the same or different mistakes. A video log would help researchers find errors and confirm results.
Video is simply a better solution.
Video is simply a better solution.
The whole scientific publishing field is still stuck in previous century. We desperately need a github for science, where any reader would be a reviwer, who would be able to rate the article and comment on it, and then journals would use these ratings combined with credibility rankings of reviewers to create indexes of interesting new article-repositories.
I have people citing research with useless AMA standard. If you want both parties to agree, they need to see the Data.
Most studies become uninteresting after you read their terrible data.
Most studies become uninteresting after you read their terrible data.
Absolutely agree, articles without data and without the sourcecode, are not science, and that's another reason why we need github for science.
Somewhat tangential, but are there any journals that will only accept a paper if someone unrelated to the author can replicate the study? Or a journal of only replicated studies?
Sure. My favourite is IPOL Journal · Image Processing On Line https://www.ipol.im/, which is doing it since 2011: «IPOL is a research journal of image processing and image analysis which emphasizes the role of mathematics as a source for algorithm design and the reproducibility of the research. Each article contains a text on an algorithm and its source code, with an online demonstration facility and an archive of experiments. Text and source code are peer-reviewed and the demonstration is controlled. IPOL is an Open Science and Reproducible Research journal».
More recently, some megajournals have joined the trend and are trying to expand it, particularly eLife: https://elifesciences.org/labs/ad58f08d/introducing-elife-s-... (also covered in https://www.nature.com/articles/d41586-019-00724-7 ).
For some context see: * https://en.wikipedia.org/wiki/Replication_crisis * https://www.frontiersin.org/articles/10.3389/fninf.2017.0006...
More recently, some megajournals have joined the trend and are trying to expand it, particularly eLife: https://elifesciences.org/labs/ad58f08d/introducing-elife-s-... (also covered in https://www.nature.com/articles/d41586-019-00724-7 ).
For some context see: * https://en.wikipedia.org/wiki/Replication_crisis * https://www.frontiersin.org/articles/10.3389/fninf.2017.0006...
We don’t.
Peer review is a very helpful component of a system that slowly filters and refines academic work. It’s like the first air intake stage of a jet turbine.
Peer review is a very helpful component of a system that slowly filters and refines academic work. It’s like the first air intake stage of a jet turbine.
If peer review in Academia is anything like code review in corporate software dev it's probably next to worthless.
Peer review never seems to catch actual bugs but does give you helpful tips like "we already have a function that does this" or "the language has a feature for this"
Oh code review done properly is really great - I'm just jaded by repeatedly seeing code go up and getting that green tick with no comments and knowing that the reviewer barely even attempted to read it.
buboard(3)
I think they're clearly correct, and that even on HN, in a community that (roughly) prides itself on being scientifically literate, there are broad misunderstandings of what peer review means (during the bogus "Sokal Squared" hoax, for instance, many commenters implied that peer review prior to publication was meant to encompass replication). Also, while I'm not a "scientist", I've gotten to do some peer review work for ACM and Usenix, and even in the little bit of review I did, I seen some shit. There is much less formality and oversight to review than you might expect.