Is Peer Review a Good Idea? (2020)(journals.uchicago.edu)
journals.uchicago.edu
Is Peer Review a Good Idea? (2020)
https://www.journals.uchicago.edu/doi/10.1093/bjps/axz029
95 comments
CS publication culture is a large part of the problem. The average CS paper is small (in terms of person-years put into the project), so people submit many papers. Most papers are submitted to conferences first, which usually accept or reject after a single round of reviews. When the typical outcome is rejection and resubmission to another conference, the negative aspects of peer review become prominent.
Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper.
Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper.
Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
This is a very thoughtful take on the issue. My personal experience corroborates with this - CS conferences have more of these culture issues than mathematics journals, where the reviews tend to more thoughtful, and at the same time, considerate.
Agree. I used to think double blind was an obvious requirement for a good peer review system. Your “theatre” take is aptly harsh I’d say.
Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have.
My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent.
More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers?
Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher.
Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty.
~~~~~~~~
[1] https://www.cos.io/initiatives/registered-reports
Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have.
My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent.
More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers?
Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher.
Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty.
~~~~~~~~
[1] https://www.cos.io/initiatives/registered-reports
> Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.
In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
Not only this, but people publish their papers months ahead on arxiv. Big labs have big publicity and thousands will read their papers. They can often do more impactful work too because the massive amount of compute that they have. There's a good chance that if the reviewers are qualified to review the material they have already read the paper. (or a public version can be found)
One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
> Not only this, but people publish their papers months ahead on arxiv.
What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
> Big labs have big publicity and thousands will read their papers.
Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
> Big labs have big publicity and thousands will read their papers.
Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
> What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities.
> Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive.
Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that.
But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work.
The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.
There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities.
> Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive.
Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that.
But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work.
The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.
The opposite issue also - conflicts of interest can automatically disqualify reviewers, and if you are conflicted with all of microsoft research, or all of google research, etc., then the number of qualified reviewers shrinks dramatically.
Who generally identifies these conflicts of interest? Is it reliant on self-reporting?
In my experience, yes, essentially all self-reporting. They might do something like automatically conflict your email address domain (*.mit.edu), which is fun when random unrelated people sign up with gmail addresses.
> This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
That misses the whole point of anonymous reviews, or even anonymous voting.
You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work?
When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Your comment reads a whole lot like "the fox and the grapes".
https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes
That misses the whole point of anonymous reviews, or even anonymous voting.
You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work?
When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Your comment reads a whole lot like "the fox and the grapes".
https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes
> You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well.
> Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc.
Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't.
> When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career.
[1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well.
> Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc.
Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't.
> When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career.
[1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
It would be interesting to see if there is a distinction between review and acceptance statistics depending on timeframe.
For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
First, journals and conferences are different. I would expect more valuable feedback from journals than from conference reviews.
Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
>Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
In some publications (mostly CS conferences), there are associate chairs or 'meta-reviewers' whose sole job is to manage the peer review for a paper. They will read the reviews and if there isn't consensus, they'll ask the reviewers to read the other reviews and have a discussion. It is usually 3 reviewers and 1 meta-reviewer per paper, so there are often 2-1 splits.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
I really hope that something like https://openreview.net/ becomes the norm for all scientific journals in near future.
The open discussions not only benefit the peer review process but also act as context and learning material for others.
The open discussions not only benefit the peer review process but also act as context and learning material for others.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this.
Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this.
Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
The one experience I had with peer review before leaving academia was actually pretty nice. One of the reviewers was a bit of a hardass, but they made good points and I think made the paper much better.
As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
Peer review is useful, no doubt. It is only reviewer anonymity that probably should be re-examined.
There would probably be a need for a mechanism to ensure objectivity. E.g., if Reviewer A rejects Reviewer B, there may be a higher chance Reviewer B will reject Review A's future work, just out of spite.
Yes, remove all the blinders.
Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing.
I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing.
I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
Peer review is done for free, maybe it needs to be compensated.
I can imagine some unintended consequences from this. If I'm a hard-going reviewer, people might choose to avoid me, and I'll be compensated less. Conversely, if I'm easy-going, people might find ways to get their papers to me.
I suppose there can be some other layer of reviewer meta-review to account for this, much like the role 'acceptance rate' has come to have for journals and conferences. But, following Goodhart's law, even that has come to be gamed now that a meaning - a proxy for journal prestige or quality - has been placed on it.
I suppose there can be some other layer of reviewer meta-review to account for this, much like the role 'acceptance rate' has come to have for journals and conferences. But, following Goodhart's law, even that has come to be gamed now that a meaning - a proxy for journal prestige or quality - has been placed on it.
That's why I use the term "credit". It needs to have real-world value of some sort. I don't think just paying people is the solution, that would just mean the richer you are, the more reviews you get, and the better your reviews are. You can argue they should be paid via a neutral source of money, but there is no such thing.
If the problem of gaming the system could be gotten around, something very similar to internet comments could work - anyone can post a public review of anyone's work, in which case reputation would become the currency. Successfully criticizing someone with higher reputation than yourself would gain more status than praising them, but also come with more risk. It would also make it a lot less meaningful to get those 'not new ideas lol' comments that others on this thread have complained of.
I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
There are efforts to give reviewers credit for their work, like Publons.
I have no serious involvement in academia. What is the proportion of work that gets reviewed by someone with an axe to grind? I don’t doubt that it happens regularly but I have no conception of the magnitude of the problem.
In experimental fields, when submitting to Nature / Science / Cell, it's likely that your work ends up getting reviewed by someone who sees you as a competitor (this mentality drives me mad).
Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications.
Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university).
For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications.
Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university).
For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
>For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
This is interesting. Has there been a write up on this saga anywhere?
This is interesting. Has there been a write up on this saga anywhere?
I can't find better writeups at the minute, but this is a good starting point: https://en.wikipedia.org/wiki/Francisco_Mojica#CRISPR_as_a_m...
In my experience, about 30-40% of the reviews I've seen are a mix between "axe to grind" and "I have no idea what I'm talking about". Some are extremely blatant, such as reviewers "suggesting" you should cite a certain paper that isn't really related to your work. Others are very strict with the "this is just practical knowledge so it doesn't advance the science", and then there's the set of "I didn't understand the paper so I'm going to criticize it based on my misconceptions of it and reject it".
I'm so glad to be out of that world, to be honest.
I'm so glad to be out of that world, to be honest.
My favorite anecdote is from Adam Grant, who once got a snarky reply from a reviewer that "the author would do well to familiarize themselves with the work of Adam Grant."
A lot of authors are also very sensitive to negative feedback. How many times did I hear "the reviewer doesn't like us" when the feedback was legitimate.
[deleted]
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
Not meant in a snarky way, but that almost feels to me as if the process worked. If they application committee did not see the value in the proposed work, it indicates they probably don't have much need for it. If they don't have much need, they shouldn't fund it.
That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
All of the process creates incredible gatekeeping effects. Science would be much more robust if published github-style: post your results, engage with the community, fix issues, respond to critics, and eventually wrap it up, but always have the priority date of your original work recorded so that there's no dispute as to who invented it first.
Just one question about that - what you are describing here - is that:
.) an unsolvable problem inherent to all peer reviews?
.) a problem that comes from specific practices, and could be solved by doing peer reviews differently?
I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?
.) an unsolvable problem inherent to all peer reviews?
.) a problem that comes from specific practices, and could be solved by doing peer reviews differently?
I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?
Your parent is not arguing against peer reviews, but against blind and double blind reviews.
They are actually proposing that peer reviews are credited:
> Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly.
However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.
They are actually proposing that peer reviews are credited:
> Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly.
However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.
It would be a much better idea if double-blind peer review were a default. See "Nobel and Novice: Author Prominence Affects Peer Review": https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4190976
Double-blind peer review sounds nice in theory, but I don't think it would hold up in practice, especially in specialized domains.
In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
So, I could be charitably described as mid-career, and I had this discussion a few times a couple decades ago -- in which I took the position you have. It's not an unreasonable position.
I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews.
I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible.
What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.
I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews.
I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible.
What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.
Worse still, many papers are basically unintelligible to anyone outside of an "inner circle".
Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!"
For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way!
The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper.
In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations.
I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with.
PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else.
Picture that.
That's modern science.
Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!"
For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way!
The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper.
In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations.
I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with.
PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else.
Picture that.
That's modern science.
> Worse still, many papers are basically unintelligible to anyone outside of an "inner circle".
Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony.
Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony.
Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
These perverse incentives (publish or perish; using big words to appear more serious) are far bigger problems than peer review being single-blind or not.
Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better.
Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results.
Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better.
Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results.
Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
It might not be perfectly effective, but I don't see any substantive drawbacks vs the current system.
It seems like in small fields, it would just be adding implausible deniability. Neutrality theater?
The field(s) I'm in use double blind and it's broken for a host of other reasons. It's really hard to build off a string of work for example. Students (eg the ones writing papers) are often bad at the anonymization requirements. On the other side, the reviewers can often form an opinion on likely authors (whether correct or not). And then up one level (at the editorial / program committee level) that anonymization goes away since they need to check for reviewer conflicts, etc.
Also in fields small enough that everyone knows each other, you know exactly what the rest is working on because they collaborate often and there aren’t that many reviewers
That would be an improvement, but the formal peer-review process itself is a problem that can and should be solved: https://jakeseliger.com/2020/05/24/a-simple-solution-to-peer...
I don't think there's much research that double blind significantly improves the process. Been a while since I read it, but IIRC this article goes into that: https://absolutelymaybe.plos.org/2015/05/13/weighing-up-anon...
Modern academics are so niche that people reviewing a paper can pretty much guess who wrote it if they are a known quantity.
Perhaps, surprisingly, it might go both ways: when I know who is on the PC, I think I can tell which person wrote which review, because I both know their work and met them personally. In fact, those people on the PC is why I am submitting to a venue in the first place, because I want reviewers who will easily understand my work.
Double-blind is not all that feasible unless preprints, conference posters, and talks were also all done anonymously.
Know of a few people that have gotten dismissive comments to review better the work from X paper of Y author... when the person submitting is that same one.
You can't do much.
You can't do much.
I think ML, at least when I was involved in it, had a much more sane way of approaching publication than other fields (I came from a biology background).
Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student.
But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important.
So I'm all for abolishing "pre-publication peer review".
Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student.
But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important.
So I'm all for abolishing "pre-publication peer review".
This vastly underestimates the esoteric nature, experiential knowledge and complexity required to review most research published in the scientific community. The common public cannot be expected to provide non-populist capable review of pharmaceutical level research
But they can do so for deep learning and mathematics? Because in those fields, as I said, pre-publication peer review is not as common.
Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.
Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.
As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work.
I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them.
According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities.
[1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them.
According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities.
[1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
I just don't see how double-blind peer review can work. To properly peer review a manuscript you often have to read the papers cited (because often details of the methods of the current manuscript are described in previously published work). These citations are almost always to the authors' own work. So it is trivial for any peer reviewer to figure out who wrote the manuscript they are reviewing, even if their names are blinded.
I hear this a lot and I just don't buy it. The whole idea of science is that you can describe methods in text, send that text around the world, and have other people use those same methods to do their own independent studies. So even if you know that lab X really likes using a particular method, it could be someone from lab Y who read the papers from lab X and wanted to use them as well.
Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
I'm sure different fields have different dynamics, and in some what you describe here might be viable and accurate.
But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy.
The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software.
Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature.
None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality.
That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process.
Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on.
An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.
But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy.
The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software.
Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature.
None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality.
That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process.
Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on.
An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.
> I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work.
At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.
At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.
Interesting to see how the good ol’ “think of the children” defense has morphed into “think of the women and minorities”.
> I have a partial solution: researchers “publish” papers to arXiv or similar, then “submit” them to the journal, which conducts peer review. The “journal” is a list of links to papers that it has accepted or verified.
This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2].
This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals?
[1] https://arxiv.org/about
[2] https://arxiv.org/help/submit
This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2].
This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals?
[1] https://arxiv.org/about
[2] https://arxiv.org/help/submit
"Official" (aka well-known/have a high "Impact Factor") journals give you a stamp of approval that help you get recognised in your career, and in some fields are where your peers go to stay up to date on new potentially relevant research.
(Disclosure: I volunteer for https://plaudit.pub, a non-profit that aims to separate that from the publication process.)
(Disclosure: I volunteer for https://plaudit.pub, a non-profit that aims to separate that from the publication process.)
Submissions are moderated, but depending on the field, anything reasonable-looking may be accepted. For example, there is more than one "proof" of the Riemann Hypothesis in the arXiv. But the moderation system certainly keeps out spam.
Edit: Here's a proof of the Riemann hypothesis submitted just last week: https://arxiv.org/pdf/2209.01890.pdf You can have a little fun exploring "proofs" of famous theorems on the arXiv over the years.
Edit: Here's a proof of the Riemann hypothesis submitted just last week: https://arxiv.org/pdf/2209.01890.pdf You can have a little fun exploring "proofs" of famous theorems on the arXiv over the years.
Where is that quote from?
Did you mean to reply to https://news.ycombinator.com/item?id=32832212 ?
No? Maybe I misinterpreted your comment. I was answering your question
> Where is that quote from?
by providing a link to the source of the quote.
Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
> Where is that quote from?
by providing a link to the source of the quote.
Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
Not ironic, that is what I meant. I didn't realize they were two different people, my bad.
The standard should be reproducibility, not peer-reviewed. Reproducibility is open loop, peer-review only is very susceptible to closed loop.
There's lots of science that can't be reproduced before publishing or where reproducing doesn't help:
* Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish)
In fact it's only small and self contained experiments where this is really practical.
* Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish)
In fact it's only small and self contained experiments where this is really practical.
>Computer simulations (re-run the simulation? Well that's not going to detect most issues
It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
There were two popular models, in rough agreement. https://publichealth.jhu.edu/2020/transparency-reproducibili...
> Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries.
Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 from May 29, 2020.
You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models.
> Running the same simulation with the same inputs produced wildly different results between runs.
As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution.
> I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here.
I can. The Reinhart-Rogoff spreadsheet error which was widely used as the excuse for "austerity" cuts, rather than the real reason of wanting to lower taxes on the rich. https://truthout.org/articles/excel-spreadsheet-error-lesson...
I wonder where that tax money could have been used. Perhaps into public health measures?
> Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries.
Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 from May 29, 2020.
You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models.
> Running the same simulation with the same inputs produced wildly different results between runs.
As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution.
> I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here.
I can. The Reinhart-Rogoff spreadsheet error which was widely used as the excuse for "austerity" cuts, rather than the real reason of wanting to lower taxes on the rich. https://truthout.org/articles/excel-spreadsheet-error-lesson...
I wonder where that tax money could have been used. Perhaps into public health measures?
If it’s not reproducible then assertions such as “the science is settled” and “trust the science” are completely inane and certainly should not be used as policy cudgels.
Different categories. Papers are what peers review. Experiments are what are reproducible.
[deleted]
The standard isn't peer review, it is unusual. https://xkcd.com/882/ applies to science and will get you publishes that green jelly beans cause cancer with no need to mention all experiments that got expected results.
Which is why reproducibility should be the most important topic.
Which is why reproducibility should be the most important topic.
Having been on both sides of the peer review process in my field (Theoretical Physics), I say we should completely abolish it. Just publish the papers on the preprint archives. Some will be bad, some will be good, and you will know the impact of a paper after 10 years, nothing different from how it is now. Currently the very big majority of the effort in the reviews (on both sides of the process) is wasted time.
Off-topic comment: it surprised me that The British Journal for the Philosophy of Science is published by The University of Chicago Press.
While knowing who the authors are can cause bias, hiding behind anonymization (as a reviewer) can as well. I think publishing the reviews (and who reviews) would help a lot (some fields do this). Science is really a dialog. The authors are putting forth part of the dialog and the reviewers are reflecting on it and giving their opinions. Hopefully publishing names next to reviews would minimize crappy reviews (often by the most senior folks). However the other problem is there are not enough qualified reviewers to go around and I could see many opting out of putting their name next to a review. But, we're likely better off.
There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.
There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.
having worked in academic publishing, i've concluded that peer review would perform best if it were embedded in the very act of reading (so post-publication, non-formalized kind of peer review) instead of having it be a formalized process riddled with issues and cracks as it is currently.
it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it).
sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it
it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it).
sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it
I like the basic principle of peer review, but not how it's done. I understand why it was done that way for so long, but we have the technology to do better now.
I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use.
Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.
I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use.
Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.
Doesn't the proposed solution already exist? Authors are allowed to publish their papers in preprint repositories before submitting to a journal.
The way Journal of Open Source (JOSS) - https://joss.theoj.org - does the reviewing process is good. Everything is out in the open, reviewers and authors names, reviews, responses, discussions etc. The whole process is out there for everyone to see.
The main issue with critical and complex code is that most if not all of your peers who are going to review your code are actually hackers..........
This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive.
Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.