Peer review: the end of an error?(the-tls.co.uk)
the-tls.co.uk
Peer review: the end of an error?
https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/
79 comments
A point not mentioned in the article is that this formal peer review system is itself quite young. I think that sending every paper out to several reviewers only became the default once we had xerox machines -- 1960s? We did a lot of serious science before that.
A bit earlier, the famous anecdote is about Einstein's offence that the Phys Rev editor would dare to send his paper for review at all -- apparently this would not have happened at a German journal of that era: http://www.preposterousuniverse.com/blog/2005/09/16/einstein...
A bit earlier, the famous anecdote is about Einstein's offence that the Phys Rev editor would dare to send his paper for review at all -- apparently this would not have happened at a German journal of that era: http://www.preposterousuniverse.com/blog/2005/09/16/einstein...
If anything, that works as a great defense of peer review given Einstein was wrong.
One hopes the present system will go away: http://sappingattention.blogspot.com/2017/09/peer-review-is-...
The problems are in some ways even worse in the humanities: http://www.chronicle.com/article/The-New-Intellectuals/23835... than the sciences. Or see https://jakeseliger.com/2013/02/12/a-lot-of-academic-researc... .
The problems are in some ways even worse in the humanities: http://www.chronicle.com/article/The-New-Intellectuals/23835... than the sciences. Or see https://jakeseliger.com/2013/02/12/a-lot-of-academic-researc... .
> Remarkably, we have arrived at a system where academics feel a moral obligation to perform the thankless task of reviewing the work of other academics, anonymously and unpaid. This undoubtedly makes the literature better than it would otherwise have been, and ensures at least one reader for each paper.
I'm afraid it doesn't ensure that all papers will be read. Some papers can attract more attention, some papers can even receive too much comments with relevant comments being drown in the sea of observations.
I'm afraid it doesn't ensure that all papers will be read. Some papers can attract more attention, some papers can even receive too much comments with relevant comments being drown in the sea of observations.
I think the "one reader" refers to the reviewer, who ostensibly should read as a part of the review process.
Exactly. It's the same fallacy with open source code. Just because it could be reviewed, doesn't mean it has been. Formal or semi-formal groups that professionally review papers are essential. How are they going to get paid?
If you want research to be publicly available (as it should be) you can either put the reviewers on government dime (which isn't a great idea but it's better than nothing) or you can follow the system that was pretty much followed until now:
Research institutions that produce papers to be reviewed, pay up the reviewers, and then we make the research publicly available, because everyone is paid and happy. No need to double-pay the reviewers via per-paper costs etc. The cost of reviewing a paper is fixed. Hosting costs are trivial compared to that. The uni can take them on. In a sense sci-hub fixed the peer-reviewed industry. Formalize it (make all journals open-access) and we're done.
Research institutions that produce papers to be reviewed, pay up the reviewers, and then we make the research publicly available, because everyone is paid and happy. No need to double-pay the reviewers via per-paper costs etc. The cost of reviewing a paper is fixed. Hosting costs are trivial compared to that. The uni can take them on. In a sense sci-hub fixed the peer-reviewed industry. Formalize it (make all journals open-access) and we're done.
Pull-request reviews work the same as peer review: it ensures that at least one person (the reviewer) reads the code or article.
The big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading service.
The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the most constructive peer review then I would put in the required effort to make peer review a worthwhile exercise.
One thing that would help the peer review process is a data dump of the raw data. Many times I have seen some figure or table in a manuscript and felt I could not provide a constructive critic without seeing the underlying data. Going back through the editor to get this data is just too much of a hassle most of the time.
The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the most constructive peer review then I would put in the required effort to make peer review a worthwhile exercise.
One thing that would help the peer review process is a data dump of the raw data. Many times I have seen some figure or table in a manuscript and felt I could not provide a constructive critic without seeing the underlying data. Going back through the editor to get this data is just too much of a hassle most of the time.
But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard?
Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.
Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There's a lot of value, even if it's not perfect, in writers being forced to respond to one or two reviewers' comments.
> But who decides what's constructive?
For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive.
A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.
For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive.
A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.
I suggested in the post either the editor or the other reviewers makes the decision of which was the most constructive review.
The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it.
In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
The suggestion of going back and doing more work exists under the current peer review system. With my proposal you might even get suggestions of what work needs to be done and why you should do it.
In all the papers I have published I have only had a few reviewers agree on anything. The normal approach is you agree to all the reasonable demands even if you don’t personally like them and argue over the important one. Most of the time if the reviewers see you have moved some of the way towards their arbitrary demands they will let you hold onto the important ones.
You're right, I completely missed that. I quite like the idea of other reviewers picking which is best, although it may take some fiddling to avoid abuse.
I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.
I wasn't suggesting reviewers currently can't tell you to go back and do more work, quite the opposite - that I think it's a valuable if often too arbitrary requirement. But if the writer isn't picking which comments to respond to my point is moot anyway.
The initial examples the author uses really do reflect the magnitude of the third problem. All of them were big breakthrough results in their fields - and hence people cared enough to do proper reviews - especially in the latter two cases. As an academic, I have participated in reviews of papers that I would have never otherwise read, but could still contribute to the process. Without something pushing all academics to participate in the system, then a more open solution risks moving science even closer to “only interesting problems/solutions/results get published”.
"Only interesting problems/solutions/results get published"
I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.
I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.
Isn't this the point of journals? They communicate new knowledge, which changes the status of what is known.
There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high.
Things which stay constant are the job of standards bodies, and textbook writers... these are important too, but different.
There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high.
Things which stay constant are the job of standards bodies, and textbook writers... these are important too, but different.
I don't think so. If you work in a hard science like medicine or psychology, where problems are still so hard we can't produce clear answers as to results of a paper, we shouldn't really be trusting papers until they've been reproduced by some unrelated team. But if these results don't contradict the previous papers, they're not really communicating new knowledge.
Sometimes this isn't a big deal (medicine is as far as I know regulated enough that reproduction is a part of the problem), but most science is characterised by not enough money distributed without caring about scientific outcomes.
So we do have a problem at the moment. A team who does important but thankless scientific work can't get published. Too much science is just first draft status. And the only way to determine that is that people keep trying and trying to build on it and eventually decades later the tower of cards collapses when clever new PhD students start down new research avenues.
Sometimes this isn't a big deal (medicine is as far as I know regulated enough that reproduction is a part of the problem), but most science is characterised by not enough money distributed without caring about scientific outcomes.
So we do have a problem at the moment. A team who does important but thankless scientific work can't get published. Too much science is just first draft status. And the only way to determine that is that people keep trying and trying to build on it and eventually decades later the tower of cards collapses when clever new PhD students start down new research avenues.
I agree there is definitely a bias towards such results - but I fear that completely free-for-all open access reviewing will make the effect even greater.
I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.
Was going to comment to say the exact same thing. OP used a paper by medical professionals to poo-poo formal peer review and then used math papers as examples of successful arxiv papers scrutinized by other scientists. After realizing that, I stopped reading the article then Googled his name, and--surprise, surprise--he's a mathematician.
It will work for mathy and theoretical sciences like CS and theoretical physics, but good luck getting experimentalists to review papers for free (or at least without being asked by a journal). I'm not sure whether peer review is necessary in those cases, but the culture around more applied sciences definitely considers peer review more important, even though we routinely post to arxiv anyway.
It will work for mathy and theoretical sciences like CS and theoretical physics, but good luck getting experimentalists to review papers for free (or at least without being asked by a journal). I'm not sure whether peer review is necessary in those cases, but the culture around more applied sciences definitely considers peer review more important, even though we routinely post to arxiv anyway.
It sounds like you're saying that experimentalists rely on peer review as a way to fraudulently cover up their shoddy experimental methods.
Hmm, if you're lab A, why would you help to do the cover up of lab B's work, especially if you might be competing with them?
Scientists at least in my field don't do each others' dirty work, we're much more disconected from each other than that. Also, peer review is anonymous.
Scientists at least in my field don't do each others' dirty work, we're much more disconected from each other than that. Also, peer review is anonymous.
I disagree. Even with the current peer review system in mathematics, false proofs have made it into the literature. Who knows how many more would creep in if we did away with peer review?
One can argue that false proofs made it into the literature because people trusted the reviewers - compare with: appeal to authority.
There are two points that I wished were addressed more often.
1. New researchers. The examples he mentions include researchers who were already important in the field, and whose prestige guarantees review. But if you're a new researcher, good luck getting noticed. Blind review doesn't have this problem.
2. The cost of journals. People say "journal" and think of Elsevier, but there are several journals and conferences that publish all papers for free. The ACL, the largest conference on computational linguistics, finances itself with the entry fee to their conference, and you can read all of their papers right now (disclaimer: I'm currently doing work for them). There is a middle ground between Elsevier and Arxiv.
1. New researchers. The examples he mentions include researchers who were already important in the field, and whose prestige guarantees review. But if you're a new researcher, good luck getting noticed. Blind review doesn't have this problem.
2. The cost of journals. People say "journal" and think of Elsevier, but there are several journals and conferences that publish all papers for free. The ACL, the largest conference on computational linguistics, finances itself with the entry fee to their conference, and you can read all of their papers right now (disclaimer: I'm currently doing work for them). There is a middle ground between Elsevier and Arxiv.
One way to look at journals are that they are "ready made" reputation for those without their own. A new author can leverage the reputation of the journal to establish their own reputation.
On that basis, maybe the mark of a truly prestigious researcher is one who can post directly to arXiv? Rather than the highest accolade being for a paper in Nature, the highest accolade should be for a widely reviewed paper on arXiv?
On that basis, maybe the mark of a truly prestigious researcher is one who can post directly to arXiv? Rather than the highest accolade being for a paper in Nature, the highest accolade should be for a widely reviewed paper on arXiv?
The academic peer review debate has strong parallels to the social-media news debate -- As traditional newspapers/ TV channels are no longer gatekeepers of publication, we have given up a level of vetting in exchange for freedom-from-censorship. But that puts an onus on consumers -- it's not right for consumer to accept everything they see with the same level of default trust they had before. Consumers still must "consider the source", and society to create an effective replacement for the editorial filters we are discarding.
see also: "web of trust"
see also: "web of trust"
I am afraid this piece is missing the most massive reason that articles are still published in journals - Impact Factor. Academics live and die not just by the number of articles that they get published, but, crucially, where they get published. The entire academic system of progression is basically controlled by the journals. The reason that the splashy websites never succeed is that they have targeted the wrong problem. If you want to fix the system then you have to come up with a new way of rating academic's publications, which would then be fed back into the career progression system in their individual institutions. Good luck with that.
While this is true in many institutions there are now useful proxies, specifically a Google Scholar profile or an individual h-index which can be calculated without publication impact factor.
You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addition because a lot of impactful work is coming out of industry and there is rapid turnover between academia and industry the culture as a whole is evolving.
You are correct that it will take a long time to update documentation and policies that reference impact scores, but informally, at least in Machine Learning, people know who the contributors are with high actual impact. In addition because a lot of impactful work is coming out of industry and there is rapid turnover between academia and industry the culture as a whole is evolving.
Of course there's an untruth in what I said. Ostensibly it's all about Impact Factor, but in reality it's all about how much money you are bringing in. Academia is in a sad state.
In some fields, such as HCI (and research through design) I think there could be value in moving to a system where instead you version papers and also publish the criticisms that led to its refinement.
If you've explored a problem, designed a piece of technology, and taken it back to that audience to gain further insights into the problem, what I can do in reviewing it isn't the same kind of analysis as a maths proof. Even if I you were to give me the tech or the code, I wouldn't have the same people to test it with, and would not expect to get exactly the same results. Instead, peer review tends to criticise the writing, missing elements in the argument, pieces of literature that also need considering, aspects of the methodology, and suggest more things you could do.
All of which is very valuable, and sometimes peer review is the best (only) mentoring you'll get if you're in a small university. But it's not quite the same as "we have to review this or a wrong claim might get the imprimatur of being science". Design questions don't make the same claims of completeness or context-independence. It's more of an argument based in the kind of muddy evidence you have to put up with when tackling muddy problems.
Which makes me think that instead of peer review being a hurdle to publication, perhaps we should just make the whole argument and refinement public. Give the reviewers more kudos as instead of being invisible entities whose thoughts are only received by the author and editor, their role in the development of that idea becomes a published item too.
If you've explored a problem, designed a piece of technology, and taken it back to that audience to gain further insights into the problem, what I can do in reviewing it isn't the same kind of analysis as a maths proof. Even if I you were to give me the tech or the code, I wouldn't have the same people to test it with, and would not expect to get exactly the same results. Instead, peer review tends to criticise the writing, missing elements in the argument, pieces of literature that also need considering, aspects of the methodology, and suggest more things you could do.
All of which is very valuable, and sometimes peer review is the best (only) mentoring you'll get if you're in a small university. But it's not quite the same as "we have to review this or a wrong claim might get the imprimatur of being science". Design questions don't make the same claims of completeness or context-independence. It's more of an argument based in the kind of muddy evidence you have to put up with when tackling muddy problems.
Which makes me think that instead of peer review being a hurdle to publication, perhaps we should just make the whole argument and refinement public. Give the reviewers more kudos as instead of being invisible entities whose thoughts are only received by the author and editor, their role in the development of that idea becomes a published item too.
I've published in CS conferences, CS journals, and on a blog.
Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you.
OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for conferences/journals.
Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you.
OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for conferences/journals.
The difference is that more often than not, you can't go to the same level of detail and depth in a blog post as you would in a scientific article (and if you do, not that many people will read it). Reading science is hard work, and properly reviewing a "journal length" paper can take upwards of a day. I don't think you can have it both ways. If you write a good vulgarization, or technical blog post, and submit it here, you will likely get many eyeballs. If you post serious science, you'll get much less feedback, if any.
My blog posts typically have more scientific content than a CS paper.
I'd be happy to see some, please link
Care to link? Now I'm curious.
Not to defend peer review as it is now, but one of the things I've come to value from editorial gatekeepers is that they put the brakes on me publishing things that in later years I might regret. I wrote some pretty mediocre papers as a grad student and I'm grateful some of them were rejected, because I wouldn't want them forever part of my legacy. As a matter of fact, I wrote some "papers" when I was a teenager, which, if I had been allowed to freely publish at that time, might have become permanent albatrosses around my neck :P
Didn’t your supervisor step in and say this is not up to scratch? When I was a supervisor a good chunk of my time was spent going through my student’s and postdoc’s manuscripts to ensure nothing was in them that might embarrass us both.
I was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor.
Fortunately, the bad ones were rejected. The good ones ended up making me well-recognized in some extremely tiny niches, and ultimately did way more for my career than my PhD work did :D I'm really glad some of the bad ones didn't see the light of day :P
Fortunately, the bad ones were rejected. The good ones ended up making me well-recognized in some extremely tiny niches, and ultimately did way more for my career than my PhD work did :D I'm really glad some of the bad ones didn't see the light of day :P
> I was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor.
This is the opposite of reckless, this is how you make a name for yourself, and how you prove that you can develop an independent research agenda. As you say, it's good for your career, and it's good for science too (if the work is good, of course). Good for you!
This is the opposite of reckless, this is how you make a name for yourself, and how you prove that you can develop an independent research agenda. As you say, it's good for your career, and it's good for science too (if the work is good, of course). Good for you!
mmm not wise :)
I guess it depends on the field, but in mine (biological sciences) I can’t think of an example where a student would submit a paper without their supervisor knowing.
Anyone thinking of doing the same as you should try and draw on their colleagues in the department to proofread the manuscript. I used to proof lots of manuscripts from my colleagues and their students.
I guess it depends on the field, but in mine (biological sciences) I can’t think of an example where a student would submit a paper without their supervisor knowing.
Anyone thinking of doing the same as you should try and draw on their colleagues in the department to proofread the manuscript. I used to proof lots of manuscripts from my colleagues and their students.
Do some of the alternative platforms offer a way to repudiate a "published" paper?
I really think we should have something like Stack Overflow for academic papers. The comments system works so well. Also, the so called impact factor of journals, which in many cases are directly or indirectly linked with academics and universities financial outcome, could be replaced by the points system from SO.
Of course I'm not proposing using SO directly for academic research but something inspired by it.
Of course I'm not proposing using SO directly for academic research but something inspired by it.
I think a hybrid between SO and Discourse for papers would be more fitting. There really needs to be an open but moderated discussion for the papers - that'll serve as the peer review.
An interesting parallel can be found in the anonymous review of grant proposals. See "Fund ideas, not pedigree, to find fresh insight", https://bit.ly/2Fy64cP. It ends with "peer review may be the worst system, except for all the others". It can be found in Nature, a peer reviewed journal.
Archived copy, which can be read without JS enabled:
https://archive.is/iPwXi
https://archive.is/iPwXi
Reddit but only academics can participate? Probably better then peer review, but it won't have the aura.
I've said it before and I'll say it again.
The focus should be on reproducing studies, not on peer-review.
Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues.
I used to hold the peer-review and journal process on a pedestal. Then when working at a genetics lab I ended up having to read a lot of papers to get my sysadmin work done, and started realizing just how bad so many papers are, even though published in "reputable" journals.
For example, since publishing is seen as so vital to some people, I noticed many of them would "consult" on another groups paper just to get their name on the paper, even though they did almost no work, and then those same people would turn around and say stuff like "I've been published X times!" as a sort of reinforcement of authoritativeness (and as advertisement).
People are gaming the system because hardly anyone is actually checking them. Beyond that, the actual peer-review process itself seems to be very lackluster in rigor on it own.
I think there should be a certain amount of public research funding that should be earmarked to go directly to independent reproducability labs.
On top of it all, all this research that the public is funding is often behind a fucking paywall... and that's not just bullshit, it's theft and misuse of public funds. It should be required that publicly funded research should be publicly published, and therefore available for more eyes to see and review.
The focus should be on reproducing studies, not on peer-review.
Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues.
I used to hold the peer-review and journal process on a pedestal. Then when working at a genetics lab I ended up having to read a lot of papers to get my sysadmin work done, and started realizing just how bad so many papers are, even though published in "reputable" journals.
For example, since publishing is seen as so vital to some people, I noticed many of them would "consult" on another groups paper just to get their name on the paper, even though they did almost no work, and then those same people would turn around and say stuff like "I've been published X times!" as a sort of reinforcement of authoritativeness (and as advertisement).
People are gaming the system because hardly anyone is actually checking them. Beyond that, the actual peer-review process itself seems to be very lackluster in rigor on it own.
I think there should be a certain amount of public research funding that should be earmarked to go directly to independent reproducability labs.
On top of it all, all this research that the public is funding is often behind a fucking paywall... and that's not just bullshit, it's theft and misuse of public funds. It should be required that publicly funded research should be publicly published, and therefore available for more eyes to see and review.
> The focus should be on reproducing studies, not on peer-review
The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.
The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.
Reproducing is a form of peer review (if you assume that a person who successfully attempts to reproduce your work is ipso facto your peer).
But the fact that your peer (i.e. another academic who works in the same area) has read your paper and commented on it in the process of publication (i.e. customary "formal peer review") does not reflect on its reproducibility. The peer, being a human, has attentional deficits and there's probably true facts that increase the chances of the result occurring by chance which are not considered worthy of publication.
Until a paper's result is reproduced, it can't be regarded as reproducible.
But the fact that your peer (i.e. another academic who works in the same area) has read your paper and commented on it in the process of publication (i.e. customary "formal peer review") does not reflect on its reproducibility. The peer, being a human, has attentional deficits and there's probably true facts that increase the chances of the result occurring by chance which are not considered worthy of publication.
Until a paper's result is reproduced, it can't be regarded as reproducible.
I agree, I just worded it slightly poorly. My point is that the biggest focus should be on reproducing studies, not that peer-review shouldn't be revamped also.
I mean, isn't reproducing a study the ultimate peer-review?
I mean, isn't reproducing a study the ultimate peer-review?
(2017)
This article says that peer review is unpaid - it isn't.
If you are an academic or work in industrial research, it's part of your paid job to do peer review as part of the community. You are paid for it - it's part of your job and it's normal to do it during work hours.
If you are an academic or work in industrial research, it's part of your paid job to do peer review as part of the community. You are paid for it - it's part of your job and it's normal to do it during work hours.
That's the way it should be seen, but it's often not.
I was at a conference not too long ago and chatting with a fellow faculty member at another university, pretty well-known in his field (in a medical school department), who told me a story about how his chair berided him for doing peer reviews at all. The colleague was like "but this is part of our job as a scientist" and the chair's response was "but what account number is that coming out of", as in, "who is paying for that?" The chair proceeded to explain that unless the department was being paid for his peer reviewing time, he shouldn't be doing it.
He subsequently explained that he feels like he's checking out of academics per se and trying to shift to public outreach activities because he's tired of trying to make a case for research integrity for its own sake, as opposed to being a vehicle for bringing funds into the department.
Things in higher education are quickly moving toward a state (or have already moved toward a state) where it is implicitly assumed that your job as a researcher is to bring in money, not to do research. That is, if you are paid by the state, you are not being paid to do a job (research) you are being paid to bring in even more money. You're a salesperson or adsperson not a laborer. Grad students are the laborers. I strongly disagree with this model, but it's where we are.
I used to think that conversations like the one with my colleague were kind of fictional dystopian kinds of conversations until I actually had one. And he was dead serious.
Universities are quickly becoming places to rent space for research, basically.
I was at a conference not too long ago and chatting with a fellow faculty member at another university, pretty well-known in his field (in a medical school department), who told me a story about how his chair berided him for doing peer reviews at all. The colleague was like "but this is part of our job as a scientist" and the chair's response was "but what account number is that coming out of", as in, "who is paying for that?" The chair proceeded to explain that unless the department was being paid for his peer reviewing time, he shouldn't be doing it.
He subsequently explained that he feels like he's checking out of academics per se and trying to shift to public outreach activities because he's tired of trying to make a case for research integrity for its own sake, as opposed to being a vehicle for bringing funds into the department.
Things in higher education are quickly moving toward a state (or have already moved toward a state) where it is implicitly assumed that your job as a researcher is to bring in money, not to do research. That is, if you are paid by the state, you are not being paid to do a job (research) you are being paid to bring in even more money. You're a salesperson or adsperson not a laborer. Grad students are the laborers. I strongly disagree with this model, but it's where we are.
I used to think that conversations like the one with my colleague were kind of fictional dystopian kinds of conversations until I actually had one. And he was dead serious.
Universities are quickly becoming places to rent space for research, basically.
> you are not being paid to do a job (research) you are being paid to bring in even more money
And research funding is going towards big hotel, travel expenses, and conference fees.
> Grad students are the laborers
Making the business highly profitable as there is no need to hire a large number of professors where you can offer tuition waivers and cheap stipends to a larger number of people who wants to get another degree
And research funding is going towards big hotel, travel expenses, and conference fees.
> Grad students are the laborers
Making the business highly profitable as there is no need to hire a large number of professors where you can offer tuition waivers and cheap stipends to a larger number of people who wants to get another degree
I tend to do most of my paper reviewing during the weekend. No one is going to complain if I do it during my normal work hours, but the demands on my time are high enough that it's hard to focus on it. So in some sense, for me, it is unpaid, outside of my paid day job.
Trust me that a minute spent reviewing a paper is one less minute you have toward your actual KPI’s. About the only value of peer review is to give the paper to your students and tell them to find all the mistakes and holes in the paper. This is actually quite a good for the education of grad students as they learn what not to do.
Although reviewership is a useful thing that gets put on CVs, I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement.
A more salient point about why it is important to note that peer review is unpaid is that academics, as authors, submit their work---for free---to a journal, where reviewers review it for free, and then the journal turns around and sells the article for $35 apiece.
A more salient point about why it is important to note that peer review is unpaid is that academics, as authors, submit their work---for free---to a journal, where reviewers review it for free, and then the journal turns around and sells the article for $35 apiece.
This is such an adversarial way to look at it.
I see it as a community. We write and share our papers as part of our jobs, we review each others' papers as part of our jobs, we help to run the conference as part of our jobs. Everyone contributes to run the system.
Nobody buys articles for $35 a piece. All major universities and companies subscribe to the ACM or whatever your field's organisation is for you. And guess who runs the ACM and other organisations? The community does! People from the community serve as leadership and help run things.
Yes the ACM gets some money to run their website, conferences and outreach programs. I don't think that's crazy.
I see it as a community. We write and share our papers as part of our jobs, we review each others' papers as part of our jobs, we help to run the conference as part of our jobs. Everyone contributes to run the system.
Nobody buys articles for $35 a piece. All major universities and companies subscribe to the ACM or whatever your field's organisation is for you. And guess who runs the ACM and other organisations? The community does! People from the community serve as leadership and help run things.
Yes the ACM gets some money to run their website, conferences and outreach programs. I don't think that's crazy.
I don't have a huge issue with the ACM or even so much with IEEE. But have you ever submitted to or reviewed for Springer or Elsevier? It's an emasculating and thankless experience, and Springer and Elsevier go get to make huge profits off of everyone's free work. (haha, we just did it for exposure, right???)
I don't mean to imply that peer review is this shitty thing that we "have" to do. I review papers regularly and generally enjoy the process. But I think it's important to point out that no, we are not being paid for it, and actually typically the incentives in our jobs mean that peer review is a "thing we have to stuff in there somewhere", and certainly not a primary task.
I don't mean to imply that peer review is this shitty thing that we "have" to do. I review papers regularly and generally enjoy the process. But I think it's important to point out that no, we are not being paid for it, and actually typically the incentives in our jobs mean that peer review is a "thing we have to stuff in there somewhere", and certainly not a primary task.
> emasculating
that's an unfortunate word.
that's an unfortunate word.
The ACM model is a good one, but it's not the norm in the other sciences. By which I mean that the main publisher of computer science work is actually a professional organization, which is community run and managed. But for people in fields where their professional organization and main publishers are different, it's not the same.
As computer scientists, we have it pretty good.
As computer scientists, we have it pretty good.
> I see it as a community.
Maybe you'd like to take a look at these members of the community, who show they care about the community by refusing to participate in journals for certain publishers: http://thecostofknowledge.com/ It even has a Wikipedia article: https://en.wikipedia.org/w/index.php?title=The_Cost_of_Knowl... And here's an example of one of them: https://www.scottaaronson.com/writings/journal.pdf
Maybe you'd like to take a look at these members of the community, who show they care about the community by refusing to participate in journals for certain publishers: http://thecostofknowledge.com/ It even has a Wikipedia article: https://en.wikipedia.org/w/index.php?title=The_Cost_of_Knowl... And here's an example of one of them: https://www.scottaaronson.com/writings/journal.pdf
The whole reason they charge money is that not everyone who wants to read the papers has access. I work for government cancer lab. We subscribe to a lot of journals. Even so, I see these paywalls come up regularly because we can't subscribe to all journals.
> I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement.
I work in an industry research lab in the bay, and while there isn't a requirement to review X papers a year, it is expected that you keep up with the literature, and actively engaging in peer review is one way to do so. A few of my colleagues are on the paper committee for our major academic conference, and they end up reviewing 40+ papers a year and using a week or so to take part in the committee meetings.
I work in an industry research lab in the bay, and while there isn't a requirement to review X papers a year, it is expected that you keep up with the literature, and actively engaging in peer review is one way to do so. A few of my colleagues are on the paper committee for our major academic conference, and they end up reviewing 40+ papers a year and using a week or so to take part in the committee meetings.
It is unpaid. A lot of reviewing for conferences is actually being done by Ph.D. students. They have to do it, whether they are on a stipend or not. Equally, a lot of reviewing is being done by (in some cases, former) academics whose current occupation does not provide resources (time) for doing peer review.
> They have to do it, whether they are on a stipend or not.
This is like saying that undergraduate students are unpaid for doing their homework.
It's part of their education to learn how to take part in the community by doing things like reviewing papers.
This is like saying that undergraduate students are unpaid for doing their homework.
It's part of their education to learn how to take part in the community by doing things like reviewing papers.
Well sure, but nobody benefits financially or reputationally from an undergraduate student's homework. There's clearly some proportion of having an academic's PhD students do this type of work for them which would stop being educational and start being an abuse. The question is, is too much of this type of work being pawned off on grad students?
> It's part of their education to learn how to take part in the community by doing things like reviewing papers.
And this "part of their education to learn" keeps really cheap labor running science. Even postdocs are considered trainees, because they will always be learning because is part of their "job"
And this "part of their education to learn" keeps really cheap labor running science. Even postdocs are considered trainees, because they will always be learning because is part of their "job"
> A lot of reviewing for conferences is actually being done by Ph.D. students. They have to do it, whether they are on a stipend or not.
Most (all?) PhD students in science and engineering are on fellowships, at least in the US. Major research universities will not admit unfunded PhD students.
Most (all?) PhD students in science and engineering are on fellowships, at least in the US. Major research universities will not admit unfunded PhD students.
Oh boy, would I love that to be the case as an unfunded PhD student, working a separate job as visiting faculty.
The point is that it's done largely out of a feeling of obligation to the community. There's a chance that if you declined all of it you'd get a reputation for being an asshole, but there would be no other consequence for you. Whoever pays your salary would probably not notice.
I have actually been paid by some journals, but only a token amount, like $30/paper.
I have actually been paid by some journals, but only a token amount, like $30/paper.
Engaging in the profession is a part of your job. Doing editorial work for a journal (many of which are for-profit) is arguable. It is certainly accurate to say that reviewers are unpaid by the journal.
https://en.wikipedia.org/wiki/The_Cost_of_Knowledge
He has practiced what he preaches with respect to publishing in open access journals.