Farewell peer reviewing

I’m saying goodbye to peer reviewing. Of course, you can’t just tap out of the system in which you work, but as much as possible, I’m saying goodbye to welcoming exploitation into my inbox.

A young academic’s CV or grant application invariably touts their service to the scientific community as a reviewer for a suite of prestigious journals. It’s a convenient way to re-assure people that you are part of a community – you put your hand up to serve and you are part of the space where decisions get made. Ideally, it shows that in your little part of the scientific world, you are a somebody.

In the past I’ve written a few blog posts that have focused on peer review. I’ve discussed my own frustrations as an author, based on firsthand experiences. I’ve also discussed peer review more broadly as a means to legitimise scientific knowledge. But in spite of several blog posts and twitter conversation, I still can’t figure out what the devil we expect to get out of peer review and what role we expect editors, reviewers and authors to play in the process.

Sometime ago I reviewed a paper for a “decent”* journal that I was surprised went out to review. I try to be a generous reviewer, but the paper was riddled with errors ranging from typographical to methodological. It was highly descriptive, the analysis and discussion were scant and the conclusions unoriginal. It wasn’t ready to be submitted.

The other two reviewers’ opinions aligned with my own report. The editor decided major revisions were in order and eventually a revised manuscript was returned to me. Only superficial changes had been made and many reviewer comments went largely ignored. I recommended a rejection, and was surprised sometime later to be thanked and notified that the paper had been accepted for publication with only cursory improvements.

More recently, I reviewed a paper for an “excellent”* journal. I was asked to review for a particular purpose and given some guiding questions to help in helping the editor. In my opinion, the paper was good – well written, nice figures, punchy and of interest. I had some qualms about a few minor analytical points and felt that there were a few unjustified bombastic statements. The other two reviewers had differing areas of expertise but had formed similar impressions of the quality of the paper.

It was later returned to me with a request for a quick turnaround review.  Such requests naturally align with a week jam-packed with meetings, workshops, dentist appointment and trips to drop off/pick a relative up from the airport. I made a decision and obliged with the request. I had some very minor comments but thought it was good to go. Reviewer 3 was more critical but agreed it was nearly ready.

A few weeks later, I received an email that the paper had been rejected due to an editorial decision. I was stumped – the paper had been assessed prior to peer review, which was positive.

A strong editor with a fair and consultative approach can cut through differing opinions (/egos) and direct an author to problem areas. A strong editor can make a paper. But editors who disregard the advice of reviewers that they have personally assigned, is this common?

In one instance, the editor was a scientist within the discipline serving a term on a journal’s editorial board in an unpaid role or possibly minimally compensated for time. In the other, the editor was in a paid position, dedicate to that role, with a strong scientific background but was not a scientist and certainly not an active expert in the field.

What’s the point of reviewers? At times, it seems like we are gratefully exploited rubber stamps. In these two instances, I voluntarily dedicated time to lucrative businesses. In the first case, the results are now considered to have met some minimum standard of quality, having undergone peer review. In the second case, a decision was made about the quality and novelty of a study that was vastly different from opinions sourced from within the scientific community.

At the moment, my capacity to review papers is diminished due to an increasing amount of time spend in service as an editor of a journal*. In this sense my frustrations are convenient, as I already have a solid basis for rejecting review assignments.

Regardless, I would have insurmountable qualms about accepting further assignments from either of these journals, or others within the same publishing houses. While not all experiences of reviewing or being reviewed reflect these two particular instances, messy review situations are commonplace.

At my most cynical, I could view a list of review assignments on my CV as an acknowledgement that I am willing to squander my taxpayer salary in service to a company with a record of thinking my opinion should be disregard.

Less cynically, reviewing is an ambiguous contract between profit making, editors, authors and the community. Reviewers are not provided with clear expectations from the editor, or with information about how their recommendations will be used. There is no way for the authors or readers to cut through this process and understand how publishing decisions are made.

Until such processes are brought into the open*, I’m saying goodbye to the weekly requests for enslavement to the review process.

* Insofar as impact factor can measure the quality of a journal

* Which does not have page or subscription charges and is published with no budget through the community’s sense of heart feels, rainbows, puppies and skittles

* Or until I decide otherwise out of guilt for expecting others to review my work

Advertisements

14 thoughts on “Farewell peer reviewing

  1. Really, I think it is stories like these that really show why the Earth sciences should adopt the approach taken by physics, mathematics and computer science, and have a completely open submission site like ArXiv. From there, the strong papers could rise to the top with an open editorial process, and the ability to make changes to the study. We live in the Internet era, there is no need for physical journals anymore, and no need to have static papers either.

  2. Peer review is meant to prevent the dross from getting any scientific kudos and also to improve papers that have something significant to say by taking account of the reviewers remarks. I was a reviewer for a well-known journal in my field and was told by the editor that if the paper was c**p just to say so and not waste any time reviewing it. I suppose I was taking the role the editor would normally assume in in giving every submitted paper a quick reality check. I knew the editor well so this tells me the editors were already overworked 20 years ago.

    Open review fulfils many of the same duties but can lead to an increased workload for the editors and authors due to the number of possible comments, all of which have to be addressed.

    IMO a bigger problem is the rise of the ‘paid for’ papers which undergo little or no review before publication in a journal based on profit rather than scientific integrity and progress.

    Having said that if you are getting weekly requests for reviews then that is far too much. One a month should be the maximum in order to give each submission sufficient attention. And as a journal editor you should not be expected to do more than 4 a year. I always found it possible to turn down a review with the excuse I was too busy to meet the deadline. It is a question of giving back something to the scientific community especially if you expect them to do the same and review your papers.

    I hope you reconsider your ‘farerwell to peer reviewing’ but you must decide what is a reasonable workload and don’t be afraid to say no!

    • Thanks for your thoughts. I’m sure I will reconsider. I guess I no longer see peer reviewing (in many circumstances) as “giving back something to the scientific community” but giving my time to a business. Also, reviewing one paper a month would equate to about a day of work a month, which is ostensibly the service commitment of a junior researcher. How much is giving back?

  3. Excellent post! I’ve resolved to submit my work to EGU journals whenever possible, as they are by far the most progressive journals in the weather/ocean/climate sciences (open access, open review etc).

  4. Great post! and you are not alone, I just recently have had almost the exact same experience as a reviewer: bad manuscript published without changes, good manuscript rejected without a good reason (editorial policy). As an author… the usual, manuscripts sent back because of small details. Students destroyed because their manuscript was rejected based on a poor review, etc. I think I agree that peer review is fine in principle , and that open peer review is probably a good option. But I would go further than that and require a complete audit-trail for a scientific project (from study design to scientific communication). I mean this is nowadays technically possible, however we still aim at publishing manuscripts keeping the process of producing them hidden from the public.

  5. I think that the peer review system basically works. And like everything that works in the real world, it has defects and you can find anecdotal evidence of major flaws. Bad reviews are common, most often in conferences where TPC chairs and reviewers are asked for quick turnaround, much less often in journal, especially serious journals. Sometimes I have had my reviews essentially ignored, but those were rare events. And sometimes I have received inconsistent reviews, but the editors corrected the problem by asking for more reviews or judging themselves. Most of the time, journal reviews helped me as an author to improve my paper, sometimes helped me really much.

    And yes, one review per week is really too much to do, I accept reviews when I have time, I do maybe ten reviews a year. And I find it a great way to give back to the community (yes, I think that’s true), to get a first-hand idea of what my peers do and the quality you can expect on average and, when I am lucky, to read up-to-date quality research in my field of expertise.

  6. Thanks for sharing your experiences! I myself have had good experience with journals. This includes also rejections of my own submissions, done directly by the editor, including comments on the major weeknesses – fair enough. It also includes fair treatment as a reviewer for journals. I am not sure if paid or unpaid work makes all the difference, but skills and routines as editor sure does. And yes, this is important, and really a soft spot in the quality ensurement system of the academic community!
    What about mentors for editors?

  7. I learned the editorial process in different roles as author,of course, reviewer, member of an editorial board and as editor. I saw, and I see, the weaknesses of the current review process. Do we, however, have a good alternative? If not, we have to continue with the existing system trying to improve it as much as possible.

  8. My guess is that the second paper you mention may have had an ethics issue. The journal may not have known this until the revised paper came in. It could have been declined for this reason but due to the confidential nature, the reviewers were not told the entire story. Thank you for writing this post. I manage peer review and I appreciate seeing where the pain point are for reviewers so we can make it better.

  9. Hi Sophie, Nice post. I’ve been on the road for ten years more than you and I’m still struggling with peer-reviewer issues (who isn’t?). Specifically about your “farewell”, on the one hand, if I accepted every editor’s request, I wouldn’t do anything else but reviewing. On the other hand, i don’t think it’s fair completely stopping reviewing (maybe out of guilt, as you wrote). In my own blog, I propose a set of fair rules to tell how many papers I should review. You may check that at https://mariobarbatti.wordpress.com/2016/08/22/how-many-papers-should-i-review/.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s