What’s the point of peer review?

The last month has really made me feel like my approach to reviewing journal papers is completely wrong. When I get a request to review a paper, I read the abstract, have a think about the content and my background, look at my diary and then respond to the editor as to whether I am in a position to review the manuscript adequately.  On the whole, I really enjoy reviewing articles.

And then when I do start the review process, I always begin with the expectation that I’m going to like the paper – it will be clearly written, use new and appropriate methods and have exciting conclusions. Of course this isn’t always the case, but most papers have something very much worth salvaging.

I have had a couple of papers snarled in the review process in recent months that have made me think that perhaps not everyone involved in peer-review starts from the same point as I do. And even, in some instances, that there is very little value in peer-review. And so, I begin my fight with peer review.

Bout 1:

The most recent paper I got back from review was a short-format paper submitted to a special issue of a good journal, requiring a strict word limit of 1500 words. Of course, when I submitted our contribution to the special issue, it inevitably came back with disparate reviews.

The first reviewer made a particularly valid comment that we had not been clear enough in discussing the limitations of our study. This was certainly worth expanding on in our revised paper.

Meanwhile, the second reviewer had many (many!) comments, of increasing strangeness.  Few of the comments seemed to even relate to our study. The reviewer was unfamiliar with the models, experiments and statistical tests we used, as well as the major climatic influences on Australia. The reviewer went onto ask for information and references that were already supplied, and to identify features in figures that no-one else could see.

I went about the necessary, but confusing, task of modifying our paper to accommodate these comments. With the strict word limit in place, it was a rather painstaking matter of ‘one word in, one word out’.

Bout 2:

My other recent battle with peer review raged over 10 months. Last year, I submitted a paper using multi-disciplinary approaches to a journal that would provide readership in both disciplinary communities. I naively thought this was a good thing to do. Oh, youth!

Since then, roughly every three months, I have periodically turned from a normal, decent human, into a raging, twitching, muttering mess.

The first round of reviews provided mixed opinions. Two reviewers with a common discipline liked the study and the approach used, but were a little ambivalent about the conclusions and thought the writing was unnecessarily muddled. The third reviewer did not like my analysis and thought there was little value in my approach. As a result, I revised the manuscript and produced a far clearer study as a result.

But the second round of reviews provided even more polarised opinions. The first two reviewers, from the previous round, were satisfied and indicated that the paper was ready for publication, after some minor tweaks. The third reviewer was a new participant in the bout. And he or she did not like my work.

This is quite a generous take on the comments provided – the reviewer really didn’t like my paper. He or she essentially suggested that I re-write my study to exclude all of my ideas or approaches and that it would be rather more useful to provide understanding solely from his/her discipline and citing only literature from this area.

And so, I re-wrote my paper. I ended up with something that would make Tolstoy proud. It was a meandering epic of textbook proportions. There were highs, there were lows, there was war, there was peace. And none of it made any sense. I re-wrote my paper again. This time, I attempted to distinguish between reviewer comments that were useful and those that were self-motivated.

I submitted my paper again, it went to review again. I submitted my paper again, it went to review again. And then when the reviewer maintained that my paper should be rejected but could no longer articulate why, the editor relented and accepted my paper.

And the winner is….

Eventually, both papers were accepted. Huzzah! But the combined experience does make me confused about the usefulness of peer review in its current configuration.

As for my first fight against peer review, after several days, I produced a revised manuscript that does not seem any clearer than the first iteration. The purpose of addressing such strange reviewer comments can hardly be to produce a better written manuscript, nor a stronger, more scientifically robust study.

As for my second fight again peer review, I still feel as though the paper has actually degraded in clarity since the first round of revision. And it certainly makes me wonder how anyone ever manages to publish anything novel, when we are compelled to rehash the work of our reviewers, instead of striking out with new approaches or interpretations.

Occasionally, excellent, skilled people with excellent, skilful ideas have reviewed my papers. They have taken time out of their tight schedules to consider my work. They have contributed thoughtful comments that have quite rightfully changed the way I have interpreted a dataset, or structured a manuscript, or they have provided invaluable insight into the appropriateness of certain statistical approach.

In these cases, they have enriched my work and my capacity as a young scientist, and I enjoy the process of improving my own work with the help of these generous souls.

But more commonly in my experience, some reviewers enjoy disparaging the work of others when it provides alternative ways of understanding issues or problems. Or perhaps they even gain satisfaction from labelling the work of others as woeful, reprehensible and of little value.

And so who won the battle? Clearly not me, and clearly not science. In one case, I have taken vast amounts of time produced a longer, more meandering, less precise manuscript that I dislike. In the other, I have spent days appeasing a (possibly marginally qualified) reviewer, with no gain in the quality of my work. The science is no stronger, nor clearer.

In some cases, peer review is simply a lazy way in which we come to self-assign authority to science, without any real investment of intellectual consideration.

Peer review shouldn’t be a fight, with author pitted against reviewer. It should be a process by which we can improve each others’ work and evaluate its usefulness to our community, not just to ourselves. Peer reviewers are not the gatekeepers of some exalted, unwritten scientific standards. Rather, reviewers provide a single opinion as to whether, and how, a piece of work is presented to the wider community for further, ongoing scrutiny.

In my experience, reviewing works best when we come from a starting point that we will enjoy each others’ work and that we are investing our time together for the improvement of our collective knowledge.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s