Much is made these days (over and over and over!) about the supposed horrors of peer review: It is said to be inefficient, biased, and to fail to serve the role that it was intended to serve, of producing a high quality archival literature. [The latest litany that I have come across appeared in the premire issue of the new Journal of Participatory Medicine (http://jopm.org/index.php/jpm), specifically by Richard W. Smith (http://jopm.org/index.php/jpm/article/view/12/25).] Although I think that everyone would agree that the peer review system could be improved, esp. in terms of its speed, I feel that these discussions miss what seems to me to be the most important value of peer review: Its role in the education of scientists, and specifically, of highly efficient, precisely targeted, and secure narrow band communication among scientists. Having received hundreds of peer reviews of my work, and certainly my fair share of negative reviews, I can only think of one review that I!
received that was not useful to me in improving my paper, and more importantly, in teaching me something important and central to my field -- usually many things! We say things in our manuscripts, and in our reviews of others' manuscripts, things that often ought probably not reach the light of day. Reviewer's catch these for us, and teach us why these are not the right things to say. On the other side, we say things in the reviews we write that we would never publish, because we are speaking to particular scientists (the authors of the paper), basically teaching them what we think they ought to know. Nowhere else, once scientists leave their final post doc, is there a similar opportunity for direct continuing eduction. Peer review is only in part a filtering system -- and to my mind that is a relatively small part of it value. It is, in addition and more importantly, a highly efficient and secure system of targeted peer education. To ignore this function in changing the way the peer review works is, to my mind, to endanger one of the pillars of scientific communication.
Jeff Shrager

Dear Jeff,

Wow, I couldn't disagree more. You're entire point of view is predicated on the assumption that the reviewer always knows more and better than the person or people writing the paper that is being reviewed. Did it ever occur to you that the reviewers might be in a position to learn from the writers of the paper. Did it ever occur to you that the purpose of peer review in socializing scientists also works against innovation by imposing paradigms on their work that may not be appropriate?

Sorry, but I've got files full of reviews that have taught me nothing except that some people aren't very smart, don't read very carefully, or are simply immune to any new idea.  One reviewer rejected a paper of mine with the following critique: "I simply don't believe it." That was the entire review and the Editor accepted it. Another told us that we couldn't do the experiments we had done -- the technique was impossible. I had to remind the journal editor that his journal had published the first paper describing the technique, and that he had published half-a-dozen articles in his journal alone using that method previously before he was willing to overrule the reviewer.   In my most recent exchange, I had a paper sent back for major revisions. I made every change requested except three. When I sent a rebuttal letter pointing out that two of the changes were contradictions of each other and another was due to a misunderstanding of the experiment (the methods now rewritten to prevent any further ambiguity), the reveiwers threw fits and rejected the manuscript outright because I had challenged their authority.  I was literally told by the editor of the journal that it was not my place to challenge the reviewers: they were the experts and I was to kowtow whether they were right or wrong.  You'll have to excuse me if I place the integrity of the science above the social needs of those who think that it is their prerogative as reviewers to automatically be in a position of authority.  How am I supposed to make changes to a manuscript that introduce a contradiction and another that will introduce an outright error? 




Let me also say that peer review has often worked very well for me, and improved some of my papers quite a bit. Here's the crux of the matter from my perspective: Peer review works great when you work within a paradigm and give reviewers what they expect to see using methods they are famiiliar with. Then reveiwers are very good about determining how well you've fulfilled expectation. Peer review stinks when reviewers are challenged with ideas that might undermine their own cherished beliefs, methods with which they are not famiilar, conclusions that contradict those of the field, etc.  Peer review promotes what Kuhn called "normal" science; it retards (viciously) revolutionary or innovative science.  THere's plenty of evidence for this from studies of responses to the emergence of new scientists and paradigm shifting work in science.....

Bob Root-Bernstein

Hi, Bob. Thanks for the thoughtful notes. We do seem to disagree, and the truth probably varies, and is clearly an empirical questions, but let me reply to some of your remarks:

> You're entire point of view is
> predicated on the assumption that the reviewer always knows more and
> better than the person or people writing the paper that is being
> reviewed.

No, this actually has nothing to do with my point. The communication can, and is, in my experience, in both directions.

> Did it ever occur to you that the reviewers might be in a
> position to learn from the writers of the paper.

Yes, in fact, I intended to say this, but it got lost in editing. I agree completely that the communication is two-way, but this only serves to strengthen, not weaken, my claim that peer review is an important channel of communication and eduction. I didn't mean to imply that the authors are the only ones being educated.

> Did it ever occur to
> you that the purpose of peer review in socializing scientists also
> works against innovation by imposing paradigms on their work that may
> not be appropriate?

I wouldn't call it "socializing", but I agree that there may be a small smoothing effect here. However, in my experience this is very small. Of course, my experience doesn't really count, so this would seem to be an interesting empirical question. However, the strong version of this logic leads one to the claim that the function of all education and communication is to impose paradigms, and we probably don't want to end up in that particular nest of snakes.

> Sorry, but I've got files full of reviews that have taught me nothing
> except that some people aren't very smart, don't read very carefully,
> or are simply immune to any new idea. One reviewer rejected a paper
> of mine with the following critique: "I simply don't believe it." That
> was the entire review and the Editor accepted it.

You always have the right to request another review. The one really stupid review I had was similar. (Actually, it was a different sentence: "The author must read <my book, including ISBN#!!>" I asked for, and received a new review.

> Another told us that
> we couldn't do the experiments we had done -- the technique was
> impossible. I had to remind the journal editor that his journal had
> published the first paper describing the technique, and that he
> had published half-a-dozen articles in his journal alone using that
> method previously before he was willing to overrule the reviewer.

Good, see, the process does work!

> In
> my most recent exchange, I had a paper sent back for major revisions.
> I made every change requested except three. When I sent a rebuttal
> letter pointing out that two of the changes were contradictions of
> each other and another was due to a misunderstanding of the experiment
> (the methods now rewritten to prevent any further ambiguity), the
> reveiwers threw fits and rejected the manuscript outright because I
> had challenged their authority. I was literally told by the editor of
> the journal that it was not my place to challenge the reviewers: they
> were the experts and I was to kowtow whether they were right or
> wrong.

I've never seen anything like this behavior. Maybe there should be a peer review ethics board for cases like this.

> You'll have to excuse me if I place the integrity of the
> science above the social needs of those who think that it is their
> prerogative as reviewers to automatically be in a position of
> authority. How am I supposed to make changes to a manuscript that
> introduce a contradiction and another that will introduce an outright
> error?

Sounds like you've had more than your fair share of poor experiences with peer review. I can't say that same. Clearly there is a range of experiences here, and interesting empricial questions. I contend that my claim is at least worthy of empirical consideration.

> Let me also say that peer review has often worked very well for me,
> and improved some of my papers quite a bit. Here's the crux of the
> matter from my perspective: Peer review works great when you work
> within a paradigm and give reviewers what they expect to see using
> methods they are famiiliar with. Then reveiwers are very good about
> determining how well you've fulfilled expectation. Peer review stinks
> when reviewers are challenged with ideas that might undermine their
> own cherished beliefs, methods with which they are not famiilar,
> conclusions that contradict those of the field, etc. Peer review
> promotes what Kuhn called "normal" science; it retards (viciously)
> revolutionary or innovative science. THere's plenty of evidence for
> this from studies of responses to the emergence of new scientists and
> paradigm shifting work in science.....

Even if one has a radical new way of thinking that isn't appreciated by the expert reviewers, at the very least the reviews give one a preview of how "the dominant paradigm" will push back on one's attempt to introduce the paradigm. This is probably extremely useful information, and it is at least plausible that it improves the efficiency of science.

Cheers,
'Jeff