Scientific Peer Review – Is it a corrupted process?

Peer review is the process by which the sciences and other professions subject the work of members of those professions to scholarly review (http://en.wikipedia.org/wiki/Peer_review). The purpose for this process is to determine the quality and scholarly depth of the researchers work. Other researchers in the person’s area of expertise usually conduct it. In the more modern scientific era where work is published in journals the process is known as refereeing.


Why have peer review?

Ostensively, the peer review process is to ensure that only work of quality makes it through to publication. It has also been used to help to improve on a workers methodology and logical arguments. Peer review is regarded as the most reliable method for the determination of the validity of the work being examined.

So. Is peer review the ultimate determiner of quality science?

Well. No. Horton (http://www.mja.com.au/public/issues/172_04_210200/horton/horton.html) states quite convincingly:

“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”

His is not a lone voice in the wilderness. There was an editorial in Nature in 1999 (http://www.nature.com/nature/journal/v401/n6755/full/401727a0.html) that concluded that an over-reliance on peer-reviewed publication:

“has disadvantages that should be countered by adequate provision of time and resources for independent assessment and, in the midst of controversies, publicly funded agencies providing comprehensive, reliable and prompt complementary information”

Robert Higgs (http://hnn.us/blogs/entries/38532.html) comments that:

“Peer review, on which lay people place great weight, varies from important, where the editors and the referees are competent and responsible, to a complete farce, where they are not. As a rule, not surprisingly, the process operates somewhere in the middle, being more than a joke but less than the nearly flawless system of Olympian scrutiny that outsiders imagine it to be.”

Can peer review be abused?

Definitely YES. Examples abound where the peer review process has been used to prevent good research from being published. It has also been used to prevent research that contravenes the establishment consensus from being published in certain journals. This has happened in AIDS research, medical and dental research, climate research and many other disciplinary areas. See: http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/ss/ss5.html for a number of cases of the misuse of the peer review process to quash research results.

Why bring this up on climate science blog?

I have introduced this issue to bring to reader’s attention that the assumption of peer reviewed research being true and accurate is not necessarily the case. I have encountered many comments on climate blogs (especially on Tamino’s site) that call to the fact that because papers are peer reviewed they are peerless. That is not true since these same sites denigrate peer-reviewed papers by non-AGW authors as if they were garbage. Peer review never has and never will guarantee that research is both good and correct. See Horton’s quote above. Any argument by either side of the debate that because a paper is peer reviewed and published in a prestigious journal it is beyond reproach is both wrong and unsustainable.

Can the process be improved?

Definitely. There are a number of possible strategies. I think that the only ones that will work are those that remove the process from the publishing journal. Peer review if it is to continue to be a useful process must be devolved back to the institution of the researcher. Some would argue that this would guarantee publication of poor quality research. That argument is not proven nor does it acknowledge the fact that institutions have greater expertise to review research. Most institutions will not allow research to be published in their name until it has gone through a thorough internal review. And they are not keen on letting dodgy research be published.

Perhaps a thorough audit process or review of the research by committees of disciplinary bodies is a viable alternative to peer review as it is practiced today. In fact this process is how peer review is really supposed to work.

Another possibility is to do away with peer review altogether. This has been done in astronomy and other disciplines to a lesser or greater degree. Leave it up to the scientists themselves to winnow out the frass.

Just some thoughts on the matter.

4 Responses to Scientific Peer Review – Is it a corrupted process?

  1. futureofscipub says:

    i agree with this view of yours: “There are a number of possible strategies. I think that the only ones that will work are those that remove the process from the publishing journal. Peer review if it is to continue to be a useful process must be devolved back to the institution of the researcher.”

    what we need is a general open post-publication peer review system with
    - any scientist allowed to review any paper
    - public availability of all reviews along with each published paper
    - reviews as open letters to the community
    - signed and unsigned reviews
    - digital author authentication
    - numerical judgments on multiple scales allowing for automatic paper selection by means of arbitrary ranking functions

    i’m exploring these ideas here:
    http://futureofscipub.wordpress.com/

    it would be great to hear your thoughts.

    –nikolaus kriegeskorte

    • saltator says:

      Thanks for your contribution on this post. Peer review is certainly an area that needs urgent attention.

      I certainly agree with most of what you are saying. In addition, it is my experience looking at climate science blogs that people argue on the level of statistical validity of their results. This means that review should be data audited, not just passed to some peer in the same area of study (who probably has no greater knowledge of statistical procedure than the person who wrote the paper) where it passes even though the analyses can be quite flawed (e.g. the Hockey Stick).

  2. Ian Forrester says:

    Richard Steckis you are broadcasting rubbish as usual. Had a paper turned down recently have you? I am not surprised

    • saltator says:

      Ian. I suggest you visit futureofscipub’s blog for more information that might enlighten you. And for your information, no paper I have been involved with has been refused publication after peer review.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: