Peer review is the process by which the sciences and other professions subject the work of members of those professions to scholarly review (http://en.wikipedia.org/wiki/Peer_review). The purpose for this process is to determine the quality and scholarly depth of the researchers work. Other researchers in the person’s area of expertise usually conduct it. In the more modern scientific era where work is published in journals the process is known as refereeing.
Why have peer review?
Ostensively, the peer review process is to ensure that only work of quality makes it through to publication. It has also been used to help to improve on a workers methodology and logical arguments. Peer review is regarded as the most reliable method for the determination of the validity of the work being examined.
So. Is peer review the ultimate determiner of quality science?
Well. No. Horton (http://www.mja.com.au/public/issues/172_04_210200/horton/horton.html) states quite convincingly:
“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”
His is not a lone voice in the wilderness. There was an editorial in Nature in 1999 (http://www.nature.com/nature/journal/v401/n6755/full/401727a0.html) that concluded that an over-reliance on peer-reviewed publication:
“has disadvantages that should be countered by adequate provision of time and resources for independent assessment and, in the midst of controversies, publicly funded agencies providing comprehensive, reliable and prompt complementary information”
Robert Higgs (http://hnn.us/blogs/entries/38532.html) comments that:
“Peer review, on which lay people place great weight, varies from important, where the editors and the referees are competent and responsible, to a complete farce, where they are not. As a rule, not surprisingly, the process operates somewhere in the middle, being more than a joke but less than the nearly flawless system of Olympian scrutiny that outsiders imagine it to be.”
Can peer review be abused?
Definitely YES. Examples abound where the peer review process has been used to prevent good research from being published. It has also been used to prevent research that contravenes the establishment consensus from being published in certain journals. This has happened in AIDS research, medical and dental research, climate research and many other disciplinary areas. See: http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/ss/ss5.html for a number of cases of the misuse of the peer review process to quash research results.
Why bring this up on climate science blog?
I have introduced this issue to bring to reader’s attention that the assumption of peer reviewed research being true and accurate is not necessarily the case. I have encountered many comments on climate blogs (especially on Tamino’s site) that call to the fact that because papers are peer reviewed they are peerless. That is not true since these same sites denigrate peer-reviewed papers by non-AGW authors as if they were garbage. Peer review never has and never will guarantee that research is both good and correct. See Horton’s quote above. Any argument by either side of the debate that because a paper is peer reviewed and published in a prestigious journal it is beyond reproach is both wrong and unsustainable.
Can the process be improved?
Definitely. There are a number of possible strategies. I think that the only ones that will work are those that remove the process from the publishing journal. Peer review if it is to continue to be a useful process must be devolved back to the institution of the researcher. Some would argue that this would guarantee publication of poor quality research. That argument is not proven nor does it acknowledge the fact that institutions have greater expertise to review research. Most institutions will not allow research to be published in their name until it has gone through a thorough internal review. And they are not keen on letting dodgy research be published.
Perhaps a thorough audit process or review of the research by committees of disciplinary bodies is a viable alternative to peer review as it is practiced today. In fact this process is how peer review is really supposed to work.
Another possibility is to do away with peer review altogether. This has been done in astronomy and other disciplines to a lesser or greater degree. Leave it up to the scientists themselves to winnow out the frass.
Just some thoughts on the matter.