Peer Review

Peer review in science is a good thing, in the sense that it works better than any other process you can think of. But how well does it work? Dilbert Blog reader Jeff points to this link about the limits of peer review.

http://www.jr2.ox.ac.uk/bandolier/band88/b88-4.html

The article fits my preconceived notions quite well. Assuming scientists are human beings, it seems to me that most peer reviewers would fall into one of these categories:

1. Asshole
2. Biased egomaniac
3. Nice person who doesn’t want to make people feel bad
4. Too busy to put any quality thought into it
5. Person with low self-esteem who doesn’t want others to succeed in his or her field
6. Coward who doesn’t want to rock the boat

I suppose some scientists have plenty of free time, no biases, and would be happy to see colleagues succeed beyond their own careers. But seriously, how many of those scientists could there be? I don’t know any non-scientists who could fit that description.

Still, I assume peer review works well enough for killing the worst ideas. I don’t have a better idea for evaluating science. It’s just important to keep things in perspective.

[Update: Here's a link to an even better article on peer review: http://www.independent.org/newsroom/article.asp?id=1963 -- Scott]

[Update 2: Reader Winston points out this article debunking the notion that scientists ever generally thought an ice age was imminent: http://www.wmconnolley.org.uk/sci/iceage/ -- Scott]

Comments

[ma546zda]

c24t

[Melly]

Looking for information and found it at this great site…9

[otuyi]

Very good web site, great work and thank you for your service.+p

[Bob]

this one is simple & nice.

[Bob]

Great tutorial.2

[Pat]

The original post is concise and insightful - there are many caveats in reply for enjoyable reading. I am a scientist and editor - the original post resonated strongly.

My two cents:

Peer review is a lovely ideal that is unachievable. There are people without peer. Most of the best ideas violate what people are comfortable accepting, therefore, many nobel laureates frequently have their papers rejected the first few times.

The trouble is in implementation. Everyone is too busy, everyone has different priorities for the journal, and the editors rarely edit but just tally reviewer scores with a quick glance.

We really need feeback in the system - reviewers and decisiosn are given without author rebuttal. Isn't it far better if the paper is published with the reviews and rebuttal added to the end?? This way the negative comments are on the record, and hte reviewer has to be kept honest.

I have no idea how we got onto that submitting something for blind review was a good idea. Case in point, the proof of Fermat's Last Theorem was oriignally riddled with errors, but the reviewers and author got together to fix them and the paper was great. Usually, if there is any one disagreeable thing the whole thing is thrown out and the author can't resubmit (or if he can, it takes forever).

[Publish on a bloglike forum and the comments can take care of some of that. -- Scott]

[Dixon]

The problem with peer review is that sometimes the truely brilliant are peerless. Case in point: Albert Einstien was unable to get a job teaching high school physics because the had an equation on his resume that noone understood. It took over 10 years for the physics community to accept that Newton didn't have it 100% right and that there was something to E=MC^2.

[Monica]

My biochemistry professor has a theory that scientists are pure (not personally biased, and impervious to manipulation), so peer review would work. Otherwise... I'd be really worried, because peer review is a HUGE part of scientific research.
I guess the theory doesn't apply to the scientists that helped tobacco companies say that smoking was good for you.

[Wendy]

I happen to know plenty of scientists who take their time to give an unbiased response

[ceolaf]

Scott,

You write, "Assuming scientists are human beings, it seems to me that most peer reviewers would fall into one of these categories..."

You've made a mistake there. You actually are assuming that scientists are representative of the entire human population. Well, you ACTUALLY are assuming that REVIEWERS are representative of the entire human population. And you have no good reason to assume that.

Reviewers are all volunteers. Reviews have to have some credentials of their own before they are accepted as reviewers.

Sure, some of them are assholes, but the major league assholes and those that really are that busy don't volunteer. Nice people who don't want to make others feel bad don't volunteer because most submissions have to be rejected.

Reviewers, actually, tend to try to be very helpful. Often too "helpful." Yes. biased egomaniacs are too common. People who don't want others to succeed tend not to be invited as reviewers, or their efforts to volunteer are rejected.

Yes, there are many cowards who don't want to rock the boat.

Reviewers are not the entire population of scientists or academics, nor are the a random subset of that population. There are a particular subset. And therefore, though you might be right about people in general, you are not right about this particular group.

[ceolaf]

There are two big issues in the articles you cite:

1) Reviewers tend not to agree.
2) Editors have incredible amounts of power when they assign the reviews.

The latter is clearly a big issue. Politics is huge in academia, no question. Luckily, there are multiple journals, there are conferences and there is the possibility of publishing books. Thus, while the editor of a tier journal has lots of power in his/her field, there are ways to get around him/her.

Former is not necessarily a problem. It depends on what your goal is. If you goals is to only publish the best work, it's ok. Of course, if your goal is to publish all the best work, it's a problem. Reviewers can disagree for many reasons, each feeling that different parts of of an article are more important. That's a good thing! That is, the article only gets published if Prof. M AND Prof. N's priorities are met.

Obviously, a problem in this system is that some some good work is not published. But there is such a surfeit of good work out there, the journals don't suffer. It makes it very hard to challenge conventional wisdom in a field, of course. But that's part of the strength of peer review, actually. Changes in conventional wisdom are hard to do, and therefore much really be well based on something. It's not whimsical.

[ceolaf]

There are two big issues in the articles you cite:

1) Reviewers tend not to agree.
2) Editors have incredible amounts of power when they assign the reviews.

The latter is clearly a big issue. Politics is huge in academia, no question. Luckily, there are multiple journals, there are conferences and there is the possibility of publishing books. Thus, while the editor of a tier journal has lots of power in his/her field, there are ways to get around him/her.

Former is not necessarily a problem. It depends on what your goal is. If you goals is to only publish the best work, it's ok. Of course, if your goal is to publish all the best work, it's a problem. Reviewers can disagree for many reasons, each feeling that different parts of of an article are more important. That's a good thing! That is, the article only gets published if Prof. M AND Prof. N's priorities are met.

Obviously, a problem in this system is that some some good work is not published. But there is such a surfeit of good work out there, the journals don't suffer. It makes it very hard to challenge conventional wisdom in a field, of course. But that's part of the strength of peer review, actually. Changes in conventional wisdom are hard to do, and therefore much really be well based on something. It's not whimsical.

[olie]

Is "scientific peer-review" really only open to some bunch of scientists at some convention who might fall into the categories you describe? I'm asking sincerely? If they used the "open source" model, and let ANYBODY look at it, they'd get this:

* A lot of the sort of thing you describe.
* A lot of wacko crap that has to be ignored/tossed
* A handful of really smart college kids with way too much time on their hands and an interest in this area finding the holes in the theories.

It turns out that this system works really-really well. At least for plugging security holes in Linux, and similar highly-complicated problems.

(Please don't start a Linux flame-war -- it was an example, not an unqualified endorsement.)

[HopefulSkeptic]

I will have to respectfully disagree with your assessment that "most" peer reviewers fall into your categories. There is a difference between doing science and/or engineering, and publishing it. The process of publishing in peer-reviewed forums can be career-building, or intended to gather funding, and in those cases there is temptation to act without honor. But often publication is a function of establishing common baselines for collaboration that other researchers can build upon as a community. In those cases, there is less incentive to bend rules, and more reason to be accurate and precise, since others will indeed test what you have presented.

In addition, while we are not doctors with hippocratic oaths, or lawyers with bar associations to report to, many scientists and engineers do take similar oaths to live by an honor code. We are not just wage-earners, but professionals, and the professional societies to which many belong require that we follow Canons of Ethics, or risk losing our professional credentials. I personally take this code quite seriously, and keep a copy posted in my office. To bend the results of my research, or present it falsely, in order to reach some ends regarding publication, would not be living up to the code I have sworn to uphold. Neither would hindering the honest publications of others. And it would be a disservice to the public I must always protect, to knowingly permit publication of false or misleading data or findings.

The oath I swore was first published in 1929, but similar codes/canons have been around for over a century, and most claim to be inspired by Hammurabi (~1750BC). Mine reads as follows:
I am an Engineer. In my profession I take deep pride but without vainglory; to it I owe solemn obligations that I am eager to fulfill.
As an engineer I will participate in none but honest enterprise to him that has engaged my services, as employer or client, I will agree the utmost of performance and fidelity.
When needed, my skills and knowledge shall be given without reservation for the public good. From special capacity springs the obligation to use it well in the service of humanity; and I accept the challenge that this implies.
Jealous of the high repute of my calling, I will strive to protect true interests and the good name of any engineer that I know to be deserving; but I will not shrink should duty dictate, from disclosing the truth regarding anyone that, by unscrupulous act, has shown himself unworthy of the profession.
Since the age of the stone, human progress has been conditioned by the genius of my professional forbearers. By them have been rendered usable to mankind nature's vast resources of material and energy. By them have been vitalized and turned to practical account the principles of science and the revelations of technology. Except for this heritage of accumulated experience, my efforts would be feeble. I dedicate myself to the dissemination of engineering knowledge, and especially, to the instruction of younger members of my profession in all its arts and traditions.
To my fellows I pledge in the same full measure I ask of them in integrity and fair dealing tolerance and respect and devotion to the standards and the dignity of our professionals with the consciousness always that our special expertness carries with the obligation to serve humanity with complete sincerity.

[bob]

I'm a scientist. This is a really good summary of peer review.

Some things that got through: Homeopathy, The orginal cold fussion, 100% fake bukyball supercondutivity.... The list goes on.

In reality with reviewers its 80% from option 4 and 19% from option 2. Take your pick for the reaming 1%

[Scruffy Dan]

That second link debunking the validity of peer-review (i know thats an exaggeration) makes a large assumption, that science is motivated by its funding agencies (mainly government), and while i don;t deny that is the case, there is a lot of US government funded climate research that agrees with the IPCC reports, while the US government has been very hostile to the IPCC and anyone who agrees with them.

By the author's logic climate change most be a much worse disaster than the current consensus claims:

http://www.sciam.com/article.cfm?chanID=sa006&colID=5&articleID=5B9E73AD-E7F2-99DF-3F71280BCE41ED77

and all we get is the sugar coated (relatively) version that the IPCC and other agencies have been promoting.

While I do believe in climate change i still feel that peer-review is the best crap filter available, and while its not perfect is seems like the logical place for those uneducated on the subject (politicians) to place their bets

[Paul]

I have reviewed many scientific papers over the years, and I don't think I fall into any of those categories (with the possible exception of asshole). Many of us peer review manuscripts fairly and diligently because that's what makes the system work -- without it our scientific discipline would fail.

But I just wanted to clear up one misconception about the peer review process. It's not an attempt to determine the "truth" or otherwise of the paper, but a quality control check on the methodology, the accuracy (numerical errors and such), and that the conclusions or assertions are supported by logical reasoning from the evidence presented.

The veracity of the reported observations are taken on trust. Let me repeat that because it's important -- the data reported are accepted on trust. The scientific experiments are not checked by the peer reviewer. Thus, outright fraud is very easy to get through the peer review process. It is only caught later when other scientists try to repeat the published experiments. Peer review cannot guarantee that that the paper is factual, it can only (at best) ensure that the methodology is sound and the data presented are complete.

[Jason]

I work doing scientific research where I have to submit papers for peer review. Although in general your comments about the 6 types of people that can ruin the peer review process is true two things happen:

1) there are typically 3+ people plus the editor reviewing the article.
2) not all of them can be categorized in the 6 types

In the first point the editor makes sure that the 3+ people reviewing the material do not have a tendency to be in the 6 types. Additionally if the editor sees that one of the reviewers are totally inballenced they drop their review.

The second point is that the best reviewers are always the people in your field and you tend to get harder responses to critique your submission.

Once the responses are addressed you then resubmit and the process is repeated until all of the concerns are addressed. Typically a much better submission is achieved with clearer goals and refined subjects.

If this fails, you typically have the choice to repeat the process in a different peer reviewed journal and hope the same person does not review your article.

[Sarah]

You need another category for the scientists who are peer reviewers. The grad student that is trying to impress their boss that is any of the 6 above.

[Patrick]

Who reviews comments before they get posted here, and which category does that person fit in? :-)

Apparently, Albert Einstein is quoted to have replied to a negative outcome of a review:

"I sent it to be published not to be criticised."

[MaxHedrm]

gr8hands:

I might have to disagree with you about economics. By many of your points, meteorology isn't a science. Neither is the study of earthquakes. Or the human brain.

The problem with predicting, for instance, stock market behavior, is the number of variables, not necessarily the methods used. Much like the weather.

[Noah Vaile]

First of all I was alive, well and paying attention in the seventies. I remember "scientists" and scientific opinion of the time and U recall there being a definite leaning towards "ice age on Thursday next" in the same publications that now hold for instant incineration in a week and a half. The mere fact that "scientists" favor one theory or another supplies no proof for any of them.
If we think the earth is getting colder with imminent ice age we should be consulting with Eskimoes for the best ways to cope. If it is heating up we need to talk with Tongans, et al. If it is staying the same we should just gabble among ourselves, whereever and whoever we are. Enough of predicting things we can't predict from inadequate data we have to make up. We should all be studying igloo making and air conditioner maintenance.
oh yeah! Peer Review?? What scientist believes he really has a peer who can properly review his work?

[Minister of Silly People in Green]

Question everything, except your core beliefs.

Great articles.

[Webar]

All human systems are inherently flawed, but thankfully they usually improve with time.

The mistake that we make is assuming that because today's scientist has access to better computers and laboratories, today's science is completely valid. This we believe, despite all personal experience to the contrary.

We believe that scientists can tell us the right foods to eat to make us healthy, and we believe that scientists can tell us what cars to drive to make the environment more stable and sustainable. We believe it, even though most of us have lived through the 'eggs are good', 'eggs are bad', 'eggs are good' cycle of advice at least twice. Our grandparents remember when automobiles were the SOLUTION for environmental pollution because they didn't leave feces all over the pubic streets.

It is arrogant and naive to imagine that we know anything for certain.

[Silvox]

If s replace by a in this post, its a totally different angle.