Explain Peer Review to Me?

Question

I have no doubt that a group of Chiropractors would review information from their peer and recommend a spinal readjustment over surgery.

I have no doubt that a paper discussing changing water into wine would pass peer review standards of Theologians.

Would ANYONE think that a dissertation that global warming was man made and real WOULDN'T pass peer review of Climatologist? I would be shocked if one was rejected!

Remember - Mann's hockey stick and Hansen's Y2K problem got by their peers. Maybe they were scared to question people of their stature. Their errors were discovered and corrected by a mining engineer.

Science shouldn't be accepted until it's independently verified. Not accepted if your buddy agrees with you. (as long as you peer review his work first.)

Answer 1

Do you mean in science in general, or in climate science?

Oh wait, we’re in the Global Warming section, in that case…


PEER REVIEW, verb, THE ACT OF GETTING YOUR BESTEST BUDDIES TO PRONOUNCE YOUR WORK IS VALID, EVEN IF IT’S UTTER NONSENSE, IN AN EFFORT TO DUPE THE REST OF THE WORLD INTO BELIEVING IN THE MYTH OF ANTHROPOGENIC GLOBAL WARMING.

Please note tongue firmly in cheek :)

OK, on a more serious note, I agree with what Michael Crichton says on the matter…

“The fact is if we required the same standard of information from climate scientists that we do from drug companies, the whole debate on global warming would be long over. We wouldn’t be talking about it. We need mechanisms to insure a much, much higher standard of reliability in information in the future.”

See- http://www.crichton-official.com/speech-ourenvironmentalfuture.html

And…

“Sooner or later, we must form an independent research institute in this country. It must be funded by industry, by government, and by private philanthropy, both individuals and trusts. The money must be pooled, so that investigators do not know who is paying them. The institute must fund more than one team to do research in a particular area, and the verification of results will be a foregone requirement: teams will know their results will be checked by other groups. In many cases, those who decide how to gather the data will not gather it, and those who gather the data will not analyze it. If we were to address the land temperature records with such rigor, we would be well on our way to an understanding of exactly how much faith we can place in global warming, and therefore what seriousness we must address this.”

See- http://www.crichton-official.com/speech-alienscauseglobalwarming.html


I couldn’t agree more.

Answer 2

Peer review works (it caught mistakes like cold fusion and polywater), and is the best alternative. Who else could "independently verify" a paper on advanced quantum mechanics but a physicist schooled in the area? Steven Milloy?

Your examples are wrong. Mann's hockey stick was basically validated by the National Academy of Sciences who said his conclusions ere right, even though the curve was overly "smoothed" (averaged). Since then it's been duplicated many times by other scientists, with more precise statistics. Here are ten examples:

http://www.globalwarmingart.com/wiki/Image:2000_Year_Temperature_Comparison_png

I don't know why you say the Y2K problem was Hansen's or what peer reviewed article he wrote about it. There was an extremely minor error in the NASA temperature data, which affected the US data by a few hundreths of a degree. It changes absolutely nothing about global warming. This graph changed by 1/1000 of a degree. You can't see that small of a change, of course.

http://www.giss.nasa.gov/research/news/20070208/2006_temp_anom.gif

Answer 3

Peer review of scientific papers is a mechanism for checking them for errors. A paper is submitted to a scientific journal by the authors. The paper is then assigned to one of the journal's editors. The editor sends the paper out to one or more scientists working in the field for review. The reviewer is supposed to make judgements about several points:
1) Does the paper have sufficient substance for publication? That is, does it contain genuinely new material?
2) If the paper describes an experiment or method, is the description sufficiently detailed that another scientist working on that subject could duplicate the results?
3) If the paper describes data, is the method for collecting that data correct? Are the potential problems or errors described?
4) Are there flaws in the analysis? Have the authors made errors in reasoning or computation?
5) Are the conclusions warranted by the substance of the paper: do they overreach, or are they perhaps overly cautious?

The reviewer writes a report, touching upon these issues. Those reports are read and judged by the editor, who then communicates with the authors. The paper may be rejected, conditionally accepted with required changes, accepted with suggested changes, or (rarely) simply accepted.

If the paper is not accepted, the authors may revise the paper and re-submit. The process then repeats. The editor usually sends the revised paper back to the same referee(s), to speed up the process. Sometimes, the referees are unreasonable, and the authors can request that the editor assign new refrees. In any case, the paper is not published until the referees are satisfied that it is "correct".

Note that the referee is not expected to duplicate the results, or rule on the correctness of the data. The peer review system does not guarantee against fraud. What it does is attempt to assure that published papers are falsifiable---there is enough information to reproduce the result, and the internal processes of the paper are without obvious error.

Of course this does not guarantee that the paper is really true. A look at scientific papers from 50 or 100 years ago shows that some of them are erroneous, and many are simply irrelevent to current scientific paradigms. There is progress, however, and a coherent scientific picture slowly emerges from the literature and is summarized in review papers and books.

Answer 4

Peer review relates mainly to publication in scientific journals. Magazines publishing scientific articles which are run by "laymen" are not well thought of. Physicians are the foremost example. They may have originated the concept. The purpose of peer review is NOT to assure that what gets published is true. The purpose is to make sure the established standards for research have been followed. These are basically the same across all branches of science, but each has a few conventions of its own.

For example, observing that something happened once is not considered a study. It can be called a case review, or case study, but not a study. Even if it is accurate, it proves only itself.

There are small studies and large studies. Small studies amount to the minimum amount of information that is significantly significant to support a conclusion. Thirty-five data points is the lower end as far as statistical significance (you'd need to ask a statistician why). That's pretty much universally accepted in all fields. The upper limit seems to vary, depending on the field of research, and the era of research involved. Math experts constantly analyze and try to improve the groundrules. When you get into thousands or millions of data points, you have a large study. What that does for you is increase the certainty of a conclusion (penicillin cures clap, for example). Within their definition, small studies normally conclude that something may be a promising avenue for further research, or it may not. They lack credibility if they are the only support of an idea.

Then there is the issue of controls, or a control group (data points that don't include the experimental factor). Small studies may lack controls altogether. Large studies are required to have them.

There is the issue of "blind" versus "double blind". When testing a new drug, do the test subjects know whether they are getting the drug or the placebo? Do the control subjects know? Do both know? This is supposed to be defined, where it applies.

There is the issue of proactive versus retroactive. Suppose a person looked at 2000 records of deaths for patients taking aspirin alone, who died of a heart attack. Suppose also that the person looked at 2000 records of patients taking aspirin and Viagra who died of heart attacks. That person could then make some statements about mortality due to heart attack for each group. To have a control group the person would need to look at 2000 patients not taking either drug who died of heart attacks. Then you'd have a large controlled study, but it would still be retroactive, as you're relying entirely on historic data you yourself did not measure or record.

A proactive version of the same study would be exactly the same, except all the subjects would be selected in advance, followed until they died, and monitored by blood testing for drug levels, etc. As you would expect, there are more retroactive and small studies out there than there are proactive.

Then there is the issue of measurement error. Every method or instrument used to measure something has a factor of error that is either known or can be measured. Two seemingly identical pH meters may differ slightly as to results. A record of testing known samples for each allows the difference (error) to be quantified. The reason this is important is that when a number like this is plugged into an equation, the error is going to be multiplied, divided, adde or whatever, depending on where it is used in the formula. When several numbers like this are involved, the errors from each can accumulate and be magnified to the point where the conclusion is rendered meaningless, unless the measurement errors are managed effectively. The same can be said of the issue of "rounding up", or "rounding down" the numbers with digits to the right of the decimal place.

So the purpose of peer review is not to rule on the truth or falsehood of the conclusion, but merely to certify that the conclusion was reached in a reasonable and acceptable way.

Proof is an issue that begins after publication, The first step is whether the data can be reproduced by another investigator. Another is whether it can be reached using a different method of measurement. A third is whether the conclusion can be reproduced using the formula when all the other variables are know.

I hope this provides some clarification on the issue of peer review.

Answer 5

I think there is going to be a "Peer Review" button right next to spell check and grammar check on the next Microsoft Word. It will make a paper no less legitimate than the current process of peer review.

Interesting you brought up Mann's hockey stick. While not exactly peer review, it certainly brings up questions as to what is independent verification. This is the "INDEPENDENT" group of researchers that "validated" Mann Bradley & Hughes' '98 hockey stick:

Bradley and Jones [1993]
Hughes and Diaz [1994]
Mann, Bradley and Hughes [1998, 1999]
Jones, Briffa and others [1998]
Briffa [2000]
Briffa, Jones and others [2001]
Mann and Jones [2003]
Bradley, Hughes and Diaz [2003]
Jones and Mann [2004]
Rutherford, Mann, Bradley, Hughes, Briffa, Jones
and Osborn [2005]
http://www.marshall.org/pdf/materials/316.pdf

Looks like "consensus" and "collusion" are interchangeable concepts.

Answer 6

Bob, I'm not sure what you mean by "an extremely minor error in the NASA temperature data, which affected the US data by a few hundreths of a degree." NASA themselves say it is .15 degree C. I guess that could be a few hundredths of a degree. Considering the US has warmed .7 degree C according to global warming alarmists, that would mean that the claimed temperature change is off by 21%. Extremely minor? Hardly.

Answer 7

Do you need someone to tell you when to take a dump? Do you have children ? Are you prepared for a catastrophe as seen around the world ? Who cares about global warming and what happened in the past. Look around fool.

I am getting tired of tasting the air that I breath. And, I don't want to JUST move to greener pastures.