Most scientific papers are probably wrong

[Guest guest]

http://www.newscientist.com/article.ns?id=dn7915

* 02:00 30 August 2005

* NewScientist.com news service

* Kurt Kleiner

Most published scientific research papers are wrong, according to a

new analysis. Assuming that the new paper is itself correct, problems

with experimental and statistical methods mean that there is less than

a 50% chance that the results of any randomly chosen scientific paper

are true.

Ioannidis, an epidemiologist at the University of Ioannina School

of Medicine in Greece, says that small sample sizes, poor study

design, researcher bias, and selective reporting and other problems

combine to make most research findings false. But even large,

well-designed studies are not always right, meaning that scientists

and the public have to be wary of reported findings.

" We should accept that most research findings will be refuted. Some

will be replicated and validated. The replication process is more

important than the first discovery, " Ioannidis says.

In the paper, Ioannidis does not show that any particular findings are

false. Instead, he shows statistically how the many obstacles to

getting research findings right combine to make most published

research wrong.

Massaged conclusions

Traditionally a study is said to be " statistically significant " if the

odds are only 1 in 20 that the result could be pure chance. But in a

complicated field where there are many potential hypotheses to sift

through - such as whether a particular gene influences a particular

disease - it is easy to reach false conclusions using this standard.

If you test 20 false hypotheses, one of them is likely to show up as

true, on average.

Odds get even worse for studies that are too small, studies that find

small effects (for example, a drug that works for only 10% of

patients), or studies where the protocol and endpoints are poorly

defined, allowing researchers to massage their conclusions after the

fact.

Surprisingly, Ioannidis says another predictor of false findings is if

a field is " hot " , with many teams feeling pressure to beat the others

to statistically significant findings.

But Snyder, senior editor at the Proceedings of the National

Academy of Sciences, and a neuroscientist at s Hopkins Medical

School in Baltimore, US, says most working scientists understand the

limitations of published research.

" When I read the literature, I'm not reading it to find proof like a

textbook. I'm reading to get ideas. So even if something is wrong with

the paper, if they have the kernel of a novel idea, that's something

to think about, " he says.

Journal reference: Public Library of Science Medicine (DOI:

10.1371/journal.pmed.0020124)

######

--

" Twenty years ago I was an extreme right-wing

Republican, a young and lone 'Neanderthal'

(as the liberals used to call us) who believed,

as one friend pungently put it, that 'Senator

Taft had sold out to the socialists. Today,

I am most likely to be called an extreme leftist,

since I favor immediate withdrawal from

Vietnam, denounce U.S. imperialism, advocate

Black Power and have just joined the new

Peace and Freedom Party. And yet my basic

political views have not changed by a single

iota in these two decades! "

Murray Rothbard