Fw: Something Rotten at the Core of Science?

[Guest guest]

From: Ilena Rose <ilena@...>

Sent: Friday, March 09, 2001 4:54 PM

Subject: Something Rotten at the Core of Science?

~~~ great article. especially with what we know about the " peer reviewed "

NEJM " studies " published by Junk$cientist Marcia Angell Medical Deity and

Gang and promoted via ... ACSH.org, junkscience.com, Fumento, Milloy,

$to$$el et al ...

ilena ~~~

OPINION

Something Rotten at the Core of Science?

by F. Horrobin

Reprinted with permission from Trends in Pharmacological Sciences, Vol.

22, No. 2, February 2001

Posted February 2, 2001 · Issue 95

-------------------------------------------------------------------------

-------

Abstract

A recent U.S. Supreme Court decision and an analysis of the peer review

system substantiate complaints about this fundamental aspect of scientific

research. Far from filtering out junk science, peer review may be blocking

the flow of innovation and corrupting public support of science.

-------------------------------------------------------------------------

The U.S. Supreme Court has recently been wrestling with the issues of the

acceptability and reliability of scientific evidence. In its judgement in

the case of Daubert v. Merrell Dow, the court attempted to set guidelines

for U.S. judges to follow when listening to scientific experts. Whether or

not findings had been published in a peer-reviewed journal provided one

important criterion. But in a key caveat, the court emphasized that peer

review might sometimes be flawed, and that therefore this criterion was not

unequivocal evidence of validity or otherwise. A recent analysis of peer

review adds to this controversy by identifying an alarming lack of

correlation between reviewers' recommendations.

The Supreme Court questioned the authority of peer review.

Many scientists and lawyers are unhappy about the admission by the top

legal authority in the United States that peer review might in some

circumstances be flawed [1].

Goodstein, writing in the Guide to the Federal Rules of Evidence -

one of whose functions is to interpret the judgement in the case of Daubert

- states that Peer review is one of the sacred pillars of the scientific

edifice [2]. In public, at least, almost all scientists would agree. Those

who disagree are almost always dismissed in pejorative terms such as

maverick, failure, and driven by bitterness.

Peer review is central to the organization of modern science. The

peer-review process for submitted manuscripts is a crucial determinant of

what sees the light of day in a particular journal. Fortunately, it is less

effective in blocking publication completely; there are so many journals

that most even modestly competent studies will be published provided that

the authors are determined enough. The publication might not be in a

prestigious journal, but at least it will get into print. However, peer

review is also the process that controls access to funding, and here the

situation becomes much more serious. There might often be only two or three

realistic sources of funding for a project, and the networks of reviewers

for these sources are often interacting and interlocking. Failure to pass

the peer-review process might well mean that a project is never funded.

Science bases its presumed authority in the world on the reliability and

objectivity of the evidence that is produced. If the pronouncements of

science are to be greeted with public confidence - and there is plenty of

evidence to suggest that such confidence is low and eroding - it should be

able to demonstrate that peer review, one of the sacred pillars of the

scientific edifice, is a process that has been validated objectively as a

reliable process for putting a stamp of approval on work that has been

done. Peer review should also have been validated as a reliable method for

making appropriate choices as to what work should be done. Yet when one

looks for that evidence it is simply not there.

Why not apply scientific methods to the peer review process?

For 30 years or so, I and others have been pointing out the fallibility of

peer review and have been calling for much more openness and objective

evaluation of its procedures [3-5]. For the most part, the scientific

establishment, its journals, and its grant-giving bodies have resisted such

open evaluation. They fail to understand that if a process that is as

central to the scientific endeavor as peer review has no validated

experimental base, and if it consistently refuses open scrutiny, it is not

surprising that the public is increasingly skeptical about the agenda and

the conclusions of science.

Largely because of this antagonism to openness and evaluation, there is a

great lack of good evidence either way concerning the objectivity and

validity of peer review. What evidence there is does not give confidence

but is open to many criticisms. Now, Rothwell and Martyn

have thrown a bombshell [6]. Their conclusions are measured and cautious,

but there is little doubt that they have provided solid evidence of

something truly rotten at the core of science.

Forget the reviewers. Just flip a coin.

Rothwell and Martyn performed a detailed evaluation of the reviews of

papers submitted to two neuroscience journals. Each journal normally sent

papers out to two reviewers. Reviews of abstracts and oral presentations

sent to two neuroscience meetings were also evaluated. One meeting sent its

abstracts to 16 reviewers and the other to 14 reviewers, which provides a

good opportunity for statistical evaluation. Rothwell and Martyn analyzed

the correlations among reviewers' recommendations by analysis of variance.

Their report should be read in full; however, the conclusions are

alarmingly clear. For one journal, the relationships among the reviewers'

opinions were no better than that obtained by chance. For the other

journal, the relationship was only fractionally better. For the meeting

abstracts, the content of the abstract accounted for only about 10 to 20

percent of the variance in opinion of referees, and other factors accounted

for 80 to 90 percent of the variance.

These appalling figures will not be surprising to critics of peer review,

but they give solid substance to what these critics have been saying. The

core system by which the scientific community allots prestige (in terms of

oral presentations at major meetings and publication in major journals) and

funding is a non-validated charade whose processes generate results little

better than does chance. Given the fact that most reviewers are likely to

be mainstream and broadly supportive of the existing organization of the

scientific enterprise, it would not be surprising if the likelihood of

support for truly innovative research was considerably less than that

provided by chance.

Objective evaluation of grant proposals is a high priority. Scientists

frequently become very angry about the public's rejection of the

conclusions of the scientific process. However, the Rothwell and Martyn

findings, coming on top of so much other evidence, suggest that the public

might be right in groping its way to a conclusion that there is something

rotten in the state of science. Public support can only erode further if

science does not put its house in order and begin a real attempt to develop

validated processes for the distribution of publication rights, credit for

completed work, and funds for new work. Funding is the most important issue

that most urgently requires opening up to rigorous research and objective

evaluation.

What relevance does this have for pharmacology and pharmaceuticals?

Despite enormous amounts of hype and optimistic puffery, pharmaceutical

research is actually failing [7]. The annual number of new chemical

entities submitted for approval is steadily falling in spite of the

enthusiasm for techniques such as combinatorial chemistry, high-throughput

screening, and pharmacogenomics. The drive to merge pharmaceutical

companies is driven by failure, and not by success.

The peer review process may be stifling innovation.

Could the peer-review processes in both academia and industry have

destroyed rather than promoted innovation? In my own field of

psychopharmacology, could it be that peer review has ensured that in

depression and schizophrenia, we are still largely pursuing themes that

were initiated in the 1950s? Could peer review explain the fact that in

both diseases the efficacy of modern drugs is no better than those

compounds developed in 1950? Even in terms of side-effects, where the

differences between old and new drugs are much hyped, modern research has

failed substantially. Is it really a success that 27 of every 100 patients

taking the selective 5-HT reuptake inhibitors stop treatment within six

weeks compared with the 30 of every 100 who take a 1950s tricyclic

antidepressant compound? The Rothwell-Martyn bombshell is a wake-up call to

the cozy establishments who run science. If science is to have any

credibility - and also if it is to be successful - the peer-review process

must be put on a much sounder and properly validated basis or scrapped

altogether.

F. Horrobin, a longtime critic of anonymous peer review. heads

Laxdale Ltd., which develops novel treatments for psychiatric disorders. In

1972 he founded Medical Hypotheses, the only journal fully devoted to

discussion of ideas in medicine. Andrzej Krauze is an illustrator, poster

maker, cartoonist, and painter who illustrates regularly for HMS Beagle,

The Guardian, The Sunday Telegraph, Bookseller, and New Statesman.