Problems with testimonials ... it worked for me!

March 2, 2010

There are many reasons why testimonials should be regarded with suspicion, and

that these accounts cannot be considered evidence:

The most obvious being you can't tell in any individual case what would have

happened if nothing was done, or something else. (The reason expensive and

time consuming controlled studies are required to win marketing approval.)

Such accounts can't inform about the number of persons who used the intervention

and did not benefit, or were harmed? (There is no denominator which is required

to provide a rate)

Reporting bias: Those who die cannot testify. Also " Response " is a subjective

term, easily influence by expectation. Those who report, can do so selectively

– when it's working, but not when it isn't. Or they may overestimate the

significance of the response, or wrongly attribute it to the intervention.

The authenticity of the report or its accuracy can't be verified.

Conflict of interest? Do those who testify on behalf of an unproven product

sell it or charge a fee for dispensing the information? (True, drug companies

have a conflict of interest, but the drug approval is determined by independent

review of controlled studies, typically with blinded independent monitoring.)

What is the natural course of the disease? Can it sometimes wax and wane without

intervention? Did the intervention cause the outcome, or was it coincidental?

Even for cancers with a very poor prognosis there are case reports in the

literature of spontaneous remissions, independent of any intervention. This

commonly than in follicular lymphoma

People sometimes win the lottery, but this does not make playing the lottery a

good bet - particularly when betting your life.

Was it an objectively measured response, or a patient reported outcome? Was it

that the patient felt better? What happened later? ... did the intervention

lead to a lasting clinical benefit?

What other medical treatments were given shortly before or after?

A CT scan will often show lesions after standard treatment that are necrotic

scar tissue. Credit might be given to an alternative practice used after this

treatment, when it was merely the resolution of a scar tissue over time, a

normal bodily process.

The accuracy of the diagnosis? Was it a false diagnosis of a cancer, or a cancer

of a type with an indolent course?

Recommended Posts