Confounding and the Language of ExperimentationPart II - Importance of Proper

Comparisons and Randomization

This video is designed to accompany

pages 13-18

in

Making Sense of UncertaintyActivities for Teaching Statistical

ReasoningVan-Griner Publishing Company

Arthroscopic Knee Surgery

Traditional Design

Patients come to doctor with knee symptoms. Range of motion measured

Arthroscopic surgery used to treat the condition

Patients asked if they felt better and range of motion re-measured.

Largely:• subjects felt better • range of motion was

better

What’s Missing

No Placebo ComparisonHow do we know that the results were not confounded by the placebo effect?

Seriously!

Source: N Engl J Med. 2002 Jul 11;347(2):81-8.

Title: A controlled trial of arthroscopic surgery for osteoarthritis of the knee.

Authors: Moseley JB, O'Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH, Hollingsworth JC, Ashton CM, Wray NP.

Place: Houston Veterans Affairs Medical Center, Baylor College of Medicine, Houston, TX 77030, USA.

Controlled Study w/ Placebo

METHODS:

A total of 180 patients with osteoarthritis of the knee were randomly assigned to receive arthroscopic débridement, arthroscopic lavage, or placebo surgery.

Patients in the placebo group received skin incisions and underwent a simulated débridement without insertion of the arthroscope. Patients and assessors of outcome were blinded to the treatment-group assignment.

Outcomes were assessed at multiple points over a 24-month period with the use of five self-reported scores--three on scales for pain and two on scales for function--and one objective test of walking and stair climbing. A total of 165 patients completed the trial.

Surprising Conclusion

CONCLUSIONS:

In this controlled trial involving patients with osteoarthritis of the knee, the outcomes after arthroscopic lavage or arthroscopic débridement were no better than those after a placebo procedure.

(N Engl J Med 2002;347:81-8.)Copyright © 2002 Massachusetts Medical Society.

Language

• Response Variables – Level of pain and function

• Explanatory Variable – Type of Procedure (Real or Placebo)

• Subjects – The participating patients with osteoarthritis of the knee

• Confounding– In the traditional design, the placebo effect created the confounding and compromised the inference.

Diagraming the Faulty Design

Compare

Measure Pain and Function

Measure Pain and Function

Before or No Surgery Surgery

Placebo Respons

e

Diagraming the Careful Design

Compare

Measure Pain and Function

Measure Pain and Function

Placebo Surgery Real

Surgery

Importance of Randomization

USE OF AN AIDS VIRUS DRUG CURBED ON THE BASIS OF DATABy ANDREW POLLACK, Special to the New York TimesPublished: April 16, 1987

SAN FRANCISCO, April 15— The Food and Drug Administration has told a California pharmaceutical company that there is not yet enough evidence to justify more widespread use of its antiviral drug ribavirin to treat patients with early symptoms of infection with the AIDS virus.

Importance of Randomization

Reported Results

Treatment Number of Patients

How Many Developed Aids

800 mg 52 0

600 mg 55 6

Placebo 56 10

Looks Convincing!

FDA Looked Closely …

There was evidence that the sickest patients had been assigned to the placebo group.

And the healthiest patients to the Ribavirin treatments.

Inference of “effective treatment” was compromised.

Language

• Response Variable – Development of AIDS (yes/no)

• Explanatory Variable – Type of Intervention (Ribavirin level, Placebo)

• Subjects – The participating patients with pre-AIDS symptoms

• Confounding– Lack of randomization put healthiest patients in the Ribavirin treatment groups.

Benefits of Randomization

Helps to keep the comparison groups – treatment groups – as much alike as possible.

In that sense, randomization addresses confounding in a very direct way.

One-Sentence Reflection

The placebo effect and lack of randomization can create very real obstacles to making credible inferences from experimental data.