randomized controlled trials in health services research morris weinberger, phd senior career...

Randomized Controlled Trials Randomized Controlled Trials in Health Services Research in Health Services Research

Morris Weinberger, PhDMorris Weinberger, PhD

Senior Career Scientist, HSR&D ServiceSenior Career Scientist, HSR&D Service

Investigator, Center for Health Services Investigator, Center for Health Services Research, Durham VAMCResearch, Durham VAMC

VA Cyber-Seminar, January 12, 2009VA Cyber-Seminar, January 12, 2009

Overview to Today’s SeminarOverview to Today’s Seminar

• Overview of randomized controlled trialsOverview of randomized controlled trials

• Minimizing threats to internal validityMinimizing threats to internal validity

• Targets of interventionsTargets of interventions

• Analytical issuesAnalytical issues

• Practical issues and advicePractical issues and advice

Overview of Randomized Overview of Randomized Controlled TrialsControlled Trials

• BiasBias– Systematic (non-random) errorSystematic (non-random) error– Bane of research, regardless of study designBane of research, regardless of study design– Can invalidate study resultsCan invalidate study results– Can occur in any phase of researchCan occur in any phase of research


• Most powerful research design to Most powerful research design to establish causality, including the establish causality, including the effectiveness of interventionseffectiveness of interventions

• Establishes causality by controlling for Establishes causality by controlling for confounding factorsconfounding factors

• Well-designed experiments minimize biasWell-designed experiments minimize bias• Not suitable for all research questionsNot suitable for all research questions


• Subjects Subjects randomizedrandomized to treatment groups to treatment groups

• Follow subjects Follow subjects prospectivelyprospectively

• Compare subjects Compare subjects across treatment across treatment groupsgroups on relevant outcomes on relevant outcomes

Required CONSORT FigureRequired CONSORT Figure


• Overview of randomized trialsOverview of randomized trials

• Minimizing threats to internal validity Minimizing threats to internal validity

ValidityValidity

• Internal validity:Internal validity: Can the observed Can the observed differences between groups be differences between groups be attributed to the intervention? attributed to the intervention? – RandomizationRandomization

• External validity:External validity: Are the observed Are the observed differences in your study representative differences in your study representative of patients/subjects in general?of patients/subjects in general?– Random samplingRandom sampling

Threats to Internal ValidityThreats to Internal Validity

• Biased assignment of patients to groupsBiased assignment of patients to groups• Biased outcome assessment Biased outcome assessment • Non-compliance with treatment protocolNon-compliance with treatment protocol• DropoutsDropouts• Co-interventionCo-intervention• ContaminationContamination

Minimizing Threats to Internal Minimizing Threats to Internal ValidityValidity

• RandomizationRandomization

• Blinding (Masking) Blinding (Masking)

• Intervention designIntervention design

• Study protocolStudy protocol

RandomizationRandomization

• Random assignment of subjects to study groups:Random assignment of subjects to study groups:– Produces study groups comparable with respect to Produces study groups comparable with respect to

measured and unmeasured characteristicsmeasured and unmeasured characteristics

– Removes investigator bias in assigning patients to Removes investigator bias in assigning patients to groupsgroups

– Increases validity of statistical tests Increases validity of statistical tests

• If allocation of subjects to groups is predictable, If allocation of subjects to groups is predictable, it may lead to bias, e.g., decision to participateit may lead to bias, e.g., decision to participate


• Randomization Randomization


Considerations When Considerations When Designing InterventionsDesigning Interventions

• What is the intervention?What is the intervention?• Is it likely to be potent?Is it likely to be potent?• Is it ethical?Is it ethical?• Is it practical and feasible in the “real Is it practical and feasible in the “real

world”?world”?• Will it be acceptable to patients?Will it be acceptable to patients?• Is it effective?Is it effective?

What is the Intervention?What is the Intervention?StandardizationStandardization

• Who did what to whom?Who did what to whom?• What was the dose?What was the dose?• How often?How often?• For how long?For how long?• Administered under what conditions?Administered under what conditions?• With what dose adjustments?With what dose adjustments?


• RandomizationRandomization



• Study protocolStudy protocol

Study ProtocolStudy Protocol

• Choice of control groupChoice of control group• Recruitment strategiesRecruitment strategies• Retention strategiesRetention strategies• Outcome measuresOutcome measures• Evaluating effectiveness of the interventionEvaluating effectiveness of the intervention

– Compliance with treatment protocolCompliance with treatment protocol

– Co-interventionCo-intervention

– ContaminationContamination

Choosing the Control GroupChoosing the Control Group

• NothingNothing

• Usual CareUsual Care

• PlaceboPlacebo

Recruitment StrategiesRecruitment Strategies

• Recruitment sourcesRecruitment sources– Community vs. patientsCommunity vs. patients– Volunteer biasVolunteer bias

• Barriers to recruitmentBarriers to recruitment– Interest in subjectInterest in subject– Distrust of researchDistrust of research– Distrust of contact from unknown personsDistrust of contact from unknown persons– TransportationTransportation– Informed consentInformed consent– Long enrollment visitsLong enrollment visits

RetentionRetention

• Impact on external validity of the trial Impact on external validity of the trial – Who completes the trial?Who completes the trial?

• Impact on internal validity of the trial Impact on internal validity of the trial – What if there is differential dropout?What if there is differential dropout?

• Impact on number of patients recruitedImpact on number of patients recruited– How will my sample size estimates be How will my sample size estimates be

affected?affected?

Design Considerations:Design Considerations:Choosing MeasuresChoosing Measures

• Properties of the measureProperties of the measure– Validity and reliabilityValidity and reliability

– Floor and ceiling effectsFloor and ceiling effects

– Sensitivity to changeSensitivity to change

• Pragmatic considerationsPragmatic considerations– SettingSetting

– Respondent burdenRespondent burden

– Appropriateness for subjectsAppropriateness for subjects

– CostCost

Evaluating Effectiveness of the Evaluating Effectiveness of the InterventionIntervention

• Dose: Dose: Was the intervention delivered?Was the intervention delivered?• Contamination:Contamination: Did the control group receive Did the control group receive

components of the intervention?components of the intervention?• Co-intervention:Co-intervention: Did the treatment group Did the treatment group

receive interventions other than what was receive interventions other than what was intended?intended?

• Key:Key: Measure what elements of the Measure what elements of the intervention were delivered to intervention were delivered to allall study groups study groups



• Minimizing threats to internal validityMinimizing threats to internal validity


Targets of Strategies to Targets of Strategies to Improve Outcomes Improve Outcomes

• PatientsPatients

• ProvidersProviders

• SystemSystem

Patient-Level StrategiesPatient-Level Strategies

• Randomization is at the patient levelRandomization is at the patient level

• Simplifies the statistical analysisSimplifies the statistical analysis

Provider-Level StrategiesProvider-Level Strategies

• Strategies to improve quality of care by Strategies to improve quality of care by intervening with providersintervening with providers

• If outcomes are at physician level, issues are If outcomes are at physician level, issues are generally similar to patient-level interventionsgenerally similar to patient-level interventions


• Often, goal is to evaluate the impact of provider Often, goal is to evaluate the impact of provider interventions on patient outcomes because interventions on patient outcomes because– Expect intervention with provider to affect patient Expect intervention with provider to affect patient

outcomes (e.g., improving patients’ glycemic outcomes (e.g., improving patients’ glycemic control)control)

– Concern that providing intervention to intervention Concern that providing intervention to intervention patients will affect providers behavior with control patients will affect providers behavior with control patients (i.e. contamination)patients (i.e. contamination)

• It seems like effective sample size should be It seems like effective sample size should be greater than the number of physicians greater than the number of physicians randomizedrandomized


• Randomizing patients assumes balance on both Randomizing patients assumes balance on both measured and unmeasured variables measured and unmeasured variables

• Analytically, assumes that patients are Analytically, assumes that patients are independently assigned to groupsindependently assigned to groups

• When unit of randomization is not the unit of When unit of randomization is not the unit of analysis:analysis:– Patients are not independent within physiciansPatients are not independent within physicians– Physicians not independent within setting (e.g., Physicians not independent within setting (e.g.,

team, hospital)team, hospital)– Complicates sample size estimatesComplicates sample size estimates

Provider-Level Strategies:Provider-Level Strategies:SummarySummary

• Often, interventions to improve outcomes target Often, interventions to improve outcomes target providers, but analyze patientsproviders, but analyze patients

• Reasonable, and perhaps only plausible strategyReasonable, and perhaps only plausible strategy• Must provide reviewers with clear justification Must provide reviewers with clear justification

(i.e., minimizes threats to internal validity that (i.e., minimizes threats to internal validity that would otherwise result) would otherwise result)

• Involve biostatisticians early, as the analyses Involve biostatisticians early, as the analyses (including sample size calculations) are complex(including sample size calculations) are complex

Design Considerations: Design Considerations: Biomedical vs. HSR TrialsBiomedical vs. HSR Trials

ConsiderationConsideration BiomedicalBiomedical HSRHSR• PatientsPatients

– EligibilityEligibility NarrowNarrow BroadBroad– RandomizationRandomization PatientPatient Patient, Patient,

Physician, Physician, Clinic, HospitalClinic, Hospital

• Intervention: Intervention: - Components- Components SingleSingle MultipleMultiple

- - TypeType Drug, device, Drug, device, Structure of careStructure of care

procedureprocedure- Uniformity- Uniformity HighHigh LowLow

Design Considerations: Design Considerations: Biomedical vs. HSR TrialsBiomedical vs. HSR Trials

ConsiderationConsideration BiomedicalBiomedical HSRHSR• InterventionIntervention

– Control groupControl group StandardStandard Usual CareUsual Care– Assess complianceAssess compliance Easy-ModerateEasy-Moderate DifficultDifficult

• Data CollectionData Collection– OutcomesOutcomes Events, test resultEvents, test result Patient-Patient-

centeredcentered– Process of careProcess of care Easy-moderateEasy-moderate DifficultDifficult– CostCost Easy-moderateEasy-moderate DifficultDifficult– BlindingBlinding PossiblePossible Not possibleNot possible



• Threats to internal validityThreats to internal validity



Analytical IssuesAnalytical Issues

• Between- versus within-group Between- versus within-group comparisonscomparisons– Primary analysis is between-groupsPrimary analysis is between-groups

• Controlling for baseline differencesControlling for baseline differences– Randomization does not achieve balance on Randomization does not achieve balance on

key factorskey factors


• Intention to Treat:Intention to Treat: Subjects analyzed as Subjects analyzed as part of original group, regardless of part of original group, regardless of compliance, dropout, or crossovercompliance, dropout, or crossover– Question answered: What is benefit of Question answered: What is benefit of

treatment/intervention as given?treatment/intervention as given?

• Per protocol: Per protocol: Focus on subjects who were Focus on subjects who were compliant with protocolcompliant with protocol– Question answered: What is benefit for Question answered: What is benefit for

people who are compliant?people who are compliant?


• Sub-group analyses: Sub-group analyses: Focus on subjects Focus on subjects with certain characteristicswith certain characteristics– Question answered: Does the treatment Question answered: Does the treatment

work for certain types of patientswork for certain types of patients

• Disaggregating complex interventions: Disaggregating complex interventions: Can we identify the effective component?Can we identify the effective component?

• Unit of assignment versus unit of analysisUnit of assignment versus unit of analysis



• Threats to internal validityThreats to internal validity


• Multi-site trials in health services Multi-site trials in health services researchresearch


• Practical issues and advicePractical issues and advice

Practical Issues and AdvicePractical Issues and Advice

• What help do I need?What help do I need?

• Am I ready to conduct a clinical trial?Am I ready to conduct a clinical trial?

• Is the intervention worth evaluating?Is the intervention worth evaluating?

• Is the project feasible?Is the project feasible?

• What outcomes can I reasonably expect What outcomes can I reasonably expect to change?to change?

What Help Do I Need?What Help Do I Need?

• Colleagues with content and Colleagues with content and methodological expertise and/or other methodological expertise and/or other specialized skills specialized skills

• BiostatisticianBiostatistician

Biostatisticians Are Your Biostatisticians Are Your FriendsFriends

• Study designStudy design

• Sample sizeSample size

• Measuring outcomesMeasuring outcomes

• Data managementData management

• Statistical analysisStatistical analysis

Am I Ready to Conduct a Am I Ready to Conduct a Clinical Trial?Clinical Trial?

• Equipoise Equipoise • Advance previous studyAdvance previous study

– Good idea, flawed studyGood idea, flawed study– Application to another patient populationApplication to another patient population– Application to another clinical venueApplication to another clinical venue

• Relevance Relevance – Policy makers Policy makers – Clinicians Clinicians – Health care organizationsHealth care organizations

Is the Intervention Worth Is the Intervention Worth Evaluating?Evaluating?

• Is the study ethical?Is the study ethical?• Is the intervention feasible and practical?Is the intervention feasible and practical?• If effective, will intervention be accepted If effective, will intervention be accepted

and useful?and useful?– Health care organizationHealth care organization– PhysiciansPhysicians– StaffStaff– PatientsPatients

Is the Project Feasible?Is the Project Feasible?Pilot Study as Dress RehearsalPilot Study as Dress Rehearsal

• Are there enough subjects?Are there enough subjects?• Do I have a practical strategy to identify, enroll Do I have a practical strategy to identify, enroll

and retain subjects in the study?and retain subjects in the study?• Do I have time to complete the project?Do I have time to complete the project?• Do I have resources to complete the study?Do I have resources to complete the study?• Can I measure the critical variables?Can I measure the critical variables?• Are data collection forms reasonable?Are data collection forms reasonable?• Do I have buy-in from the organization and Do I have buy-in from the organization and

personnel (physicians, nurses, clerks, etc.)?personnel (physicians, nurses, clerks, etc.)?

What Outcomes Can I What Outcomes Can I Reasonably Expect to Change? Reasonably Expect to Change? • What should my outcomes be? What should my outcomes be?

– Mortality and/or morbidity Mortality and/or morbidity – Clinical parametersClinical parameters– Health-related quality of lifeHealth-related quality of life– Health services utilization/costHealth services utilization/cost

Final Advice Final Advice

• Specify the hypothesesSpecify the hypotheses• Write early, write often: Write early, write often:

– methodsmethods– dummy tablesdummy tables

• Monitor what is happeningMonitor what is happening– EnrollmentEnrollment– RetentionRetention– Delivery of the interventionDelivery of the intervention

• Review CONSORT statement on reporting Review CONSORT statement on reporting randomized trialsrandomized trials

randomized controlled trials in health services research morris weinberger, phd senior career...

Documents

internal validity slide

validity internal validity

phase of research slide

study groups comparable

advice slide

study design

random sampling slide

study representative