Outcome MeasurementsStudy Design

Liz Lingard BPhty, MPhil, MPHLiz Lingard BPhty, MPhil, MPHResearch CoordinatorResearch Coordinator

Department of Orthopaedics Department of Orthopaedics Freeman HospitalFreeman Hospital

Orthopaedic Clinical Research

Traditionally focused on reporting clinical and Traditionally focused on reporting clinical and radiographic resultsradiographic results

Rates of complications following proceduresRates of complications following procedures

Survivorship analysis of the life of a prosthesisSurvivorship analysis of the life of a prosthesis

Many studies are observationalMany studies are observational

Need For Outcome Measures

Technological advances, epidemiologic and Technological advances, epidemiologic and demographic trends and rising public demand had demographic trends and rising public demand had rapidly increased health care expenditure over the rapidly increased health care expenditure over the past three decadespast three decades

As health care resources are limited they need to As health care resources are limited they need to be allocated according to clinical effectiveness and be allocated according to clinical effectiveness and cost-effectivenesscost-effectiveness

Orthopaedic Interventions

Surgical treatment for one or more of the Surgical treatment for one or more of the following musculoskeletal problemsfollowing musculoskeletal problems PainPain Loss of range of motionLoss of range of motion InstabilityInstability Functional limitationFunctional limitation DeformityDeformity TraumaTrauma TumourTumour

International Classification of Functioning, Disability and Health (ICF)

Functioning and DisabilityFunctioning and Disability Body function and structuresBody function and structures Activities and participationActivities and participation

Contextual FactorsContextual Factors Environmental factorsEnvironmental factors Personal factorsPersonal factors

Interaction Between ICF Components

H e a lth c o nd ition(d isord er o r d isea se)

B o d y F un c tio nsa n d S tru c tu res

A c tiv ities P a r tic ipa tion

E n v iro n m e n talF a c to rs

P e r so n alF a c to rs

Uses of Outcome Measures

Clinical researchClinical research

Audit and quality assuranceAudit and quality assurance

Assess health care needs of populationsAssess health care needs of populations

Assist in individual patient careAssist in individual patient care

Outcome Measures

Assessment of disease or injury severityAssessment of disease or injury severity Measure change over time due to:Measure change over time due to:

Natural historyNatural history InterventionIntervention

Supplement mortality, clinical, radiological Supplement mortality, clinical, radiological and laboratory measuresand laboratory measures

Outcome instrumentsOutcome instruments Observer administeredObserver administered Patient administeredPatient administered

Observer Administered

Functional TestsFunctional Tests Timed Up & GoTimed Up & Go 6 minute walk6 minute walk Activity MonitoringActivity Monitoring

Global rating scoresGlobal rating scores Knee Society ScoreKnee Society Score Harris Hip ScoreHarris Hip Score Foot and Ankle ScoreFoot and Ankle Score

ActivPal Monitor

Ambulatory activity monitor worn for 7 days

Activity Monitoring

matched walking periods

Subject A: 72 year old THR patient Subject B: 65 year old spouse

Activity SummaryActivity Summary

sitting standing stepping (slow) Stepping (moderate) Stepping (vigerous)

Time sittng (h) Time upright (h)

A

B

03 3 66

Stepping SummaryStepping Summary

2554

7202

6552

8956

0 5000 10000 15000 20000

A

B

Steps per day

stepping (slow) Stepping (moderate) Stepping (vigerous)

Global Rating Scores

May combine measures of pain, joint range May combine measures of pain, joint range of motion, stability, muscle power, of motion, stability, muscle power, deformity, function and even complications deformity, function and even complications and investigationsand investigations

Weight of scoring of each item variesWeight of scoring of each item varies Inter / intra-rater reliability & observer biasInter / intra-rater reliability & observer bias

Pitfalls of global scores

Different scoring systems assign different Different scoring systems assign different weights to parts of the score and the weights to parts of the score and the summary results are often inconsistent summary results are often inconsistent giving contrary measures of success in the giving contrary measures of success in the same patient. same patient. (Bryant et al. JBJS-Br 1993)(Bryant et al. JBJS-Br 1993)

Patient Administered

Report summary scores for different health Report summary scores for different health dimensionsdimensions Physical FunctionPhysical Function Symptoms – painSymptoms – pain Global judgements of healthGlobal judgements of health Psychological well-beingPsychological well-being Social well-beingSocial well-being Cognitive functioningCognitive functioning Role activitiesRole activities Personal constructsPersonal constructs SatisfactionSatisfaction

Patient Administered

Disease-specific Disease-specific (WOMAC, AIMS)(WOMAC, AIMS)

Site or region-specific Site or region-specific (KOOS, DASH, ODI)(KOOS, DASH, ODI)

Dimension-specific Dimension-specific (McGill Pain Questionnaire)(McGill Pain Questionnaire)

General health status General health status (SF-36, NHP)(SF-36, NHP)

Health state utility Health state utility (EQ-5D)(EQ-5D)

Satisfaction Satisfaction (Management / Outcome)(Management / Outcome)

WOMAC

Western Ontario & McMaster University OA indexWestern Ontario & McMaster University OA index Tri-dimensional scaleTri-dimensional scale (Likert and VAS versions)(Likert and VAS versions)::

Pain (5 items)Pain (5 items) Stiffness (2 items)Stiffness (2 items) Function (17 items)Function (17 items)

Validated on patients with OA of hip and knee Validated on patients with OA of hip and knee Sensitive and responsive for detecting change Sensitive and responsive for detecting change

over time due to NSAIDs and joint arthroplastyover time due to NSAIDs and joint arthroplasty Translated into Translated into over over 5500 different languages different languages

WOMAC

Responses for each item are none, mild, moderate, Responses for each item are none, mild, moderate, severe or extreme (pain, stiffness or difficulty)severe or extreme (pain, stiffness or difficulty)

Scores transformed to 0-100 (100 best)Scores transformed to 0-100 (100 best) 100 = none100 = none 75 = mild75 = mild 50 = moderate50 = moderate 25 = severe25 = severe 0 = extreme0 = extreme

Preoperative WOMAC

36 36 3734 32 3636 38 35

0102030405060708090

100

Pain Function Stiffness

Primary TKAPrimary THAPrimary BHR

None

Mild

Moderate

Severe

Extreme

Note: Data on Primary Joints – 1,061 TKA, 760 THA & 174 BHR

1-Year WOMAC

8174 72

8879 80

9085 84

0102030405060708090

100

Pain Function Stiffness

Primary TKAPrimary THAPrimary BHR

None

Mild

Moderate

Severe

Extreme

Note: Data on Primary Joints - 490 TKA, 387 THA & 75 BHR

Developed and validated by Ewa RoosDeveloped and validated by Ewa Roos

Meniscus injury, ACL injury and/or OA Meniscus injury, ACL injury and/or OA

Adds dimensions to the WOMAC scale:Adds dimensions to the WOMAC scale: Sport and recreation functionSport and recreation function Knee-related quality of lifeKnee-related quality of life

Improves sensitivity of evaluation of Improves sensitivity of evaluation of younger or more physically active personsyounger or more physically active persons

Knee injury and OsteoarthritisOutcome Score (KOOS)

SquattingSquatting

RunningRunning

JumpingJumping

Twisting/pivotingTwisting/pivoting

KneelingKneeling

KOOS – Sports/Recreation

How often are you aware of your knee How often are you aware of your knee problems?problems?

Have you modified your lifestyle to avoid Have you modified your lifestyle to avoid potentially damaging activities to your knee?potentially damaging activities to your knee?

How much are you troubled with lack of How much are you troubled with lack of confidence in your knee?confidence in your knee?

In general, how much difficulty do you have In general, how much difficulty do you have with your knee?with your knee?

KOOS – Knee Related QOL

KOOS Profile

0

20

40

60

80

100

Pain

Stiffn

ess

Function

Sport/R

ecQOL

ControlsOA

Roos EM et al, Osteoarthritis & Cartilage 1999; 7:216-221Roos EM et al, Osteoarthritis & Cartilage 1999; 7:216-221

WOMACWOMAC

KOOSKOOS

KOOS Profile

0

20

40

60

80

100

Pain Symptoms ADL Sport/Rec QOL

Preop3 mo postop6 mo postop12 mo postop

Roos EM et al, JOSPT 1998; 78:88-96Roos EM et al, JOSPT 1998; 78:88-96

WOMAC Pain and Function

8879

010

2030

405060

7080

90100

Pain Function

Kinemax 10-Year

None

Mild

Moderate

Severe

Extreme

Patients at 10 years after primary TKR

How often are you aware of your knee problem?

40

156

28

11

0

10

20

30

40

50

60

70

80

90

100

Never Monthly Weekly Daily Constantly

% p

atie

nts

Kinemax 10-Year

Patients at 10 years after primary TKR

In general, how much difficulty do you have with your knee?

40

2920

91

0

10

20

30

40

50

60

70

80

90

100

None Mild Moderate Severe Extreme

% p

atie

nts

Kinemax 10-Year

Patients 10 years after primary TKR

Short-Form 36 (SF-36)

Reports on 8 health dimensionsReports on 8 health dimensions Physical FunctioningPhysical Functioning Bodily PainBodily Pain Role Physical Role Physical General Health PerceptionGeneral Health Perception Mental HealthMental Health Role EmotionalRole Emotional VitalityVitality Social FunctioningSocial Functioning

Each subscale given a 0-100 score (100 best)Each subscale given a 0-100 score (100 best)

Physical component

score

Mental component

score

SF-36 – Primary TKA

2327 29

58

40

50

59

69

4851 54

60

51

71 7075

4750

5560

50

70 6974

0

10

20

30

40

50

60

70

80

90

100

PhysicalFunctioning

Role Physical Bodily Pain GeneralHealth

Vitality SocialFunctioning

RoleEmotional

Mental Health

Preoperative 1-Year 2-Year US Norm (65-74)

Note: Data on 1,061 at preop, 490 1-year & 243 at 2-years

SF-36 – Primary THA

1822 23

58

3642

55

66

5257 60

63

54

77 7875

49

57 5862

53

77 78 76

0

10

20

30

40

50

60

70

80

90

100

PhysicalFunctioning

Role Physical Bodily Pain GeneralHealth

Vitality SocialFunctioning

RoleEmotional

Mental Health

Preoperative 1-Year 2-Year US Norm (65-74)

Note: Data on 760 at preop, 387 at 1-year & 186 at 2-years

SF-36 – Primary BHR

2528

24

57

36

44

63 63

73 7570 68

58

82 8277

83 8678

73

63

9195

80

0

10

20

30

40

50

60

70

80

90

100

PhysicalFunctioning

Role Physical Bodily Pain GeneralHealth

Vitality SocialFunctioning

RoleEmotional

Mental Health

Preoperative 1-Year 2-Year US Norm (45-54)

Note: Data on 174 at preop, 75 at 1-year & 35 at 2-years

Satisfaction with outcome

Asked on follow-up questionnairesAsked on follow-up questionnaires Patient rates how satisfied they are with the results Patient rates how satisfied they are with the results

of surgery:of surgery: OverallOverall Pain reliefPain relief Ability to do ADLAbility to do ADL Ability to do recreational activitiesAbility to do recreational activities

Responses are very satisfied, somewhat satisfied, Responses are very satisfied, somewhat satisfied, somewhat dissatisfied and very dissatisfiedsomewhat dissatisfied and very dissatisfied

Satisfaction – Primary TKA

71

20

72

75

17

62

53

33

104

46

34

146

0

10

20

30

40

50

60

70

80

90

100

VerySatisfied

SomewhatSatisfied

SomehwatDissatisfied

VeryDissatisfied

% p

atie

nts Overall

Pain ReliefADLRecreation

Note: Data on 490 primary TKA patients at 1-Year

Satisfaction – Primary THA

83

13

3 1

85

12

2 1

63

28

73

58

29

8 5

0

10

20

30

40

50

60

70

80

90

100

VerySatisfied

SomewhatSatisfied

SomewhatDissatisfied

VeryDissatisfied

% p

atie

nts Overall

Pain ReliefADLRecreation

Note: Data on 388 primary THA patients at 1-Year

ICF Quiz

H e a lth c o nd ition(d isord er o r d isea se)

B o d y F un c tio nsa n d S tru c tu res

A c tiv ities P a r tic ipa tion

E n v iro n m e n talF a c to rs

P e r so n alF a c to rs

Choosing an Outcome Measure

Formulate a precise study questionFormulate a precise study question Know your patient populationKnow your patient population Know what independent variables may Know what independent variables may

affect the outcomeaffect the outcome Commonly used outcome measure does not Commonly used outcome measure does not

necessarily mean it is the best measure to necessarily mean it is the best measure to use for your studyuse for your study

Independent Variable Examples

Socio-demographicsSocio-demographics Body mass indexBody mass index Comorbid medical conditionsComorbid medical conditions Comorbid musculoskeletal conditionsComorbid musculoskeletal conditions Mental health statusMental health status Common geriatric problemsCommon geriatric problems

Appropriateness

Disease / injury to be studiedDisease / injury to be studied Patient populationPatient population Intervention Intervention Dimension of health likely to be affectedDimension of health likely to be affected AcceptabilityAcceptability FeasibilityFeasibility

Acceptability

Minimise burden to patients:Minimise burden to patients: Avoid distress to patientsAvoid distress to patients Reduce refusal rateReduce refusal rate Reduce incomplete dataReduce incomplete data

Need to consider:Need to consider: Literacy of patientsLiteracy of patients Legibility and layout of questionnaireLegibility and layout of questionnaire Time taken to completeTime taken to complete

Feasibility

Impact on clinical staff / researchersImpact on clinical staff / researchers Staff attitudes are key determinant of data Staff attitudes are key determinant of data

collectioncollection Data collectionData collection

Brief questionnairesBrief questionnaires Lengthy questionnaires with interviewsLengthy questionnaires with interviews Postal / telephonePostal / telephone

Coding and entering dataCoding and entering data Analysis of dataAnalysis of data

Validity

Measure what it says it doesMeasure what it says it does Content – appropriate range of itemsContent – appropriate range of items Face – measures what they aim to measureFace – measures what they aim to measure Construct – correlates to other measuresConstruct – correlates to other measures Criterion – correlates to a ‘gold standard’Criterion – correlates to a ‘gold standard’

Reliability

Measures accuratelyMeasures accurately Internal consistencyInternal consistency Test – re-testTest – re-test Inter-raterInter-rater Intra-raterIntra-rater

Responsiveness

Sensitive to detecting changes over time that are Sensitive to detecting changes over time that are important to the patientimportant to the patient

Correlates with changes in disease severityCorrelates with changes in disease severity

Effect size index = Effect size index =

((mean1 – mean2)/ pooled SD

Standardised response mean = Standardised response mean =

((mean1 – mean2)/ SD of change in scoreSD of change in score

Precision

Ceiling and floor effectsCeiling and floor effects Same amount of change across different Same amount of change across different

parts of the scale may not be equalparts of the scale may not be equal Instruments that make more distinctions are Instruments that make more distinctions are

not more precisenot more precise

Interpretation

Outcome measures produce numeric scoresOutcome measures produce numeric scores Score ranges need to be clearScore ranges need to be clear Need to know what is the smallest clinical Need to know what is the smallest clinical

meaningful changes that are perceptible to a meaningful changes that are perceptible to a patient for that particular instrumentpatient for that particular instrument

Study Design

Study Design

RetrospectiveRetrospective Deliberate interventionDeliberate intervention Observational studiesObservational studies

ProspectiveProspective Deliberate interventionDeliberate intervention Observational studiesObservational studies

Study Design

Cross-sectional studiesCross-sectional studies Disease descriptionDisease description Diagnosis and stagingDiagnosis and staging

Abnormal rangesAbnormal rangesDisease severityDisease severity

Disease processDisease process

Study Design

Longitudinal studiesLongitudinal studies ProspectiveProspective

RandomisedRandomisedNon-randomisedNon-randomised

Randomised vs. non-randomised

RCT is seen as the ‘gold standard’ RCT is seen as the ‘gold standard’ Should ensures patients differ only by their Should ensures patients differ only by their

exposure to the interventionexposure to the intervention Often difficult to generalise results due to Often difficult to generalise results due to

inclusion/exclusion criteriainclusion/exclusion criteria Blinded trials in orthopaedics almost Blinded trials in orthopaedics almost

impossible introducing observer/patient biasimpossible introducing observer/patient bias Non-randomised trials Non-randomised trials

Cohort and case-controlled studiesCohort and case-controlled studies More inclusive but may have selection biasMore inclusive but may have selection bias

Study design

For studies that compare health care For studies that compare health care interventions to be valid they must produce interventions to be valid they must produce results that are:results that are: Generalisable to to other potential Generalisable to to other potential

patients (high external validity)patients (high external validity) Estimate outcome effects that can be Estimate outcome effects that can be

reliably attributed to the intervention reliably attributed to the intervention (high internal validity)(high internal validity)

Solutions to threats of validity

Internal validityInternal validity Allocation bias (confounding) – risk adjustment Allocation bias (confounding) – risk adjustment

and subgroup analysisand subgroup analysis Patient preference – preference arms or Patient preference – preference arms or

adjustment for preferenceadjustment for preference External validityExternal validity

Exclusions – expand inclusion criteriaExclusions – expand inclusion criteria Non-participation – multi-centre, pragmatic Non-participation – multi-centre, pragmatic

designdesign Non-participation – less rigorous consentNon-participation – less rigorous consent

Potential Problems

RecruitmentRecruitment Need to know what proportion of the reference Need to know what proportion of the reference

population have been recruitedpopulation have been recruited

Reasons for non-participation – does this group Reasons for non-participation – does this group of patients differ from participants. Knowing of patients differ from participants. Knowing baseline details about this group strengthens baseline details about this group strengthens your studyyour study

Volunteerism likely to be associated with age, Volunteerism likely to be associated with age, gender, socioeconomic status, education etc. gender, socioeconomic status, education etc. which may influence outcomeswhich may influence outcomes

Potential Problems

ComplianceCompliance Depends on how well informed patients are at Depends on how well informed patients are at

start of trial of what is expectedstart of trial of what is expected

If intervention is ongoing (e.g. physical vs. If intervention is ongoing (e.g. physical vs. surgical intervention) then poor compliance surgical intervention) then poor compliance with intervention can affect statistical power with intervention can affect statistical power

If patients are non-compliant with parts of If patients are non-compliant with parts of study this is preferable to total non-compliancestudy this is preferable to total non-compliance

Need to track all patients and document reasons Need to track all patients and document reasons for lost to follow-upfor lost to follow-up

Statistical Support

Speak to a statisticianSpeak to a statistician After you have reviewed the literature, selected After you have reviewed the literature, selected

your primary outcome measure and drafted a your primary outcome measure and drafted a protocolprotocol

Sample size calculationSample size calculationMethod of data collectionMethod of data collectionAdvice about coding of dataAdvice about coding of dataAnalysis of dataAnalysis of data

BUTBUT Before you submit to either R&D or LRECBefore you submit to either R&D or LREC