1

Subjects and Subjects and Measurements—The Measurements—The

Nuts & Bolts of Nuts & Bolts of ResearchResearch

Optimizing Subjects & VariablesOptimizing Subjects & Variables

Doug Bauer, M.D.Doug Bauer, M.D.Depts. of Medicine, Epidemiology and BiostatisticsDepts. of Medicine, Epidemiology and Biostatistics

UCSFUCSF

August 6, 2008August 6, 2008

2

UCSF Resident Reseach UCSF Resident Reseach ProgarmProgarm

http://www.ctst.ucsf.edu/residents.htmhttp://www.ctst.ucsf.edu/residents.htm Reseach ElectiveReseach Elective

Counts towards ATCR, Masters of Clinical ResearchCounts towards ATCR, Masters of Clinical Research Research FundingResearch Funding

Up to $2,000/year. Funded 17 residents 12/07, 4/08Up to $2,000/year. Funded 17 residents 12/07, 4/08 Next deadline for applications about 11/08Next deadline for applications about 11/08

Resident Research Symposia (May 2009)Resident Research Symposia (May 2009)Abstracts, awards, keynote (e.g. Gene Washington)Abstracts, awards, keynote (e.g. Gene Washington)

The future: resident seminars, ATCR, PathwaysThe future: resident seminars, ATCR, Pathways

3

Today’s ObjectivesToday’s Objectives

Selecting the SubjectsSelecting the Subjects Develop systematic approach to optimize Develop systematic approach to optimize

subject selectionsubject selection Choosing the MeasurementsChoosing the Measurements

Understand the implications of exposure Understand the implications of exposure & outcome variable/measurement & outcome variable/measurement choiceschoices

Application to a Real Research Application to a Real Research Question: The Osteoporosis Question: The Osteoporosis in Men (MrOS) Studyin Men (MrOS) Study

4

After deciding a great research After deciding a great research question, figuring out question, figuring out WHOWHO you you want to study and want to study and WHATWHAT you want you want to measure are the next key steps…to measure are the next key steps…

Subjects and Variables: The Subjects and Variables: The Nuts and Bolts of the Nuts and Bolts of the

Research QuestionResearch Question

5

Selecting Your Selecting Your Subjects…Subjects…

6

Optimizing Subject Optimizing Subject Selection: Selection: A Delicate A Delicate

Balancing ActBalancing Act

FeasibilityFeasibility

AccessibilityAccessibility

CostCost

Time/EfficiencyTime/Efficiency

GeneralizabilityGeneralizability

AccuracyAccuracy

Diversity Diversity

Adequate SizeAdequate Size

At the end…Will I believe the findings and will I care?At the end…Will I believe the findings and will I care?

7

Subject Selection: The Subject Selection: The Nitty GrittyNitty Gritty

ExplicitlyExplicitly Define Inclusion Criteria Define Inclusion Criteria Demographic features (e.g., age, gender, Demographic features (e.g., age, gender,

race)race) Clinical criteriaClinical criteria Geographic/administrative characteristics Geographic/administrative characteristics Sampling time frameSampling time frame

ExplicitlyExplicitly Define Exclusion Criteria Define Exclusion Criteria Minimum number necessary to be feasible Minimum number necessary to be feasible

yet yet maintainmaintain generalizability to target generalizability to target populationpopulation

8

Subject Sampling Subject Sampling Techniques:Techniques:

Convenience SamplesConvenience Samples True convenience True convenience (e.g., 25 clinic patients I (e.g., 25 clinic patients I

know well)know well) Consecutive Consecutive (e.g., next 100 patients (e.g., next 100 patients

undergoing liposuction)undergoing liposuction)

Probability SamplesProbability Samples Simple random Simple random (e.g., using random number (e.g., using random number

table)table) Stratified or weighted random Stratified or weighted random (e.g., (e.g.,

by gender)by gender) Cluster Cluster (e.g., by clinic or neighborhood)(e.g., by clinic or neighborhood)

9

Subject Recruitment:Subject Recruitment:

Successful Recruitment Generally Means…Successful Recruitment Generally Means… response, generalizable sample, adequate sizeresponse, generalizable sample, adequate size

For database only studies—For database only studies—Not usually a big Not usually a big problemproblem

For hands-on studies For hands-on studies ((e.g., surveys, cohorts, trialse.g., surveys, cohorts, trials)) Expect that it will be harder than you think!Expect that it will be harder than you think! Use reasonable inclusion/exclusion criteriaUse reasonable inclusion/exclusion criteria Acceptable subject burden/potential benefitsAcceptable subject burden/potential benefits Efforts to minimize subject non-responseEfforts to minimize subject non-response

10

Applying These Principles Applying These Principles to Answer a Real Research to Answer a Real Research

Question:Question:

What is the association between PPI What is the association between PPI use and fracture risk in older men?use and fracture risk in older men?

Elaine Wu, UCSF medicine residentElaine Wu, UCSF medicine resident

11

PPI Use and PPI Use and OsteoporosisOsteoporosis

Proton pump inhibitors (PPIs) are commonly used to Proton pump inhibitors (PPIs) are commonly used to reduce gastric acid secretionreduce gastric acid secretion

Over 82 million PPI prescriptions written in 2004Over 82 million PPI prescriptions written in 2004 PPI over-the-counter since 2003PPI over-the-counter since 2003

A recent cross-over trial of 18 postmenopausal women A recent cross-over trial of 18 postmenopausal women found that short-term PPI use (omeprazole, 20 mg/d) found that short-term PPI use (omeprazole, 20 mg/d) was associated with a 41% reduction in intestinal was associated with a 41% reduction in intestinal absorption of calcium (O’Connell, 2005)absorption of calcium (O’Connell, 2005)

MrOS: prospective study of risk factors for MrOS: prospective study of risk factors for osteoporosis among 5,595 men >65osteoporosis among 5,595 men >65

12

Older men

PPI

Fracture

MrOS StudyMrOS Study

13

MrOS Study: SubjectsMrOS Study: Subjects

All older men in U.S.

Ambulatory men >65 at 6 US

clinical centers

14

MrOS Study: Inclusion MrOS Study: Inclusion CriteriaCriteria

Sampling FrameSampling Frame: All ambulatory, : All ambulatory, independently living menindependently living men

Inclusion criteriaInclusion criteria:: DemographyDemography: : >>65 years, all race/ethnic 65 years, all race/ethnic

groupsgroups Geography/AdministrativeGeography/Administrative: Recruited : Recruited

from population-based listing (e.g. DMV from population-based listing (e.g. DMV records) at 6 US clinical centersrecords) at 6 US clinical centers

Time PeriodTime Period: Enrolled 2000-2006, one : Enrolled 2000-2006, one follow-up visit after 4.6 yrfollow-up visit after 4.6 yr

15

MrOS Study: Exclusion MrOS Study: Exclusion CriteriaCriteria

Exclusion criteria:Exclusion criteria: Bilateral hip replacement (precludes Bilateral hip replacement (precludes

DXA)DXA) Life-threatening malignancy, dementiaLife-threatening malignancy, dementia Planning to move within 2 yearsPlanning to move within 2 years

16

MrOS: Baseline MrOS: Baseline CharacteristicsCharacteristics

Mean age (yr)Mean age (yr) 73.5 73.5BMIBMI 27.327.3Dietary calcium (mg/d)Dietary calcium (mg/d) 797 797Calcium supplementsCalcium supplements 35 %35 %HypertensionHypertension 50 %50 %Diabetes mellitusDiabetes mellitus 18 %18 %Hip BMD (g/cmHip BMD (g/cm22)) 0.9580.958PPI use at baselinePPI use at baseline 8%8%PPI use at follow-upPPI use at follow-up 15%15%

17

Making the Making the Measurements:Measurements:

Implications for Exposure Implications for Exposure & Outcome& Outcome Variable Variable

ChoicesChoices

18

““The most elegant The most elegant design of a clinical design of a clinical

study will study will notnot overcome the overcome the

damage caused by damage caused by unreliable or unreliable or

imprecise imprecise measurement.”measurement.”J.L. Fleiss (1986)J.L. Fleiss (1986)

Fleiss, JL. The design and analysis of clinical experiments. pp. 1-5. 1986. John Wiley and Sons, New York.

19

““Accuracy must be balanced Accuracy must be balanced against practical against practical considerations, and that considerations, and that method chosen which will method chosen which will provide the provide the maximal accuracy maximal accuracy within the bounds of the within the bounds of the investigator’s resources and investigator’s resources and other practical limitationsother practical limitations.”.”J.H. Abramson (1984)J.H. Abramson (1984)

Abramson, JH. Survey methods in community medicine (3rd Ed.), p. 121. 1984. Churchill Livingstone, Edinburgh.

20

PredictorPredictor** OutcomeOutcome

Confounding Confounding VariablesVariables**

Effect ModifiersEffect Modifiers**

Planning the Planning the MeasurementsMeasurements

*Often generally categorized as “exposures”

(interaction)

21

Additional “Exposure” Additional “Exposure” ConsiderationsConsiderations

Dose IssuesDose Issues Cumulative Cumulative

exposureexposure Exposure rateExposure rate

Time IssuesTime Issues Start of exposureStart of exposure When it endedWhen it ended Exposure Exposure

distributiondistribution

PPI UsePPI Use Total # of pillsTotal # of pills # pills/day# pills/day

Date of first useDate of first use Date of last useDate of last use Daily vs. Daily vs.

intermittentintermittentMrOS did not collect medication dose or duration of use

22

General Variable TypesGeneral Variable Types

ContinuousContinuous Quantitative Quantitative

intervals with intervals with typical rankingtypical ranking

Examples:Examples: Cholesterol levelCholesterol level Number of drinksNumber of drinks Day supply of drugDay supply of drug Waist sizeWaist size BMDBMD

CategoricalCategorical Dichotomous (yes/no) Dichotomous (yes/no)

(e.g., death, fracture, DM)(e.g., death, fracture, DM)

Nominal (no order) Nominal (no order) (e.g., ethnicity, (e.g., ethnicity, occupation)occupation)

Ordinal (ordered rank) Ordinal (ordered rank) (e.g., NYHA HF Class I-(e.g., NYHA HF Class I-IV)IV)

23

Typical Data SourcesTypical Data Sources

Survey/Survey/questionnairequestionnaire

Interviews Interviews DiariesDiaries Direct Direct

observationobservation Environmental Environmental

measurementsmeasurements

Databases/registriesDatabases/registries Medical recordsMedical records Physiologic Physiologic

measuresmeasures Biomarkers Biomarkers (e.g., DNA, (e.g., DNA,

sera)sera)

Imaging testsImaging tests PathologyPathology

Goal: choose the source that gives Goal: choose the source that gives data closest to the “gold standard” data closest to the “gold standard”

while being feasible to collectwhile being feasible to collect

24

General Measurement General Measurement Goals…Goals…

You get the same result when You get the same result when measured repeatedly (i.e., measured repeatedly (i.e., maximize maximize precisionprecision)—within the same subject, )—within the same subject, between subjects, and over timebetween subjects, and over time

It represents what it’s really supposed It represents what it’s really supposed to be (i.e., to be (i.e., maximize accuracy/validitymaximize accuracy/validity) ) + high sensitivity & specificity+ high sensitivity & specificity

25

The Measurement The Measurement SpectrumSpectrum

After deciding the exposure/outcome After deciding the exposure/outcome of interest, “measurement” includes:of interest, “measurement” includes: Preparing written instructions for Preparing written instructions for

applying the method for measuring the applying the method for measuring the variablevariable

Actually doing the measurement method Actually doing the measurement method itselfitself

Delineation of steps of preparing the Delineation of steps of preparing the collected data for analysiscollected data for analysis

Implementing quality control procedures Implementing quality control procedures throughout throughout (i.e., making sure you get what you meant (i.e., making sure you get what you meant to get)to get)

26

Improving Precision and Improving Precision and Accuracy of Variables & Accuracy of Variables &

Reducing BiasReducing Bias Standardize methodsStandardize methods Pretest, pretest, Pretest, pretest,

pretestpretest Refine/automate Refine/automate

instrumentinstrument Train & evaluate Train & evaluate

staffstaff Timely editing, Timely editing,

coding & correcting coding & correcting of formsof forms

Multiple Multiple measurementsmeasurements

Use or validate against Use or validate against “gold standard” “gold standard”

Less obtrusive Less obtrusive measuresmeasures

For outcomes, blinding For outcomes, blinding to exposure statusto exposure status

Institute quality Institute quality control measures control measures during data collection, during data collection, processing, and processing, and analysisanalysis

27

Applying These Principles Applying These Principles to Answer a Real to Answer a Real

Research Question:Research Question:

What is the association between What is the association between PPI use and fracture risk in older PPI use and fracture risk in older

men?men?

28

MrOS Study: MrOS Study: MeasurementsMeasurements

All older men in U.S.

Ambulatory men recruited to MrOS

PPI

Fracture

-Baseline and F/U PPI use

-Documented non-spine facture

29

PredictorPredictor OutcomeOutcome

Confounding Confounding VariablesVariables

Effect ModifiersEffect Modifiers

MrOS MeasurementsMrOS Measurements

-Demographic features-Osteoporosis risk factors-Other medication use

(calcium intake)

30

Exposure: PPI UseExposure: PPI Use PPI Use (main predictor)PPI Use (main predictor)

Baseline “PPI useBaseline “PPI use”—”— Self-reported use (any) in preceding 1 Self-reported use (any) in preceding 1

monthmonth Longitudinal “PPI useLongitudinal “PPI use”—time-”—time-

dependent exposure based on PPI at dependent exposure based on PPI at follow-up visitfollow-up visit

Validation study of method for Validation study of method for baseline usebaseline use Chart review of random sample of Chart review of random sample of

“users” & “non-users”: 94% raw “users” & “non-users”: 94% raw agreement (agreement (=0.90)=0.90)

31

Outcome: FractureOutcome: Fracture Non-spine fracture (primary outcome)Non-spine fracture (primary outcome)

Identification methodIdentification method: self-report: self-report Post card reporting of falls and fracture every 4 Post card reporting of falls and fracture every 4

months months Validation method:Validation method: masked review and masked review and

adjudication of objective documentationadjudication of objective documentation Obtain pre-operative radiology reports, medical Obtain pre-operative radiology reports, medical

recordsrecords Central MD and/or radiologist reviewCentral MD and/or radiologist review Obtain original radiographs if neededObtain original radiographs if needed Questionable or uncertain events reviewed by Questionable or uncertain events reviewed by

Endpoints CommitteeEndpoints Committee

Hip BMD using DXA (secondary Hip BMD using DXA (secondary outcome)outcome)

32

What Did We Find?What Did We Find? Among 5,595 ambulatory men, Among 5,595 ambulatory men,

longitudinal use of PPI therapy was longitudinal use of PPI therapy was associated with…associated with… Lower hip BMD among PPI users (0.946 vs. Lower hip BMD among PPI users (0.946 vs.

0.958, p=0.04)0.958, p=0.04) Trend towards increased fracture risk among Trend towards increased fracture risk among

PPI users (RH=1.26, 95% CI: 0.88, 1.75)PPI users (RH=1.26, 95% CI: 0.88, 1.75) Significant increase in fracture risk Significant increase in fracture risk

(RH=1.78, CI: 1.16, 2.73) among men with (RH=1.78, CI: 1.16, 2.73) among men with low calcium intake (interaction between low calcium intake (interaction between calcium intake and effect of PPI use on calcium intake and effect of PPI use on fracture risk)fracture risk)

33

Not True, Questions Not True, Questions Welcome!Welcome!