1

Evaluating Health Information Evaluating Health Information Technology: A PrimerTechnology: A Primer

Eric Poon, MD MPHEric Poon, MD MPHClinical Informatics Research and Development,Clinical Informatics Research and Development,

Partners Information SystemsPartners Information Systems

Davis Bu, MD MADavis Bu, MD MACenter for Information Technology Leadership,Center for Information Technology Leadership,

Partners Information SystemsPartners Information Systems

AHRQ National Resource Center for Health AHRQ National Resource Center for Health Information TechnologyInformation Technology

2

Pre-Conference Logistics Pre-Conference Logistics

To Access Slides:To Access Slides: Go to http://extranet.ahrq.gov/rcGo to http://extranet.ahrq.gov/rc

Login with username and passwordLogin with username and password

Follow the links to download slidesFollow the links to download slides

Problems? Email ResourceCenter@norc.org Problems? Email ResourceCenter@norc.org

Q&A Session at the End Q&A Session at the End Dial *1 to ask a questionDial *1 to ask a question

Please pick up handset (not speakerphone)Please pick up handset (not speakerphone)

Note that this teleconference is being recordedNote that this teleconference is being recorded

3

OutlineOutline Why evaluate?Why evaluate?

General Approach to EvaluationGeneral Approach to Evaluation

Deciding what to MeasureDeciding what to Measure

Study Design TypesStudy Design Types

Analytical issues in HIT evaluationsAnalytical issues in HIT evaluations

Some practical advice on specific Some practical advice on specific evaluation techniquesevaluation techniques

4

Why Measure Impact of HIT?Why Measure Impact of HIT?

Impact of HIT often hard to predict Impact of HIT often hard to predict Many “slam dunks” go awryMany “slam dunks” go awry

Understand how to clear barriers to effective Understand how to clear barriers to effective implementationimplementation Understand what works and what doesn’tUnderstand what works and what doesn’t

Justify enormous investmentsJustify enormous investments Return on investmentReturn on investment Allow other institutions to make tradeoffs intelligentlyAllow other institutions to make tradeoffs intelligently

Use results to win over late adoptersUse results to win over late adopters You can’t manage/improve what isn’t measuredYou can’t manage/improve what isn’t measured Good publicity for organizationGood publicity for organization

5

General Approach to Evaluating HITGeneral Approach to Evaluating HIT

Understand your interventionUnderstand your intervention

Select meaningful measuresSelect meaningful measures

Pick the study designPick the study design

Validate data collection methodsValidate data collection methods

Data analysisData analysis

6

Getting Started: Get to know your Getting Started: Get to know your interventionintervention

Clarify the question: What problem does it Clarify the question: What problem does it address?address?

Think about intermediate processesThink about intermediate processes

Identify potential barriers to successful Identify potential barriers to successful implementationimplementation

Identify potential managerial and Identify potential managerial and behavioral process to overcome behavioral process to overcome implementation barriersimplementation barriers

7

Array of MeasuresArray of Measures

Quality and SafetyQuality and Safety Clinical OutcomesClinical Outcomes Clinical ProcessesClinical Processes

KnowledgeKnowledge Patient knowledgePatient knowledge Provider knowledgeProvider knowledge

SatisfactionSatisfaction Patient satisfactionPatient satisfaction Provider satisfactionProvider satisfaction

Resource utilizationResource utilization Costs and chargesCosts and charges LOSLOS Employee time/workflowEmployee time/workflow

8

Introducing the Evaluation ToolkitIntroducing the Evaluation Toolkit

Rough guides on general approach, costs and Rough guides on general approach, costs and potential pitfallspotential pitfalls

Major domains:Major domains: Clinical OutcomesClinical Outcomes Clinical ProcessClinical Process Provider Adoption & Attitudes Provider Adoption & Attitudes

Measure Characteristics:Measure Characteristics: IOM DomainIOM Domain Data SourceData Source Relative CostRelative Cost

Would love to hear your feedbackWould love to hear your feedback

Patient Knowledge & AttitudesPatient Knowledge & Attitudes Workflow ImpactWorkflow Impact Financial ImpactFinancial Impact

Potential PitfallsPotential Pitfalls General NotesGeneral Notes

9

Selecting Evaluation Measures Selecting Evaluation Measures for HIT: for HIT:

Three ExamplesThree Examples

10

Computerized Provider Order Entry Computerized Provider Order Entry (CPOE) Example(CPOE) Example

Clarify the primary question:Clarify the primary question: Does CPOE improve quality of care?Does CPOE improve quality of care?

Competing questions:Competing questions: Does CPOE save money?Does CPOE save money? What are the barriers to physician What are the barriers to physician

acceptance?acceptance? Does CPOE introduce new errors?Does CPOE introduce new errors?

11

CPOE: How can it affect quality?CPOE: How can it affect quality?

Think about intermediate processesThink about intermediate processes Patient data is presented to ordering physicianPatient data is presented to ordering physician ADE alerts may be triggered and presented at the ADE alerts may be triggered and presented at the

point of care (which alerts?)point of care (which alerts?) Guideline reminders may be triggered an presented at Guideline reminders may be triggered an presented at

the point of care (which guidelines?)the point of care (which guidelines?) Medication order is enteredMedication order is entered Medication order is executed by pharmacyMedication order is executed by pharmacy Medication order is executed by nursing staffMedication order is executed by nursing staff

12

Does CPOE Improve Quality of Care?Does CPOE Improve Quality of Care?

Identify measuresIdentify measures

Process MeasureData Presentation (Redundant test ordering)

ADE Alert Alert frequency, ADE frequency

Guideline Reminder

Guideline compliance, clinical outcome

Order Entry Ordering errors

Pharmacy (Time to process order)

Administration Time to administration

13

Evaluating CPOE’s Impact on QualityEvaluating CPOE’s Impact on Quality

Select Appropriate MethodologySelect Appropriate Methodology Does existing data exist that can be Does existing data exist that can be

leveraged? (e.g. ongoing QA activities)leveraged? (e.g. ongoing QA activities) Does concurrent control exist?Does concurrent control exist? How will the data be analyzed?How will the data be analyzed?

14

Electronic Medical Records (EMR) Electronic Medical Records (EMR) ExampleExample

Clarify the primary question:Clarify the primary question: What are the barriers and facilitators to What are the barriers and facilitators to

effective EMR implementation?effective EMR implementation?

Competing questions:Competing questions: Do EMRs save money?Do EMRs save money? Do EMRs improve quality of care?Do EMRs improve quality of care? Do EMRs introduce new errors?Do EMRs introduce new errors?

15

EMR: Dissecting the EMR EMR: Dissecting the EMR Implementation ProcessImplementation Process

Identify stakeholdersIdentify stakeholders Providers, et al.Providers, et al.

Catalogue stakeholder interests and valuesCatalogue stakeholder interests and values Workflow efficiencyWorkflow efficiency

Clarify stakeholder role in implementationClarify stakeholder role in implementation Users of system, clinical leaders, administrative leadersUsers of system, clinical leaders, administrative leaders

Clarify impact of Implementation on clinical Clarify impact of Implementation on clinical processesprocesses User interface optimization, workflow re-engineeringUser interface optimization, workflow re-engineering

Define implementation success criteriaDefine implementation success criteria Provider buy-in, provider use and acceptanceProvider buy-in, provider use and acceptance

16

EMR: Understanding the Barriers EMR: Understanding the Barriers and Facilitators to Implementationand Facilitators to Implementation

Identify measuresIdentify measuresProcess MeasureStakeholder attitudes

Attitude/Satisfaction surveys Readiness Assessment Staff Turnover

Workflow Efficiency metrics

Process improvements

Staffing levels Patient flow Practice productivity Implementation participation from staff

Success Criteria Usage data Training attendance Measures listed above

17

EMR: Understanding the Barriers EMR: Understanding the Barriers and Facilitator to Implementationand Facilitator to Implementation

Select Appropriate MethodologySelect Appropriate Methodology Combination of quantitative and qualitative Combination of quantitative and qualitative

studiesstudies Example: efficiency measures:Example: efficiency measures:

Time motion studies: how did the system affect provider Time motion studies: how did the system affect provider efficiency?efficiency?

Attitude Surveys: How did the system affect provider Attitude Surveys: How did the system affect provider perception of efficiency?perception of efficiency?

Semi-structured interviews: How did the implementation Semi-structured interviews: How did the implementation affect stakeholder workflow? Did that effect change over time affect stakeholder workflow? Did that effect change over time and why?and why?

18

Local Health Information Local Health Information Infrastructure (Laboratory)Infrastructure (Laboratory)

Clarify the primary questionClarify the primary question Can LHIIs for labs generate a positive ROI?Can LHIIs for labs generate a positive ROI?

Competing questions:Competing questions: Can LHIIs for labs improve quality of care?Can LHIIs for labs improve quality of care? Which architecture is best suited for LHIIs for Which architecture is best suited for LHIIs for

labs?labs? How do LHIIs for labs affect provider and How do LHIIs for labs affect provider and

patient perception of the health care system?patient perception of the health care system?

19

LHII (Laboratory): Defining the ROILHII (Laboratory): Defining the ROI

Specify intermediate processesSpecify intermediate processes Data is pulled from local laboratoriesData is pulled from local laboratories

Previous labs pulledPrevious labs pulled Lab order enteredLab order entered Lab order transmittedLab order transmitted Administrative handlingAdministrative handling Lab results reportedLab results reported Lab results recordedLab results recorded

Data is pulled from primary providerData is pulled from primary provider Authorization and payment is coordinated with payerAuthorization and payment is coordinated with payer Implementation of LHIOImplementation of LHIO

20

LHII (Laboratory): Defining the ROILHII (Laboratory): Defining the ROI

Identify associated measuresIdentify associated measures

Process MeasureProvider requests data

Volume of requests

Data is pulled Chart pulls, time for chart pulls, administrative costs

Provider interprets data

Amount of missing information

Provider Orders test Volume of redundant tests

21

LHII (Laboratory): Evaluating the ROILHII (Laboratory): Evaluating the ROI

Select Appropriate MethodologySelect Appropriate Methodology Does concurrent control exist?Does concurrent control exist? Are there ongoing trends over time?Are there ongoing trends over time? How will the data be analyzed?How will the data be analyzed?

22

Selecting Outcome Measures: Selecting Outcome Measures: General CommentsGeneral Comments

Generally want to pick 1-3 outcomes of primary interestGenerally want to pick 1-3 outcomes of primary interest If choose more, need to make correction (e.g. Bonferroni)If choose more, need to make correction (e.g. Bonferroni)

Outcome must be sufficiently frequent to be detectableOutcome must be sufficiently frequent to be detectable Rare events such as adverse events due to errors particularly Rare events such as adverse events due to errors particularly

challengingchallenging

Important enough to provoke interestImportant enough to provoke interest Whether study is positive or negativeWhether study is positive or negative How would the results change policy (local or national)?How would the results change policy (local or national)?

Process vs. outcomeProcess vs. outcome Legitimate to measure processLegitimate to measure process

Outcome often takes too longOutcome often takes too long In many situations link between process, outcome clearIn many situations link between process, outcome clear

23

Study TypesStudy Types

Commonly used study types:Commonly used study types: Before-and-after time series TrialsBefore-and-after time series Trials Randomized Controlled TrialsRandomized Controlled Trials Factorial DesignFactorial Design

Study design often influenced by Study design often influenced by implementation planimplementation plan

24

Time Series vs. Randomized Time Series vs. Randomized Controlled TrialsControlled Trials

Before-and-after trial common in informaticsBefore-and-after trial common in informatics Concurrent randomization is hardConcurrent randomization is hard Don’t lose the opportunity to collect baseline data!Don’t lose the opportunity to collect baseline data!

Off-On-Off trial design possibleOff-On-Off trial design possible But may not be politically/ethically acceptable to turn But may not be politically/ethically acceptable to turn

off a highly used featureoff a highly used feature

RCT preferable if feasibleRCT preferable if feasible Eliminates the issue of secular trendEliminates the issue of secular trend Balance of baseline confoundingBalance of baseline confounding

25

Randomization ConsiderationsRandomization Considerations

Justifiable to have a control arm as long as Justifiable to have a control arm as long as benefit not already demonstrated (usual care)benefit not already demonstrated (usual care)

Want to choose a truly random variable Want to choose a truly random variable Not day of the weekNot day of the week Legitimate to stratify on baseline variables (e.g. Legitimate to stratify on baseline variables (e.g.

education for pt, computer experience for providers)education for pt, computer experience for providers) Minimal number of armsMinimal number of arms

More arms, less powerMore arms, less power Strongest possible interventionStrongest possible intervention

26

Unit of RandomizationUnit of Randomization

PatientsPatients PhysiciansPhysicians Practices/wardsPractices/wards

27

Randomization Unit:Randomization Unit:How to Decide?How to Decide?

Small units (patients) vs. Large units (practices Small units (patients) vs. Large units (practices wards)wards) Contamination across randomization unitsContamination across randomization units If risk of contamination is significant, consider If risk of contamination is significant, consider

larger unitslarger units Effect contamination-can underestimate impactEffect contamination-can underestimate impact

However, if you see a difference, impact is presentHowever, if you see a difference, impact is present

Randomization by patient generally undesirableRandomization by patient generally undesirable ContaminationContamination Ethical concernEthical concern

28

Randomization Schemes:Randomization Schemes:Simple RCTSimple RCT

Burn-in periodBurn-in period Give target population time to get used to new Give target population time to get used to new

intervention intervention Data not used in final analysisData not used in final analysis

XX Clinics

Baseline Period

Baseline Data Collection Data Collection for RCT

No Intervention

Intervention Period

3 month burn-in period

Intervention Deployed

Intervention

arm

Control

arm

Control arm gets intervention

Post- Intervention

Period

29

Randomization schemes: Randomization schemes: Factorial DesignFactorial Design

May be used to May be used to concurrently evaluate concurrently evaluate more than one more than one intervention:intervention: Assess interventions Assess interventions

independently and in independently and in combinationcombination

Loss of statistical powerLoss of statistical power

Usually not practical for Usually not practical for more than 2 interventionsmore than 2 interventions

Control (no interventions)

A

B

A+B

30

Randomization Schemes:Randomization Schemes:Staggered DeploymentStaggered Deployment

AdvantagesAdvantages Easier for user education and trainingEasier for user education and training Can fix IT problems up frontCan fix IT problems up front

Need to account for secular trendNeed to account for secular trend Time variable in regression analysisTime variable in regression analysis

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Pilot Practice B x Practice A-Intervention B x Practice B-Control B Practice C-Intervention B x Practice D-Control B Practice E-Intervention B x Practice F-Control B Practice G-Intervention B x Practice H-Control B

2003 2004 2005

Deployment of InterventionB Baseline Data Collectionx Burn-in period

31

Randomization Schemes:Randomization Schemes:Multiple InterventionsMultiple Interventions

Time efficient designTime efficient design Every clinic gets something. (Keeps clinics and IRB Every clinic gets something. (Keeps clinics and IRB

happy)happy) Watch out for cross-arm intervention contaminationWatch out for cross-arm intervention contamination

Arm 2

Arm 1

12clinics

6 clinics

6 clinics

18 mo

Randomize

Medication Tracking; Diabetes Care

Prev Care Reminders;Family History

Control for Arm 1

Control for Arm 2

4 Interventions involving patient’s use of shared online medical records:

• Medication Tracking• Diabetes Care• Prev. Care Reminders• Family History

32

Inherent Limitations of RCTs in Inherent Limitations of RCTs in InformaticsInformatics

Blinding is seldom possibleBlinding is seldom possible

Effect on documentation vs. clinical actionEffect on documentation vs. clinical action

People always question generalizabilityPeople always question generalizability Success is highly implementation independentSuccess is highly implementation independent Efficacy-effectiveness gap: ‘Invented here’ Efficacy-effectiveness gap: ‘Invented here’

effecteffect

33

Data CollectionData Collection

Electronic data abstractionElectronic data abstraction Convenient and time-saving, but…Convenient and time-saving, but…

Some chart review (selected) to get Some chart review (selected) to get information not available electronicallyinformation not available electronically

Get ready for nasty surprisesGet ready for nasty surprises

Pilot your data collection protocol earlyPilot your data collection protocol early And then pilot some more…And then pilot some more…

34

Data Collection Issue: Data Collection Issue: Baseline DifferencesBaseline Differences

Randomization schemes Randomization schemes often lead to imbalance often lead to imbalance between intervention and between intervention and control arms:control arms: Need to collect baseline Need to collect baseline

data and adjust for data and adjust for baseline differences baseline differences

Interaction term ( Time * Interaction term ( Time * Allocation Arm) gives Allocation Arm) gives effect for intervention in effect for intervention in regression analysisregression analysis

35

Data Collection Issue: Data Collection Issue: Completeness of FollowupCompleteness of Followup

The higher the better:The higher the better: Over 90%Over 90% 80-90%80-90% Less than 80%Less than 80%

Intention to treat analysisIntention to treat analysis In an RCT, should analyze outcomes In an RCT, should analyze outcomes

according to the original randomization according to the original randomization assignmentassignment

36

A Common Analytical Issue A Common Analytical Issue The Clustering EffectThe Clustering Effect

Occurs when your observations are not Occurs when your observations are not independent:independent: Example: Each physician treats multiple patients:Example: Each physician treats multiple patients:

Intervention Group Control Group

Physicians

Patient -> Outcome assessed

37

Options for Dealing with the Options for Dealing with the Clustering EffectClustering Effect

Analyze at the level of clinicianAnalyze at the level of clinician Example: Analyze % of MD’s patients in compliance with Example: Analyze % of MD’s patients in compliance with

guideline, and make MD unit of analysisguideline, and make MD unit of analysis Huge drop in statistical power.Huge drop in statistical power. Not recommended.Not recommended.

Generalized Estimating Equations Generalized Estimating Equations PROC GENMOD in SAS, or PROC RLOGIST in SUDAAN PROC GENMOD in SAS, or PROC RLOGIST in SUDAAN Allows you to randomize at one level (e.g. physician) and Allows you to randomize at one level (e.g. physician) and

then do analysis at another (e.g. patient) then do analysis at another (e.g. patient) Accounts for correlation of behaviors within a single physician (i.e. Accounts for correlation of behaviors within a single physician (i.e.

adjusts for the fact that observations across patients are NOT adjusts for the fact that observations across patients are NOT independent)independent)

38

A Word About SurveysA Word About Surveys

Survey of user believes, attitude and Survey of user believes, attitude and behaviorsbehaviors Response rate – responder bias: Aim for Response rate – responder bias: Aim for

response rate > 50-60%response rate > 50-60% Keep the survey conciseKeep the survey concise Pilot survey for readability and clarityPilot survey for readability and clarity Need formal validation if you want plan to Need formal validation if you want plan to

develop a scaledevelop a scale

39

Looking at Usage DataLooking at Usage Data

Great way to tell how well the intervention Great way to tell how well the intervention is goingis going Target your trouble-shooting effortsTarget your trouble-shooting efforts

In terms of evaluating HIT:In terms of evaluating HIT: Correlate usage to implementation/training Correlate usage to implementation/training

strategystrategy Correlate usage to stakeholder characteristicsCorrelate usage to stakeholder characteristics Correlate usage to improved outcomeCorrelate usage to improved outcome

40

Studies on Workflow and UsabilityStudies on Workflow and Usability

How to make observations?How to make observations? Direct observationsDirect observations Stimulated observationsStimulated observations

Random paging methodRandom paging method Subjects must be motivated and cooperativeSubjects must be motivated and cooperative

Usability LabUsability Lab What to look for?What to look for?

Time to accomplish specific tasks:Time to accomplish specific tasks: Need to pre-classify activitiesNeed to pre-classify activities Handheld/Tablet PC tools may be very helpfulHandheld/Tablet PC tools may be very helpful

Workflow analysisWorkflow analysis Asking users to ‘think aloud’Asking users to ‘think aloud’

Unintended consequences of HITUnintended consequences of HIT

41

Qualitative MethodologiesQualitative Methodologies

Major techniquesMajor techniques Direct observationsDirect observations Semi-structured interviewsSemi-structured interviews Focus groupsFocus groups

Adds richness to the evaluationAdds richness to the evaluation Explains successes and failures. Generate Lessons learnedExplains successes and failures. Generate Lessons learned Captures the unexpectedCaptures the unexpected Great for forming hypothesesGreat for forming hypotheses People love to hear storiesPeople love to hear stories

Data analysisData analysis Goal is to make sense of your observationsGoal is to make sense of your observations Iterative & interactiveIterative & interactive

42

Cost Benefit AnalysisCost Benefit Analysis

Cost DataCost Data Generally availableGenerally available Caveat: allocation of indirect costsCaveat: allocation of indirect costs

Financial Benefit DataFinancial Benefit Data Revenue EnhancementRevenue Enhancement Cost AvoidanceCost Avoidance

Benefit AllocationBenefit Allocation Benefits may accrue to multiple partiesBenefits may accrue to multiple parties Are benefits realizable (e.g. labor savings)?Are benefits realizable (e.g. labor savings)? Calculation of benefits to external parties may be of Calculation of benefits to external parties may be of

interest, even if it does not impact on ROIinterest, even if it does not impact on ROI

43

Cost Benefit AnalysisCost Benefit Analysis

Activity Based Costing ExampleActivity Based Costing Example Simply put, a method for assigning costs to particular activitiesSimply put, a method for assigning costs to particular activities Alternate method of assigning indirect costs to the projectAlternate method of assigning indirect costs to the project Also, may serve as a framework for capturing cost savingsAlso, may serve as a framework for capturing cost savings

Step* Example

Identify activities Paper chart maintenance

Determine cost for each activity Cost data for medical records

Determine cost drivers Number of chart pulls

Obtain activity data How many charts were pulled

Calculate total cost Savings from decreased pulls

* http://www.pitt.edu/~roztocki/abc/abctutor/

44

Concluding RemarksConcluding Remarks

Don’t bite off more than what you can chewDon’t bite off more than what you can chew Pick a few study outcomes and study them well. Pick a few study outcomes and study them well.

It’s a practical worldIt’s a practical world Balancing operational and research needs is Balancing operational and research needs is

always a challenge.always a challenge. Life (data collection) is like a box of Life (data collection) is like a box of

chocolates…chocolates… You don’t know what you’re going to get until you You don’t know what you’re going to get until you

look, so look early!look, so look early!

45

Thank youThank you

Eric Poon, MD MPHEric Poon, MD MPH Email: Email: epoon@partners.orgepoon@partners.org

Davis Bu, MD MADavis Bu, MD MA Email: Email: dbu@partners.orgdbu@partners.org

46

Give Us Feedback! Give Us Feedback!

We are eager to hear your feedback!We are eager to hear your feedback!

Go to http://extranet.ahrq.gov/rcGo to http://extranet.ahrq.gov/rc

Login with username and passwordLogin with username and password

Follow the links to provide feedback-thanks!Follow the links to provide feedback-thanks!

Want to hear this teleconference again?Want to hear this teleconference again?

Dial 1-800-486-4195Dial 1-800-486-4195 to replay until 5/4/05 to replay until 5/4/05