For the Practice Change Fellows ProgramSeptember 28, 2007

Washington, DC

Dennis A. Ehrich, MD, FACCVice President for Medical Affairs

St. Joseph’s Hospital Health CenterSyracuse, New York

Measurement for Improvement

Agenda for the Morning1-Why Do We Measure in Health care?2-The Model for Improvement3-Selecting one’s measures7-Time ordered statistics and understanding

variation 8-The run chart10-Practical exercises11-The control chart (if time permits)

Why We Measure in Health Care

Measuring for Research

Measuring for Judgment

Measuring for Improvement

Purpose To discover new knowledge

To compare to others, to rank

To bring new knowledge into daily

practice

Tests One large trial Public reporting quarterly or with

12 month running averages

Many sequential, observable tests

Biases Control for as many as possible

Severity or risk adjustment where

available

Stabilize the biases from test to test

Data Gather as much data as possible,

just in case

Reports structure, process or outcomes

Small tests of significant changes, accelerates the rate

of improvement

Duration Can require large numbers of

patients and long periods of time to obtain

results

Ongoing data collection and

quarterly public reporting

Short iterative cycles in a limited number of subjects, followed by

spread

Set aims that are measurable, time-specific, and apply to a defined population

The Model for Improvement

Establish measures to determine if a specific change leads to improvement

Select changes most likely to result in improvement

Test the changes

T. Nolan et al. www.ihi.org

The Use of Iterative PDSA Cycles

Implementing the Changes

“Rapid-cycle CQI”

T. Nolan et al. www.ihi.org

Spreading the Change

1-Planning and set-up 2-Spread within the target population 3-Continuous monitoring and feedback on the spread process  

T. Nolan et al. www.ihi.org

Donabedian’s Quality Triangle-It’s Relevance to Process Improvement

-Avedis Donabedian, MD, MPH (1919-2000)

Donabedian’s TriadStructure

OrganizationPeopleEquipment/Technology

ProcessThe actual steps taken in accomplishing the

changeResults must be client-focusedMust deliver results reliably

OutcomesClinicalClient perception or satisfactionFinancial

Selecting Your Measures

The Three Domains of MeasurementStructural MeasuresProcess measuresOutcomes Measures

Balancing measures

Donabedian

The Three Domains of MeasurementStructural Measures

Describe the environment. How many?Square footage of a clinical unitNumber of staffStaff qualifications and competenciesPresence or absence of technology and its

characteristicsProcess Measures

Process cycle timeThe percentage of patients for whom the process

achieves its desired result

Donabedian

The Three Domains of MeasurementOutcome Measures

The impact of the change initiative on mortality, readmissions to the hospital, ED visits

The satisfaction scores of clients and staff The cost per case, average LOS, revenue per case

Balancing Measures Unintended outcomes that are consequences

of the new programUnanticipated mortality, morbidity or cost Has the shifting of resources in an

organization compromised other client or patient populations?

Donabedian

ACTION

Aim

Selecting A Measure

Operational Definitions

Data Collection Plan

Data Collection

Data Analysis

The Quality Measurement

Roadmap

Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”

Selecting a Measure:

-When selecting a measure, have clarity as to whether the measure is one of structure, process or outcome

-And select a balanced panel of indicators that reflect the dimensions of performance being evaluated and the change concept(s) being employed

What Dimension of Performance do You Want to Measure?Appropriateness AvailabilityContinuityEffectivenessEfficiencyRespect and caringFinancial/ViabilitySafetyTime lines

Joint Commission (1996)

What Dimension of Performance do You Want to Measure?SafetyEffectivenessPatient-centerednessTimelinessEfficiencyEquity

IOM: Crossing the Quality Chasm (2001)

What is the “Change Concept”?Eliminate wasteImprove work flowShorten a waiting listChange the work environmentImprove the Provider/Client interfaceManage timeFocus on variationError proofing a processFocusing on product or service

The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass

Relating a Change Concept to a Specific Measure

Concept Potential Indicators for this processPatient scheduling •The average number of days between the call

for an appointment and the actual appointment date•The percentage of appointments made within 3 days of the call for an appointment•The number of appointments scheduled each day

Home care visits •The number of home care visits•The average time spent during a home care visit•The percentage of time spent traveling during each home care visit•The number of visits per home care nurse

CQI Training •The number of participants attending a class•The percentage of cancellations•The percentage of no-shows•The information recall scores at 30 and 60 days

Establishing Operational Definitions That Are Agreed Upon By All Stakeholders

Operational DefinitionsIs clear and unambiguousSpecifies the measurement method,

procedures and equipment when appropriateClinical data (chart reviews) vs. administrative dataClient logs vs. a computer database

Define specific criteria for the data to be collectedDefine all inclusions and exclusionsFor percentages or rates, or ratios, define the criteria

for inclusion in the numerator and denominatorAlways ask “How might somebody be confused

by this definition?”

Lloyd, R. Quality Health Care (2004) Jones and Bartlett

Examples of Unclear DefinitionsTimely completion of the screening processA complete medication listThe readmission rateMedication errorCost impactFrom the acute care hospital

A patient fallSurgical start time

Lloyd, R. Quality Health Care (2004) Jones and Bartlett

Data Analysis What descriptive statistics will be used?

Mean, median, modeMinimum, maximum, range, standard

deviationQuantities, proportions (percentages), rates

How will data be displayed?Bar chart, histogram, line chart, pie chart, Pareto

diagramRun chart, control chart

External Benchmarking

Joint Commission

CMS

Data ReportingData reporting plan

Who will receive the resultsHow often will they receive the resultsHow will the data be disseminated?

Printed reportsEmailDashboardSpider diagram

Displaying Time-Ordered Statistics and Understanding Variation

Tools for Displaying Time-ordered DataRun charts

Plot of data over time with the median of the data set plotted as a center line

Control chartsPlot of data over time with the mean as the

center line and with upper and lower control limits

Run ChartsEasily constructed by hand or in available

spreadsheet programsProvides a good idea of improvement in a

change initiativeLess sensitive to significant changes

(special cause variation) than the control chart

Control Charts More sensitive to special cause variation

than a run chart Requires specialized computer software to

create There are 9 types of control charts used in

health care, depending upon whether the data collected is distributed normally, is continuous (numerical) or discreet (attributes) and whether the events measured are frequent or infrequent

Have their own set of rules to identify special cause variation

Understanding VariationAll data, collected over time, variesRandom variation (common cause)

The changes occurring are intrinsic to the process being measured

Non-random variation (special cause)The changes are being imposed on the

system by some external factorMay be unintended and un anticipated or

may be by design

Common Cause (Random) Variation

Special Cause Variation

Creating a Run Chart

Hand-Drawing a Run Chart

Plot data points as a line graph on x-y axes, where “x” is the increment of time and “y” is the measurement.

Calculate the median value of the data set and draw that line on the chart Sort the data from smallest to largest value Count the data points. That count=N If “N” is an odd number: Median=N+1/2. Begin

counting from smallest to the largest number. When the count reaches N+1/2, that number is the median

If “N” is an even number: Median=The average of N/2 and the next number in the series. Begin counting from smallest to the largest number. When the count reaches N/2, stop and take the average of that number and the next number in the series. That average is the median

Calculating the Median

Odd Number of Data Points 1 N=722371112

Median=N+1/2=7+1/2=4The median is the 4th number in the series, which is 3

Even Number of Data Points1 N=634558

Median=The avg. of the number that is N/2 and the next number in the series. =[4 (the third number in the series) +5 (the next number in the series)] / 2=4.5

Balestracci, D., and Barlow, J, Quality Improvement. 1998 Center for Research in Ambulatory Health Care Administration

Definitions A run is 1 or more consecutive data points on the same

side of the median line A useful observation is one that does not fall on the

median line

•Sixteen of the eighteen observations are useful•There are 10 runs on this run chart

Testing for Special Cause Variation on a Run ChartTest 1. Are any runs longer than expected? If

so, then that run represents a special cause.If there are fewer than 20 useful

observations, then 7 or more data points in a run indicate a special cause.

If there are 20 or more useful observations, then 8 or more data points in a run indicate a special cause.

Testing for Special Cause Variation on a Run ChartTest 2. Is there a trend? A trend is an

excessively long series of consecutive increases or decreases in the data.

Total Number of Data Points on the Chart

Number of ConsecutiveAscending or Descending Points Indicating a Special

Cause

5 to 89 to 20

21 to 100

5 or more6 or more7 or more

Applying Tests 1 and 2

Total number of data points=18 Number of useful observations=16

Test 1-Since there are < 20 useful observations it will take ≥ 7 data points in

a run to cause a run to be “too long” defining special cause variation

Test 2-Is there a trend? For 18 total data points, it will take ≥6 consecutive

ascending or descending data points to define a trend.

Testing for Special Cause Variation on a Run ChartTest 3. Are there too few or too many runs

in the data?Determine the number of useful observations

in your data set. Use the following table to determine whether

the number of runs in your data are within the expected range. If the number of runs is above or below the expected range, the data suggest special cause variation

Useful Observatio

ns

Lower

Limit

Upper

Limit

1516171819202122232425262728

455666778899910

1212131314151516161717181919

UsefulObservatio

ns

Lower

Limit

Upper Limit

29303132333435 3637383940

101111111112131313141415

202021222223232425252626

Expected Number of Runs

Applying Test 3

Are there too many runs? Useful observations= 24. Number of runs= 8. Expected number of runs = 8-17. Therefore there is no evidence for special cause variation.

Testing for Special Cause Variation on a Run ChartTest 4. Fourteen or more points alternating above

and below the median line is a saw tooth pattern indicate a special cause.

When this pattern is seen, it indicates either that two different processes are operating at the same time and have been measured together; in which case stratification of the data would be helpful. Or, it may indicate tampering

KQC=Key Quality Characteristic