expanding research and evaluation designs…for qii carolyn m. clancy, md director, ahrq september...
TRANSCRIPT
Expanding Research and Expanding Research and Evaluation Designs…for QIIEvaluation Designs…for QII
Carolyn M. Clancy, MDCarolyn M. Clancy, MD
Director, AHRQDirector, AHRQ
September 13, 2005September 13, 2005
Are We Making Any Are We Making Any Progress?Progress?
Second annual reports focus on quality of Second annual reports focus on quality of and disparities in health care in Americaand disparities in health care in America
–Quality ReportQuality Report finds finds that quality is improving that quality is improving and identifies areas and identifies areas which are in need of which are in need of major improvementsmajor improvements
–Disparities ReportDisparities Report indicates that there are indicates that there are pervasive disparities pervasive disparities related to race, related to race, ethnicity, and ethnicity, and socioeconomic statussocioeconomic status
Preliminary NHQR Highlights Preliminary NHQR Highlights 20052005
Health Care Quality Continues To Improve at a Health Care Quality Continues To Improve at a Modest Pace Across Most Measures of QualityModest Pace Across Most Measures of Quality– 10:1 ratio of measures improved to declined10:1 ratio of measures improved to declined
– Overall improvement rate: 2.8%Overall improvement rate: 2.8%
Health Care Quality Improvement Is Variable, With Health Care Quality Improvement Is Variable, With Notable Areas of High Performance Notable Areas of High Performance – Patient safety: 10.2%Patient safety: 10.2%
– QIO Measures: 9.2%QIO Measures: 9.2%
– Effectiveness measures: 2.8%Effectiveness measures: 2.8% High improvement subset: 5.4%High improvement subset: 5.4%
RAND Study: Quality of Health RAND Study: Quality of Health Care Often Not OptimalCare Often Not Optimal
Doctors provide appropriate health Doctors provide appropriate health care only about half the timecare only about half the time
Percentage of timePercentage of time
Alcohol dependenceAlcohol dependenceHip fractureHip fracturePeptic ulcerPeptic ulcer
DiabetesDiabetesLow back painLow back painPrenatal carePrenatal careBreast cancerBreast cancer
CataractsCataracts
11%11%23%23%
33%33%45%45%
69%69%73%73%76%76%79%79%
E. McGlynn, S. Asch, J. Adams, et al., The Quality of Health Care Delivered E. McGlynn, S. Asch, J. Adams, et al., The Quality of Health Care Delivered to Adults in the United States, to Adults in the United States, N Engl J MedN Engl J Med, 2003, 2003
55%
44%
Public Perceptions
2004 2000*
Percent who say they are dissatisfied with the quality of health care in this country…
4%
40%
38%
17%
Has the quality of health care in this country…
Gotten worse
Gotten better
Stayed about the same
Don’t Know
Source: Kaiser Family Foundation / Agency for Healthcare Research and Quality / Harvard School of Public Health Source: Kaiser Family Foundation / Agency for Healthcare Research and Quality / Harvard School of Public Health National Survey on Consumers’ National Survey on Consumers’ Experiences with Patient Safety and Quality Information, Experiences with Patient Safety and Quality Information, November 2004 (Conducted July 7 – September 5, 2005).November 2004 (Conducted July 7 – September 5, 2005).
* * Gallup Poll conducted September 11-13, 2000 with 1,008 U.S. adults.Gallup Poll conducted September 11-13, 2000 with 1,008 U.S. adults.
AHRQ’s Mission AHRQ’s Mission
To improve the safety and To improve the safety and quality, efficiency and quality, efficiency and
effectiveness of health care effectiveness of health care for all Americansfor all Americans
Answers must be:Answers must be:
TimelyTimely ValidValid ConvincingConvincing PracticalPractical
Publication
Bibliographic databases
Submission
Reviews, guidelines, textbook
Negative results
variable
0.3 year
6. 0 - 13.0 years50%
46%
18%
35%
0.6 year
0.5 year
9.3 years
It takes 17 years to turn 14 per cent of original research to the benefit of patient care
Dickersin, 1987
Koren, 1989
Balas, 1995
Poynard, 1985
Kumar, 1992
Kumar, 1992
Poyer, 1982
Antman, 1992
Negative results
Lack of numbers Expert
opinion
Inconsistentindexing
17:14
Original research
Acceptance
Implementation
Balas, 2002Balas, 2002
Treatments Thought to Work Treatments Thought to Work but Shown Ineffectivebut Shown Ineffective
Sulphuric acid for scurvySulphuric acid for scurvy Leeches for almost anythingLeeches for almost anything Insulin for schizophreniaInsulin for schizophrenia Vitamin K for myocardial Vitamin K for myocardial
infarctioninfarction HRT to prevent HRT to prevent
cardiovascular diseasecardiovascular disease Flecainide for ventricular Flecainide for ventricular
tachycardiatachycardia Routine blood tests prior to Routine blood tests prior to
surgerysurgery ABMT for late stage Breast ABMT for late stage Breast
CACABMJFebruary 28 2004; 324:474-5.BMJFebruary 28 2004; 324:474-5.
Expanding Research and Expanding Research and Evaluation Designs…for QII Evaluation Designs…for QII
Carolyn M. Clancy, M.D.Carolyn M. Clancy, M.D.Director, AHRQDirector, AHRQ
September 13, 2005September 13, 2005DRAFT – Denise D and David Atkins – 9/7/05DRAFT – Denise D and David Atkins – 9/7/05
““Health services research has not yet been Health services research has not yet been sufficiently helpful in meeting the sufficiently helpful in meeting the challenge of improving care in part challenge of improving care in part because it has over-constrained both its because it has over-constrained both its methods and its favorite topics….HSR methods and its favorite topics….HSR should become more effectively part of should become more effectively part of the solution. To do that will require that the solution. To do that will require that we enrich our portfolio of methods and we enrich our portfolio of methods and broaden our area of inquiry.”broaden our area of inquiry.”
--Berwick, HSR, 40:2 (April 2005)--Berwick, HSR, 40:2 (April 2005)
Challenges to the RCT Challenges to the RCT ParadigmParadigm
The intervention targets are not patients The intervention targets are not patients directly, directly,
Interventions are complex Interventions are complex Interventions change, and Interventions change, and Where the setting is an essential Where the setting is an essential
component of the question and component of the question and intervention. intervention.
AHRQ Research Study: Computerized AHRQ Research Study: Computerized Physician Order Entry SystemsPhysician Order Entry Systems
Major Finding:Major Finding: While computerized physician order entry (CPOE) While computerized physician order entry (CPOE) is expected to significantly reduce medication errors, systems must is expected to significantly reduce medication errors, systems must be implemented thoughtfully to avoid facilitating certain types of be implemented thoughtfully to avoid facilitating certain types of errorserrors
Study looked at clinicians’ experience in using one CPOE system at Study looked at clinicians’ experience in using one CPOE system at a major urban teaching hospitala major urban teaching hospital
Implementation problems can be minimized through testing before Implementation problems can be minimized through testing before products are marketed and through adaptation to meet the needs of products are marketed and through adaptation to meet the needs of individual clinical settingsindividual clinical settings
R. Koppel, J. Metlay, A. Cohen, et al., Role of computerized physician order entry systems in R. Koppel, J. Metlay, A. Cohen, et al., Role of computerized physician order entry systems in facilitating medication errors,facilitating medication errors, Journal of the American Medical AssociationJournal of the American Medical Association, March 9, 2005, March 9, 2005
Current QI evaluation questions Current QI evaluation questions AHRQ and others are askingAHRQ and others are asking
Can a regional health information organization Can a regional health information organization improve interoperability of health information improve interoperability of health information technology systems and improve patient safety technology systems and improve patient safety and quality of care?and quality of care?
Can pay for performance improve quality?Can pay for performance improve quality? Do changes in hospital culture reduce medical Do changes in hospital culture reduce medical
errors?errors? What QI strategies work for reducing disparities?What QI strategies work for reducing disparities?
– Different strategies for different populations and Different strategies for different populations and settings?settings?
– Same strategies? Same strategies?
Current QI evaluation questions Current QI evaluation questions AHRQ and others are asking - 2AHRQ and others are asking - 2
When should coverage be linked with When should coverage be linked with development of better evidence (‘decision-based development of better evidence (‘decision-based evidence making’)?evidence making’)?
[In an action-oriented world how to identify a [In an action-oriented world how to identify a control group?]control group?]
Can evidence for the effectiveness of team Can evidence for the effectiveness of team training in obstetrics be developed as training is training in obstetrics be developed as training is provided?provided?
For which questions is a registry useful? What For which questions is a registry useful? What methods will help us take advantage of new methods will help us take advantage of new sources of electronic data?sources of electronic data?
Galaxies of Evaluation Designs That May Inform Galaxies of Evaluation Designs That May Inform QII Evaluation Designs and MethodsQII Evaluation Designs and Methods
[1] For purposes of this chart, intervention evaluation research is defined as studies designed to Answer the Questions: Does the Intervention Work? For Whom or What? Under What Conditions?
RCT
HSR: 2° data Analysis
Medical/clinical/HSR Galaxy
Behavioral/Social Science Galaxy
Action Research—
K Lewin
PDSA/SPCDeming/
Shewhart
RCT
Quasi-exp
Case reports
Case studies
Qualita-tive
methods
Systems Science
Case-Control
ITS/MBS
QIIEvaluation Research
Cohort study
RD
GRT
