educational outcomes measurement: what we’ve learned, what’s ahead
DESCRIPTION
Educational Outcomes Measurement: What We’ve Learned, What’s Ahead. Derek T. Dietze, MA President. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/1.jpg)
1
Educational Outcomes Measurement:
What We’ve Learned, What’s Ahead
Derek T. Dietze, MAPresident
Presentation given at the 2006 Alliance for CME Annual Meeting as: Executive Director of CME Veritas Institute for Medical Education, Inc. Hasbrouck Heights, New Jersey
![Page 2: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/2.jpg)
2
Session Objectives
1. Evaluate you organization’s educational outcomes measurement (EOM) in the context of others’ efforts
2. Identify opportunities to advance EOM3. Describe the potential impact of EOM
on the future of CME
You will be better able to:
![Page 3: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/3.jpg)
3
Who Is in the Audience Today?
1. Medical education/communication company (MECC)
2. Hospital3. Medical school4. Medical specialty society5. Health system6. Commercial interest7. Other
1. What is your affiliation?
![Page 4: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/4.jpg)
4
Who Is in the Audience Today?
2. I have been involved in CME for _____
1 = less than 1 year
2 = 1 to 3 years
3 = 3 to 5 years
4 = 5 to 10 years
5 = more than 10 years
![Page 5: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/5.jpg)
5
Audience Opinion Poll
1 = Strongly disagree2 3456 = Strongly agree7 = Not applicable
3. To what extent do you agree with the following statement? Compared with other organizations, my organization is doing a great job of educational outcomes measurement.
![Page 6: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/6.jpg)
6
Audience Opinion Poll
4. To what extent do you agree with the following statement?
Educational outcomes measurement is a fad that will pass with time.
1 = Strongly Disagree2345 6 = Strongly Agree
![Page 7: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/7.jpg)
7
Audience Opinion Poll
1 = Strongly disagree2 3456 = Strongly agree7 = Not applicable
5. To what extent do you agree with the following statement? Implementing EOM is essential to the long-term success of my organization.
![Page 8: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/8.jpg)
8
Audience Opinion Poll
1 = Strongly disagree2 3456 = Strongly agree7 = Not applicable
6. To what extent do you agree with the following statement? I need to learn a lot more about EOM.
![Page 9: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/9.jpg)
9
CME should be a strategic asset for improving performance and enhancing patient care.
CME in the public interest.
![Page 10: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/10.jpg)
10
From the CEO of the ACCME (Dec. 2005)
“On the research front, primary studies and syntheses no longer need to ask if CE, in general, improves practice or other outcomes because there is so much evidence that many kinds and combinations can do so.”
From Robertson, Umble and Cervero, Journal for Continuing Education in the Health Professions 23, 146, 2003
![Page 11: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/11.jpg)
11
EOM: The Past
• Early-mid 1990s: discussion focused on the nature and definitions of “educational outcomes”
• Alliance for CME conference on educational outcomes
• Early-adopters and innovators• Since 2000: “we need to do EOM!”
![Page 12: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/12.jpg)
12
The Value of CME
“A CME evaluation should focus on identifying, measuring, and describing the value provided by an intervention that leads to enhanced performance, improved healthcare quality, and reduced costs.”
Moore DE. A framework for outcomes evaluation in the continuing professional development of physicians. In: Davis D, Barnes BE, Fox R, eds. The Continuing Professional Development of Physicians: From Research to Practice. Chicago, Ill: American Medical Association; 2003.
![Page 13: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/13.jpg)
13
Levels of Educational Outcomes
Learning Changes in self-reported knowledge of the participants; development of competence
PerformanceChanges in observed practice performance; the application of learning; the application of competence
Patient Health
Changes in the health status of patients due to changes in participant practice behavior
Population Health
Changes in the health status of a population of patients due to changes in widespread practice behavior
Participation The number of people who registered and/or participated
SatisfactionThe degree to which participants’ expectations about the setting and delivery of the CME activity were met
Adapted from: Moore DE. A framework for outcomes evaluation in the continuing professional development of physicians. In: Davis D, Barnes BE, Fox R, eds. The Continuing Professional Development of Physicians: From Research to Practice. Chicago, Ill: The American Medical Association; 2003.
![Page 14: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/14.jpg)
14
Measurement: Alliance for CME Competencies Survey*
• Competency 2.12, “…offer educational consultation that emphasizes measurement in physician organizations to identify goals for education that are specific to the practice and measurable…,”
• Rated the lowest in terms of “responsibility” and “attainment” by CME leaders, managers, and coordinators (2.6/2.6).
*King R, North S. Alliance for CME Competency Areas Research Project Report. Alliance for CME; 2004.
![Page 15: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/15.jpg)
15
Forces Driving EOM
• Accountability for effectiveness, impact, and use of resources
• Desire for “exemplary” status• Performance improvement movement• MOC and MOL requirements/initiatives• Pay for performance movement• Increasingly a requirement of
commercial support• ACCME is “raising the bar” for providers• CME community’s desire for continuous
improvement
![Page 16: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/16.jpg)
16
Performance Improvement Movement
October 6, 2004Performance Improvement Activities approved for creditStage A: Learning from current practice performance assessmentStage B: Learning from the application of PI to patient careStage C: Learning from the evaluation of the PI effort
May 23, 2005Internet Point of Care now eligible for credit> Review original clinical question(s).> Identify the relevant sources from among those consulted.> Describe the application of their findings to practice.
American Medical Association
![Page 17: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/17.jpg)
17
Performance Improvement Movement (cont.)• American Academy of Family Physicians:
Performance Improvement in Practice
![Page 18: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/18.jpg)
18
MOC and MOL Initiatives
• July 2003: ABFM introduced new MOC requirements for family physicians.
• Part IV requirement replaced the traditional clinical chart review. Focus on improving quality of care, rather than on record-keeping.
• Each FP must complete 1 QI project during his or her recertification cycle to comply with Part IV.
American Board of Medical Specialties: Part IV Maintenance of Certification
“Evidence of evaluation of performance in practice”
![Page 19: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/19.jpg)
19
Board of Directors Action Plan Fiscal 2004
“Assure ACCME accreditation remains relevant by addressing the needs of medical boards to quantify learning and practice outcomes of individual physician learners.”
![Page 20: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/20.jpg)
20
Pay for Performance Movement
Phase in of Medicare Pay for Performance
Q. Does the Academy (AAFP) have anything to help me see what P4P might be like?A. Yes. METRIC -- Measuring, Evaluating and Translating Research Into Care -- lets you earn 20 Prescribed CME credits in your office as you complete performance measurement and improvement projects.
![Page 21: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/21.jpg)
21
Commercial Interests’ Expectations
Increasingly, commercial interests are requiring CME providers to:
• Include a sound outcomes measurement plan in grant requests
• Become more familiar with EOM methodologies and tools
• Report results for evaluation and future planning
![Page 22: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/22.jpg)
22
Current Status of EOM
• Increasing understanding of EOM among some CME providers and MECCs
• Some measurement methodologies gaining acceptance and popularity
• Few standards or best practices• Some measurement beyond
“satisfaction” into “learning” and “performance”
• Lack of scientific rigor• Much talk• Limited experimentation• Fear
![Page 23: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/23.jpg)
23
EOM Challenges• EOM competence, skills, and resources• Time• High level support & resources• Practical EOM methodologies/tools• Funding/support for EOM from
commercial interests• Actual/perceived lack of access to “the
data”• “Rigor mortis” and “perfection
paralysis”• Fear of the results
![Page 24: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/24.jpg)
24
Current EOM Methodologies & Tools
• Follow-up surveys• Immediate pre- and post-activity
surveys/tests• Control group surveys• Focus groups• Clinical assertions• Clinical/case vignettes• Commitment to change• Pre- and post-activity prescription
measurement• Chart audits• QA data
![Page 25: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/25.jpg)
25
Pre/Post Clinical Assertion with Control
“’Early aggressive therapy’ is treatment initiated within the first 6 months of RA onset.”
4.91
5.43
4.91
1 2 3 4 5 6
Pre-Activity
Post-Activity
Control
Strongly
Disagree
Strongly
Agree
n=117
n=129
n=33
Single-factor ANOVA:Significant at P < 0.01
Data courtesy of Veritas Institute for Medical Education, Inc.
![Page 26: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/26.jpg)
26
Pre/Post Case Vignette: Patient With AS
• How would you treat a 28-year-old male patient exhibiting the following symptoms:– Morning stiffness– X-ray shows bilateral (sacroiliitis)– Swollen, painful knee– ESR 10– CBC normal
![Page 27: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/27.jpg)
27
Case: AS PatientPercent of Pre-/Post-Discussion Responses
“How would you treat?”
35.8
22.5
58.8
4.8
5.2
8.7
10.6
12.4
21.1
20.2
Pre-discussion (n=111)Post-discussion (n=106)
Combination
Anti-TNF
MTX
SSZ
NSAIDs
Percent of ResponsesData courtesy of Veritas Institute for Medical Education, Inc.
![Page 28: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/28.jpg)
28
Case: AS PatientPercent Change in Pre-/Post-Discussion Response
“How would you treat?”
Combination
Anti-TNF
MTX
SSZ
NSAIDs
-15.4
-5.3
-27.1
+10.2
+37.6
-40 -30 -20 -10 0 10 20 30 40 50Percent Change in Responses
Data courtesy of Veritas Institute for Medical Education, Inc.
![Page 29: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/29.jpg)
29
Case Vignette: Post-activity with Control
Outcomes, Inc. CME Metrics Report, July 25, 2005. Challenging Cases in Gastroenterology—Then & Now, Jobson Education Group. Satellite symposium at Digestive Disease Week, May 15, 2005.
N=46 for both groups P=0.05
![Page 30: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/30.jpg)
30
Only patients with severe and/or refractory cases of urticaria should be managed with steroids.
Pre/Post/Follow-up Clinical Assertions with Control
4.66 4.81
5.64 5.725.33
5.75
4.74
1
2
3
4
5
6
Baseline Live Pre Live Post Live + EMPost
Live OnlyFU
Live + EMFU
BaselineFU
Strongly agree
Strongly disagree
*ANOVA P <
0.001
n=390 n=1,753 n=1,374n=335 n=229 n=67 n=372FU = Follow-up survey
EM = Enduring material
Mean RatingLive only
participants do not maintain learning
Live + EM participants maintain better learning
through follow up
Dietze DT, Magazine HI. Enhancing and Sustaining Learning: Report of a Comprehensive CME Initiative and Educational Outcomes Measurement Plan (forum). Alliance for CME 31st Annual Conference; January 24, 2006; San Francisco, California.
![Page 31: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/31.jpg)
31
Performance Improvement ProjectPurpose:
Decrease the risk of recurrent vascular events in patients with AMI and CAD by initiating therapy with lipid-lowering agents during hospitalization.
Measurement Point:Percentage of defined AMI and CAD patients who received a lipid lowering agent charge during stay.
Multidisciplinary Team on PI Project:Cardiology Physicians, Telemetry Nursing Staff, Quality Services, CME Department, Pharmacy Decision Support (data specialist)Information courtesy of Susan G. McAlexander, MPA, CHES, CME Coordinator,
Providence St. Peter Hospital, Olympia, WA, February 2006.
![Page 32: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/32.jpg)
32
Measurement, Data Collection & Analysis• Lipid use rates compared monthly and
quarterly for changes after interventions.• Education – CME and other• Forms adjustments• In-services• Monthly project task force meetings to project
planning, discuss interventions and results.
• Aggregate and specific rate collection performed in the Quality Department utilizing charge data and chart review.
• Assessment of knowledge post CME single activity intervention performed by CME department.
• Monthly tends reported on graphs.
![Page 33: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/33.jpg)
33
Results and Application to Future CME• Goal percentage attained in 3rd quarter for AMI
patients and 4th quarter for ICS patients.• Greatest change noted in the cardiology and
internal medicine sections. Family medicine N too low to have statistical significance.
• Final report to CIC included a recommendation to tie CME into future PI projects for maximal impact.
• CME was seen to have a positive and reinforcing effect on physician learning and practice change.
![Page 34: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/34.jpg)
34
Measurement Is in Your Future!
![Page 35: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/35.jpg)
35
From the CEO of the ACCME (Dec. 2005)“The Board wants to make it clear that the ACCME values an accreditation system that rewards providersa) for establishing a mission that identifies improvements in physician performance and/or patient or population health as organizational goals, and b) for having a process in place to measure and increase its effectiveness in meeting this mission.”
![Page 36: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/36.jpg)
36
From the CEO of the ACCME (Dec. 2005)“Over the next few months…presentation of draft new compliance criteria and policy through which ACCME will reward providers for having processes in place:• To facilitate changes in knowledge, strategy, performance-in-practice, healthcare delivery or patient health.• To facilitate life-long learning based on individualized needs assessment where effectiveness is measured by the learning or improvements that occur.• To deliver education in support of the ACGME/ABMS Core Competencies, around specialty-specific curricula and in the context of the IOM Competencies.”
![Page 37: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/37.jpg)
37
Audience Opinion Poll
1 = Strongly disagree2 3456 = Strongly agree
7. To what extent do you agree with the following statement? New ACCME compliance criteria for measurement of CME effectiveness would be a positive step forward for the CME community.
![Page 38: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/38.jpg)
38
Audience Opinion Poll
1 = Strongly disagree2 3456 = Strongly agree7 = Not applicable
8. To what extent do you agree with the following statement? Compared to other organizations, my organization is doing a great job of EOM.
![Page 39: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/39.jpg)
39
Audience Opinion Poll
1 = Strongly disagree2 3456 = Strongly agree7 = Not applicable
9. To what extent do you agree with the following statement? Implementing EOM is essential to the long-term success of my organization.
![Page 40: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/40.jpg)
40
Vision for EOM
• EOM standards and benchmarks
• EOM best practices• Proven methodologies and
tools• More efficient and effective
CME• Improved patient care• Improved patient outcomes• Improved population health
![Page 41: Educational Outcomes Measurement: What We’ve Learned, What’s Ahead](https://reader036.vdocuments.net/reader036/viewer/2022062809/5681574d550346895dc4ef87/html5/thumbnails/41.jpg)
41
Thank You for Your Participation!
Derek T. Dietze, MAPresident
Queen Creek, Arizona