performance outcomes in eap national behavioral consortium september 6, 2007 loews coronado, san...
TRANSCRIPT
Performance Outcomes in EAP
National Behavioral ConsortiumSeptember 6, 2007Loews Coronado, San Diego
Russ Hagen, [email protected] Sharar, Managing Director [email protected]
Agenda
Overview of Issues with Performance Outcomes in EAP (slides 3-13)
Some example Chestnut Outcome Projects (15-28)
Where do we go as a field? (29-30)
1.2.
3.
The Conceptual Dilemma – What is EAP?
• A “theory" of workplace intervention • A "specialized field" or profession • A "place" where other professions ply their
trade • A set of "common components" or a "practice
model" • A type of "funding mechanism" for counseling
or mental health services• “EAPs have enlarged their scope: Can we align
metrics with relevant attributes of EAP definition?”
The Research & Empirical Base of EAP • Citations of effectiveness are often
obsolete/flawed (about 39 studies in past 10-12 years)
• Existing studies are often unpublished or proprietary
• No agreed upon markers of success, or true comparisons across vendors, models, programs
• No identified academic discipline or plan to build researchers
• Little transparency or collaboration among vendors
Performance is in the Eye of the Beholder
Purchaser: Are my EAP dollars being spent wisely?
Provider: (primary vendor or affiliate): Does good service or results produce rewards?
Client (employee or family member): Is the EAP responsive to my personal needs?
State of performance measurement in EAP
• Substandard performance is largely invisible
• Many measures are blunt, incomplete, distorting, exaggerated (e.g. Utilization rates)
• We lack common definitions & standard markers of success
• Under capitation, “marginal” performance receives the same rate as “optimal”
Why measure performance?
• Describe the effects or results of our interventions (outcome measures)
• Improve an aspect of the process of care (process measure), which in theory leads to better outcomes
• Make comparisons across vendors or program models
• Counterforce to deflation of EAP rates
Issues with Employers
• Many employers don’t know a good measure from a bad one
• Lack of senior management engagement• Over-reliance on consultants and brokers
who do not understand EAP• Competing goals between finance, HR,
benefits, occupational medicine, etc.• Good performance measurement is
difficult-and this difficulty is not always appreciated
Why are outcomes so difficult?
Natural Problems• Confounding factors beyond your control• Sample size too small to produce effect• Long delays when measuring over time• Low frequency of interesting outcomes
Why are outcomes so difficult?
Human Problems• Inadequate information systems• No extra funding (vendor bears cost)• Accessing employer data• Point of measurement complexity• Insufficient level of clinical detail• How does one address poor outcomes?
Criteria for selecting measures:
• Does the measure serve to enhance the productivity or well-being of employees? (e.g. is it “mission critical”?
• Is the measure based on science or opinion? (and if opinion, is there consensus)?
• Is it feasible? (resource availability, automated data collection, and statistically meaningful comparisons)
Practical Suggestions
• Implement new P4P pricing • Align better outcomes with higher
payments• Educate employers to buy based on
results, not price
Why P4P in EAP?
• Financial incentives in EAP (under capitation) are perverse, flat, almost non-existent
• Many contracts use “penalties” (withhold rather than bonus)
• So, delivering high quality EAP does not usually pay
"The larger issue is not whether EAPs are effective, but which EAPs are."
Ken Collins. "EAP Cost/Benefit Analysis: The Last Word." in EAPA Exchange (2000, Nov/Dec, p. 31).
Started in 1985 and grew to 90 full/part time staff grossing $9 Million a year in external funds (NIH, SAMHSA, Foundations)LI-Research: Several major experiments, quasi-experiments and major surveysLI-Training and Publications: 100s of training days and largest collection of evidence-based treatment manualsEBTx Coordinating Center---Supports training, certification, and coaching of clinicians and clinical supervisors learning A-CRA and ACC GAIN Coordinating Center – supports training, certification and use of the GAIN to support diagnosis, placement, treatment planning, and research
Chestnut strategy - link researchcapability with funded EAP outcome studies
Types of EAP Outcomes
• General Clinical Outcomes• Work Productivity Ratings• Relationships between clinical
outcomes and workplace variables
Global Appraisal of Individual Needs (GAIN) Short Screener (SS) embedded into WebMD's Health Risk Assessment for ADM
• A scientifically valid, 3 minute behavioral health screener for use in general populations
• Identifies who has a disorder and who does not with 95% accuracy
• Approximates the type of problem and severity • Guides further assessment & can be used as a
measure of change
Website: http://www.chestnut.org/LI/gain/GAIN_SS/index.html
ADM Pilot (n=1469)
69%
77%
83%
52%
25%
21%
16%
32%
7%
17%
1%
2%
0% 20% 40% 60% 80% 100%
Internal Disorder
External Disorder
Substance UseDisorder
Total DisorderScreener
Low Moderate High
Source: Collected as part of ADM health risk assessments from 11/05 to 8/06; Total Disorder Screener is based on 14 of 20 GSS items (one item in the internal not asked, and the violence and crime screener were not asked).
ADM Pilot (n=1469) Internal and Disorder Screener Items by Total Screener Score
24%
10%
11%
8%
19%
11%
3%
2%
1%
0%
1%
16%
3%
1%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Significant problems sleeping
Feeling trapped, lonely, sad
Feeling anxious, nervous
Distressed/upset by memories
Hard time paying attention
Hard time listening
Threatened other people
You lied or conned to get things
Have you hit someone
Tried to Hide AOD Use
Fam/co-workers complain about use
Used AOD weekly
Kept using though got you in trouble
Spent lot of time getting/using AOD
Internalizing Disorder Screener
Externalizing Disorder Screener
Substance Disorder Screener
76%
90%
35%
82%
31%
15%
3%
8%
14%
17%
9%
53%
4%
7%
56%
0% 20% 40% 60% 80% 100%
Internal Disorder
External Disorder
Substance UseDisorder
Violence and Crime
Total DisorderScreener
Low Moderate High
Clinical Sample of Workers 12 Months AfterSA Treatment (n=115)
Source: ERI (Dennis & Scott) Interviews conducted between 1/05 and 5/05
$0
$3,000
$6,000
$9,000
$12,000
$15,000
$18,000
$21,000
$24,000
$27,000
$30,000
$33,000
Low (0) Moderate (1-2) High (3+)
$0
$3,000
$6,000
$9,000
$12,000
$15,000
$18,000
$21,000
$24,000
$27,000
$30,000
$33,000
Costs of Service Utilization in the NEXT 12 Months by Total Disorder Screener (n=115)
Source: ERI (Dennis & Scott) Interviews conducted between 1/05 and 5/05; p<.05
Each has a sharp right
skew
Higher Median Costs
Median Hours Absent by ADM Employees by Total Disorder Screener Score
108
156
131
0
20
40
60
80
100
120
140
160
180
200
Low (n=221) Moderate (n=123) High (n=74)
Hou
rs A
bsen
t in
a Y
ear
Based on those with any absenteeism
Next Steps with the GAIN SS
1. Link ADM GAIN SS scores to ADM medical & pharmacy claims, workers' compensation, and other measures to determine if we can predict future claims or expenses
2. Implement a process to proactively "reach out" to ADM members with moderate or high risk scores
3. Use the GAIN SS as a longitudinal follow-up tool to measure reductions in symptoms
ADM Formal Management Referrals in 2003
Supervisor Ratings of Work Performance (before and
after EAP) N=317 (% rated satisfactory or above)
41
82
0
10
20
30
40
50
60
70
80
90
Before EAP 3 Months after EAP
Workers' Compensation Study 2005
629,000
285,000
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
Pre-EAP1999-2001
Post-EAP2003-2005
Examined financial effect of ADM’sEAP on workers’ compensation claim dollars
Pre and Post EAP
N= 217
3 years pre- and post-intervention
55% reduction in claims
Alcohol, drugs & depression cases in 2002
Survey Data
Percent reporting dissatisfaction with . . .
12%
7%
13%
7%
14%
5%6%
3%
13%
8%
18%
9%
Emotionalstate
Socialrelationships
Maritalrelationships
Familyrelationships
J ob Kids'educationalexperience
Pre IAP After first year of IAP
22.0%
9.4%
19.0%
9.7%
31.0%
15.8%
Considered early repatdue to personal problem
Considered early repatdue to work problem
Family considered earlyrepatriation
Pre IAP
1-yearfollow up
Survey DataPercent who have considered early repatriation
Survey Data“How has your personal life been affected while living
abroad?” Percent reporting a “negative” impact.
25.00%
12.80%
Pre IAP 1 year follow up
39% decrease49% decrease
• Tension between price and performance cannot be resolved without measurement
• Obtain agreement on core set of performance measures
• All vendors/program should report on same measures
• Compete on who is best at addressing employee problems
Where do we go from here?
GOOD TOGREAT CONCEPT
BUSINESS SECTOR SOCIAL SECTORS
Defining and Measuring “Great”
Widely agreed-upon financial metrics of performance. Money is both an input (a means to success) and an output (a measure of success).
Fewer widely agreed-upon metrics of performance. Money is only an input, not an output. Performance relative to mission, not financial returns, is the primary definition to success.
Summary difference between business and social sectors thought the good-to-great framework (Jim Collins)