roi measurement: finding the fallacies. roi how roi is calculated some examples of what rois are how...
TRANSCRIPT
![Page 1: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/1.jpg)
ROI measurement: Finding the Fallacies
![Page 2: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/2.jpg)
ROI
• How ROI is calculated
• Some examples of what ROIs are
• How to know when it is calculated wrong, as it usually is– Best way to understand ROI is to spot
examples of how it is calculated wrong• Like learning to drive by getting behind the wheel
rather than watching someone else
![Page 3: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/3.jpg)
ROI
• Take the total population with the disease
• Trend it forward by the plan’s inflation rate
• That gives you the “adjusted baseline.”
![Page 4: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/4.jpg)
ROI
• Take the total population with the disease
• Trend it forward by the plan’s inflation rate
• That gives you the “adjusted baseline.”
• Calculate the actual spending on that disease and compare it to the adjusted baseline
![Page 5: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/5.jpg)
ROI
• Take the total population with the disease• Trend it forward by the plan’s inflation rate • That gives you the “adjusted baseline.”• Calculate the actual spending on that
disease and compare it to the adjusted baseline
• TOTAL savings on claims/TOTAL fees = ROI*
• Savings are usually calculated wrong
![Page 6: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/6.jpg)
*
• Total savings/total fees = ROI
• Total savings – total fees = net savings
• Example:– Program cost $100– Program savings $400– ROI is 4:1– Net savings is $300
![Page 7: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/7.jpg)
In pre-post
• You find everyone in the baseline year(s) with claims for the disease
• You develop a baseline year cost/disease person
• You trend that cost forward and compare the “actual” cost for the disease-managed population
• Example: Asthma
![Page 8: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/8.jpg)
In the example
• Assume no inflation, no claims other than asthma – These assumptions just simplify. They don’t
distort
![Page 9: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/9.jpg)
Example from AsthmaFirst asthmatic has a claim in 2004
2004 (baseline)
2005 (contract)
Asthmatic #1
1000 0
Asthmatic #2
Cost/asthmatic
![Page 10: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/10.jpg)
Example from AsthmaSecond asthmatic has a claim in 2005
2004 (baseline)
2005 (contract)
Asthmatic #1
1000 0
Asthmatic #2
0 1000
Cost/asthmatic
![Page 11: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/11.jpg)
Question
• What is your cost/asthmatic during the baseline year?
![Page 12: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/12.jpg)
Cost/asthmatic in baseline?
2004 (baseline)
2005 (contract)
Asthmatic #1
1000 0
Asthmatic #2
0 1000
Cost/asthmatic
![Page 13: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/13.jpg)
Cost/asthmatic in baseline?
2004 (baseline)
2005 (contract)
Asthmatic #1
1000 0
Asthmatic #2
0 1000
Cost/
asthmatic
$1000
![Page 14: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/14.jpg)
In pre-post – remember this slide?
• You find everyone in the baseline year(s) with claims for the disease
• You develop a baseline year cost/disease person
• You trend that cost forward and compare the “actual” cost for the disease-managed population
• Example: Asthma
![Page 15: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/15.jpg)
Cost/asthmatic in contract period?
2004 (baseline)
2005 (contract)
Asthmatic #1
1000 0
Asthmatic #2
0 1000
Cost/
asthmatic
$1000
![Page 16: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/16.jpg)
Cost/asthmatic in contract period?
2004 (baseline)
2005 (contract)
Asthmatic #1
1000 0
Asthmatic #2
0 1000
Cost/asthmatic
$1000 $500
![Page 17: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/17.jpg)
Congratulations
• You just “saved” 50% by doing nothing• This is called the “zeroes” or “planes on the
ground” fallacy (planes on the ground aren’t spotted by radar)– The claims extraction methodology only works if
someone with the disease has disease-specific claims in the baseline year
• Note that the way you can spot this fallacy is that prevalence rises dramatically, reductions in cost bears no resemblance to quality improvements, and claims fall without any “nexus” to the program
![Page 18: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/18.jpg)
ROI By Disease: Impact of the fallacy
0
0.5
1
1.5
2
2.5
ROI calculation
asthma
CADdiabetes
CHF
Vendor Measurement
Valid measurement
![Page 19: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/19.jpg)
Examples of invalid calculations
• ROI mistake #1: Let’s go to the web…
![Page 20: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/20.jpg)
ROI Mistake #2
• In the following example, utilization figures were multiplied by the (assume to be correct) cost figures to get a savings
• Assume (correctly) no other changes were talking place
• Why is this obviously incorrect?
![Page 21: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/21.jpg)
Savings by Category of Utilization per 1000 members
$0
$500
$1,000
$1,500
$2,000
$2,500
IPAdmits
ERVisits
OPFacility
MDVisits
Drug Other
2003
2004
![Page 22: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/22.jpg)
Issue-spotter #3Can you critically analyze these presented
numbers from a major national health plan?*Disease Category
Admission Reduction
Cost Reduction
Asthma 2% 12%
cardiology 5% 15%
*The name of this health plan will not be provided – you had to be there
![Page 23: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/23.jpg)
AsthmaEmergency Room Visits
0
5
10
15
20
25
30
pe
r 1
00
0 m
em
be
rs
Baseline 26 16 26
Reporting 10 16 10
HMO Total High Risk Low Risk
Total N = 781 High Risk N = 61 Low Risk N = 720
Issue-Spotter #4: What is wrong with this slide
Note: name of vendor can be shared upon NDA
![Page 24: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/24.jpg)
Issue-Spotter #5
• Look at the next two slides together
![Page 25: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/25.jpg)
AsthmaHospital Admissions
0
20
40
60
80
100
120
140
pe
r 1
00
0 m
em
be
rs
Baseline 77 131 72
Reporting 33 115 26
HMO total High Risk Low Risk
57%
63%
13%
![Page 26: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/26.jpg)
AsthmaHospital Days
0
100
200
300
400
500
600
700
pe
r 1
00
0 m
em
be
rs
Baseline 223 656 186
Reporting 114 197 107
HMO total High Risk Low Risk
Total N = 781 High Risk N = 61 Low Risk N = 720
-48%
-70%
-43%
![Page 27: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/27.jpg)
AsthmaHospital Days and Admissions
0
100
200
300
400
500
600
700
per 1
000
mem
bers
Baseline 223 656 186
Reporting 114 197 107
HMO total High Risk Low Risk
-48%
-70%
-43%0
20
40
60
80
100
120
140
per 1
000 m
embe
rsBaseline 77 131 72
Reporting 33 115 26
HMO total High Risk Low Risk
DAYSADMISSIONS
![Page 28: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/28.jpg)
Issue-Spotter #6
• Look at the next two slides together
![Page 29: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/29.jpg)
CHF Group #1Emergency Room Visits/Year
0
1,000
2,000
3,000
4,000
5,000
6,000
per
1000 m
em
bers
Baseline 3,081 4,940 2,526
R#1 2,739 4,366 2,254
R#2 2,801 4,918 2,169
HMO total High Risk Low Risk
Total N = 1166 High Risk N = 268 Low Risk N = 898
Care to try your luck at this one?
![Page 30: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/30.jpg)
CHF Group #1Inpatient Admissions/Year
0
100
200
300
400
500
600
per 1
000 m
embe
rs
Baseline 350 494 252
R#1 273 478 195
R#2 280 491 216
HMO total High Risk Low Risk
Total N = 1166 High Risk N = 268 Low Risk N = 898
Care to try your luck at this one vs. the other one?
![Page 31: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/31.jpg)
Emergency Room Visits and IP stays/Year
0
1,000
2,000
3,000
4,000
5,000
6,000
per 1
000
mem
bers
Baseline 3,081 4,940 2,526
R#1 2,739 4,366 2,254
R#2 2,801 4,918 2,169
HMO total High Risk Low Risk
Care to try your luck at this one?
0
100
200
300
400
500
600
per 1
000
mem
bers
Baseline 350 494 252
R#1 273 478 195
R#2 280 491 216
HMO total High Risk Low Risk
ER Visits IP Stays
![Page 32: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/32.jpg)
Issue-Spotter #7
• Can you find the mistake which a major actuarial firm missed?
![Page 33: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/33.jpg)
Pre-post comparison: Asthma Medicaid Disabled Population
Baseline Period 1/03-12/03 paid through 6/30/04
Study Period 1/04-12/04, paid through 2/28/05
Member-months
15047 31884
Cost PDMPM $432 $391
Gross savings & ROI
$2,400,125
2.72 – to -1
![Page 34: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/34.jpg)
Issue Spotter #8
• What do you see in these two slides?– Within either, or comparing the two?– This was presented by Blue Cross of
Minnesota
![Page 35: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/35.jpg)
Cohort Study Results (all claims, all members)
![Page 36: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/36.jpg)
ROI and PMPM reductions at 6 Months
• Reporting Period – July - December 2002
• Base Period – July - December 2001
• Total ROI 2.48 : 1– Extended Conditions
4.23 : 1– Core Conditions
1.86 : 1
• “Our Auditors validated a $42 PMPM reduction due to this program”
![Page 37: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/37.jpg)
Combined
• Reporting Period – July - Dec 2002
• Base Period – July - Dec 2001
• Total ROI 2.48 : 1– Extended Conditions
4.23 : 1– Core Conditions
1.86 : 1
• Auditors validated
a $42 PMPM savings
![Page 38: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/38.jpg)
#9 – What’s wrong with these outcomes?
![Page 39: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/39.jpg)
Program Year One – Clinical IndicatorsProgram Year One – Clinical Indicators
Clinical Outcomes:
Base Post Year 1 Improvement
% of CHD Members with an LDL screen 75.0% 77.0% 2.0%
% of CHD Members with at least one claim for a Statin 69.0% 70.5% 1.5%
% of CHD Members receiving an ACE inhibitor or alternative 43.5% 44.7% 1.2%
% of CHD Members post-MI with at least one claim for a beta-blocker
0.89 0.89 0.0%
Hospitalizations/1,000 CHD Members for a primary diagnosis of Myocardial Infarction*
47.60 24.38 -48.8%
*measure based on total membership, not just "continuously enrolled" membership
Percentage of Continuously Enrolled Members
![Page 40: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/40.jpg)
Issue-Spotter #10
• What kills the credibility of the savings reported on the next slide?
![Page 41: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/41.jpg)
Top Ten Diagnoses—admissions per 100 Cardio Disease Management Members
(pre- and post-DM)
0
50
100
150
200
250
300
350
Angina
Sympto
ms
CAD
Dorsopat
hies
Hyperte
nsion
Arthro
pathie
s
persons
without r
eporte
d Dx
Oth
er m
etab
olic
rheu
amtis
m, e
xcl.
back
2003--pre
2004--post
![Page 42: ROI measurement: Finding the Fallacies. ROI How ROI is calculated Some examples of what ROIs are How to know when it is calculated wrong, as it usually](https://reader033.vdocuments.net/reader033/viewer/2022061306/55146ebc550346494e8b5f48/html5/thumbnails/42.jpg)
Conclusion
• Don’t assume that it’s right just because it’s written down
• 75% of reporting has major, invalidating mistakes in it
• Who feels better equipped to look for them?