Development and Evaluation of CMS-HCCConcurrent Risk Adjustment Models

Presented byEric Olmsted, Ph.D.Gregory Pope, M.S.John Kautter, Ph.D.RTI International

Presented atAcademy HealthJune 26, 2005

411 Waverley Oaks Road ■ Suite 330 ■ Waltham, MA 02452-8414

Concurrent Risk AdjustmentIntroduction

OverviewRisk Adjustment/HCC ModelConcurrent v. Prospective

Project Goals and Challenges

Model Development

Model Evaluation

Summary and Conclusion

OverviewRisk Adjustment Introduction

Population Risk Adjustment:The process by which the health status of a

population is taken into consideration when setting capitation rates or evaluating patterns or outcomes of practice

Risk adjustment is used to create “apples to apples” comparisons Risk adjustment removes the effect of health

status differences Reduces or eliminates the problem of selection

OverviewRisk Adjustment Model

Model calibrated on 5% national sample of Medicare fee-for-service beneficiaries

Expenditures are regressed on HCC (& demographic) risk markers to estimate incremental impact of each diagnosis on expenditures

Annualized Expenditures = Σαi + Σβi + Єi

αi = demographic markers

βi = HCC markers

OverviewHCC Model

Full model contains 184 HCCs

CMS-HCC model contains 70 HCCs CMS-HCCs:

Cover a broad spectrum of health disorders Have well-defined diagnostic criteria Exclude highly discretionary diagnoses Include conditions with significant expected health expenditures

Demographic Markers Age, Gender, Medicaid, & Originally Disabled Status Ensure means for demographic populations correctly estimated

OverviewConcurrent vs. Prospective

Prospective risk adjustment uses current year diagnoses to predict next year’s expendituresChronic conditions are more important

Concurrent risk adjustment uses current year diagnoses to predict this year’s expendituresAcute conditions are more important

OverviewConcurrent vs. Prospective

AMI:Prospective Coefficient = $1,838Concurrent Coefficient = $12,211

63% of HCC coefficients with >$1,000 difference

R-squared:Concurrent - .4811Prospective - .0981

Project Goals

Concurrent Risk Adjustment Project Goals:Develop payment model for Pay-for-Performance

demonstrationDevelop model for use in profiling physiciansMake model consistent with prospective CMS-HCC

model that is being used for MA payment, and its data collection requirements

Improve prediction across the spectrum of patient cost

Concurrent Modeling Challenges

Applied standard HCC modelResulted in negative predictions and coefficients

Concurrent HCC coefficients fit high-cost beneficiariesThis forces age-sex coefficients down and they

sometimes become negativeAge-sex coefficients reflect the average beneficiaryNegative age-sex coefficients can lead to negative

predictions

Model Challenges Standard Regression

-$2,000

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

0 0.5 1 1.5 2 2.5 3

Risk Score

Exp

end

itu

res

($)

Beneficiaries

Regression Line

Model ChallengesSplit Sample Regression

-$2,000

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

0 0.5 1 1.5 2 2.5 3

Risk Score

Exp

end

itu

res

($)

High-CostBeneficiaries

Low-CostBeneficiaries

Low-Cost Regression

High-Cost Regression

Model ChallengesRegression through the Origin

-$2,000

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

0 0.5 1 1.5 2 2.5 3

Risk Score

Exp

end

itu

res

($)

High-CostBeneficiaries

Low-CostBeneficiaries

Regression Line

Model ChallengesNonlinear Regression

-$2,000

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

0 1 2 3

Risk Score

Exp

end

itu

res

($)

High-CostBeneficiaries

Low-CostBeneficiaries

Project GoalsModel Selection

Criteria for Model SelectionAvoid negative predictions, which lack face validityAvoid negative coefficientsMaintain correct age-sex means to prevent age

and sex selection by providersPrefer simple models to complex modelsSelect model with good ‘performance’ among

model evaluation measures

Model DevelopmentSample Statistics

1.4 million FFS Medicare beneficiaries with mean expenditures of $5,214

Beneficiaries with at least one CMS-HCC represent 61% of the population, but provide 94% of all Medicare expenditures

Model Development Standard Models

Full HCC Model184 HCCs & demographics

CMS-HCC Model70 HCCs & demographics

Interaction and Topcoding ModelsCreated disease and demographic interactions

to tease out high-expense beneficiariesCreated topcoded models to reduce impact of

outliers

Model DevelopmentAlternative Models

Nonlinear Models Log model Square root model

Split Sample Models Designed separate models for populations with different expected

expenditures Community/Institutional High Cost/Low Cost HCC Catastrophic HCC Multi-stage models including two-part and four-part logit models Simple two-stage model with demographic multipliers Segmentation

Model EvaluationStandard Model Results

Full HCC model suffers not only from 30% negative predictions, but also contains negative HCC coefficients

CMS-HCC model explains 92% of the variation that the Full HCC model explains CMS-HCC model eliminates negative HCC

coefficientsCMS-HCC model has only 10% negative predictions

Interaction and Topcoding ModelsDid not sufficiently reduce negative predictions

Model EvaluationAlternative Model Results

Nonlinear ModelsLog model and square root model did not produce

reasonable predictions

Split Sample ModelsSplitting sample by community/institutional did not eliminate

negative predictionsSplitting sample by disease burden eliminated negative

predictions

Model EvaluationMeasures of Model Performance

R2 within .04 for all modelsR2 did not differentiate models

Predictive Ratio = Average of model’s predictions

Average of actual expenditures

Where each of the two averages is taken over the individuals in the subgroup Predicted expenditure deciles Number of HCCs for a beneficiary

Model EvaluationPredictive Ratios by Expenditure Percentile

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Percentile Predicted Expenditures

Pre

dic

ted

/Act

ual

Exp

end

itu

res

CMS-HCC Model

Multiplier Model

High Cost/LowCost Model

Four Part Model

Model EvaluationPredictive Ratios by Expenditure Percentile

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Percentile Predicted Expenditures

Pre

dic

ted

/Act

ual

Exp

end

itu

res

CMS-HCC Model

Catastrophic/Non-CatastrophicModel

Two-Stage SampleSegmentationModel

Topcoded at$50,000 Model

Model EvaluationPredictive Ratios by # of HCCs

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1.10

1.20

Observed Number of HCCs

Pre

dic

ted

/Act

ual

Exp

end

itu

res

CMS-HCCModel

MultiplierModel

High-Cost/Low-Cost Model

Four PartModel

Model EvaluationPredictive Ratios by # of HCCs

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1.10

1.20

Observed Number of HCCs

Pre

dic

ted

/Actu

al E

xp

en

dit

ure

s

CMS-HCCModel

Catastrophic/Non-CatastrophicModel

Two-StageSampleSegmentationModel

Concurrent Model EvaluationModel Summary

High Cost & Catastrophic Models performs well Some face validity problems with splitting HCCs into “high-cost”

and “low-cost” Still has negative predictions

Four Part Model also performs well Computationally advanced and hard to interpret intuitively No negative predictions

Sample Segmentation Model performs very well Also computationally advanced

Two-Stage Multiplier Model performs adequately No face validity problems

Concurrent Model EvaluationConclusion

Nonlinearities cause difficulties in concurrent risk adjustment model calibration Negative coefficients and predictions

These difficulties can be addressed with: Nonlinear models Split sample models

But nonlinear/split sample models add complexity Difficult to estimate Difficult to interpret Adds instability

Two-Stage Multiplier Model Good face validity, avoids negative coefficients and predictions Simpler to estimate and interpret