friends don't let friends deploy black-box models ...€¦ · friends don’t let friends...
TRANSCRIPT
Friends Don’t Let Friends Deploy Black-Box Models:Detecting and Preventing Bias via Transparent Modeling
Rich CaruanaMicrosoft Research
Joint Work withYin Lou & Sarah Tan
Johannes Gehrke, Paul Koch, Marc Sturm, Noemie Elhadad
Thanks toGreg Cooper MD PhD, Mike Fine MD MPH, Eric Horvitz MD PhD
Nick Craswell, Tom Mitchell, Jacob Bien, Giles Hooker, Noah Snavely
August 14, 2017Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 1 / 41
When is it Safe to Use Machine Learning in Healthcare?
data for 1M patients
1000’s great clinical features
train state-of-the-art machine learning model on data
accuracy looks great on test set: AUC = 0.95
is it safe to deploy this model and use on real patients?
is high accuracy on test data enough to trust a model?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 2 / 41
When is it Safe to Use Machine Learning in Healthcare?
data for 1M patients
1000’s great clinical features
train state-of-the-art machine learning model on data
accuracy looks great on test set: AUC = 0.95
is it safe to deploy this model and use on real patients?
is high accuracy on test data enough to trust a model?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 2 / 41
When is it Safe to Use Machine Learning in Healthcare?
data for 1M patients
1000’s great clinical features
train state-of-the-art machine learning model on data
accuracy looks great on test set: AUC = 0.95
is it safe to deploy this model and use on real patients?
is high accuracy on test data enough to trust a model?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 3 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
LOW Risk: outpatient: antibiotics, call if not feeling better
HIGH Risk: admit to hospital (≈10% of pneumonia patients die)
One goal was to compare various ML methods:logistic regressionrule-based learningk-nearest neighborneural netsBayesian methodshierarchical mixtures of experts...
Most accurate ML method: multitask neural nets
Safe to use neural nets on patients?
No — we used logistic regression instead...
Why???
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 4 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
LOW Risk: outpatient: antibiotics, call if not feeling better
HIGH Risk: admit to hospital (≈10% of pneumonia patients die)
One goal was to compare various ML methods:logistic regressionrule-based learningk-nearest neighborneural netsBayesian methodshierarchical mixtures of experts...
Most accurate ML method: multitask neural nets
Safe to use neural nets on patients?
No — we used logistic regression instead...
Why???
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 4 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
LOW Risk: outpatient: antibiotics, call if not feeling better
HIGH Risk: admit to hospital (≈10% of pneumonia patients die)
One goal was to compare various ML methods:logistic regressionrule-based learningk-nearest neighborneural netsBayesian methodshierarchical mixtures of experts...
Most accurate ML method: multitask neural nets
Safe to use neural nets on patients?
No — we used logistic regression instead...
Why???
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 4 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
RBL learned rule: HasAsthma(x) => LessRisk(x)
True pattern in data:
asthmatics presenting with pneumonia considered very high riskreceive agressive treatment and often admitted to ICUhistory of asthma also means they often go to healthcare soonertreatment lowers risk of death compared to general population
If RBL learned asthma is good for you, NN probably did, too
if we use NN for admission decision, could hurt asthmatics
Key to discovering HasAsthma(x)... was intelligibility of rules
even if we can remove asthma problem from neural net, whatother ”bad patterns” don’t we know about that RBL missed?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 5 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
RBL learned rule: HasAsthma(x) => LessRisk(x)
True pattern in data:
asthmatics presenting with pneumonia considered very high riskreceive agressive treatment and often admitted to ICUhistory of asthma also means they often go to healthcare soonertreatment lowers risk of death compared to general population
If RBL learned asthma is good for you, NN probably did, too
if we use NN for admission decision, could hurt asthmatics
Key to discovering HasAsthma(x)... was intelligibility of rules
even if we can remove asthma problem from neural net, whatother ”bad patterns” don’t we know about that RBL missed?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 5 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
RBL learned rule: HasAsthma(x) => LessRisk(x)
True pattern in data:
asthmatics presenting with pneumonia considered very high riskreceive agressive treatment and often admitted to ICUhistory of asthma also means they often go to healthcare soonertreatment lowers risk of death compared to general population
If RBL learned asthma is good for you, NN probably did, too
if we use NN for admission decision, could hurt asthmatics
Key to discovering HasAsthma(x)... was intelligibility of rules
even if we can remove asthma problem from neural net, whatother ”bad patterns” don’t we know about that RBL missed?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 5 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
RBL learned rule: HasAsthma(x) => LessRisk(x)
True pattern in data:
asthmatics presenting with pneumonia considered very high riskreceive agressive treatment and often admitted to ICUhistory of asthma also means they often go to healthcare soonertreatment lowers risk of death compared to general population
If RBL learned asthma is good for you, NN probably did, too
if we use NN for admission decision, could hurt asthmatics
Key to discovering HasAsthma(x)... was intelligibility of rules
even if we can remove asthma problem from neural net, whatother ”bad patterns” don’t we know about that RBL missed?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 5 / 41
Lessons Learned
Risky to use data for purposes it was not designed for
Most data has unexpected landminesNot ethical to collect correct data for asthma
Much too difficult to fully understand the data
Our approach is to make the learned models as intelligible as possible
Must be able to understand models used in healthcare
Otherwise models can hurt patients because of true patterns in dataIf you don’t understand model it will make bad mistakes
Same story for race, gender, socioeconomic bias
The problem is in data and training signals, not learning algorithm
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 6 / 41
Lessons Learned
Risky to use data for purposes it was not designed for
Most data has unexpected landminesNot ethical to collect correct data for asthma
Much too difficult to fully understand the data
Our approach is to make the learned models as intelligible as possible
Must be able to understand models used in healthcare
Otherwise models can hurt patients because of true patterns in dataIf you don’t understand model it will make bad mistakes
Same story for race, gender, socioeconomic bias
The problem is in data and training signals, not learning algorithm
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 6 / 41
Lessons Learned
Risky to use data for purposes it was not designed for
Most data has unexpected landminesNot ethical to collect correct data for asthma
Much too difficult to fully understand the data
Our approach is to make the learned models as intelligible as possible
Must be able to understand models used in healthcare
Otherwise models can hurt patients because of true patterns in dataIf you don’t understand model it will make bad mistakes
Same story for race, gender, socioeconomic bias
The problem is in data and training signals, not learning algorithm
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 6 / 41
Lessons Learned
Risky to use data for purposes it was not designed for
Most data has unexpected landminesNot ethical to collect correct data for asthma
Much too difficult to fully understand the data
Our approach is to make the learned models as intelligible as possible
Must be able to understand models used in healthcare
Otherwise models can hurt patients because of true patterns in dataIf you don’t understand model it will make bad mistakes
Same story for race, gender, socioeconomic bias
The problem is in data and training signals, not learning algorithm
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 6 / 41
All we need is an accurate, intelligible model
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 7 / 41
Problem: The Accuracy vs. Intelligibility Tradeoff
Intelligibility
Acc
ura
cy
Logistic Regression
Naive Bayes
Single Decision Tree
Neural Nets
Boosted Trees
Random Forests
Decision Lists
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 8 / 41
Problem: The Accuracy vs. Intelligibility Tradeoff
???
Intelligibility
Acc
ura
cy
Logistic Regression
Naive Bayes
Single Decision Tree
Neural Nets
Boosted Trees
Random Forests
Decision Lists
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 9 / 41
Model Space from Simple to Complex
Linear Model: y = β0 + β1x1 + ...+ βnxn
Additive Model: y = f1(x1) + ...+ fn(xn)
Additive Model with Interactions: y =∑
i fi (xi ) +∑
ij fij(xi , xj) +∑
ijk fijk(xi , xj , xk) + ...
Full Complexity Model: y = f (x1, ..., xn)
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 10 / 41
Model Space from Simple to Complex
Linear Model: y = β0 + β1x1 + ...+ βnxn
Additive Model: y = f1(x1) + ...+ fn(xn)
Additive Model with Interactions: y =∑
i fi (xi ) +∑
ij fij(xi , xj) +∑
ijk fijk(xi , xj , xk) + ...
Full Complexity Model: y = f (x1, ..., xn)
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 10 / 41
Model Space from Simple to Complex
Linear Model: y = β0 + β1x1 + ...+ βnxn
Additive Model: y = f1(x1) + ...+ fn(xn)
Additive Model with Interactions: y =∑
i fi (xi ) +∑
ij fij(xi , xj) +∑
ijk fijk(xi , xj , xk) + ...
Full Complexity Model: y = f (x1, ..., xn)
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 10 / 41
Model Space from Simple to Complex
Linear Model: y = β0 + β1x1 + ...+ βnxn
Additive Model: y = f1(x1) + ...+ fn(xn)
Additive Model with Interactions: y =∑
i fi (xi ) +∑
ij fij(xi , xj) +∑
ijk fijk(xi , xj , xk) + ...
Full Complexity Model: y = f (x1, ..., xn)
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 10 / 41
Add ML-Steroids to old Stats Method: GAMs → GA2Ms
Generalized Additive Models (GAMs)
Developed at Stanford by Hastie and Tibshirani in late 80’sRegression: y = f1(x1) + ...+ fn(xn)Classification: logit(y) = f1(x1) + ...+ fn(xn)Each feature is “shaped” by shape function fi
T. Hastie and R. Tibshirani.Generalized additive models.Chapman & Hall/CRC, 1990.
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 11 / 41
Skip technical details of algorithm and jump to results
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 12 / 41
Motivation: Predicting Pneumonia Risk Study (mid-90’s)
Pneumonia Data (dataset from early 1990’s)
14,199 pneumonia patients70:30 train:test split (train=9847; test=4352)46 featurespredict POD (probability of death)10.86% of patients (1542) died
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 13 / 41
Pneumonia Dataset (mid-90’s): 46 Features
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 14 / 41
What GA2Ms on Steroids Learn About Risk vs. Age
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 15 / 41
Some of the things the intelligible model learned:
Age 105 is safer than Age 95We should have a retirement variableHas Asthma => lower riskHistory of chest pain => lower riskHistory of heart disease => lower risk
Good we didn’t deploy neural net back in 1995
But can understand, edit and safely deploy intelligible GA2M model
Intelligible/transparent model is like having a magic pair of glasses
Model correctness depends on how model will be used
this is a good model for health insurance providersbut needs to be repaired to use for hospital admissions
Important: Must keep potentially offending features in model!
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 16 / 41
Some of the things the intelligible model learned:
Age 105 is safer than Age 95We should have a retirement variableHas Asthma => lower riskHistory of chest pain => lower riskHistory of heart disease => lower risk
Good we didn’t deploy neural net back in 1995
But can understand, edit and safely deploy intelligible GA2M model
Intelligible/transparent model is like having a magic pair of glasses
Model correctness depends on how model will be used
this is a good model for health insurance providersbut needs to be repaired to use for hospital admissions
Important: Must keep potentially offending features in model!
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 16 / 41
Some of the things the intelligible model learned:
Age 105 is safer than Age 95We should have a retirement variableHas Asthma => lower riskHistory of chest pain => lower riskHistory of heart disease => lower risk
Good we didn’t deploy neural net back in 1995
But can understand, edit and safely deploy intelligible GA2M model
Intelligible/transparent model is like having a magic pair of glasses
Model correctness depends on how model will be used
this is a good model for health insurance providersbut needs to be repaired to use for hospital admissions
Important: Must keep potentially offending features in model!
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 16 / 41
Some of the things the intelligible model learned:
Age 105 is safer than Age 95We should have a retirement variableHas Asthma => lower riskHistory of chest pain => lower riskHistory of heart disease => lower risk
Good we didn’t deploy neural net back in 1995
But can understand, edit and safely deploy intelligible GA2M model
Intelligible/transparent model is like having a magic pair of glasses
Model correctness depends on how model will be used
this is a good model for health insurance providersbut needs to be repaired to use for hospital admissions
Important: Must keep potentially offending features in model!
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 16 / 41
Some of the things the intelligible model learned:
Age 105 is safer than Age 95We should have a retirement variableHas Asthma => lower riskHistory of chest pain => lower riskHistory of heart disease => lower risk
Good we didn’t deploy neural net back in 1995
But can understand, edit and safely deploy intelligible GA2M model
Intelligible/transparent model is like having a magic pair of glasses
Model correctness depends on how model will be used
this is a good model for health insurance providersbut needs to be repaired to use for hospital admissions
Important: Must keep potentially offending features in model!
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 16 / 41
Some of the things the intelligible model learned:
Age 105 is safer than Age 95We should have a retirement variableHas Asthma => lower riskHistory of chest pain => lower riskHistory of heart disease => lower risk
Good we didn’t deploy neural net back in 1995
But can understand, edit and safely deploy intelligible GA2M model
Intelligible/transparent model is like having a magic pair of glasses
Model correctness depends on how model will be used
this is a good model for health insurance providersbut needs to be repaired to use for hospital admissions
Important: Must keep potentially offending features in model!
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 16 / 41
Pairwise Interactions?
0 10 0 1
1 1 0
Parity is the classic (extreme) interaction
For N-bit parity, need all N bits at same time to calculate parityNo correlation between any of the bits and parity signalNo information in any subset of the bits
Interactions can’t be modeled as sum of independent effects
Interactions important on some problems, less on others
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 17 / 41
Age vs. Cancer Pairwise Interaction (Pneumonia-95)
no cancer has cancer
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 18 / 41
30-day Hospital Readmission (joint work with NYP)
30-day Hospital Readmission Data
larger, modern datasetrecords from NYP 2011-2014train=195,901 (2011-12); test=100,823 (2013)3,956 features for each patientgoal: predict probability patient will be readmitted within 30 days8.91% of patients readmitted within 30 days
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 19 / 41
Age Plot for 30-Day Hospital Readmission
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 20 / 41
Age Plot for 30-Day Readmission: Zomm in on Age 0-2
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 21 / 41
Quick look at two 30-day Readmission Patients
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 22 / 41
GA2Ms for FAT/ML: Detecting and Removing Bias
Bias is in the data, not the learning algorithm
Models trained on data will learn any biases in the dataML for resume processing will learn gender bias if data has a gender biasML for recidivism prediction will learn race bias if data has a race bias...
How to deal with bias using transparent models:must keep bias features in data when model is trainedremove what was learned from these bias features after training
If offending bias variables are eliminated prior to training:often can’t tell you still have a problemmakes it harder to correct the problem
EU General Data Protection Regulation (goes into effect 2018):Article 9 makes it more difficult to use personal data revealingracial or ethnic origin and other “special categories”
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 23 / 41
GA2Ms for FAT/ML: Detecting and Removing Bias
Bias is in the data, not the learning algorithm
Models trained on data will learn any biases in the dataML for resume processing will learn gender bias if data has a gender biasML for recidivism prediction will learn race bias if data has a race bias...
How to deal with bias using transparent models:must keep bias features in data when model is trainedremove what was learned from these bias features after training
If offending bias variables are eliminated prior to training:often can’t tell you still have a problemmakes it harder to correct the problem
EU General Data Protection Regulation (goes into effect 2018):Article 9 makes it more difficult to use personal data revealingracial or ethnic origin and other “special categories”
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 23 / 41
GA2Ms for FAT/ML: Detecting and Removing Bias
Bias is in the data, not the learning algorithm
Models trained on data will learn any biases in the dataML for resume processing will learn gender bias if data has a gender biasML for recidivism prediction will learn race bias if data has a race bias...
How to deal with bias using transparent models:must keep bias features in data when model is trainedremove what was learned from these bias features after training
If offending bias variables are eliminated prior to training:often can’t tell you still have a problemmakes it harder to correct the problem
EU General Data Protection Regulation (goes into effect 2018):Article 9 makes it more difficult to use personal data revealingracial or ethnic origin and other “special categories”
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 23 / 41
GA2Ms for FAT/ML: Detecting and Removing Bias
Bias is in the data, not the learning algorithm
Models trained on data will learn any biases in the dataML for resume processing will learn gender bias if data has a gender biasML for recidivism prediction will learn race bias if data has a race bias...
How to deal with bias using transparent models:must keep bias features in data when model is trainedremove what was learned from these bias features after training
If offending bias variables are eliminated prior to training:often can’t tell you still have a problemmakes it harder to correct the problem
EU General Data Protection Regulation (goes into effect 2018):Article 9 makes it more difficult to use personal data revealingracial or ethnic origin and other “special categories”
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 23 / 41
GA2Ms for FAT/ML: Detecting and Removing Bias
Bias is in the data, not the learning algorithm
Models trained on data will learn any biases in the dataML for resume processing will learn gender bias if data has a gender biasML for recidivism prediction will learn race bias if data has a race bias...
How to deal with bias using transparent models:must keep bias features in data when model is trainedremove what was learned from these bias features after training
If offending bias variables are eliminated prior to training:often can’t tell you still have a problemmakes it harder to correct the problem
EU General Data Protection Regulation (goes into effect 2018):Article 9 makes it more difficult to use personal data revealingracial or ethnic origin and other “special categories”
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 23 / 41
But we have yet another transparency trick up our sleeve...
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 24 / 41
ProPublica COMPAS Recidivism Data
Is COMPAS model biased?
Is recidivism training data biased?
Is COMPAS model more biased than training data might warrant?
Transparent Modeling Trick:
train transparent model #1 on raw recidivism datatrain transparent model #2 on COMPAS model predictionsReal vs. Memorex:compare what is learned in model #1 from raw data to model #2 from COMPAS mimic
one caveat: ProPublica data doesn’t have all of the COMPAS features
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 25 / 41
ProPublica COMPAS Recidivism Data
Is COMPAS model biased?
Is recidivism training data biased?
Is COMPAS model more biased than training data might warrant?
Transparent Modeling Trick:
train transparent model #1 on raw recidivism datatrain transparent model #2 on COMPAS model predictionsReal vs. Memorex:compare what is learned in model #1 from raw data to model #2 from COMPAS mimic
one caveat: ProPublica data doesn’t have all of the COMPAS features
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 25 / 41
ProPublica COMPAS Recidivism Data
Is COMPAS model biased?
Is recidivism training data biased?
Is COMPAS model more biased than training data might warrant?
Transparent Modeling Trick:
train transparent model #1 on raw recidivism datatrain transparent model #2 on COMPAS model predictionsReal vs. Memorex:compare what is learned in model #1 from raw data to model #2 from COMPAS mimic
one caveat: ProPublica data doesn’t have all of the COMPAS features
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 25 / 41
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 26 / 41
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 27 / 41
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 28 / 41
black asian white hispanic natAmer other
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 29 / 41
female male
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 30 / 41
female male
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 31 / 41
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 32 / 41
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 33 / 41
not felony felony
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 34 / 41
Advantages of Transparent Modeling for Bias Detection and Elimination
Don’t have to know what biases to look for in advance
Don’t have to design statistical tests for biases in advance
Just train model, and look at what it learned — model can (and will!) surprise you
Modularity of GAMs makes bias easier to recognize
Modularity of GAMs makes bias easier to correct
Accuracy of GA2Ms means less is missing — GA2M model often is as accurate as anyother model black-box we could train on data
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 35 / 41
Advantages of Transparent Modeling for Bias Detection and Elimination
Don’t have to know what biases to look for in advance
Don’t have to design statistical tests for biases in advance
Just train model, and look at what it learned — model can (and will!) surprise you
Modularity of GAMs makes bias easier to recognize
Modularity of GAMs makes bias easier to correct
Accuracy of GA2Ms means less is missing — GA2M model often is as accurate as anyother model black-box we could train on data
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 35 / 41
Advantages of Transparent Modeling for Bias Detection and Elimination
Don’t have to know what biases to look for in advance
Don’t have to design statistical tests for biases in advance
Just train model, and look at what it learned — model can (and will!) surprise you
Modularity of GAMs makes bias easier to recognize
Modularity of GAMs makes bias easier to correct
Accuracy of GA2Ms means less is missing — GA2M model often is as accurate as anyother model black-box we could train on data
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 35 / 41
Comments
GA2Ms are not causal models
because they’re simple and transparent, often find causal effectsbut it’s up to the user to figure out what’s really going on
GA2Ms do not cure the curse of dimensionality and correlation
GA2Ms are intelligible only if features are intelligible
GA2Ms are not a replacement for deep learning on raw signals
does not work as well as deep nets on pixels, speech signals, ...works best on features crafted by humans
GA2Ms are not perfect yet...
we’re still doing research to make the GA2Ms betterbut they’re now good enough to be used instead of logistic regression
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 36 / 41
Comments
GA2Ms are not causal models
because they’re simple and transparent, often find causal effectsbut it’s up to the user to figure out what’s really going on
GA2Ms do not cure the curse of dimensionality and correlation
GA2Ms are intelligible only if features are intelligible
GA2Ms are not a replacement for deep learning on raw signals
does not work as well as deep nets on pixels, speech signals, ...works best on features crafted by humans
GA2Ms are not perfect yet...
we’re still doing research to make the GA2Ms betterbut they’re now good enough to be used instead of logistic regression
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 36 / 41
Comments
GA2Ms are not causal models
because they’re simple and transparent, often find causal effectsbut it’s up to the user to figure out what’s really going on
GA2Ms do not cure the curse of dimensionality and correlation
GA2Ms are intelligible only if features are intelligible
GA2Ms are not a replacement for deep learning on raw signals
does not work as well as deep nets on pixels, speech signals, ...works best on features crafted by humans
GA2Ms are not perfect yet...
we’re still doing research to make the GA2Ms betterbut they’re now good enough to be used instead of logistic regression
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 36 / 41
Comments
GA2Ms are not causal models
because they’re simple and transparent, often find causal effectsbut it’s up to the user to figure out what’s really going on
GA2Ms do not cure the curse of dimensionality and correlation
GA2Ms are intelligible only if features are intelligible
GA2Ms are not a replacement for deep learning on raw signals
does not work as well as deep nets on pixels, speech signals, ...works best on features crafted by humans
GA2Ms are not perfect yet...
we’re still doing research to make the GA2Ms betterbut they’re now good enough to be used instead of logistic regression
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 36 / 41
Summary
High accuracy on test set is not enough
There are land mines hidden in the data
You need magic glasses to see the landmines
It’s critical to understand model before deploying it
Model correctness depends on how model will be used
New GA2Ms give us accuracy and intelligibility at same time
Important to keep potentially offending variables in model so biascan be detected and then removed after training
If you eliminate offending variables before training you:
can’t tell you have a problemmake it harder to correct the problem
Transparency allows you to detect problems you didn’t anticipate in advance
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 37 / 41
Intelligible Models or Black Box?
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 38 / 41
References
Y. Lou, R. Caruana, and J. Gehrke.Intelligible Models for Classification and Regression.In KDD, 2012.
Y. Lou, R. Caruana, J. Gehrke, and G. Hooker.Accurate Intelligible Models With Pairwise Interactions.In KDD, 2013.
R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, Noemie Elhadad.Intelligible Models for Healthcare.In KDD, 2015.
T. Hastie and R. Tibshirani.Generalized additive models.Chapman & Hall/CRC, 1990.
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 39 / 41
Thank You!
Thanks to MSR, after 5+ years of research we now havesomething no one else has that’s important for healthcareand also for regulatory transparency and dealing with bias.
The world is beginning to take note and this work alreadyhas been mentioned in half a dozen press articles.
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 40 / 41
GA2M Algorithm Sketch
Stage 1: build best additive model using only 1-dim components
Additive effects are now modeledIf Stage 1 done perfectly, only have interaction (and noise) in residuals
Stage 2: fix the one-dimensional functions
Detect pairwise interactions on residuals (new FAST algorithm)
Stage 3: build shape models for most important pairwise interactions on residuals
Stage 4: post-process shape plots
center average prediction of each plot to improve modularitysort terms by importance to aid intelligibility
Bag (repeat) process 10-100 times to create pseudo-confidence intervals and furtherreduce overfitting
Rich Caruana (Microsoft Research) FAT/ML 2017: Intelligible Models August 14, 2017 41 / 41