a user study on the effect of aggregating explanations for ... · a user study on the effect of...
TRANSCRIPT
![Page 1: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/1.jpg)
A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models
Josua Krause*, Adam Perer**, Enrico Bertini*
* **
Mon, August 20th 2018
[work in progress]
![Page 2: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/2.jpg)
2"Why Should I Trust You?" Explaining the Predictions of Any Classifier Marco Riberio, Sameer Singh, Carlos Guestrin International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD 2016)
Instance Explanations
![Page 3: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/3.jpg)
3"Why Should I Trust You?" Explaining the Predictions of Any Classifier Marco Riberio, Sameer Singh, Carlos Guestrin International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD 2016)
Finding Data Biases
![Page 4: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/4.jpg)
4
Problem:Inspecting single instances
does not scale well
![Page 5: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/5.jpg)
5
Solution:Aggregating data and explanations
![Page 6: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/6.jpg)
6
Solution:Aggregating data and explanations
Correct Incorrect
Negative
Negative
Positive
Positive
Prediction
Ground Truth
![Page 7: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/7.jpg)
7
Solution:Aggregating data and explanations
Living Area (numeric)
Correct Incorrect
Negative
Negative
Positive
Positive
Prediction
Ground Truth
![Page 8: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/8.jpg)
8
Living Area (numeric)
Correct Incorrect
Negative
Negative
Positive
Positive
Prediction
Ground Truth
Feature Value
![Page 9: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/9.jpg)
9
Living Area (numeric)
Correct Incorrect
Negative
Negative
Positive
Positive
Prediction
Ground Truth
Feature Value
Concentration Within Subset
![Page 10: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/10.jpg)
10
Living Area (numeric)
Correct Incorrect
Negative
Negative
Positive
Positive
Prediction
Ground Truth
Feature Value
Concentration Within Subset
Feature Importance
![Page 11: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/11.jpg)
Sorted by Importance
![Page 12: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/12.jpg)
What is the impact of aggregation?
What is the impact of instance-level explanations?
How do those settings affect the ability to detect biases in the data?
12
![Page 13: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/13.jpg)
13
Four ConditionsTable Histogram
No Explanation
Explanation
![Page 14: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/14.jpg)
14
Four ConditionsTable Histogram
No Explanation
Explanation
![Page 15: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/15.jpg)
15
Four ConditionsTable Histogram
No Explanation
Explanation
![Page 16: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/16.jpg)
16
Four ConditionsTable Histogram
No Explanation
Explanation
![Page 17: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/17.jpg)
17
Four ConditionsTable Histogram
No Explanation
Explanation
![Page 18: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/18.jpg)
18
Two Data Sets
![Page 19: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/19.jpg)
19
Two Data Sets
Living Area (numeric)
High Price Low Price
![Page 20: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/20.jpg)
20
Two Data Sets
Model Accuracy: 81.959%
Model Accuracy: 88.325%
![Page 21: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/21.jpg)
21
Individual models:• Do you think the predictions of the model make sense?
5 point Likert scale (Not at all – Very much) • How well does the model perform in terms of accuracy?
5 point Likert scale (Not much – Very well) • How much do you trust the model?
5 point Likert scale (Not at all – Very much) • Why do you trust or not trust this model?
Free text answer Summary:
Which model do you prefer?Multiple choice and text answer
Questions
![Page 22: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/22.jpg)
22
100 participants
4 conditions (25 each): • Table without Explanations (T/N) • Table with Explanations (T/E) • Histogram without Explanations (H/N) • Histogram with Explanations (H/E)
Random model order Correctly identified more accurate model
Evaluation metrics:Model preference (trust) Bias detection
Study
![Page 23: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/23.jpg)
23
Participants Who Trusted the Correct Model
40%
30%
20%
10%
00%T/E H/N H/E
T: Table H: Histogram E: Explanation N: No Explanation
![Page 24: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/24.jpg)
40%
30%
20%
10%
00%T/E H/N H/E
24
Participants Who Trusted the Correct Model
Significant improvement!
p-value 0.0477 < 0.05
vs.
T: Table H: Histogram E: Explanation N: No Explanation
![Page 25: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/25.jpg)
40%
30%
20%
10%
00%T/E H/N H/E
25
Participants Who Trusted the Correct Model
p-value 0.0982 > 0.05
vs.
T: Table H: Histogram E: Explanation N: No Explanation
![Page 26: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/26.jpg)
40%
30%
20%
10%
00%T/E H/N H/E
vs.
Participants Who Trusted the Correct Model
26
"It has higher accuracy so should be more trustworthy than the other one. However some of the results don’t make sense to me. Maybe this is just an atypical property
market."
"It is accurate, yet the predictions do not make much sense. Higher quality houses having a larger amount of low priced houses, percentage-wise? More rooms, area, or
stories resulting in lower prices? The logic does not work out."
"larger houses are valued lower than others which are smaller"
T: Table H: Histogram E: Explanation N: No Explanation
![Page 27: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/27.jpg)
40%
30%
20%
10%
00%T/E H/N H/E
Participants Who Trusted the Correct Model
vs.
27T: Table H: Histogram E: Explanation N: No Explanation
"If the data says it’s true, then it’s true I suppose and it’s more trustworthy than my common sense."
"I feel like the results of [the biased model] where strange even though they where correct according to the dataset."
"I’m drawn to trusting the model which was more accurate even though it didn’t entirely make sense to me."
25% of the participants who found the biasdid not change their mind!
![Page 28: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/28.jpg)
40%
30%
20%
10%
00%T/E H/N H/E
50%
28
Participants Who Detected the Bias
p-value 0.0359 < 0.05
vs.
Significant improvement!
T: Table H: Histogram E: Explanation N: No Explanation
![Page 29: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/29.jpg)
29
Participants Who Detected the Bias
T: Table H: Histogram E: Explanation N: No Explanation
40%
30%
20%
10%
00%T/E H/N H/E
50%
T/N
p-value 0.0311 < 0.05
![Page 30: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/30.jpg)
30Bootstrapped 95% Confidence Intervals
T: Table H: Histogram E: Explanation N: No Explanation
Number of Hovered Cells
Number of Hovered Bars
H/E
H/N
T/E
T/N
0 100 200 300 400 500
0 200 400 600 800 1000
![Page 31: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/31.jpg)
Number of Hovered Cells
Number of Hovered Bars
H/E
H/N
T/E
T/N
0 100 200 300 400 500
0 200 400 600 800 1000
Bootstrapped 95% Confidence Intervals
Number of Hovered Bars
31
![Page 32: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/32.jpg)
40%
30%
20%
10%
00%T/E H/N H/E
50%
T/N 32
Participants Who Detected the BiasSimilar performance!
T: Table H: Histogram E: Explanation N: No Explanation
![Page 33: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/33.jpg)
33
vs.
Note that the task was chosen in a way that under all conditions it was possible to find the bias.
Histograms scale better to larger data sets or more complex errors in the data.
In tables you have to extrapolate...
![Page 34: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/34.jpg)
Lessons LearnedPeople trust accuracy (too much).
Aggregating instance-level explanations significantly helps detecting biases
compared to individual explanations.
Individual instance-level explanations may hurt performance.
34
![Page 35: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/35.jpg)
Further Work
More targeted studies to confirm hypotheses
Different results for expert users?
35
![Page 36: A User Study on the Effect of Aggregating Explanations for ... · A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models Josua Krause*, Adam](https://reader033.vdocuments.net/reader033/viewer/2022042120/5e9a1c5a450ebb731c096c14/html5/thumbnails/36.jpg)
A User Study on the Effect of Aggregating Explanations for Interpreting Machine Learning Models
Josua Krause*, Adam Perer**, Enrico Bertini*
* **
Thank You!
[work in progress]