manipulating and measuring model...
TRANSCRIPT
ManipulatingandMeasuringModelInterpretability
JennWortmanVaughanMicrosoftResearch
BasedonjointworkwithForough Poursabzi-Sangdeh,DanGoldstein,JakeHofman,&HannaWallach
Approach1:DesignSimpleModels
PointSystems(Jungetal.,2017;Ustun &Rudin,2015,etc.)
Classicmethods:decisiontrees,rulelists(if-then-else),rulesets,(sparse)linearmodels,…
GeneralizedAdditiveModels(Lou,Caruana,etal.,2012&2013)
𝑦 = 𝑓$ 𝑥$ +…+𝑓) 𝑥)
Approach2:DesignSimpleExplanationsfor(PotentiallyComplex)Models
InterpretableLocalApproximations
(Ribeiroetal.,2016;LundbergandLee,2017)
ModelVisualizations(e.g.,workatGooglePAIR)
Interpretability?
ALegalNecessity
Thedatacontrollershallprovide“meaningfulinformationaboutthelogicinvolved,aswellasthesignificanceandtheenvisagedconsequencesofsuchprocessing.”
InterpretabilityisaLatentProperty
Interpretability
trust
abilitytosimulate
abilitytodebug
…
numberoffeatures
clearvs.blackbox
linear?
UI
…
DifferentUsersandDifferentNeeds
Explainsin
gle
pred
ictio
n
Und
erstand
mod
elglobally
Makebe
tter
decisio
ns
Debu
gmod
els
Assessbias
Inspire
trust
CEOs
Datascientists
Laypeople
Regulators
ApproachA
ApproachB
ApproachC
Interpretability
trust
abilitytosimulate
abilitytodebug
…
numberoffeatures
clearvs.blackbox
linear?
UI
…
Propertiesofthesystemdesign
Propertiesofhumanbehavior
Interpretabilityisnotapurelycomputationalproblem.
Weneedinterdisciplinaryapproachestoaddressit.
Canwelearnfrompsychology?
– Psychologistsandsocialscientistshavebeenstudyingtrustinmodelssincethe1950s• E.g.,literatureonalgorithmaversion
– Generalapproach:Runrandomizedhumansubjectexperimentstoisolateandmeasuretheimpactofdifferentfactorsontrust
Ourgoal:Applythisapproachtounderstandthefundamentalpropertiesofhumanbehaviorpertinenttointerpretability
Interpretability
trust
abilitytosimulate
abilitytodebug
…
numberoffeatures
clearvs.blackbox
linear?
UI
…
Propertiesofthesystemdesign
Propertiesofhumanbehavior
InitialExperiment
– Weranarandomizedhumansubjectexperimenton1250participantsfromMechanicalTurk
– Wevaried• Thenumberoffeatures• Black-boxvs.visible(“clear”)internals
– Wemeasured• Trustinthemodel• Simulatability• Erroroftheenduser’spredictions
FocusonLayPeople
Explainsin
gle
pred
ictio
n
Und
erstand
mod
elglobally
Makebe
tter
decisio
ns
Debu
gmod
els
Assessbias
Inspire
trust
CEOs
Datascientists
Laypeople
Regulators
? ? ?
PredictiveTask
– ParticipantsaskedtopredictthepricesofUpperWestSideapartmentswiththehelpofamodel
ExperimentalConditions2-feature,black-box
2-feature,clear
8-feature,black-box
8-feature,clear
ExperimentalConditions2-feature,black-box
2-feature,clear
8-feature,black-box
8-feature,clear
ExperimentalConditions2-feature,black-box
2-feature,clear
8-feature,black-box
8-feature,clear
ExperimentInterface
ExperimentInterface
Pre-registeredHypotheses
1. Theclear,2-featuremodelwillbeeasiestforparticipantstosimulate.
2. Participantswillfollowtheclear,2-featuremodelmorethantheblack-box,8-featuremodel.
3. Behaviorwillvaryacrossconditionswhenanunusualexampleleadsamodeltomakeahighlyinaccurateprediction.
BB2 Clear2 BB8 Clear8
SimulationError
|modelprediction– guessofmodelprediction|
Pre-registeredHypotheses
1. Theclear,2-featuremodelwillbeeasiestforparticipantstosimulate.
2. Participantswillfollowtheclear,2-featuremodelmorethantheblack-box,8-featuremodel.
3. Behaviorwillvaryacrossconditionswhenanunusualexampleleadsamodeltomakeahighlyinaccurateprediction.
BB2 Clear2 BB8 Clear8
DeviationfromtheModel
|modelprediction– participant’sprediction|
BB2 Clear2 Clear8BB8 None
PredictionError
|actualprice– participant’sprediction|
Pre-registeredHypotheses
1. Theclear,2-featuremodelwillbeeasiestforparticipantstosimulate.
2. Participantswillfollowtheclear,2-featuremodelmorethantheblack-box,8-featuremodel.
3. Behaviorwillvaryacrossconditionswhenanunusualexampleleadsamodeltomakeahighlyinaccurateprediction.
ABadPrediction
BB2 Clear2 BB8 Clear8
Dopeopledeviatewhentheyshould?
|modelprediction– participant’sprediction|
Potentialconcern:MaybeNewYorkCitypricesaretoocrazy.
ScaledDownPrices:Hypotheses
1. Theclear,2-featuremodelwillbeeasiestforparticipantstosimulate.
2. Participantswillfollowtheclear,2-featuremodelmorethantheblack-box,8-featuremodel.
3. Participantswillfollowtheclearmodelsmorethanblack-boxwhenanunusualexampleleadsamodeltomakeahighlyinaccurateprediction.
BB2 Clear2 BB8 Clear8
ScaledDownPrices:SimulationError
|modelprediction– guessofmodelprediction|
ScaledDownPrices:Hypotheses
1. Theclear,2-featuremodelwillbeeasiestforparticipantstosimulate.
2. Participantswillfollowtheclear,2-featuremodelmorethantheblack-box,8-featuremodel.
3. Participantswillfollowtheclearmodelsmorethanblack-boxwhenanunusualexampleleadsamodeltomakeahighlyinaccurateprediction.
ScaledDownPrices: Deviation
|modelprediction– participant’sprediction|BB2 Clear2 BB8 Clear8
BB2 Clear2 Clear8BB8 None
ScaledDownPrices:PredictionError
|actualprice– participant’sprediction|
ScaledDownPrices:Hypotheses
1. Theclear,2-featuremodelwillbeeasiestforparticipantstosimulate.
2. Participantswillfollowtheclear,2-featuremodelmorethantheblack-box,8-featuremodel.
3. Participantswillfollowtheclearmodelsmorethanblack-boxwhenanunusualexampleleadsamodeltomakeahighlyinaccurateprediction.
BB2 Clear2 BB8 Clear8
Dopeopledeviatewhentheyshould?
|modelprediction– participant’sprediction|
Self-reportedTrust
BB2 Clear2 BB8 Clear8
Summaryofresults
– Ashypothesized,participantsarebetterabletosimulatetheclear,2-featuremodelcomparedwiththeblack-box,8-featuremodel.
– Thereisnosignificantdifferenceinparticipants’deviationfromthemodelacrossconditions.
– Whengivena“weird”exampleinwhichthemodeliswrong,participantsintheclearconditionsdeviatelessthanthoseinblack-box.
Interpretability
trust
abilitytosimulate
abilitytodebug
…
numberoffeatures
clearvs.blackbox
linear?
UI
…
Propertiesofthesystemdesign
Propertiesofhumanbehavior
DifferentUsersandDifferentNeeds
Explainsin
gle
pred
ictio
n
Und
erstand
mod
elglobally
Makebe
tter
decisio
ns
Debu
gmod
els
Assessbias
Inspire
trust
CEOs
Datascientists
Laypeople
Regulators
? ? ?
Interpretabilityisnotapurelycomputationalproblem.
Weneedinterdisciplinaryapproachestoaddressit.
