the role of explanations in interacting with intelligent ... · the role of explanations in...

The role of explanations in interacting with intelligent machines

Dr Simone Stumpf Centre for Human-Computer Interaction Design [email protected] @DrSimoneStumpf

Overview •  Why explain? •  Effects of explanations? •  How to explain? •  The problem…

What are intelligent systems?

•  Intelligent systems learn what to do by analysing user input –  Learn behaviour before system is deployed (offline learning) –  Update what is learned based on new data (online learning) –  Take user feedback into account (interactive machine learning, active learning)

•  Amazon Recommendations, Gmail Priority Inbox, Fraud detection / Network Intrusion, Clinical Decision Support Systems

•  Fairly accurate but they do make mistakes –  Correcting them costly, by providing more training examples

CDSS

Uncertainty

Misuse Over-Reliance

Medical observations

Suggestions

Stored Knowledge

Insufficient or incorrect:

Uncertainty

Poor data or algorithm:

Uncertainty

Uncertainty

Reliability? Trust?

Disuse Self-Reliance

or

Feedback

Why explain? •  Explanations help with intelligibility (Lim & Dey 2010,

Kulesza et al. 2012)

•  Explanations help with user experience (Tintarev and Mastoff 2011)

CDSS

Suggested diagnosis: Vestibular Migraine Confidence: 94% Suggested because: Patient’s Gaze test and Smooth Pursuit exam results, history of recent falls, and vertigo triggered by diet match disease profile

“Explanatory debugging” for machine learning

•  Debugging relatively well understood in programming

•  How can we give end users access to information that helps with debugging machine learning?

EndUser

IntelligentAgent

MentalModel

FeedbackExplanation

Effects of explanations

•  Intelligibility helps with correcting machine learning (Stumpf et al. 2009, Stumpf et al. 2008)

•  Explanations of system increased mental model correctness, perceived user experience and self-efficacy (Kulesza et al. 2012)

•  Explanations help corrections: increased accuracy with less training data (Kulesza et al. 2015)

•  Explanations considered harmful?

EndUser

IntelligentAgent

MentalModel

FeedbackExplanation

On the way to correcting machine learning

•  How can we best explain system behaviour to end users? –  Explanation styles –  Components of explanations

•  How can we support them to feedback their knowledge to the system? –  Understand debugging behaviour –  Ways to debug

Explanation styles

Machine learning algorithms do not work like people think they do –  Complex statistical approaches –  Usually not “rules”

•  Switch from one “style” to another to explain an algorithm?

Explanations styles (Stumpf et al. 2009)

•  Enron email dataset folders (farmer-d): Personal, Resume, Bankrupt, Enron News (122 messages)

•  Lo-fi prototypes with 3 explanation styles of 3 different algorithms –  Rule-based –  Similarity-based –  Keyword-based

•  13 participants •  Think-aloud •  Follows the “Natural Programming”

approach [Pane and Myers 2002]

Results

•  Understanding of explanations: –  Rule-based best understood –  Keyword-based also good but negative weights problematic

(absence of features) –  Serious understandability problems with Similarity-based

•  Preference: –  No clear overall winner, very individual

•  Factors that mattered: –  Perception of algorithm’s goodness –  Clear communication of how algorithm works

Explanation components

What do we need to explain about how a machine learning system works?

–  Existing Guidelines e.g. CDSS •  Provide Certainty/Confidence •  Provide Why?

–  Intelligibility types for context-aware systems (Lime & Dey 2009; Lim 2012)

•  What? Why? Why Not? What If? How to? •  Inputs, Outputs, Model

•  How can we explain these components best?

Explanation components (Kulesza et al. 2010)

Results: what information is useful?

4 software versions, each with different combinations of components to explain parts of system’s behaviour, 74 participants, each used two versions of the prototype

•  Impacts counts and Absence widgets not understood or liked

•  Popularity Bar and Confidence well received by participants (but careful about unintended effects)

Explanation components (Kulseza et al. VLHCC13)

��

JURXQGHG� WKHRU\� PHWKRGV� >��@�� 7KH� UHVXOWLQJ� FRGH� VHW� LV�SUHVHQWHG�LQ�7DEOH�,,��

:H�WUDQVFULEHG�SDUWLFLSDQW�XWWHUDQFHV�GXULQJ�HDFK�VRQJ�DQG�DSSOLHG� WKH� FRGHV� WR� WKHVH� XWWHUDQFHV� �HDFK� FRGH� FRXOG� EH�DSSOLHG�� DW� PRVW�� RQFH� SHU� VRQJ�� 7ZR� UHVHDUFKHUV�

LQGHSHQGHQWO\�FRGHG�D�VPDOO�SRUWLRQ�RI�WKH�WUDQVFULSWV�DQG�WKHQ�GLVFXVVHG� DUHDV�RI� GLVDJUHHPHQW��2QFH� WKH� UHVHDUFKHUV� DJUHHG�RQ� WKH� FRGLQJ� RI� WKHVH� WUDQVFULSWV�� WKH\� LQGHSHQGHQWO\� FRGHG�WZR� FRPSOHWH� WUDQVFULSWV� ��RI� WKH�GDWD�²WKHLU� DJUHHPHQW��DV� FDOFXODWHG� E\� WKH� -DFFDUG� LQGH[� �WKH� LQWHUVHFWLRQ� RI� DOO�DSSOLHG�FRGHV�RYHU� WKH�XQLRQ�RI� DOO� DSSOLHG� FRGHV��ZDV��*LYHQ� WKLV� DFFHSWDEOH� OHYHO� RI� DJUHHPHQW�� D� VLQJOH� UHVHDUFKHU�FRGHG�WKH�UHPDLQLQJ�WUDQVFULSWV�DQG�SRVW�VWXG\�TXHVWLRQQDLUHV��

:H�DVVHVVHG�SDUWLFLSDQWV¶�PHQWDO�PRGHOV�XVLQJ�WKH�QXPEHU�RI� FRUUHFW� DQG� LQFRUUHFW� VWDWHPHQWV� WKH\� PDGH� GXULQJ� WKH�H[SHULPHQW�DQG�RQ�WKH�SRVW�VWXG\�TXHVWLRQQDLUH��7DEOH�,,�VKRZV�ZKLFK� W\SHV� RI� YHUEDOL]DWLRQV�UHVSRQVHV� ZHUH� FRUUHFW� YV��LQFRUUHFW��:H�WKHQ�VXPPHG�WKH�SDUWLFLSDQWV¶�FRUUHFW�UHVSRQVHV��VXEWUDFWHG�WKH�QXPEHU�RI�LQFRUUHFW�UHVSRQVHV��DQG�WUDQVODWHG�WKH�UHVXOW� WR�EH� D� VFRUH�EHWZHHQ�� ORZHVW� ILGHOLW\�PHQWDO�PRGHO��DQG� �� KLJKHVW� ILGHOLW\� PHQWDO� PRGHO�� 3DUWLFLSDQWV¶�YHUEDOL]DWLRQV� GXULQJ� WKH� VWXG\� DQG� SRVW�VWXG\� TXHVWLRQQDLUH�UHVSRQVHV�ZHUH�ERWK�ZHLJKWHG�HTXDOO\��

9�� 5(68/76�

$�� 54��DQG�54��6RXQGQHVV��&RPSOHWHQHVV��DQG�7\SHV�++�SDUWLFLSDQWV�KDG�WKH�KLJKHVW�ILGHOLW\�PHQWDO�PRGHOV��$V�

)LJ�� VKRZV��++� SDUWLFLSDQWV� DFKLHYHG� WKUHH� RI� WKH� WRS� IRXU�VFRUHV�� ,Q�FRQWUDVW�� DOO�EXW�RQH�RI� WKH�SDUWLFLSDQWV� LQ� WKH�RWKHU�WUHDWPHQWV� FOXVWHUHG� DURXQG� ORZHU� VFRUHV�� 7KLV� VXUSULVHG� XV�EHFDXVH� ZH� KDG� H[SHFWHG� WKH� ++� WUHDWPHQW� WR� RYHUORDG�SDUWLFLSDQWV� WR� WKH� SRLQW� ZKHUH� WKH\� ZRXOG� QRW� DWWHQG� WR� VR�PXFK� FRPSOH[� LQIRUPDWLRQ�� ,QVWHDG�� ZH� H[SHFWHG� WKH� 00�WUHDWPHQW� WR� EH� D� ³VZHHW� VSRW´� LQ� WKH� WUDGH�RII� EHWZHHQ�LQIRUPDWLYHQHVV� DQG� VLPSOLFLW\²EXW� PRVW� RI� WKH� 00�SDUWLFLSDQWV�FOXVWHUHG�DURXQG�WKH�ORZHVW�VFRUHV��

)XUWKHU�� ++� SDUWLFLSDQWV¶� PHQWDO� PRGHO� VFRUHV� ZHUH�FRQVLVWHQWO\� KLJK� DFURVV� IHDWXUHV� DQG� SURFHVVHV�� DV� )LJ�� ¶V�UHVXOWV� IURP� WKH� SRVW�WDVN� TXHVWLRQQDLUH� VKRZ�� ,Q� IDFW�� DV� WKH�WKLUG� FROXPQ�RI� )LJ�� VKRZV��++�SDUWLFLSDQWV�ZHUH� WKH�RQO\�RQHV�WR�FRUUHFWO\�GHVFULEH�WKH�VRQJ�VHOHFWLRQ�SURFHVV��FRGHG�DV�SHU�7DEOH�,,��DQG�RQO\�RQH�++�SDUWLFLSDQW�PDGH�DQ\�LQFRUUHFW�SRVW�WDVN�REVHUYDWLRQV�DW�DOO��DV�WKH�ULJKW�KDOI�RI�)LJ��VKRZV��

��&RPSOHWHQHVV�DQG�,QWHOOLJLELOLW\�7\SHV��

7ZR� RI� WKH� LQWHOOLJLELOLW\� W\SHV�� ZK\� DQG� LQSXW�� UHODWH� WR�IHDWXUHV�� DQG�SDUWLFLSDQWV� WHQGHG� WR�GR�EHWWHU�DW�XQGHUVWDQGLQJ�

�

��

�

�

� �

�

)LJ��([FHUSWV�IURP�WKH�:K\�WKLV�6RQJ�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH��7RS�OHIW��7KH�KLJK�VRXQGQHVV�VKHHW�VKRZHG�D�UDQGRP�VDPSOH�RI�GHFLVLRQ�WUHHV�IURP� WKH�EDJJLQJ� HQVHPEOH��7RS� ULJKW��(DFK� WUHH�ZDV� UHSUHVHQWHG�DV� D� VHW� RI�RUGHUHG�IHDWXUHV�ZLWK�DOORZHG�UDQJHV�RI�YDOXHV��7KH�PHGLXP�VRXQGQHVV�VKHHW�ZDV�VLPLODU��EXW�RQO\�VKRZHG�RQH�GHULYHG�GHFLVLRQ�WUHH�WKDW�DSSUR[LPDWHG�WKH�EDJJLQJ� HQVHPEOH¶V� UHDVRQLQJ�� %RWWRP� ULJKW�� 7KH� ORZ� VRXQGQHVV� VKHHW� ZDV�DOVR� VLPLODU�� EXW� RQO\� VKRZHG� RQH� GHULYHG� GHFLVLRQ� VWXPS� �VLQJOH�IHDWXUHG�WUHH�� %RWWRP� OHIW�� )RU� WKH� ++�� /6+&�� DQG�00� WUHDWPHQWV�� WKLV� VKHHW� DOVR�LQFOXGHG�WKH�FHUWDLQW\�LQWHOOLJLELOLW\�W\SH��

�)LJ��([FHUSW�IURP�WKH�:K\�WKLV�$UWLVW�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH��ZKLFK�VKRZHG�WKH�DUWLVWV�VHOHFWHG�E\�WKHLU�XVHU�QHLJKERUKRRG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��EXW�ZLWK�GLIIHUHQW�QHLJKERUKRRG�VL]HV��VHH�7DEOH�,��

�)LJ�� ([FHUSW� IURP� WKH� :KDW� WKH� &RPSXWHU� .QRZV� H[SODQDWLRQ� �LQSXW�LQWHOOLJLELOLW\� W\SH�� SURYLGLQJ� D� FRPSUHKHQVLYH� OLVW� RI� IHDWXUHV� WKDW� WKH�UHFRPPHQGHU�XVHG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��

�)LJ�� ([FHUSW� IURP� WKH� +RZ� LW� $OO� :RUNV� 7RJHWKHU� H[SODQDWLRQ� �PRGHO�LQWHOOLJLELOLW\�W\SH��ZKLFK�VKRZHG�KRZ�WKH�SDUWLFLSDQWV¶�OLVW�RI�DUWLVWV�ZDV�XVHG�WR� PDNH� VRQJ� UHFRPPHQGDWLRQV�� 3RVLWLYH� DQG� QHJDWLYH� WUDLQLQJ� VHWV� ZHUH�FRORU�FRGHG� IRU�SDUWLFLSDQWV� WKURXJKRXW� WKH� IORZ�FKDUW��2QO\�++�DQG�/6+&�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��

�

%UHHG�E\�1LUYDQD��

�

Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�GLG�WKLV��WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ��,Q�93�RI�WKH��WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��

Prediction #1 Prediction #2

Danceability �

Danceability� �

Key and mode �

Key and mode��

Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��


Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��

Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Certainty

7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��

�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%

��ESP ��ESP159 bpm

�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH

6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR


�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty

7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ� �

�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�

Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�GLG�WKLV��WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ��

,Q�93�RI�WKH��WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��


Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH


,Q�<RXU�/LJKW�E\�*RW\H��

Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��

Danceability� �Duration 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Energy 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Key and mode 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Loudness 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Beat grouping 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�

Certainty


�

�

�� 27%

��

��

/LNH'LVOLNH


�

�

How it all works together

�

<RXU�OLVW�RI�PXVLFLDQV

*HW�WKH��PRVW�SRSXODU�VRQJV�E\�HDFK�RI�WKHVH�

DUWLVWV

8VH�WKHVH�UHODWLRQVKLSV�WR�SUHGLFW�ZKHWKHU�HDFK�QHZ�VRQJ�LV�OLNH�WKH�JUHHQ�VRQJV��JRRG��RU�WKH�UHG�VRQJV��EDG�

+DYH�WKH�FRPSXWHU�ORRN�IRU�UHODWLRQVKLSV�EHWZHHQ�D�VRQJV�GHWDLOV�DQG�DQG�ZKHWKHU�LWV�LQ�WKH�JUHHQ�JURXS�RU�UHG�JURXS�

*HW�GHWDLOV�DERXW�DOO��VRQJV

)LQG��SHRSOH�ZKR�OLVWHQ�WR�\RXU�OLVW�RI�PXVLFLDQV�6RUWHG�E\�KRZ�PDQ\�PXVLFLDQV�\RX�ERWK�OLNH�

*HW�HDFK�SHUVRQV��PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�

\HDU

*HW�WKH��RWKHU�DUWLVWV�WKHVH�SHRSOH�PRVW�RIWHQ�

OLVWHQ�WR

��/DVW�IP�OLVWHQHUV

)LQG��SHRSOH�ZKR�GRQW�OLVWHQ�WR�DQ\�RI�WKH�PXVLFLDQV�\RX�OLVWHG


\HDU



6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR

/HJHQG

6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR

7$%/(�,��7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��

7UHDWPHQW� ++� 00� +6/&� /6+&��5HODWLYH�6RXQGQHVV� +LJK� 0HGLXP� +LJK� /RZ�

5HODWLYH�&RPSOHWHQHVV� +LJK� 0HGLXP� /RZ� +LJK�

,QWHOOLJLELOLW\�W\SHV�

:K\��VRQJ�� %DJJHG�GHFLVLRQ�WUHH�

'HFLVLRQ�WUHH�

%DJJHG�GHFLVLRQ�WUHH�

'HFLVLRQ�VWXPS�

:K\��DUWLVW�� 1HDUHVW�QHLJKERU��N ��

1HDUHVW�QHLJKERU��N ��



&HUWDLQW\� <HV� <HV� 1R� <HV�0RGHO� <HV� 1R� 1R� <HV�,QSXW� <HV� <HV� <HV� <HV�

�

��





9�� 5(68/76�






�

��

�

�

� �

�





�


�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�




Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH








Certainty


�

�

�� 27%

��

��

/LNH'LVOLNH


�

�


�



DUWLVWV






\HDU


OLVWHQ�WR




\HDU




/HJHQG


7$%/(�,��7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��





'HFLVLRQ�WUHH�


'HFLVLRQ�VWXPS�






�

��





9�� 5(68/76�






�

��

�

�

� �

�





�


�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�




Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH








Certainty


�

�

�� 27%

��

��

/LNH'LVOLNH


�

�


�



DUWLVWV






\HDU


OLVWHQ�WR




\HDU




/HJHQG


7$%/(�,��7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��





'HFLVLRQ�WUHH�


'HFLVLRQ�VWXPS�






�

��





9�� 5(68/76�






�

��

�

�

� �

�





�


�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�



Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH



�




Danceability �

Danceability� �

Key and mode �

Key and mode��






Danceability� �

Danceability� �

Beat grouping��

Key and mode��

Key and mode��



Certainty


�

�� 27%

�� 27%

��G% ��G%-14 dB

��G% ��G%-14 dB

�� 2:57

�� 89%


�� 27%

�� 27%

��G% ��G%-14 dB

��

��

/LNH'LVOLNH








Certainty


�

�

�� 27%

��

��

/LNH'LVOLNH


�

�


�



DUWLVWV






\HDU


OLVWHQ�WR




\HDU




/HJHQG


7$%/(�,��7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��





'HFLVLRQ�WUHH�


'HFLVLRQ�VWXPS�






�

Whythissong?(whyintelligibility)

Whythisar3st?(whyintelligibility)

Whatthecomputerknows?(Inputintelligibility) Howitworks?(modelintelligibility)

Results: how much information is useful?

•  Low-fi prototype with pre-generated recommendations, 4 conditions, 17 participants, think aloud protocol

•  Features used (inputs) better understood than process (model) •  Both high completeness and soundness did better in terms of

correctness of mental models but completeness seems to matter more

•  In terms of satisfaction, completeness again matters more but can trade off against soundness

•  However, beware that reducing soundness may cost users’ trust in system

Principles of explanatory debugging (Kulesza et al. 2015)

•  Explainability –  Be iterative –  Be sound –  Be complete –  Don’t overwhelm

•  Correctability –  Be actionable –  Be reversible –  Always honor feedback –  Incremental changes matter

Explanation Components

Figure 1. The EluciDebug prototype. (A) List of folders. (B) List of messages in the selected folder. (C) The selected message.(D) Explanation of the selected message’s predicted folder. (E) Overview of which messages contain the selected word. (F) Completelist of words the learning system uses to make predictions.

them. Our EluciDebug prototype uses a multinomial naiveBayes model (MNB) [20] with feature selection [47] to meetthese constraints. Evaluating the suitability of—or changesnecessary to—other models remains an open question.

The Multinomial Naive Bayes Classifier: A Brief ReviewBefore describing how we integrated MNB with ExplanatoryDebugging, we first summarize how MNB operates. An MNBclassifier computes the probability that a given input (e.g., thedocument being classified) has of belonging to each output(e.g., the possible labels). The output with the highest prob-ability “wins” and becomes the predicted label for the input.For example, if MNB calculates that a document has a 70%probability of being junk mail and a 30% probability of notbeing junk mail, the document will be labeled as junk mail.The equations for computing probability, as defined in [20],are shown below. We use c to represent an individual classin the collection of potential output classes C, di to representan individual document to classify, and assume that the onlyfeatures the classifier uses are individual words in the set ofknown documents:

Pr(c|di) =Pr(c)Pr(di|c)

Pr(di)(1)

The term Pr(c) represents the probability that any given docu-ment belongs to class c and can be estimated by dividing thenumber of documents in c by the total number of documentsin the training set. The term Pr(di|c) represents the probability

of document di given class c and can be estimated as:

Pr(di|c) =’n

Pr(wn|c) fni (2)

The term fni is the number of instances of word n in documentdi and the term Pr(wn|c) is the probability of word n givenclass c, estimated with the equation

Pr(wn|c) =pnc+Fnc

NÂ

x=1pxc+

NÂ

x=1Fxc

(3)

where Fnc is the number of instances of word n in all of thetraining documents for class c, N is the number of uniquewords in the training documents for all classes, and pnc isa smoothing term (usually 1) to prevent the equation fromyielding 0 if no documents from class c contain word wn.The Explanatory Debugging Principles in EluciDebugBeing IterativeTo help end users iteratively build better mental models, Eluci-Debug employs two strategies: (1) each explanation focuseson an individual aspect of the learning system, and (2) layeredexplanations are available if the user wants more information.For example, the Why explanation (introduced later) primarilytells users about the system’s reasons for a specific prediction.Over time, however, the user may learn about the system’sgeneral operation by observing commonalities in the reasonsfor these specific predictions. Further, this Why explanationis layered. Part of the explanation shows users which words

The Problem…

•  Explanations do not seem to matter much for “low-risk” systems (Bunt et al. 2012)

•  In high-risk systems, they might have undesirable effects

Explanations considered harmful… (Bussone et al. 2015)

Introduced to CDSS

Rate trust in CDSS

Provided clinical vignette to input into CDSS

Receive suggestion, accept or decline

Rate trust in CDSS

Comprehensive Group

Selective Group

Comprehensive

Selective

Results: confidence and explanations

Comprehensive explanations imply more knowledge and better ability of system (Over-reliance) Selective explanations suggest inferior reasoning (self-reliance)

What else do they want explained?

•  Certainty/Confidence: What does it mean? How derived?

•  Features and their weights

•  Counter-evidence

‘So when it says the certainty is 19%, um… What does that mean?’

‘I’d want to see other things that would cause it. …So any other risk factors – patient weight, history of heart disease, history of angina?’

‘So, in medicine it’s not just the positives, it’s the negatives that you look out for as well. If I have a very clear positive, if that was the only thing I had about the patient then that’s fine, but I’d have to have the strong negatives. Only then would I accept a diagnosis.’

Further work

•  Explanations largely hand-crafted to fit algorithms –  No guide how to explain systems

•  Effects of corrections difficult to explain –  Particular components need attention: What if?

•  Explanations can be overwhelming –  How much is needed?

•  Explanations and its effects on trust –  Misuse and disuse

Conclusions

•  Involving users in interactive machine learning through explanations is a viable approach

•  Explanations at the moment are heavily domain and system dependent

•  In a way, we haven’t gone far enough –  Need better understanding of how to construct explanations –  Need better understanding of effects of explanations

the role of explanations in interacting with intelligent ... · the role of explanations in...

Documents