InterpretableDiscoveryinLargeImageDataSets
KiriL.WagstaffandJakeLee
JetPropulsionLaboratory,CaliforniaInstituteofTechnology
December7,2017
NIPSInterpretableMachineLearningSymposium
©2017,California InstituteofTechnology.Governmentsponsorshipacknowledged.ThisworkwasperformedinpartattheJetPropulsionLaboratory,CaliforniaInstituteofTechnology,underacontractwithNASA.
DiscoveryinLargeDataSets• Scientificdiscoveriesoftencomefromoutliers
WagstaffandLee- NIPS2017 2
ByFlickruserKlaus
DiscoveryinLargeImage DataSets
• Challenges:Representation,Explanations
WagstaffandLee- NIPS2017 3
HiRISE– 1.4MimagesCredit:NASA/JPL-Caltech/Univ.ofArizona
HumanfacesCredit:Pixabay userGeralt
Surveillance PlanetaryScience
NoveltyDetectionMethods• Clustering• IsolationForest[Liuetal.,2008]• Density-based(e.g.,LocalOutlierFactor[Breunig etal.,2000])• SVD• DEMUD:SVD-based+explanations• Explanation:SVDresidual;informationthemodelcouldnotexplain[Wagstaffetal.,2013]
WagstaffandLee- NIPS2017 4
WhyDEMUD?
0 5 10 15 20 25 30 351
2
3
4
5
6
Number of selected examples
Num
ber o
f cla
sses
disc
over
ed
DEMUDCLOVERNNDMSEDERInterleaveStatic SVDRandom
Rareclassdiscovery– UCIglassdata
WagstaffandLee- NIPS2017 5
Explanations– UCIglassdata
• IncrementaldiscoveryusingSVDmodelofselections• Especiallygoodfordiscoveringrareclasses• Explanationsjustifyselections
WhyDEMUD?• IncrementaldiscoveryusingSVDmodelofselections• Especiallygoodfordiscoveringrareclasses• Explanationsjustifyselections• Explanationshelpusersclassifyitems
0 5 10 15 200
10
20
30
40
50
60
70
80
90
100
Selection number
Cum
ulat
ive
accu
racy
DEMUD explanationsNo explanationsRandom
ChemCam expertclassificationperformance
WagstaffandLee- NIPS2017 6
200 300 400 500 600 7000
2
4
6
8
10
12
14
16
18
20 x 10−4
Wavelength (nm)
Inte
nsity
+Mn 259.34−Fe 273.91−Fe 274.62−Fe 274.67−Fe 274.88−Fe 274.93−Fe 275.53−Fe 275.58+Mn 293.28+Mn 293.86+Mn 293.91+Mn 294.88−Mg 516.73−Mg 517.19−Mg 518.34
Explanations:ChemCam spectra
Rhodochrosite:MnCO3
DEMUDforImages• Representation• Rawpixels• SIFT[Lowe,2004],HOG[Dalal &Triggs,2005]• CNNfeatures[Razavian etal.,2014]
WagstaffandLee- NIPS2017 7
DEMUD+CNNRepresentations
Classprobs
WagstaffandLee- NIPS2017 8
Images
[Krizhevsky etal.,2012]
DEMUD+CNNRepresentations
Images
Classprobs
WagstaffandLee- NIPS2017 9
DEMUD
Features
[Krizhevsky etal.,2012]
DEMUDExplanationswithCNNFeatures
WagstaffandLee- NIPS2017 10
DEMUDFeatures
Selection
Explanation
?
Invertresidualstogetvisualexplanations
DEMUDExplanationswithCNNFeatures
WagstaffandLee- NIPS2017 11
DEMUDFeatures
Selection
Explanation
?
DeepGoggle:Generateinputthatyieldsfeaturevalues
(Mahendran&Vedaldi,2015)
CNNFeatureInversionMethods
Invertresidualstogetvisualexplanations
DEMUDExplanationswithCNNFeatures
WagstaffandLee- NIPS2017 12
DEMUDFeatures
Selection
Explanation
?
DeepGoggle:Generateinputthatyieldsfeaturevalues
(Mahendran&Vedaldi,2015)
Up-Conv:Predictoriginalimage
withsecondNN(Dosovitskiy &Brox,2016)
CNNFeatureInversionMethods
Invertresidualstogetvisualexplanations
Experiments– ImageNet
WagstaffandLee- NIPS2017 13
• 1000images• 10classes• Evenlydistributed
Experiments– ImageNet
WagstaffandLee- NIPS2017 14
DEMUD-CNNSVD-CNNDEMUD-pixelSVD-pixelRandom
WagstaffandLee- NIPS2017 15
Selection
Deep
Goggle
Up-Con
vExplanations– ImageNetBassoon Dial Foodpacket Dogsled Zucchini
Experiments– MSLRoverimages
WagstaffandLee- NIPS2017 16
• 6737images• 26classes• Unevendistribution
Experiments– MSLRoverimages
WagstaffandLee- NIPS2017 17
DEMUD-CNNSVD-CNNDEMUD-pixelSVD-pixelRandom
WagstaffandLee- NIPS2017 18
Explanations– MSLRoverimagesSelection
Deep
Goggle
Up-Con
v
Chemin inlet REMSUVsensor MAHLIcal target Turret Ground
Summary• DEMUD+CNNfeatures+CNNfeatureinversion• Fastdiscoveryofnovelimages
• Withvisualexplanations
• Whatwillyoufindinyourimagedataset?
WagstaffandLee- NIPS2017 19
Thankyou:NASAPlanetaryDataSystem(PDS)ImagingNode