deep learning and explainable ai for spatio-temporal
TRANSCRIPT
![Page 1: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/1.jpg)
Deep Learning and Explainable AIfor spatio-temporal drought monitoringMiguel-Ángel Fernández-Torres, PhD
Image and Signal Processing (ISP) group, Image Processing Laboratory (IPL)
Universitat de València
Second ITU/WMO/UNEP Workshop on AI for Natural Disaster Management
23rd June 2021
Contact: [email protected] | Web: htpps://miguelangelft.github.io htpps://isp.uv.es | Twitter: @miguelangelft @isp_uv_es
![Page 2: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/2.jpg)
Contents
1. Motivation and objectives
2. Case study: Drought monitoring during the Russian heat-wave in 2010
2.1 Essential Climate Variables (ECVs)
2.2 Deep Learning for drought detection
- SlowFast Convolutional Autoencoder for ECVs representation
- Gaussianization Flows for extreme event detection
2.3 EXplainable AI (XAI) for drought monitoring
- Integrated Gradients
3. Take-home messages for the application of AI4NDM
2
![Page 3: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/3.jpg)
1. Motivation and objectives
• Challenge in climate change: Anticipation and detection of extreme events
• Machine learning approaches…
– have excelled in the detection of anomalies in Earth data cubes,
– but are typically both computationally costly and supervised
• Objectives
– Develop an unsupervised, generic, efficient, generative approach for extreme event detection
– The model not only detects extreme events, but also explains why they were produced
3
![Page 4: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/4.jpg)
2. Case study: Drought monitoring
• Objective: Europe, severe Russian heat-wave between April-August 2010
• Databases:1. International Disaster Database (EM-DAT, 2008)
• Drought annotations at spatio-temporal location-level• Only used for evaluation
2. Earth System Data Lab (ESDL) (Mahecha et al., 2020)• Essential Climate Variables (ECVs) • ECVs for droughts in 2003-2015• 3 different time periods for training, validation (non-
drought periods) and test (drought+non-drought period) stages.
Mirny, a Russian village ravaged by wildfires during the drought in 2010. Credit: Yuri Kochetkov/European Pressphoto
Agency
A fire near the village of Golovanovo, in the Ryazan region of Russia, during the heat-wave in 2010. Credit: Natalia
Kolesnikova/Agence France-Presse — Getty Images
Russia counts the cost of drought and wildfires. Credit: BBC
4
![Page 5: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/5.jpg)
2.1 Essential Climate Variables (ECVs) for droughts① INPUT: 𝐿 data cubes [𝑓1, 𝑓2, … , 𝑓𝐿]𝑇,𝑁
𝑇 timesteps, 𝑁 spatial locations per timestep
Evaporation
Transpiration
Surface Moisture
Root Moisture
Air Temperature 2m
Gross Primary Productivity
FAPAR Leaf Area Index
5
![Page 6: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/6.jpg)
2.2 Deep Learning for drought detection
𝑓1: Surface moisture 𝑓2: Root moisture 𝑓3: GPP 𝑓4: Transpiration
𝑓5: Air Temperature 2m 𝑓6: Evaporation 𝑓7: FAPAR 𝑓8: Leaf Area Index
① INPUT: 𝐿 Essential Climate Variables (ECVs) [𝑓1, 𝑓2, … , 𝑓𝐿]𝑇,𝑁𝑇 timesteps, 𝑁 spatial locations per timestep
OUTPUT: ECVs Reconstruction
[ መ𝑓1, መ𝑓2, … , መ𝑓𝐿]1:𝑇
ENCODER
Appearance
Motion
DECODER
Latent Representation[𝑥1, 𝑥2, … , 𝑥𝐶]𝑇,𝑁
② SlowFast Convolutional Autoencoder
OUTPUT: Anomaly score𝐷𝑡𝑛 = −log(𝑝 𝒙𝑡𝑛 )
③ Gaussianization Flows (Unsupervised Deep Generative Model)
6
![Page 7: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/7.jpg)
Credit: Resuly’s Blog, http://resuly.me/
SlowFast Convolutional Autoencoder
Appearance
INPUT: ECVs[𝑓1, 𝑓2, … , 𝑓𝐿]𝑇,𝑁𝑇 timesteps,𝑁 spatial
locations per timestep
𝛽𝐶 𝛼𝑇
𝐶𝑇
MotionENCODER DECODER
OUTPUT: ECVs Reconstruction
[ መ𝑓1, መ𝑓2, … , መ𝑓𝐿]1:𝑇𝐶𝑇𝑇 𝑇
𝛼𝑇 3D Max Pooling
Trilinear Upsampling
𝐶′𝑇
𝐿
𝐿
𝐿
3D Convolution
SlowFast Networks for Video Recognition (Feichtenhofer et al., 2019)• Slow or appearance pathway: Low frame rate and temporal resolution 𝛼𝑇, 𝛼 < 1;
high number of channels (spatial resolution) 𝛽𝐶, 𝛽 > 1• Fast or motion pathway: High frame rate and temporal resolution 𝑇; 𝐶 channels• Lateral connections between Slow and Fast pathways• Skip connections between the encoder and the decoder
7
![Page 8: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/8.jpg)
Gaussianization Flows (Meng et al., 2020)
• Deep generative, trainable model: Efficient PDF estimation via maximum likelihood + sample generation• Bijective, invertible mapping from data distribution to a standard Gaussian distribution (target)• Anomaly score: Negative Log-Likelihood (NLL) → 𝐷𝑡𝑛= −log(𝑝 𝒙𝑡𝑛 )• Visualization of training phase through epochs:
Input samples 𝒙 Gaussianized PDF 𝑝(𝒛) Real (blue) vs. estimated (orange) PDF 𝑝(𝐱)
8
![Page 9: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/9.jpg)
Dro
ught
dete
cti
on
9
![Page 10: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/10.jpg)
2.3 EXplainable AI (XAI) for drought monitoring• Integrated Gradients (IG) (Sundararajan et al., 2017): Integral of gradients of the output 𝐷 with respect to inputs
along the path from a given baseline 𝑓1:𝐿′ to input 𝑓1:𝐿.
𝐼𝐺 𝑓1:𝐿 ≔ (𝑓1:𝐿 − 𝑓1:𝐿′ ) ∙ න
𝛼=0
1 𝜕𝐷(𝑓1:𝐿′ + 𝛼 × 𝑓1:𝐿 − 𝑓1:𝐿
′ )
𝜕𝑓1:𝐿𝑑𝛼
• Two fundamental axioms of IG:• Sensitivity: “For every input and baseline that differ in one feature but have different predictions, the
differing feature should be given a non-zero attribution”• Implementation invariance: “Attributions are always identical for two functionally equivalent networks
(i.e. their outputs are equal for all inputs, despite having very different implementations)”
10
![Page 11: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/11.jpg)
Dro
ught
monit
ori
ng
11
![Page 12: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/12.jpg)
3. Take-home messages for the
application of AI4NDM
1. Fully unsupervised model
– Semi-supervised/supervised models: Annotations at spatio-temporal location-level might be inaccurate
2. Generic vs. context-aware framework
– Should we use different models for drought detection in Russia/Europe/Worldwide?
3. Generic model
– Extension to other extreme events
4. XAI for the study and analysis of compound events
12
![Page 13: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/13.jpg)
Deep Learning and Explainable AIfor spatio-temporal drought monitoringMiguel-Ángel Fernández-Torres, PhD
Image and Signal Processing (ISP) group, Image Processing Laboratory (IPL)
Universitat de València
Second ITU/WMO/UNEP Workshop on AI for Natural Disaster Management
23rd June 2021
Contact: [email protected] | Web: htpps://miguelangelft.github.io htpps://isp.uv.es | Twitter: @miguelangelft @isp_uv_es
![Page 14: Deep Learning and Explainable AI for spatio-temporal](https://reader030.vdocuments.net/reader030/viewer/2022012810/61c0f9389798ec4c9d0f29c3/html5/thumbnails/14.jpg)
References
• EM-DAT (2008). EM-DAT: The International Disaster Database (Available at http://www.emdat.be/Database/Trends/trends.html)
• Mahecha, M. D., Gans, F., Brandt, G., Christiansen, R., Cornell, S. E., Fomferra, N., ... & Reichstein, M. (2020). Earth system data cubes unravel global multivariate dynamics. Earth System Dynamics, 11(1), 201-234.
• Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019). Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6202-6211).
• Meng, C., Song, Y., Song, J., & Ermon, S. (2020, June). Gaussianization flows. In International Conference on Artificial Intelligence and Statistics (pp. 4336-4345). PMLR.
• Sundararajan, M., Taly, A., & Yan, Q. (2017, July). Axiomatic attribution for deep networks. In International Conference on Machine Learning (pp. 3319-3328). PMLR.
14