deep representation and reinforcement learning for anomaly...
TRANSCRIPT
Soumalya Sarkar, PhD Senior Research scientist, UTRC
Deep Representation and Reinforcement Learning for Anomaly Detection and Control
in Multi-modal Aerospace Applications
This document contains no technical data subject to the EAR or the ITAR.
May 9 @ GTC 2017
2
“UTRC is where
you bring your
toughest problems.”
Our
business units
This document contains no technical data subject to the EAR or the ITAR.
IIoT
52% 36%
12%
Military Aerospace
& Space
Commercial
Aerospace
Commercial
& Industrial
END MARKETS
56%
44%
Original
Equipment
Manufacturing
Aftermarket
SALES BY TYPE SALES BY GEOGRAPHY
38%
27%
20%
15%
Asia
Pacific
Europe
United
States
Other
3
A
global leader
$56B 2015 UTC Sales
$3.9B invested in R&D
This document contains no technical data subject to the EAR or the ITAR.
4
Cork, Ireland Established in 2010,
focuses on energy, security
and aerospace systems
Shanghai, China Established in 1997, focuses on
integrated buildings, fluid and
mechanical systems
Rome, Italy Joined UTC in 2012,
focuses on model-based
design and embedded
systems engineering
East Hartford, CT Founded in 1929, focuses on a
broad range of system engineering,
thermal, fluid, material, and
informational sciences
Berkeley, CA Established in 2009, focuses
on cyber physical systems
and embedded intelligence
A
global presence
This document contains no technical data subject to the EAR or the ITAR.
5
Focused on
performance
Physical Sciences Systems
Thermal & Fluid
Sciences
– Advanced Materials
– Applied Physics
– Environmental Science
– Materials Chemistry
– Measurement Science
– Solid Mechanics
– Surface Mechanics
– Advanced Laboratory for Embedded Systems
– Control Systems
– Cyber-Physical Systems
– Decision Support & Machine Intelligence
– Electromagnetics & Networks
– Power Electronics
– Software Systems
– System Dynamics & Optimization
– Acoustics
– Aerodynamics
– Aero-Thermal Testing
– Combustion Science
– Propulsion Technology
– Thermo-Fluid Dynamics
– Thermal Management
This document contains no technical data subject to the EAR or the ITAR.
Topics
This document contains no technical data subject to the EAR or the ITAR.
Deep Representation Learning - DAE Big PHM (Prognostics & Health Monitoring) SHM (Structural Health Monitoring)
Deep Reinforcement Learning (DRL) for additive manufacturing
Deep Auto-Encoder (DAE)
• Parameter learning by Stochastic gradient descent (Hinton&Salakhutdinov, 2006; Bengio et al., 2007)
• Variants: De-noising (RBM), variational etc
• Static DAE instead of LSTM AE due to ease / speed of training
Multi-layer neural network based learner of non-linear representation of the data
W
Input:
Hidden representation:
Sigmoid connecting two layers:
Parameters:
Sigmoid function at reverse mapping of reconstruction layers:
Where
Cost function for back-propagation:
This document contains no technical data subject to the EAR or the ITAR.
,𝑊′= 𝑊𝑇
Topics
This document contains no technical data subject to the EAR or the ITAR.
Deep Representation Learning - DAE Big PHM (Prognostics & Health Monitoring) SHM (Structural Health Monitoring)
Deep Reinforcement Learning (DRL) for additive manufacturing
Motivation
Big, multi-modal & heterogeneous data; unsupervised visualization
This document contains no technical data subject to the EAR or the ITAR.
Data: ~100 sensors, ~200 dimensional condition data Size ~ TB Zero/ few labels Problem: Understanding / separating different missions or faults Challenges: Low-dimensional visualization, robust separation of faults (FDI) / mission, real-time application and generalizability
FDI approaches and challenges
• Lack of high-fidelity non-linear models,
• Tedious hand-crafting (domain knowledge) of fault features,
• Lack of scalability to large data,
• Insufficient robustness to noise and
• The presence of various operating modes,
• Presence of multi-modal sensors for fault disambiguation
Previous Work and Challenges
Methods of fault detection and identification
Model based 1. Residual methods 2. Parity based
3. Kalman filter based
Data-driven 1. Time, frequency, symbolic domain
features 2. SVM, k-NN, artificial neural net based
learning systems
Hybrid 1. Parity Equation Approach and wavelet based signal features 2. PCA based system models
Deep Learning
This document contains no technical data subject to the EAR or the ITAR.
Database* for Validation
Apparatus: A set of electromechanical actuators (EMA), constructed by Moog Corporation, were used by Balaban et. al. (Balaban et al., 2009, 2015). To increase the horizon of available operating conditions, flyable electromechanical actuator (FLEA) testbed was also constructed.
13 multi-modal sensors @100Hz: Actuator Z Position, Measured Load, Motor Current X-Y-Z, Motor Voltage X-Y, Motor Temperature X-Y-Z, Nut X-Y Temperature, Ambient Temperature.
Baseline an 2 fault classes:
1. A jam fault injected via a mechanism mounted on the return channel of the ball screw that can stop circulation of the bearing balls through the circuit.
2. A spall fault injected by introducing cuts of various geometries via a precise electrostatic discharge process. The initial size and subsequent growth of these cuts were confirmed by using an optical inspection and measurement system.
*open database available at NASA Dashlink, collected by Balaban et. al. (Balaban et al., 2009, 2015)
Fault Detection and Identification
This document contains no technical data subject to the EAR or the ITAR.
DAE architecture
650 (50x13) -> 256 -> 196 -> 136 -> 76 -> 14 -> 76 -> 136 -> 196 -> 256 -> 650
Window size = 0.5 seconds, shifted by each time point
11-layer DAE
This document contains no technical data subject to the EAR or the ITAR.
DAE Reconstruction Error
Actual signals and reconstructed signals for Motor X voltage, Motor Y temperature, and load sensors (from top to bottom) with bottleneck layer of 14 dimension
Multi-modal Reconstruction Error Fault Detection and Identification
This document contains no technical data subject to the EAR or the ITAR.
Training and Parameter Learning
Variation of normalized RMS error at the reconstructed output layer with increasing dimension of the bottleneck
Individual sensor-wise reconstruction errors at the output layer for 3 different bottleneck layer dimensions
This document contains no technical data subject to the EAR or the ITAR.
Tuning bottleneck Layer
Fault Diagnostics
ROC curves via varying detection threshold on testing data for different bottleneck dimensions of 11-layer DAE and few single layer AE models
Precision-Recall curves for the same conditions as data is unbalanced
This document contains no technical data subject to the EAR or the ITAR.
ROC and Precision-Recall curves
Unsupervised Fault Disambiguation
Spider charts showing the NRMS error across different sensors during testing phase for nominal and fault scenarios
Disambiguation by Multi-dimensional Reconstruction Error
This document contains no technical data subject to the EAR or the ITAR.
Why Deep Architecture?
Spider charts of the average (over nominal and fault scenarios) NRMS error across different sensors during testing phase for
(a) single hidden-layer AE with 512-dimensional bottleneck (b) proposed 11-layer DAE with 14-dimensional bottleneck
DAE Reconstruction Error increases fault separability with low over-fitting
This document contains no technical data subject to the EAR or the ITAR.
Why Deep Architecture?
Clusters of two largest principal components obtained from PCA on 13-dimensional NRMS error distributions
Multi-dimensional NRMS from DAE increases inter-fault distance at low dimension
single hidden-layer AE model with 512-dimensional bottleneck layer proposed 11-layer DAE with 14-dimensional bottleneck layer
This document contains no technical data subject to the EAR or the ITAR.
Discussions
• trained directly on raw time series from heterogeneous sensors without feature hand-crafting and extensive data preprocessing.
• A high fault detection rate ( ~97.8%) along with zero false alarm on a large set of realistic data (available on NASA DASHlink)
• Disambiguation among different types of faults with high confidence in an unsupervised way
• Proposed DAE more robust than single-hidden layer AE
This document contains no technical data subject to the EAR or the ITAR.
Fault separation even at a low dimension in an unsupervised way
Topics
This document contains no technical data subject to the EAR or the ITAR.
Deep Representation Learning - DAE Big PHM (Prognostics & Health Monitoring) SHM (Structural Health Monitoring)
Deep Reinforcement Learning (DRL) for additive manufacturing
Autonomous SHM from Imaging
Background
This document contains no technical data subject to the EAR or the ITAR.
Vision-based SHM is required to get details of the damage such as size, configuration, shape, topological networking, geometrical statistics for material characterization, damage prognostics , RUL analysis etc
• Image processing as important tool in material/structural characterization for over three decades (Krakow, 1982; Duval et al., 2014; Robertson et al., 2011; Leach, 2013).
• Texture analysis (Comer & Delp, 2000) and segmentation (Ruggiero, Ross, & Porter, 2015; Park, Huang, Ji, & Ding, 2013) are applied image processing techniques to SHM.
• Pre-processing steps like filtering and enhancement techniques (Tomasi & Manduchi, 1998; Angulo & Velasco-Forero, 2013; Buades, Coll, & Morel, 2005) used to denoise image and perform alignment and artifact correction.
• Recent breakthroughs of deep learning are mostly in image processing because it models multiple levels of abstractions (low-level features to higher-order representations, Erhan, Courville, & Bengio, 2010).
• Broad applications to medical imaging (similar to vision-based SHM from computer vision perspective), recent application to video-based combustion PHM (Sarkar et al., 2015)
Why DAE for SHM?
Not explored enough yet!
This document contains no technical data subject to the EAR or the ITAR.
Methodological challenges: Computational Challenges:
Extensive heuristics for parameter tuning in existing tools
Limited availability of annotation causing small number of training labels (no CNN)
Robustness issue in computer vision (segmentation) techniques
Seamless incorporation of domain expert in the loop
• Lengthy and tedious process of manual annotation by domain experts on large number of samples
• Human error (expert bias) in labelling the ground truth
• Process changes significantly with new experimental setup and material
Use case: Damage characterization
Experimental setup: Variable load-induced cracks on a composite
(a) Scheme of coupon testing and (b) Representative damage pattern
This document contains no technical data subject to the EAR or the ITAR.
Thick multi-layer composite sub-elements used in numerous rotorcraft and aircraft applications.
Usually under conditions of multi-axial loading with dominant influence of bending, generating complex patterns of internal damages
Carbon fiber polymer-matric 55 layered composite (IM7/977-3 materials with lay-up [+454 / -454 / 03]2S[03/- 454 /+ 454] representing thickness of 0.290 in) coupons were considered under conditions of five-point bending.
Video starts with a straight coupon and slowly it is bent under monotonically increasing displacement-controlled load till full fracture.
Framework for Damage Characterization
Image frame to crack-length distribution
Distribution of crack lengths
Video frames with dynamic crack and non-linear bending
This document contains no technical data subject to the EAR or the ITAR.
Patching DAE and Guided Segmentation
Modeling nominal surface and segmenting cracks from reconstruction error map
Input layer Output layer
Multiple Hidden layers
Frame with NO crack
Learning Nominal Surface via Reconstructing DAE (patch size 3 by 7 pixels)
𝑆 𝑥, 𝑦 = 𝐼 𝑥, 𝑦 − 𝑐 1 − 𝑝 𝑥, 𝑦
Similarity measure Raw image intensity of pixel 𝑥, 𝑦
Probability (intensity of reconstruction error map) of a pixel 𝑥, 𝑦 being a crack
This document contains no technical data subject to the EAR or the ITAR.
Patching DAE Guided region-growing segmentation
Mean intensity of current region
A = true crack zone B = detected crack zone A ∩ B = correctly detected crack area
1. 𝐷𝑖𝑐𝑒 𝑠𝑐𝑜𝑟𝑒 = 2(𝐴 ∩ 𝐵)/(𝐴 ∪ 𝐵)
A
B A ∩ B
Performance metrics
3. Average minimum distance between true crack and detected crack areas
2. Distance between histograms, d
4. Number of cracks
This document contains no technical data subject to the EAR or the ITAR.
For characterization number of cracks and d are the most significant metrics
Tuning thresholds (on intensity of reconstruction error map) on medium level load
Refining parameter
Chosen threshold
This document contains no technical data subject to the EAR or the ITAR.
Tuning thresholds based on medium load = 0.55
Crack detection at medium load from Tuned parameter
This document contains no technical data subject to the EAR or the ITAR.
This document contains no technical data subject to the EAR or the ITAR.
Actual
Medium level load
High level load
Low level load
Crack detection across various load levels Predicted
Shows better robustness (to different load level) even crack thickness of only 2-3 pixels than sophisticated contour detector, edge detectors, morphological segmentations and single step region growing segmentation.
Discussions
• High characterization accuracy and satisfactory performance over a wide range of loading conditions with limited number labeled training image data.
• Meets required robustness (to different load level) via DAE error map in comparison to other benchmarks
• This approach can applied to field inspection and borescope inspection on complex surface structures.
This document contains no technical data subject to the EAR or the ITAR.
Less heuristics, Validation on real data, Robustness to varying condition
Topics
This document contains no technical data subject to the EAR or the ITAR.
Deep Representation Learning - DAE Big PHM (Prognostics & Health Monitoring) SHM (Structural Health Monitoring)
Deep Reinforcement Learning (DRL) for additive manufacturing (AM)
Noisy robot state Surface imaging
Nozzle Trajectory
𝑜𝑡
Global feedback controller, No need to solve online MPC
AM in aerospace domain: high precision standards & high variability of tasks
No need for state estimation
• 3D printing • Cold Spray • Arc welding • Powder deposition
Reduce reliance on expert (rather expert guided) Self-learning/adaptation to optimal behavior Closing the loop with real time perception
Cost reduction, easier commissioning, improved performance & scalability, high precision geometric and material property
for printing and repair
DRL for AM
This document contains no technical data subject to the EAR or the ITAR.
Cold Spray
Robot
Scanner
Sensors Perception/
State Estimation
Policy /Control
Low Level
Control
Actuator
Actions/ Control
Sensing
Explore recent advances in deep learning to address • End-to-end sensinglearning/control • Incorporate expert/prior knowledge
Automate
Expert engineered on a case by case basis
What are we trying to do?
This document contains no technical data subject to the EAR or the ITAR.
DRL for AM
Different Flavors
End-to-end perception to learning/control
End-to-end guided policy search (Levine et. al. 2015)
Deep Q Learning (Mnih et. al., 2013, Lillicrap et. al., 2016)
Learning features/dynamics to use in control/RL Deep Q Fitted Network (Lang et. al, 2012) Embed to control-iLQR (Watter et. al., 2015) Deep Dynamical Models-NMPC (Assael et. al, 2015)
Learning value function/policy in RL
Guided Policy Search Levine e.t al., 2014)
This document contains no technical data subject to the EAR or the ITAR.
DRL for AM
How it works?
NN to represent value function Purely exploratory No need for generative model Gaming applications / simulated environments
NN to represent policy Exploit trajectory optimization to guide search Need (approx.) generative model Robotics application
Agent
Simulated/Real Environment
Action, 𝑢𝑡 ∼ 𝜋(. |𝑠𝑡) State, 𝑠𝑡+1
Policy: 𝜋: 𝑆 → 𝑃(𝑈)
Value/Q function: 𝑄(𝑠, 𝑢) = E𝜋[J(𝜏)|s1 = s, u1 = u]
𝑠𝑡+1 = 𝑓(𝑠𝑡, 𝑢𝑡)
min𝜋𝐸𝜋[𝐽 𝜏 ]
𝐽 𝜏 = 𝑐(𝑠𝑡, 𝑢𝑡)
𝑇
𝑡=1
𝜏 = {𝑠1, 𝑢1, 𝑠2, 𝑢2, … . , 𝑠𝑇 , 𝑢𝑇}
Cost, 𝑐(𝑠𝑡, 𝑢𝑡)
Key advance: Stable training
NN for function approximation
Deep Q Learning Guided Policy Search
This document contains no technical data subject to the EAR or the ITAR.
DRL for AM
*(Levine et. al. 2015)
Guided Policy Search Problem Formulation
Approach I: One Shot “Demonstrator” Trajectory optimizer to generate optimal guiding
sample NN trained to match samples
Approach II: Incremental “Good teacher” Provides training to NN in small steps Modified cost for trajectory optimizer, so that it solves the
control problem similar to how student (fails to) solve it
min𝜃𝐸𝜋𝜃[𝐽(𝜏)]
Policy Search
Guided Policy Search = Supervised Learning
Traj
ect
ory
O
pti
miz
atio
n
Guiding samples
𝑞(𝑢𝑡|𝑠𝑡)
Simulated/Real Environment
𝜃
𝜋𝜃(𝑢𝑡|𝑠𝑡)
This document contains no technical data subject to the EAR or the ITAR.
DRL for AM
Deep Reinforcement Learning (DRL)
Guided Policy Search: Incremental Approach
Dynamic Model
Approx.
Cost Function Approx.
Iterative LQR
Sample
Trajectory
NN
Training
𝑞𝑖(𝑢𝑡|𝑠𝑡) 𝜏𝑖𝑗
𝑖 = 1, . . , 𝑀 𝑗 = 1, . . , 𝑁
Lagrange Multiplier Update
𝜏𝑖𝐼 , 𝑖 = 1, . , 𝑀
Simulator/ Real System
Quadratic Approx.
𝜋𝜃,Σ 𝑢𝑡 𝑠𝑡
𝜇(. ; 𝜃)
This document contains no technical data subject to the EAR or the ITAR.
DRL for AM: Cold Spray for high precision repair
Guided DRL for cold spray control
Challenges Complex physics:
Coupled geometric/structural properties
High dimensional nonlinear system
Path planning:
Laborious manual process ~multiple hours per part
Difficult to assess frustration points
Part machined to simplify path planning
Dynamics:
Objective: Compute 𝑢1, 𝑢2, … , 𝑢𝑇 to minimize
𝑠𝑡+1 = 𝑓 𝑠𝑡, 𝑢𝑡 𝑠𝑡 = (𝐷 𝑟1, 𝑡 , …𝐷 𝑟𝑁, 𝑡 , 𝑥𝑠 𝑡 , 𝛼(𝑡))
𝑢𝑡 = (𝑣𝑠, 𝜔𝑡)
𝐽(𝜏) =1
2 𝑠𝑡 1:𝑁 − 𝐷𝑟𝑒𝑓
∗𝑅 𝑠𝑡(1:𝑁) − 𝐷𝑟𝑒𝑓 + 𝑢𝑡
∗𝑄𝑢𝑡
𝑇
𝑡=1
─ distribution of particles in spray cone ─ deposit efficiency function ─ total deposit in the point r during one run ─ position of the nozzle at time t
𝜑 tan𝛼
𝑓 cot 𝛽
𝐷 𝑟, 𝑡
xs(t), hs
𝜕𝐷 𝑟, 𝑡
𝜕𝑡= 𝜑
𝑟 − 𝑥𝑠 𝑡
ℎ𝑠 − 𝐷 𝑟, 𝑡
𝑇
0
𝑓𝑟 − 𝑥𝑠 𝑡 − ℎ𝑠 − 𝐷 𝑟, 𝑡
𝜕𝜕𝑟 𝐷 𝑟, 𝑡
ℎ𝑠 − 𝐷 𝑟, 𝑡 + 𝑟 − 𝑥𝑠 𝑡𝜕𝜕𝑟 𝐷 𝑟, 𝑡
d𝑡
α
β
γ
D(r,t)
α
β
γ
xs(t) r
hs
𝐷𝑟𝑒𝑓
Goal: Automate cold spray control
This document contains no technical data subject to the EAR or the ITAR.
Results: GPS leads to optimal behavior as MPC
DRL for AM: Cold Spray for high precision repair
500 randomly generated surfaces
Much Faster!
This document contains no technical data subject to the EAR or the ITAR.
Co
st
Better Training!
machining
Constant Speed DRL Based Control
Generalizability to various surface topology with limited training and benchmarking against constant speed nozzle control (state-of-the-art in industry)
DRL for AM: Cold Spray for high precision repair
Generalizability to various surface imperfection
This document contains no technical data subject to the EAR or the ITAR.
DRL for AM: Conclusions
Cost reduction from time and material saving,
easier commissioning due to generalizability,
improved performance & scalability,
high precision geometric and material property
Questions?
Explainability & Certification of these approaches for safety-critical systems…
Publications
• Amit Surana, Soumalya Sarkar, and Kishore K. Reddy, “Guided Deep Reinforcement Learning for Additive Manufacturing Control Application”, NIPS 2016 Deep Reinforcement Learning Workshop, December 2016.
• Soumalya Sarkar, Kishore K. Reddy, Michael Giering, and Mark Gurvich, “Deep Learning for Structural Health Monitoring: A Damage Characterization Application”, Conference of the Prognostics and Health Management Society, August 2016.
• Kishore K. Reddy, Soumalya Sarkar, Vivek Venugopalan, and Michael Giering, “Anomaly Detection and Fault Disambiguation in Large Flight Data: A Multi-modal Deep Autoencoder Approach”, Conference of the Prognostics and Health Management Society, August 2016.
This document contains no technical data subject to the EAR or the ITAR.