David Katz, Data Scientist, TIBCO Software
Mike Alperin, Manufacturing Industry Consultant, TIBCO Software
July, 2018
Deep Learning for Anomaly Detection in Manufacturing
© Copyright 2000-2018 TIBCO Software Inc.
2
•Anomaly Detection in Manufacturing•Deep Learning Autoencoder• Neural Networks• Autoencoders• Software Tools
•Virtual Demo
Agenda
© Copyright 2000-2018 TIBCO Software Inc.
3
Anomaly Detection in Manufacturing
© Copyright 2000-2018 TIBCO Software Inc.
4
• Detecting new problems• Supervised vs. Unsupervised Learning
• Some Types of Anomalies• Facilities Equipment - sensor & environmental data • Process Equipment – sensor FDC data• Process Results – Process history and measurements• Physical Defects – defect images & characteristics• Device and Product – PCM and Sort data
• A General Method1. Detect anomalies2. Cluster them3. Classify with fingerprints or signatures 4. Determine causes of anomaly classes 5. Develop Action Plans to address causes 6. Predict cluster for new material and intervene to mitigate potential problems
Anomaly Detection in Manufacturing
© Copyright 2000-2018 TIBCO Software Inc.
5
Univariate Statistical Process Control
© Copyright 2000-2018 TIBCO Software Inc.
Detect changes from baseline –one variable at a time
Shewhart Process Control Charts• Statistically derived Control Limits• Western Electric or Nelson rules
• Automated Alerting
Individual – Moving Range Control Charts
• Suppose we measured 2 parameters y1 and y2 (e.g., person’s height & 1/weight)• Univariate charts would not detect some obvious outliers• This happens in many real applications
The Power of Multivariate Control Charts
Bad Tester
© Copyright 2000-2018 TIBCO Software Inc.
7
Univariate & Multivariate Methods
© Copyright 2000-2018 TIBCO Software Inc.
8
Real-time equipment anomaly prediction & clustering
© Copyright 2000-2018 TIBCO Software Inc.
High Tech Manufacturing Accelerator
https://community.tibco.com/modules/high-tech-manufacturing-accelerator
M
6
9
Deep Learning Autoencoder
© Copyright 2000-2018 TIBCO Software Inc.
10
• Frank Rosenblatt, Cornell, inventor of the Perceptron.• Brain mechanisms and models.
• Why the explosion?• New algorithms and techniques
• Convolutional NN, Recursive NN, Generative Adversarial NNs
• New Hardware capabilities• GPU
• Multicore
• Clusters
• More Data
• New Tools from the Open Source world.
From Neural Networks to Deep Learning
© Copyright 2000-2018 TIBCO Software Inc.
cs231n.github.io/neural-networks-1
11
• Create an identity transformation with constraints• Analogy to Principal Components – but much more
flexible/accurate.• Anomalies – the output is the reconstructed input, but it
does not fully match the original input => Reconstruction Error• Reconstruction Error:
• By component
• By sample.
Autoencoders
© Copyright 2000-2018 TIBCO Software Inc.
12
• Fraud
• Credit• Natural Language
• Speech
• Video
• Manufacturing
© Copyright 2000-2018 TIBCO Software Inc.
AbstractSmart manufacturing refers to using advanced data analytics to complement physical science for improving system performance and decision making. With the widespread deployment of sensors and Internet of Things, there is an …
Autoencoder Applications
13
•Sparse Autoencoders•Denoising Autoencoders•Generative Adversarial Networks•Variational Autoencoders
Autoencoders – Types and Variants
© Copyright 2000-2018 TIBCO Software Inc.
14
• H2O DeepLearning• Simple Structure of networks – just specify number of fully-
connected layers (and optionally dropout)• Settings for Sparse data can outperform GPU.• H2O Deep Water Project –
• uses GPU but no longer being developed;• H2O recommends Keras for new projects.
• Keras • Front end for Tensorflow, CNTK, Theano, MXNet• Specify complex network topologies• Use different types of layers – CNN, RNN,…• Can leverage GPU
Deep Learning Software
© Copyright 2000-2018 TIBCO Software Inc.
15
Virtual Demo
© Copyright 2000-2018 TIBCO Software Inc.
Industrial Plant: Raw Time series Data
© Copyright 2000-2018 TIBCO Software Inc.
Industrial Plant: Raw Time series Data
© Copyright 2000-2018 TIBCO Software Inc.
Industrial Plant: Raw Time series Data
© Copyright 2000-2018 TIBCO Software Inc.
19
Tag Training, Validation & Test Data Sets
© Copyright 2000-2018 TIBCO Software Inc.
Variable Selection
© Copyright 2000-2018 TIBCO Software Inc.
Model Configuration & Evaluation
Validation Error has clear minimumNote Distribution of Reconstruction Error
© Copyright 2000-2018 TIBCO Software Inc.
Problems converging
Model Configuration & Evaluation
Problems Converging
© Copyright 2000-2018 TIBCO Software Inc.
Model Configuration & Evaluation
© Copyright 2000-2018 TIBCO Software Inc.
Anomalies & Component Signatures
© Copyright 2000-2018 TIBCO Software Inc.
Anomalies & Component Signatures
© Copyright 2000-2018 TIBCO Software Inc.
Anomalies & Component Signatures
© Copyright 2000-2018 TIBCO Software Inc.
Anomalies & Component Signatures
Incident not detected on Univariate Chart
© Copyright 2000-2018 TIBCO Software Inc.
Reconstruction Error
Identify Incidents Programmatically
© Copyright 2000-2018 TIBCO Software Inc.
Elbow Plot of Cluster Config
© Copyright 2000-2018 TIBCO Software Inc.
Cluster Similar Incidents, View Signatures
© Copyright 2000-2018 TIBCO Software Inc.
31
Visit the TIBCO Industry 4.0page
To Learn & Do More
© Copyright 2000-2018 TIBCO Software Inc.
Visit the TIBCO Community Manufacturing Solutions page
Download AI & Machine Learning Manufacturing Solutions from the
TIBCO Exchange
32
• Thanks to Dr. Thomas Hill, Venkata Jagannath, Glenn Hoskins and Nico Rode for their contributions to this work
• Thanks to Michael O’Connell, Steven Hillion and HeleenSnelting for their support and encouragement.
Acknowledgments
© Copyright 2000-2018 TIBCO Software Inc.
Contacts:Mike Alperin
Manufacturing Industry [email protected]
David KatzData Scientist
Thank you!
Visit us in Booth 1021See this and other Manufacturing Demos
Learn more about the Technology
© Copyright 2000-2018 TIBCO Software Inc.