big data expo - machine learning in the elastic stack
TRANSCRIPT
Dr Stephen Dodson, Machine Learning, Elastic
Machine Learning in the Elastic Stack
2
Elastic Stack
Beats
Logstash
Kibana
Elasticsearch
X-PackX-Pack X-Pack
Security
Alerting
Monitoring
Reporting
Graph
Machine Learning
3
Extracting useful, valuable information is hard
4
Search
Aggregations
Visualization
Machine Learning
Extracting useful, valuable information is hard
Machine Learning1Algorithms and methods for data driven prediction, decision making, and modelling
Supervised Learning
Prediction based on examples of correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal to optimize the consequences of
actions
Other! …
Machine Learning1Algorithms and methods for data driven prediction, decision making, and modelling
Supervised Learning
Prediction based on examples of correct behavior
1Machine Learning Overview, Tommi Jaakkola, MIT
Unsupervised Learning
No explicit target, only data, goal to model/discover
Semi-supervised Learning
Supplement limited annotations with unsupervised learning
Active Learning
Learn to query the examples actually needed for learning
Transfer Learning
How to apply what you have learned from A to B
Reinforcement Learning
Learning to act, not just predict; goal to optimize the consequences of
actions
Other! …
Time Series Anomaly Detection
Machine Learning - Technical Debt
7
• Machine Learning Systems generally involved a large amount of plumbing and complexity that incur large ongoing maintenance costs
From Google Paper: Sculley, D., et al. "Machine learning - The high-interest credit card of technical debt." (2014).
© Elasticsearch BV
Has my order rate dropped significantly?
Has my order rate dropped significantly?
• Learn models from past behaviour (training, modelling)
• Use models to predict future behaviour (prediction)
• Use predictions to make decisions
Expected value @ 15:05 = 1859
Actual value @ 15:05 = 280
Probability = 0.0000174025
Demo: Simple Time Series
Has my system changed behaviour?
i-5cfd3dcb
...
…
i-f1e94994
i-ece626ff i-ebc323df
Has my system changed behaviour?
i-5cfd3dcb i-f1e94994
i-ece626ff i-ebc323df
...
…
Demo: Multiple Time Series
Do my application logs contain unusual messages?
Do my application logs contain unusual messages?Classify unstructured log messages by clustering similar messages
Nor
mal
Log
Mes
sage
sU
nusu
al lo
g M
essa
ges
Demo: Log Messages
17
Ingest, Enrich, Visualize, Analyse, Alert
Elasticsearch
X-pack
Master Nodes (3)
Ingest Nodes (X)
Data Nodes - Hot (X)
Data Nodes - Warm (X)
Beats
Log Files Metrics
Wire Data your(beat)
Filebeat Module
NGINX
Kibana
X-pack
Instances (X)
18
Coming soon - forecasting…and more…
Exceptwhereotherwisenoted,thisworkislicensedunderhttp://creativecommons.org/licenses/by-nd/4.0/
CreativeCommonsandthedoubleCinacircleareregisteredtrademarksofCreativeCommonsintheUnitedStatesandothercountries.
Thirdpartymarksandbrandsarethepropertyoftheirrespectiveholders.
Please attribute Elastic with a link to elastic.co