monitoring the unknown, 1000*100 series a day - devoxx morocco 2017
TRANSCRIPT
![Page 1: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/1.jpg)
Monitoring the unknown, 1000*100 series a day
Quentin ADAM - @waxzce
2017
![Page 2: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/2.jpg)
Who am I ?Quentin ADAM from the Clever Cloud
@waxzce on twitter – github- soundcloud – instagram ….
![Page 3: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/3.jpg)
My day to day work : , the IT automation company
![Page 4: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/4.jpg)
Keep your apps online. made with node.js,scala, java, ruby, php, python, go…
Cloud & on premise
![Page 5: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/5.jpg)
And learn a lot of things about your code, apps, and good/bad design…
![Page 6: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/6.jpg)
Give back to the community
NEVER GONNA LET YOU DOWN
clever-cloud.com
![Page 7: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/7.jpg)
Automate infrastructure
![Page 8: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/8.jpg)
What are we talking about?
![Page 9: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/9.jpg)
Immutable infrastructureFR : https://www.youtube.com/watch?v=WrZCbgQsPVU
EN : https://www.infoq.com/presentations/server-cloud
![Page 10: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/10.jpg)
Ephemeral instances
![Page 11: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/11.jpg)
Pet!=Cattle
![Page 12: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/12.jpg)
Monitoring service!=Monitoring host
![Page 13: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/13.jpg)
Monitoring the unknowPart 1
![Page 15: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/15.jpg)
Monitoring the unknowPart 2
![Page 16: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/16.jpg)
There is no ready solution…
![Page 17: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/17.jpg)
Let’s build something
![Page 18: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/18.jpg)
Data needs
![Page 19: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/19.jpg)
Store measure in time
![Page 20: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/20.jpg)
Analyze itMost of the time using a micro batching approach
![Page 21: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/21.jpg)
Traditional needs not required
• No ACID
• No Transaction
• Just a few full text
• Eventual consistency if enough
![Page 22: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/22.jpg)
SCALE
![Page 23: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/23.jpg)
Time Series DB
![Page 24: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/24.jpg)
Influx DB
![Page 25: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/25.jpg)
Prometeus
![Page 26: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/26.jpg)
Open TSDB
![Page 27: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/27.jpg)
Cassandra + Akka persistence/stream
![Page 28: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/28.jpg)
Warp10
![Page 29: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/29.jpg)
Standalone Java fat jar
![Page 30: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/30.jpg)
Distributed
• Consensus using Zookeepers
• Store and analysis on Hadoop
• Event bus with Kafka
• Several roles in a single JAR, all stateless
![Page 31: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/31.jpg)
So, We can now store data
![Page 32: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/32.jpg)
Agent: Telegraf
![Page 33: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/33.jpg)
Agent goals
![Page 34: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/34.jpg)
Collect data
• System
• Many connector to specific softs
• Open protocols…
![Page 35: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/35.jpg)
JMX
![Page 36: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/36.jpg)
Statsd
![Page 37: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/37.jpg)
Prometeus
![Page 38: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/38.jpg)
Buffer to send batched data
~100 metric point / minutes
![Page 39: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/39.jpg)
Token based security
![Page 40: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/40.jpg)
More tokens: JWT and Macaroon
EN : https://www.youtube.com/watch?v=SO5G0Gsm5-w
FR : https://www.youtube.com/watch?v=mvKeCsxGZhE
![Page 41: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/41.jpg)
✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
❔ Analyze data
❔ Trigger data
❔ Display data
![Page 42: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/42.jpg)
WarpScript, distributed on hadoop
![Page 43: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/43.jpg)
Warp10 roles
• Ingress
• Directory
• Store
• Egress
• Webcall
• Plasma
• Runner
• Fetcher
![Page 44: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/44.jpg)
µBatching WarpScriptµBatchingWarpScriptµBatchingWarpScript
![Page 45: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/45.jpg)
Build a new series
![Page 46: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/46.jpg)
Listen on Kafka or Plasma for triggers
![Page 47: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/47.jpg)
Immutable DataBase
![Page 48: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/48.jpg)
✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
✅ Analyze data
✅ Trigger data
❔ Display data
![Page 49: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/49.jpg)
Quantum
Tool
+
PolymerJS lib
![Page 50: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/50.jpg)
Quantum
Tool
+
PolymerJS lib
![Page 51: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/51.jpg)
Which data expose?
![Page 52: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/52.jpg)
Templating the data
![Page 53: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/53.jpg)
Templating the data
![Page 54: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/54.jpg)
Templating the data
![Page 55: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/55.jpg)
![Page 56: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/56.jpg)
![Page 57: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/57.jpg)
Give access to the API
![Page 58: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/58.jpg)
Enhance developer work, happiness and productivity
![Page 59: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/59.jpg)
Give access to metrics and logs
![Page 60: MONITORING THE UNKNOWN, 1000*100 SERIES A DAY - DEVOXX MOROCCO 2017](https://reader034.vdocuments.net/reader034/viewer/2022052606/5a64b6857f8b9a900f8b4733/html5/thumbnails/60.jpg)
Thank you
find me on twitter
@waxzce
Gift coupon for clever-cloud.com:
devoxxMA2017