latency: the silent monitoring system killer
DESCRIPTION
Slides from a lighting talk presented at the January Sydney DevOps Meetup.TRANSCRIPT
![Page 1: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/1.jpg)
LATENCYTHE SILENT
MONITORING SYSTEM KILLER
Saturday, 21 January 2012
![Page 2: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/2.jpg)
#MONITORINGSUCKS
Saturday, 21 January 2012
![Page 3: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/3.jpg)
SCALABILITY
Saturday, 21 January 2012
![Page 4: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/4.jpg)
execute large volumes of monitoring checks under a variety of conditions
(good + bad) with a consistent throughput
Saturday, 21 January 2012
![Page 5: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/5.jpg)
CONSISTENT THROUGHPUT
Saturday, 21 January 2012
![Page 6: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/6.jpg)
SHORT CHECK EXECUTION TIME
Saturday, 21 January 2012
![Page 7: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/7.jpg)
WHAT INTRODUCES VARIABILITY?
Saturday, 21 January 2012
![Page 8: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/8.jpg)
LATENCY INTRODUCED c SYNCHRONOUS CALLS
Saturday, 21 January 2012
![Page 9: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/9.jpg)
Saturday, 21 January 2012
![Page 10: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/10.jpg)
= 1 monitoring check
Saturday, 21 January 2012
![Page 11: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/11.jpg)
Saturday, 21 January 2012
![Page 12: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/12.jpg)
150 monitoring checkseach executed every 300 secondseach takes 1 secondchecks are executed serially
Saturday, 21 January 2012
![Page 13: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/13.jpg)
all checks executed in 150 secondsmonitoring system at 50% capacity
Saturday, 21 January 2012
![Page 14: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/14.jpg)
DOUBLE THE CHECKS
Saturday, 21 January 2012
![Page 15: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/15.jpg)
Saturday, 21 January 2012
![Page 16: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/16.jpg)
all checks executed in 300 secondsmonitoring system at 100% capacity
Saturday, 21 January 2012
![Page 17: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/17.jpg)
DOUBLE THE EXECUTION TIME
Saturday, 21 January 2012
![Page 18: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/18.jpg)
Saturday, 21 January 2012
![Page 19: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/19.jpg)
all checks executed in 600 secondsmonitoring system at 200% capacityonly 50% of the checks are “on time”
Saturday, 21 January 2012
![Page 20: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/20.jpg)
CHECK LATENCY
Saturday, 21 January 2012
![Page 21: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/21.jpg)
Saturday, 21 January 2012
![Page 22: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/22.jpg)
HOW DO WE FIX THIS!?
Saturday, 21 January 2012
![Page 23: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/23.jpg)
PERFORMANCE ANALYSIS!
Saturday, 21 January 2012
![Page 24: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/24.jpg)
AN ANALOGY
Saturday, 21 January 2012
![Page 25: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/25.jpg)
MONITORING CHECK ==
“ACTION” ON MVC WEB APP
Saturday, 21 January 2012
![Page 26: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/26.jpg)
Saturday, 21 January 2012
![Page 27: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/27.jpg)
Saturday, 21 January 2012
![Page 28: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/28.jpg)
SEPARATE DATA COLLECTION
FROM
THRESHOLDINGf NOTIFICATIONS
Saturday, 21 January 2012
![Page 29: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/29.jpg)
Saturday, 21 January 2012
![Page 30: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/30.jpg)
THIS SHIFTS LATENCY
Saturday, 21 January 2012
![Page 31: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/31.jpg)
IT DOES NOT ELIMINATE IT!
Saturday, 21 January 2012
![Page 32: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/32.jpg)
Saturday, 21 January 2012
![Page 33: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/33.jpg)
RRDTOOL IS EVIL
Saturday, 21 January 2012
![Page 34: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/34.jpg)
USE SOMETHING BETTER!
Saturday, 21 January 2012
![Page 35: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/35.jpg)
USE
OPENTSDBOR
GANGLIAWITH
CHECK_TSDBOR
CHECK_GMOND
Saturday, 21 January 2012
![Page 36: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/36.jpg)
Saturday, 21 January 2012
![Page 37: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/37.jpg)
A DIFFERENT SET a PROBLEMS
Saturday, 21 January 2012
![Page 38: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/38.jpg)
STORAGE WILL GO AWAY
Saturday, 21 January 2012
![Page 39: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/39.jpg)
CHAOS WILL ENSUE
Saturday, 21 January 2012
![Page 40: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/40.jpg)
PAGERS WILL MELT
Saturday, 21 January 2012
![Page 41: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/41.jpg)
SET UP “META-
PARENTING”
Saturday, 21 January 2012
![Page 42: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/42.jpg)
BUILD A KILL SWITCH
Saturday, 21 January 2012
![Page 43: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/43.jpg)
Saturday, 21 January 2012
![Page 44: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/44.jpg)
READ MORE ABOUT THIS:
Saturday, 21 January 2012
![Page 45: Latency: The Silent Monitoring System Killer](https://reader033.vdocuments.net/reader033/viewer/2022052900/555a4135d8b42a83368b5070/html5/thumbnails/45.jpg)
bit.ly/yN4mdy
Saturday, 21 January 2012