velocity ny 2014: signal through the noise
DESCRIPTION
In recent years it’s become evident that alerting is one of the biggest challenges facing modern Operations Engineers. Conference talks, hallways tracks, meetups, etc are rife with discussions about poor signal/noise in alerts, fatigue from false positives, and general lack of actionability. Our talk (informed by real-world experience designing, building and maintaining our distributed, multi-tenant metrics/alerting service) takes a fundamental approach and examines alerting requirements and practices in the abstract. We put forth a comprehensive abstract model with best practices that should be followed and implemented by your team regardless of your tool of choice. This talk is equal parts cultural and technical, encompassing both computational capabilities as well as social practices, like: Defining organizational policy about where and when to set alerts. Ensuring the on-call engineer is armed with the proper information to take action Best practices for configuring an alert Fire-fighting after an alert has triggered Performing analysis across your organization wide history of alertsTRANSCRIPT
![Page 1: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/1.jpg)
hi.github: djosephsen
![Page 5: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/5.jpg)
@davejosephsen
github: djosephsen
Signal Through the Noise
![Page 6: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/6.jpg)
![Page 7: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/7.jpg)
![Page 8: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/8.jpg)
![Page 9: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/9.jpg)
![Page 10: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/10.jpg)
![Page 11: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/11.jpg)
![Page 12: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/12.jpg)
WAT?
![Page 13: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/13.jpg)
WAT?
![Page 14: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/14.jpg)
AAAGHHHHH!!!
![Page 15: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/15.jpg)
ALERTS AREN’T FREE
![Page 16: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/16.jpg)
Business Projects
IT Projects
Changes
Unplanned Work
![Page 17: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/17.jpg)
Unplanned Work
(eeew Comic Sans)
![Page 18: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/18.jpg)
Unplanned Work
![Page 19: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/19.jpg)
Unplanned Work
![Page 20: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/20.jpg)
Alerting
![Page 21: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/21.jpg)
Tax the Ammunition
![Page 22: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/22.jpg)
![Page 23: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/23.jpg)
![Page 24: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/24.jpg)
![Page 25: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/25.jpg)
THE CONTENT OF YOUR ALERTS MATTERS
![Page 26: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/26.jpg)
What did he just say?
•Notifications are expensive, they hurt people and productivity
•Make people work harder to send them by requiring run books
•Run books add context to alerts. Other types of context are awesome too
•Like graphs
![Page 27: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/27.jpg)
WHY do we Monitor?
![Page 28: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/28.jpg)
![Page 29: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/29.jpg)
![Page 30: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/30.jpg)
![Page 31: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/31.jpg)
![Page 32: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/32.jpg)
Telemetry Data
Command Signal
![Page 33: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/33.jpg)
![Page 34: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/34.jpg)
1. Identify Operational LimitationsY<160bpm
X<7m km/h
![Page 35: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/35.jpg)
2. Monitor those limitations1. Identify Operational Limitations
Y<160bpmX<7m km/h
![Page 36: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/36.jpg)
![Page 37: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/37.jpg)
A Balancer ?!
![Page 38: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/38.jpg)
Balancer
>66% Host Availability
![Page 39: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/39.jpg)
Balancer
>66% Host Availability
![Page 40: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/40.jpg)
% IO per instance
![Page 41: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/41.jpg)
%hosts alive
% IO per instanceVS
(Hint: one of these things measures balancing)
![Page 42: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/42.jpg)
%hosts alive
% IO per instance
Does not measure balancing Measures balancing
66 .2VSX
![Page 43: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/43.jpg)
![Page 44: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/44.jpg)
![Page 45: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/45.jpg)
![Page 46: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/46.jpg)
![Page 47: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/47.jpg)
![Page 48: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/48.jpg)
IT Monitoring != Feedback
![Page 49: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/49.jpg)
IT Monitoring != Feedback
![Page 50: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/50.jpg)
some silly balancer!=
![Page 51: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/51.jpg)
WE CAN REDUCE ALERTS BY IMPROVING OUR TELEMETRY
SIGNAL
![Page 52: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/52.jpg)
What did he just say?•Monitoring isn't a thing. It’s just part of the engineering process
•We’re treating it like a thing that only some types of engineers might want to do, and that’s giving us broken feedback
•Aerospace engineers are rad, they don’t do that.
•Fix your monitoring and your alerts will follow
![Page 53: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/53.jpg)
![Page 54: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/54.jpg)
![Page 55: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/55.jpg)
![Page 56: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/56.jpg)
![Page 57: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/57.jpg)
![Page 58: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/58.jpg)
![Page 59: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/59.jpg)
![Page 60: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/60.jpg)
![Page 61: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/61.jpg)
![Page 62: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/62.jpg)
![Page 63: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/63.jpg)
![Page 64: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/64.jpg)
![Page 65: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/65.jpg)
Own YOUR problem
![Page 66: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/66.jpg)
Own YOUR problem
![Page 67: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/67.jpg)
Some Graph in the War Room
![Page 68: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/68.jpg)
![Page 69: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/69.jpg)
![Page 70: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/70.jpg)
![Page 71: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/71.jpg)
![Page 72: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/72.jpg)
![Page 73: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/73.jpg)
Some Graph in the War Room
![Page 74: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/74.jpg)
Some Graph in the War Room
![Page 75: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/75.jpg)
WHAT YOU MONITOR MATTERS
![Page 76: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/76.jpg)
![Page 77: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/77.jpg)
![Page 78: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/78.jpg)
![Page 79: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/79.jpg)
a } < x
C
![Page 80: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/80.jpg)
} < x
b
kxa
![Page 81: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/81.jpg)
xk
xk
xk
![Page 82: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/82.jpg)
EVERYBODY OWNS MONITORING
![Page 83: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/83.jpg)
![Page 84: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/84.jpg)
![Page 85: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/85.jpg)
![Page 86: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/86.jpg)
![Page 87: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/87.jpg)
![Page 88: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/88.jpg)
![Page 89: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/89.jpg)
![Page 90: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/90.jpg)
![Page 91: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/91.jpg)
![Page 92: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/92.jpg)
![Page 93: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/93.jpg)
![Page 94: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/94.jpg)
![Page 95: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/95.jpg)
![Page 96: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/96.jpg)
What did he just say?• Choose metrics that tell you about the things you care about.
•Alert when the things you care about hit limits you understand
•All alerts < critical go to chatrooms, ticket systems or dashboards
•Critical alers use an automated escalation service that enforces on call policy
•Escalated alerts require acknowledgement
•Escalated alerts require run book url’s and/or links to graphs of the metric
![Page 97: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/97.jpg)
ALERT ON WHAT YOU SEE
![Page 98: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/98.jpg)
![Page 99: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/99.jpg)
![Page 100: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/100.jpg)
![Page 101: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/101.jpg)
EVERYONE OWNS ALERTS(and dashboards)
![Page 102: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/102.jpg)
![Page 103: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/103.jpg)
![Page 104: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/104.jpg)
![Page 105: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/105.jpg)
![Page 106: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/106.jpg)
The Ultimate Recap• Enforce a notification policy that requires context
• Make monitoring an engineering process
• Use the same signal for all metrics introspection and notification
• Encourage everyone to rely on telemetry data (graphs or it didn’t happen!)
• Everyone who collects a metric, gets keys to dashboard and alert design
![Page 107: Velocity NY 2014: Signal through the noise](https://reader038.vdocuments.net/reader038/viewer/2022110302/5485efe1b4af9fd0668b4619/html5/thumbnails/107.jpg)
Questions?Office Hours: 1:15pm