![Page 1: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/1.jpg)
#PDSummit16#PDSummit16
The Journey of Chaos EngineeringBegins with a Single Step
![Page 2: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/2.jpg)
#PDSummit16#PDSummit16
Bruce WongSenior Engineering Manager
Twilio@bruce_m_wong
https://www.linkedin.com/in/brucemwong
![Page 3: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/3.jpg)
#PDSummit16#PDSummit16
![Page 4: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/4.jpg)
#PDSummit16#PDSummit16
2009
2012
2014http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html https://github.com/Netflix/SimianArmy http://techblog.netflix.com/2015/09/chaos-engineering-upgraded.html
![Page 5: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/5.jpg)
#PDSummit16http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone/ http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html
![Page 8: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/8.jpg)
#PDSummit16#PDSummit16
The journey of a thousand miles begins with a single
step.
-Lao Tzu
![Page 9: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/9.jpg)
#PDSummit16#PDSummit16
James BurnsTech Lead
Twilio@1mentat
#PDSummit16 https://www.linkedin.com/in/james-burns-7816a82
![Page 10: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/10.jpg)
#PDSummit16#PDSummit16
PreparationPre-Launch Log Aggregation System -Stage env -Synthetic Traffic
![Page 11: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/11.jpg)
#PDSummit16
The Master of Disaster•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load
![Page 12: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/12.jpg)
> sudo halt
#PDSummit16
![Page 13: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/13.jpg)
Incident Start
#PDSummit16
![Page 14: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/14.jpg)
Impact?
#PDSummit16
![Page 15: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/15.jpg)
Post-Mortem
#PDSummit16
![Page 16: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/16.jpg)
#PDSummit16#PDSummit16
![Page 17: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/17.jpg)
#PDSummit16
Round 2•Network Issues
•Partitions
•Thundering Herds
•Cascading Failures
•Resource Starvation
•CPU
•Memory
•Disk IO
•Network IO
•Application Load
![Page 18: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/18.jpg)
> sudo halt
#PDSummit16
![Page 19: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/19.jpg)
Third-Party API Failure
#PDSummit16
![Page 20: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/20.jpg)
#PDSummit16
Well, that’s not what I expected to see
![Page 21: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/21.jpg)
#PDSummit16
Outcomes
Instrument
Instrument
Instrument
API SLAs
Architectural Change!
![Page 22: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/22.jpg)
#PDSummit16
Recap
• Start Simple • Instrumentation
Gaps • Understand your
dashboards • Prevent outages
![Page 23: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/23.jpg)
#PDSummit16http://www.crisistextline.org/ http://polarisproject.org/befree-textline http://trekmedics.org/ https://www.twilio.org/
![Page 24: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/24.jpg)
#PDSummit16
When you wish upon a blue moon…
![Page 25: The Journey of Chaos Engineering Begins with a Single Step](https://reader033.vdocuments.net/reader033/viewer/2022051318/588258fa1a28abd40e8b52ff/html5/thumbnails/25.jpg)
#PDSummit16#PDSummit16
Please provide feedback for this
session by filling out the feedback survey