lessons from iraq - building & running sre teams
TRANSCRIPT
![Page 1: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/1.jpg)
Building andRunning
SRE Teams
Lessons from Iraq for
SREcon16 Europe
![Page 2: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/2.jpg)
Site Reliability @
2013 2014 2015 20160100200300400500600700800
![Page 3: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/3.jpg)
Site Reliability @
![Page 4: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/4.jpg)
Site Reliability @
![Page 5: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/5.jpg)
Site Reliability @
![Page 6: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/6.jpg)
Site Reliability @
AB C D E F
AB
B1 B2 B3B31B32
CC1C2
DD1 D2
D21 D22D22
1D222
D23
EE1E2E3E4
FF1 F2 F3 F4
F41
![Page 7: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/7.jpg)
![Page 8: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/8.jpg)
![Page 9: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/9.jpg)
![Page 10: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/10.jpg)
![Page 11: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/11.jpg)
![Page 12: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/12.jpg)
![Page 13: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/13.jpg)
![Page 14: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/14.jpg)
![Page 15: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/15.jpg)
Taylorist Management• Planning, • Organizing, • Commanding, • Coordinating, and • Controlling
• Incentivizes: …gather and centralize more information in order to push more and more efficient directives to the organization
![Page 16: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/16.jpg)
Taylorist Worker•Feed the cycle, and •Await the next commands
![Page 17: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/17.jpg)
Scientific Management…is premised upon:1. Absolutely rigid and inflexible standards throughout
your establishment2. That each employee of your establishment should
receive every day clear-cut, definite instructions as to just what he is to do and how he is to do it, and these instructions should be exactly carried out, whether they are right or wrong.
![Page 18: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/18.jpg)
![Page 19: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/19.jpg)
![Page 20: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/20.jpg)
![Page 21: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/21.jpg)
![Page 22: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/22.jpg)
SilosTo each unit, the piece of the war that really mattered was the piece inside their box on the org chart
![Page 23: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/23.jpg)
What’s your job?If you play defense, . . .?• Most people say their job is to prevent the other team from
scoring goals.
If you play offense, . . .?• Most people say their job is to score goals.
Most people are wrong.
![Page 24: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/24.jpg)
• Adaptability is more characteristic of small interactive teams than large top-down hierarchies
• Shifting from tactical to strategic adaptability requires organizing teams into teams of teams instead of command-delegated teams
![Page 25: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/25.jpg)
SRE Tactics for Teams of Teams• Virtual teams and working groups• great for bootstrapping new technologies
and driving adoption
![Page 26: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/26.jpg)
SRE Tactics for Teams of Teams
![Page 27: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/27.jpg)
SRE Tactics for Teams of Teams• PM Roll Up
![Page 28: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/28.jpg)
SRE Tactics for Teams of Teams• Inter-team rotations
![Page 29: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/29.jpg)
Most organizations are more concerned with how best to control information than how best to share it
![Page 30: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/30.jpg)
![Page 31: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/31.jpg)
![Page 32: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/32.jpg)
”It is the idea flow within a community that builds the intelligence that makes it
successful.”
—Alex Pentland
![Page 33: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/33.jpg)
Engagement
Exploration
![Page 34: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/34.jpg)
![Page 35: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/35.jpg)
![Page 36: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/36.jpg)
![Page 37: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/37.jpg)
Manager: “What if I spend all this money training and developing my staff and they leave?”
Consultant: “What if you don’t and they stay?”
![Page 38: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/38.jpg)
![Page 39: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/39.jpg)
Push decision making as close to the action
as possible
![Page 40: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/40.jpg)
Effective Incident Response… 1400, Friday afternoon … pager has just exploded … service has stopped serving any traffic in an entire datacenter. A few minutes in, … another alert: a second datacenter has stopped
serving. Then the third of your five datacenters fails …
Is this going to be one of those days or weekends?
• Technical Focus• Poor
Communication• Freelancing
• Recursive Separation of Duties• Incident Command• Operational Response• Communication• Planning
or
![Page 41: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/41.jpg)
“When we find that things go right under difficult circumstances, it’s
mostly because of people’s adaptive capacity; their ability to recognize, adapt to, and absorb
changes and disruptions, some of which might fall outside of what the
system is designed or trained to handle.”
– Sidney Dekker
![Page 42: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/42.jpg)
Task Force findings•With local authority to make decisions:
quality & speed increased
•Old way (”try harder”)10 18 raids per month
•New way: 300 per month (17x)
![Page 43: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/43.jpg)
![Page 44: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/44.jpg)
![Page 45: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/45.jpg)
![Page 46: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/46.jpg)
![Page 47: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/47.jpg)
![Page 48: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/48.jpg)
13
1030
100300
1000
![Page 49: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/49.jpg)
What are you
inspiring others
toward?
![Page 50: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/50.jpg)
![Page 51: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/51.jpg)
Relationships
matter
![Page 53: Lessons from Iraq - Building & Running SRE Teams](https://reader036.vdocuments.net/reader036/viewer/2022062522/58a5cc5c1a28ab6c2a8b58cb/html5/thumbnails/53.jpg)
Resources• Team of Teams, Stanley McChrystal
ISBN-13: 978-1591847489• Social Physics, Alex Pentland
ISBN-13: 978-0143126331• Conscious Business, Fred Kofman
ISBN-13: 978-1622032020• Peter Siebel’s blog: gigamonkeys.com• Rule of 3 & 10:
https://www.sequoiacap.com/article/the-rule-of-3-and-10/