![Page 1: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/1.jpg)
10 deploys per dayDev & ops cooperation at Flickr
John Allspaw & Paul HammondVelocity 2009
![Page 2: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/2.jpg)
3 billion photos
http://flickr.com/photos/jimmyroq/415506736/
40,000 photos per second
![Page 3: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/3.jpg)
Dev versus Ops
![Page 4: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/4.jpg)
“It’s not my machines, it’s your code!”
![Page 5: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/5.jpg)
“It’s not my code, it’s your machines!”
![Page 6: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/6.jpg)
Spock ScottyLittle bit weird
Sits closer to the bossThinks too hard
Pulls levers & turns knobsEasily excitedYells a lot in emergencies
![Page 7: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/7.jpg)
![Page 8: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/8.jpg)
Says “No” all the timeAfraid that new fangled things will break the site
Fingerpointy
![Page 9: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/9.jpg)
They say “NO” all the time
Because no one tells them anything
BecauseThey say “NO” all the time
Because the site breaks unexpectedly
Ops stereotype
![Page 10: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/10.jpg)
Traditional thinking
Dev’s job is to add new featuresOps’ job is to keep the site stable and fast
http://www.flickr.com/photos/stewart/461099066/
![Page 11: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/11.jpg)
Ops’ job is NOT to keep the site stable and fast
![Page 12: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/12.jpg)
Ops’ job is to enable the business(this is dev’s job too)
![Page 13: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/13.jpg)
The business requires change
![Page 14: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/14.jpg)
But change is the root cause of most outages!
![Page 15: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/15.jpg)
Discourage change in the interests of stabilityor
Allow change to happen as often as it needs to
![Page 16: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/16.jpg)
Lowering risk of change through tools and culture
![Page 17: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/17.jpg)
♥Dev and Ops
![Page 18: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/18.jpg)
Ops who think like devsDevs who think like ops
![Page 19: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/19.jpg)
“But that’s me!”
![Page 20: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/20.jpg)
You can always think more like them
![Page 21: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/21.jpg)
Tools
![Page 22: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/22.jpg)
1. Automated infrastructureIf there is only one thing you do…
![Page 23: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/23.jpg)
1. Automated infrastructureIf there is only one thing you do…
Chef
Puppet
CFengine
FAI
System ImagerCobbler
BCfg2
![Page 24: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/24.jpg)
☁Role &configurationmanagement
OS imaging
![Page 25: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/25.jpg)
2. Shared version control
![Page 26: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/26.jpg)
Everyone knows where to lookhttp://www.flickr.com/photos/thunderchild5/1330744559/
![Page 27: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/27.jpg)
3. One step build
![Page 28: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/28.jpg)
![Page 29: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/29.jpg)
3. One step buildand deploy
![Page 30: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/30.jpg)
![Page 31: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/31.jpg)
![Page 32: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/32.jpg)
[2009-06-22 16:03:57] [harmes] site deployed (changes...)
Who? When? What?
![Page 33: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/33.jpg)
![Page 34: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/34.jpg)
Small frequent changeshttp://www.flickr.com/photos/mauren/2429240906/
![Page 35: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/35.jpg)
4. Feature flags(aka branching in code)
![Page 36: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/36.jpg)
1.0 1.1 1.2
1.0.1
1.1.1
1.0.2
Desktop software
![Page 37: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/37.jpg)
r2301 r2302
Web software
r2306
![Page 38: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/38.jpg)
Always ship trunk
http://www.flickr.com/photos/8720628@N04/2188922076/
![Page 39: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/39.jpg)
Everyone knows exactly where to lookhttp://www.flickr.com/photos/thunderchild5/1330744559/
![Page 40: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/40.jpg)
#phpif ($cfg['enable_feature_video']){ …}
{* smarty *}{if $cfg.enable_feature_beehive} …{/if}
Feature flags
![Page 41: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/41.jpg)
Private betas
http://www.flickr.com/photos/healthserviceglasses/3522809727/
![Page 42: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/42.jpg)
Bucket testing
http://www.flickr.com/photos/davidw/2063575447/
![Page 43: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/43.jpg)
Dark launches
http://www.flickr.com/photos/jking89/3031204314/
![Page 44: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/44.jpg)
Freecontingency switches
http://www.flickr.com/photos/flattop341/260207875/
![Page 45: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/45.jpg)
5. Shared metrics
![Page 46: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/46.jpg)
![Page 47: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/47.jpg)
Application level metrics
![Page 48: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/48.jpg)
Application level metrics
![Page 49: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/49.jpg)
Adaptive feedback loops
App System Metrics
RU ok?
maybe?
![Page 50: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/50.jpg)
![Page 51: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/51.jpg)
6. IRC and IM robots
![Page 52: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/52.jpg)
Dev, Ops, and Robots Having a conversation
IRC
searchengine
alertsmonitors
deploylogs
buildlogs
![Page 53: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/53.jpg)
Culture
![Page 54: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/54.jpg)
1. RespectIf there is only one thing you do…
![Page 55: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/55.jpg)
Don’tstereotype(not all developers are lazy)
http://www.flickr.com/photos/aaronjacobs/64368770/
![Page 56: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/56.jpg)
Respect other people’s expertise, opinions and responsibilities
http://www.flickr.com/photos/chrisdag/2286198568/
![Page 57: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/57.jpg)
Don’t just say “No”
http://www.flickr.com/photos/jwheare/2580631103/
![Page 58: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/58.jpg)
Don’t hide things
http://www.flickr.com/photos/alancleaver/2661424637/
![Page 59: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/59.jpg)
Developers: Talk to ops about the impact of your code:
• what metrics will change, and how?• what are the risks?• what are the signs that something is going wrong?• what are the contingencies?
This means you need to work this out before talking to ops
![Page 60: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/60.jpg)
2. Trust
![Page 61: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/61.jpg)
Ops needs to trust dev to involve them on feature discussions
Dev needs to trust ops to discuss infrastructure changes
Everyone needs to trust that everyone else is doing their best for the business
http://www.flickr.com/photos/85128884@N00/2650981813/
![Page 62: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/62.jpg)
Shared runbooks & escalation plans
http://www.flickr.com/photos/flattop341/224176602/
![Page 63: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/63.jpg)
Provide knobs and levers
http://www.flickr.com/photos/telstar/2861103147/
![Page 64: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/64.jpg)
Ops: Be transparent,give devs access to systems
http://www.flickr.com/photos/williamhook/3468484351/
![Page 65: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/65.jpg)
3. Healthy attitude about failure
![Page 66: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/66.jpg)
Failure will happen
http://www.flickr.com/photos/pinksherbet/447190603/
![Page 67: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/67.jpg)
If you think you can prevent failure thenyou aren’t developing your ability to respond
http://www.flickr.com/photos/toms/2323779363/
![Page 68: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/68.jpg)
http://www.flickr.com/photos/changereality/2349538868/
![Page 69: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/69.jpg)
Fire drills
http://www.flickr.com/photos/dnorman/2678090600
![Page 70: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/70.jpg)
4. Avoiding Blame
![Page 71: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/71.jpg)
No fingerpointing
http://www.flickr.com/photos/rocketjim54/2955889085/
![Page 72: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/72.jpg)
Fingerpointyness
problem!!!argggh!
time
freaking out,not talking,finding fault
blaming,covering
ass
fixin
g th
ings
fixed.
whining,hiding.
hurt egos
figuring it out
![Page 73: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/73.jpg)
Being productive
problem!!!argggh!
time
fixin
g th
ings
fixed.
feeling guilty
figuring it out
move on with
life
![Page 74: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/74.jpg)
Developers: Remember that someone else will probably get woken up when your code breaks
http://www.flickr.com/photos/alex-s/353218851/
![Page 75: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/75.jpg)
Ops: provide constructive feedback on current achesand pains
http://www.flickr.com/photos/allspaw/2819774755/
![Page 76: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/76.jpg)
1. Automated infrastructure 2. Shared version control 3. One step build and deploy 4. Feature flags5. Shared metrics 6. IRC and IM robots
1. Respect 2. Trust 3. Healthy attitude about failure 4. Avoiding Blame
![Page 77: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/77.jpg)
This is not easyYou could just carry on shouting at each other…
![Page 78: 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr](https://reader035.vdocuments.net/reader035/viewer/2022081715/540d140e7bef0a84278b9b3e/html5/thumbnails/78.jpg)
(Thank you)