misadventures in continuous delivery...you know there’s a problem with continuous delivery when…...
TRANSCRIPT
![Page 1: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/1.jpg)
Misadventures in Continuous Delivery Dan DeMeyere, Chief Product Officer @ thredUP
![Page 2: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/2.jpg)
2
BACKGROUND
● thredUP: the world’s largest online consignment store
● Joined thredUP in 2010 as a full-stack engineer
● Responsible for engineering + product (~60)
● Cross-functional pod system
● Isomorphic React/Apollo SPA, GraphQL orchestration, Ruby/Java/Node on the back-end
● Migrated to k8s three years ago
![Page 3: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/3.jpg)
Confidential 3
![Page 4: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/4.jpg)
4
CONTINUOUS DELIVERY
“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble
![Page 5: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/5.jpg)
5
ENGINEERING MISADVENTURES
![Page 6: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/6.jpg)
6
WHY FOCUS ON MISADVENTURES?
● To avoid making the same mistakes we did
● Everyone can relate
● Learnings in how to scale are hard to get
![Page 7: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/7.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
![Page 8: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/8.jpg)
RAISE YOUR HAND IF:
Have you ever worked in E-Commerce?
![Page 9: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/9.jpg)
RAISE YOUR HAND IF:
Have you ever worked in E-Commerce? Have you ever heard of “Cyber Week”?
![Page 10: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/10.jpg)
RAISE YOUR HAND IF:
Have you ever worked in E-Commerce? Have you ever heard of “Cyber Week”? Have you had to do any dev work for anything related to “Cyber Week”?
![Page 11: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/11.jpg)
11
CYBER WEEK IN E-COMMERCE
![Page 12: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/12.jpg)
12
CYBER WEEK IN E-COMMERCE
![Page 13: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/13.jpg)
13
CYBER WEEK IN E-COMMERCE
![Page 14: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/14.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
![Page 15: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/15.jpg)
15
HOW TO PICK THE ENGINEER
![Page 16: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/16.jpg)
16
OUR DEPLOY PIPELINE
![Page 17: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/17.jpg)
17
CI PIPELINE CHECKLIST
● Is your end-to-end deploy pipeline quick?
● Is it safe? Does it gracefully degrade?
● Can you roll back with ease?
● Is the deployment process sustainable?
● Do engineers have confidence when deploying?
![Page 18: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/18.jpg)
18
CI PIPELINE CHECKLIST
● Is your end-to-end deploy pipeline quick?
● Is it safe? Does it gracefully degrade?
● Can you roll back with ease?
● Is the deployment process sustainable?
● Do engineers have confidence when deploying?
![Page 19: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/19.jpg)
19
LAZY ENGINEERS
“I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.” - Bill Gates
![Page 20: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/20.jpg)
20
CLEVER ENGINEERING
![Page 21: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/21.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
![Page 22: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/22.jpg)
22
CONTINUOUS DELIVERY
“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble
![Page 23: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/23.jpg)
RAISE YOUR HAND IF:
Have you ever tried to decompose a monolith?
![Page 24: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/24.jpg)
24
MONOLITH DECOMPOSITION Has anyone ever done something small and easy like decompose a monolith?
![Page 25: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/25.jpg)
25
MONOLITH DECOMPOSITION ● Migrate RethinkDB to AuroraDB
● Pull out user personalization into separate service
● Pull out shopping cart responsibilities into separate service
● Introduce RabbitMQ so the back-ends can talk to each other
● Consolidate all client API requests to GraphQL orchestration layer
● Pull out auction responsibilities into separate service
![Page 26: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/26.jpg)
26
MIGRATION DEPLOYS (IN THEORY - BEFORE)
![Page 27: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/27.jpg)
27
MIGRATION DEPLOYS (IN THEORY - DURING)
![Page 28: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/28.jpg)
28
MIGRATION DEPLOYS (IN THEORY - AFTER)
![Page 29: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/29.jpg)
29
MIGRATION DEPLOYS (IN PRACTICE - BEFORE)
![Page 30: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/30.jpg)
30
MIGRATION DEPLOYS (IN PRACTICE - DURING)
![Page 31: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/31.jpg)
31
MIGRATION DEPLOYS (IN PRACTICE - AFTER)
![Page 32: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/32.jpg)
32
WHY HOLE-SHOT DEPLOYS ARE PROBLEMATIC
● Staging != Production ● Extended QA in staging -> elongates the development process
● Risk for the business -> creates unnecessary stress for the team
● Repeated stress -> recipe for burn out
![Page 33: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/33.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
![Page 34: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/34.jpg)
34
FEATURE BRANCH WORKFLOW (IN THEORY)
![Page 35: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/35.jpg)
35
FEATURE BRANCH WORKFLOW (IN PRACTICE)
![Page 36: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/36.jpg)
36
WHEN YOU HAVE TO ROLL BACK THE FEATURE YOU DEPLOYED
![Page 37: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/37.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
![Page 38: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/38.jpg)
38
CONTINUOUS MERGE WORKFLOW
![Page 39: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/39.jpg)
39
CODE REVIEW FOR LONG PULL REQUESTS
![Page 40: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/40.jpg)
40
CONTINUOUS WORKFLOW BENEFITS
● Bite-sized pull requests -> easier code review ● Continuous deploys -> earlier production feedback
● More granular time estimations -> consistency -> predictability -> sustainable
● Last 10% of the project is actually the last 10%
![Page 41: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/41.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
5. Engineers think velocity and stability are mutually exclusive
![Page 42: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/42.jpg)
42
CONTINUOUS DELIVERY
“The ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble
![Page 43: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/43.jpg)
43
“SHOP HER CLOSET” LAUNCH
![Page 44: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/44.jpg)
44
LAST MINUTE CHANGES
![Page 45: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/45.jpg)
45
WHEN YOU REALIZE YOU HAVEN’T ROLLED BACK ON K8S BEFORE
![Page 46: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/46.jpg)
46
WHEN YOU’RE TRYING TO THINK THROUGH A CI BYPASS
![Page 47: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/47.jpg)
47
LAST MINUTE CHANGES
![Page 48: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/48.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
5. Engineers think velocity and stability are mutually exclusive
6. You have Devs who don’t Ops
![Page 49: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/49.jpg)
Confidential
‹#›
You know there’s a problem with continuous delivery when…
1. Your team issues code freezes
2. You see your team implementing delivery workarounds
3. Large system migrations are stressful
4. Exposing a feature to customers is harder than flipping a switch
5. Engineers think velocity and stability are mutually exclusive
6. You have Devs who don’t Ops
7. You’re satisfied with continuous delivery
![Page 50: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/50.jpg)
50
CLOSING THOUGHTS
“The most important characteristic of high performing teams is that they are never satisfied: they always strive to get better.” - Jez Humble
![Page 51: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/51.jpg)
THANK YOU
![Page 52: Misadventures in Continuous Delivery...You know there’s a problem with continuous delivery when… 1. Your team issues code freezes 2. You see your team implementing delivery workarounds](https://reader034.vdocuments.net/reader034/viewer/2022052001/6013d1a43fac4c684d374da3/html5/thumbnails/52.jpg)
Misadventures in Continuous Delivery Dan DeMeyere, Chief Product Officer @ thredUP Twitter: @dandemeyere Medium: medium.com/@dandemeyere Email: [email protected]