devops days tel aviv 2013: real world strategies in continuous delivery - aviran mordo
DESCRIPTION
Continuous delivery entails many concerns that you need to take care of; from changing the company culture to development methodologies. You have to follow Test Driven Development, perform A/B tests on every new feature you deploy to production; The product managers' is feature centric with feature toggles to turn features on and off; You have to perform gradual deployments, and more. In this talk Aviran will discuss some of the strategies we use at Wix to address these concerns, letting Wix to deploy hundreds of times every month. You will get a behind the scene look of how Wix manages A/B tests, how they changed their development methodology and the impact it had on their company culture. You will also learn what are feature toggles and how they use them to solve real life problems, and of course how they manage to do hundreds of deployments and evolve their system with no downtime. Speaker: Aviran Mordo, Wix Aviran Mordo leads Wix's back-end group. He has over 20 years of experience in the software industry and has filled many engineering roles and leading positions, from designing and building the US national Electronic Records Archives (http://www.archives.gov/era/) to building the infrastructure that powers indeed.com, the leading job search engine in the world. Aviran is a tech-savvy and a technology blogger since 2000, with vast knowledge of internet technologies, software development and continuous delivery. Twitter: @aviranm LinkedIn: http://www.linkedin.com/in/aviran Blog: http://www.aviransplace.com"TRANSCRIPT
![Page 1: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/1.jpg)
Strategies In Continuous Delivery
Learn the “magic” behind the scenes
DEVOPS Days Tel Aviv 2013
Aviran MordoHead Of Back-End Engineering @ Wix
@aviranmhttp://www.linkedin.com/in/aviran
00:00
![Page 2: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/2.jpg)
About Wix
00:00
![Page 3: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/3.jpg)
Wix in Numbers
• Over 39,000,000 users– Adding over 1,000,000 new users each month
• Static storage is over 150TB of data– Adding over 1TB of files every day
• 3 Data centers + 2 Clouds (Google AE, Amazon)– Around 300 servers
• Over 100,000,000 Server API calls per day• ~450 people work at wix
– ~ 150 people in R&D
00:00
![Page 4: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/4.jpg)
Do You Have The Guts To Deploy 10 Times A Day?
00:00
![Page 5: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/5.jpg)
00:00
From Jan’ 2013 – Jun’ 2013
• 1500• 470• 200
![Page 6: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/6.jpg)
00:00
From Jan’ 2013 – Jun’2013
• 1500 Deployments• 470 A/B Tests• 200 Feature Toggles
![Page 7: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/7.jpg)
Continuous Delivery – Key points
• Abandon the “VERSION” paradigm – move to a feature centric methodology
• Make small and frequent release as soon as possible • Automate everything – TDD/CI/CD• Measure Everything–A/B test every new feature–Monitor real KPIs (business, not CPU)
• Deploy without downtime
00:00
![Page 8: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/8.jpg)
Test Driven Development
• No new code is pushed to Git without being fully tested–We currently have around 10,000 automated tests
• Before fixing a bug first write a test to reproduce the bug
• Cover legacy (untested) systems with Integration tests
00:00
![Page 9: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/9.jpg)
Test Driven Development
• Actual impact on development–We develop products faster–Removes fear of change–Easier to enter some one else’s project–Do we really need QA? (Yes, they code tests)–Writing a feature is 10-30% slower, 45-90%
less bugs–50% faster to reach production.–Considerably less time to fix bugs00:00
![Page 10: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/10.jpg)
Feature Toggles
00:00
![Page 11: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/11.jpg)
Feature Toggles
• Everyone develops on the Trunk
• Every piece of code can get to production at anytime
00:00
![Page 12: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/12.jpg)
Code branch
00:00
New Code Old Code
FT Opene
d
Yes No
![Page 13: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/13.jpg)
Usage example
Simple “if” statement in your code
00:00
![Page 14: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/14.jpg)
DB Schema Changes Without Downtime
• Adding columns– Use another table link by primary key– Use blob field for schema flexibility
• Removing fields– Stop using. Do not make any DB
schema changes
00:00
![Page 15: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/15.jpg)
New DB schema with data migration
• Plan a lazy migration path controlled by feature toggle
1. Write to old / Read from old2. Write to both / Read from old 3. Write to both / Read from new, fallback to old4. Write to new / Read from new, fallback to
old5. Eagerly migrate data in the background6. Write to new / Read from new
00:00
![Page 16: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/16.jpg)
Feature Toggle Strategies (gradual expose users)
• Company employees• Specific users or group of users• Percentage of traffic• By GEO • By Language• By user-agent• User Profile based• By context (site id or some kind of hash on site id)
00:00
![Page 17: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/17.jpg)
Feature Toggle Override
• By specific server– Used to test system load– New database flows/migration– Refactoring that may affect performance and memory usage
• By Url parameter– Enable internal testing– Product acceptance– Faking GEO
• By FT cookie value– Testing– When working with API on a single page application
00:00
![Page 18: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/18.jpg)
Feature Toggles Management
00:00
![Page 19: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/19.jpg)
A/B Tests
00:00
![Page 20: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/20.jpg)
A/B Test
• Every new feature is A/B tested• We open the new feature to a % of
users– Define KPIs to check if the new feature is
better or worse – If it is better, we keep it– If worse, we check why and improve– If we find flaws, the impact is just for %
of our users (kind of Feature Toggle)
00:00
![Page 21: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/21.jpg)
An interesting site effect on product
• How many times did you have the conversion “what is better”?– Put the menu on top / on the side
• Well, how about building both and A/B Testing?
00:00
![Page 22: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/22.jpg)
Marking users with toss value in a cookie
• Anonymous user– Toss is randomly determined– Can not guarantee persistent experience if changing
browser
• Registered User– Toss is determined by the user ID– Guarantee toss persistency across browsers– Allows setting additional tossing criteria (for example
new users only)– Only use this for sections that a user has to be
authenticated
00:00
![Page 23: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/23.jpg)
• Do not mix anonymous and registered tests
• AB test parentage of users with optional filters–New Users Only (Registered users only)–By language –By GEO–By Browser –user-agent –OS–Any other criteria you have on your users
00:00
![Page 24: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/24.jpg)
A/B Test Features
• A/B Test Override– Allows to set a value of a test for validation– Helps support experience what users experiencing
• Override methods– Via URL parameter– Via cookie
• Start/Stop Test• Pause tests• Bots always get “A”00:00
![Page 25: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/25.jpg)
Manage your tests
00:00
![Page 26: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/26.jpg)
Self Test / Post Deployment Test
After each server deployment run a self test before deploying the next server.
• Checking server configuration and topology– Make sure database is accessible (DB connection string)– Is the schema the one I expect– Access required local resources (data files, other config files,
templates, etc’)– Access remote resources– RPC / REST endpoints reachable and operational
• Server will refuse requests unless it passes the self test• Allow a way to skip self test (and continue deployment)00:00
![Page 27: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/27.jpg)
Tools - App-info
00:00
![Page 28: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/28.jpg)
Where are we today?
• We have re-written our flash editor product as an HTML 5 editor– In just 4 months
• Introduced Wix 3rd party applications (developers API)– In just 6 weeks
• We are easily replacing significant parts of our infrastructure
• And we are doing ~10 releases a day!
00:00 1/1/2013 1/14/2013 1/28/2013 2/10/2013 2/24/2013 3/10/2013 3/24/2013 4/14/2013 4/29/2013 5/13/2013 5/28/2013 6/9/20130
5
10
15
20
25
30
35
Total
Total
![Page 29: DevOps Days Tel Aviv 2013: Real world strategies in continuous delivery - Aviran Mordo](https://reader033.vdocuments.net/reader033/viewer/2022061206/5481cf46b47959dd0c8b45fa/html5/thumbnails/29.jpg)
00:00
Read more: The Road To Continuous Delivery: http://goo.gl/K6zEK
http://www.slideshare.net/aviranwix/strategies-in-continuous-delivery
Aviran Mordo
@aviranmhttp://www.linkedin.com/in/aviranhttp://www.aviransplace.com