Changing Etsy's
Architectural Foundation
with
Continuous Deployment
Matt GrahamCore Engineer @ EtsyContinuous Deployer#surgeconSeptember 28, 2012
I'm here to talk about continuous deployment and how it helps with BIG architectural changes.I'm an engineer on Etsy's Core Team and I'm coming at this from the perspective of an engineer involved with making fundamental changes to Etsy's infrastructure.I've found continuous deployment to be a very good way to work and I'm here to spread the good word and hopefully trigger some ideas about how it could help you and how to get started with it.
Marketplace forHandmade Goods
Gross Sales 2011: $537 millionTotal Members: 19 millionItems For Sale: 15 millionUniques / month: 40 millionPage Views / month: 1.4 billion
As the business grows, there is change pressuring the software from all over.
Architecture is Relative
Start with an axiom: that good architecture is not static.The same company had 2 different architectures at 2 points in its history.While the 2005 architecture never would have scaled to 2012, the 2012 architecture would have been complete overkill in 2005. Etsy wouldn't have had time to build the features that got it started.
Organic Architecture
Here's a very not to scale graph of how the business and architecture grew together.
Premature Architecture
Here's what it would look like if they had shot straight to the 2012 architecture back in 2005.First, they probably wouldn't have gotten it right, but let's assume they did.We're done right?
Premature Architecture
What now?We overshot the 2005 architecture to be able to handle 2012, but now we're still not prepared for 2017. What do we do now?The point here is that you can't escape architectural change. All you can do is try to make it easier.Continuous deployment makes architectural change easier.
Passing Time => Change
Scale
Product
Technology
Engineering Team
As the business grows, there is change pressuring the software from all over.
Passing Time => Change
Scale
Product
Technology
Engineering Team
The Correct Architecture Changes
Ultimately, the correct architecture needs to change too.
Architectural Change Antipattern
We have to be able to make big architectural changes, and we need them to go better than this
The key to success here is breaking down big changes into many smaller changes
When we write code, we break it down into manageable modules. But when it comes time to deploy it, we mash it back together into an unmanageable chunk.This limits the scale of changes. With continuous deployment we remove that limit.
A Brief History of Deployment
Let's step back and look at how deployment got to where it is.And we'll start here, the 80s.In the olden days... software had to be copied onto floppy disks, put in a box, shipped to a store and then finally purchased from the storeAnd you wouldn't want to give people updates for free, so what happened? They'd all be batched up in an upgrade release or a new version altogether.Deploys were understandably rare
The Internet
And then this happened.Likewise, we even started distributing our thick client software over the internet: Windows Update is a good exampleWe took applications that used to be physically distributed thick clients and made them web applicationsFor the most part we were still using the old school deploy cycle
Agility
But some people wanted to go faster.We realized, Hey, it's a website, we can deploy it every month.
Continuous
But... why can't we just deploy and deploy and deploy?Well, we can.We can invert the unit of measurement from days per deploy, to be deploys per day
What it's all about
Reduce Failure Time
Start with Culture
Tools Help
Enable the Unfeasible
Makes it possible to do what might otherwise be too risky
A Tale of Six Bugs
Six Bugs with
Monthly Deploys
4 caught--->
2 missed