Changing Etsy's
Architectural Foundation
with
Continuous Deployment

Matt GrahamCore Engineer @ EtsyContinuous Deployer#surgeconSeptember 28, 2012

I'm here to talk about continuous deployment and how it helps with BIG architectural changes.I'm an engineer on Etsy's Core Team and I'm coming at this from the perspective of an engineer involved with making fundamental changes to Etsy's infrastructure.I've found continuous deployment to be a very good way to work and I'm here to spread the good word and hopefully trigger some ideas about how it could help you and how to get started with it.

Marketplace forHandmade Goods

Gross Sales 2011: $537 millionTotal Members: 19 millionItems For Sale: 15 millionUniques / month: 40 millionPage Views / month: 1.4 billion

As the business grows, there is change pressuring the software from all over.

Architecture is Relative

Start with an axiom: that good architecture is not static.The same company had 2 different architectures at 2 points in its history.While the 2005 architecture never would have scaled to 2012, the 2012 architecture would have been complete overkill in 2005. Etsy wouldn't have had time to build the features that got it started.

Organic Architecture

Here's a very not to scale graph of how the business and architecture grew together.

Premature Architecture

Here's what it would look like if they had shot straight to the 2012 architecture back in 2005.First, they probably wouldn't have gotten it right, but let's assume they did.We're done right?

Premature Architecture

What now?We overshot the 2005 architecture to be able to handle 2012, but now we're still not prepared for 2017. What do we do now?The point here is that you can't escape architectural change. All you can do is try to make it easier.Continuous deployment makes architectural change easier.

Passing Time => Change

Scale

Product

Technology

Engineering Team

As the business grows, there is change pressuring the software from all over.

Passing Time => Change

Scale

Product

Technology

Engineering Team

The Correct Architecture Changes

Ultimately, the correct architecture needs to change too.

Architectural Change Antipattern

We have to be able to make big architectural changes, and we need them to go better than this

The key to success here is breaking down big changes into many smaller changes

When we write code, we break it down into manageable modules. But when it comes time to deploy it, we mash it back together into an unmanageable chunk.This limits the scale of changes. With continuous deployment we remove that limit.

A Brief History of Deployment

Let's step back and look at how deployment got to where it is.And we'll start here, the 80s.In the olden days... software had to be copied onto floppy disks, put in a box, shipped to a store and then finally purchased from the storeAnd you wouldn't want to give people updates for free, so what happened? They'd all be batched up in an upgrade release or a new version altogether.Deploys were understandably rare

The Internet

And then this happened.Likewise, we even started distributing our thick client software over the internet: Windows Update is a good exampleWe took applications that used to be physically distributed thick clients and made them web applicationsFor the most part we were still using the old school deploy cycle

Agility

But some people wanted to go faster.We realized, Hey, it's a website, we can deploy it every month.

Continuous

But... why can't we just deploy and deploy and deploy?Well, we can.We can invert the unit of measurement from days per deploy, to be deploys per day

What it's all about

Reduce Failure Time

Start with Culture

Tools Help

Enable the Unfeasible

Makes it possible to do what might otherwise be too risky

A Tale of Six Bugs

Six Bugs with
Monthly Deploys

4 caught--->

2 missed