gluecon 2013 - dark architecture and how to forklift upgrade your system - dyn inc

Dark Architecture & How to Forklift Upgrade Your Infrastructure with Zero Downtime

Cory von WallensteinChief Technology Officer,

Dyn Inc.@cvonwallenstein

@cvonwallenstein from @DynInc at #gluecon

But First, Who Is Dyn?• Internet Infrastructure as a Service

– Managed DNS and Email Delivery

• 230 Global Employees (we bootstrapped to 170)• Headquarters in Manchester, NH (offices in SFO & UK too)• Raised first financing in Oct 2012: $38MM from NorthBridge


Problem We Are Trying To Solve

InputsBlack Magic

(Your Current System Architecture)Outputs

Different Black Magic (Your New System Architecture)

Inputs

Inputs

Inputs

Outputs

Outputs

Outputs

Scalex10, x102, etc.

Performance(t2 - t0) <= (t1 - t0)

t1

t2

t0

t0

CouplingTight -> Loose


Pragmatic Engineering over Unicorn Marketing


Why Things Get This Way• Time to market reigns supreme

– MVP was very… minimum… on… everything– Sooner is better than perfect

• Prototype to production to scale without architectural rigor– Skillset for system engineering in high demand

• Seen more often in small teams who find product market fit faster than expected– Inexperience, but we’ve all been there


Dark Architecture• A way of thinking about, and technical

approach to, solving the scale/performance/coupling problem while enabling the business to succeed and keeping (some) of your hair

• We stand on shoulders of giants– Fowler, Amazon, Netflix, etc.


High Level of Dark Architecture• Legacy approach: Flag Day Upgrade/Deploy

– Scope out 3 month upgrade to swap architecture A to B, turns into 6 months, don’t get to anything else, cross fingers on flag day, fight fires where broken, gain weight, lose hair, girlfriend breaks up with you, team quits, FML…

• Evolved approach: Fowler’s Blue/Green Deploy– Two copies of system, load balancing to rapidly

deploy new system version, rapidly fail back to legacy on failure (only one active at a time)


High Level of Dark Architecture• Dark Architecture Approach

– Two copies of system, both active, send inputs for a workflow to both, compare outputs and throw one away (the one you threw the output away from is the “dark architecture”), log and inspect output differences, gain confidence in new system when differences go away, swap which output you throw away (effectively bringing the “dark” architecture “light”), achieve equilibrium on what workflows get processed by what system so your business has flexibility, high five everyone, onward and upward.


Tangible Examples• Scaling Global DNS Stats beyond 17 POPs

– MySQL to Cassandra, Log file rsync to agg counts


Tangible Examples• Scaling Email Delivery beyond 1 billion/month

– Cron to daemon (2011), Perl to Node.js (now)

Dark Architecture Manifesto1. Clear definition of success over ambiguity

– Likely scale/performance measured, may get blank stares on coupling

2. Continuously deliver value over months of no visible progress

3. Confidence in functional equivalence over scope creep

4. ^5’s over finger pointing5. Plan for failure over cross fingers


Dark Architecture Manifesto6. Customer impact over elegant system

diagrams7. System flows over system components8. Operational confidence and familiarity over

trial by fire9. Having a ten item list over a nine item list10. Architecture evolution over architecture

revolution


Scope and Priority

• Prioritize a backlog of input/output workflows by amount of pain– Don’t think on a system component level

• “swap MySQL for Cassandra”

– Think on a system workflow level• “retrieve query logs and render *.example.com graphs”

– This exercise will force you to hone scope to exactly where the pain is so you can focus on delivering the solution to this pain first and save others for later.


Legacy Approach


Legacy Approach: Week 0

Input

Legacy System

100% of functionality enabled

100% of functionality consumed

Output



Input

Legacy System



Output

New System





Input

Legacy System



Output

New System



Most people start with easy pieces under a misguided “crawl walk run” philosophy. Quick wins on easy stuff while saving hard problems

for later rarely ends well.



Input

Legacy System



Output

New System



Progress slows as harder problems encountered



Input

Legacy System



Output

New System



80% of projects spend 80% of their calendar time at 80% perceived completion. I’m 80% sure.



Input

Legacy System



Output

New System



Other fires came up, things took longer than expected, you know… business. Morale never

been lower


Legacy Approach: Flag Day!

Input

Legacy System



Output

New System




Dark Architecture Approach


Dark Architecture Approach: Week 0

Input

Legacy System



Output



Input

Legacy System



Output

New System





Input

Legacy System



Output

New System



InputOutput

No functionality yet, just dark architecture

framework for two inputs and two

outputs (throwing one output away)


Input

Legacy System



Output

New System


2% of functionality consumed (dark)

InputOutput

Throw one away, but log and inspect differences!


Input

Legacy System



Output

New System



InputOutput

Gain confidence operating with two

equal outputs, switch which one is thrown

away for that workflow. Goes horribly wrong?

Switch back.


Input

Legacy System



Output

New System



InputOutput

Where do we stand at expected 3 months? Most painful 20% of problems resolved…

now we have flexibility for what to

do next.

Customer impact over elegant system diagrams

• Your customers are not paying you to have pretty whiteboards of elegant system architectures

• Your customers are paying you to make their pain go away. This gets priority.

• It’s OK to have different workflows handled by different systems to give your team agility– Other priorities came up? System is stable.– Have technical debt time? Continue arch migration


Parting Takeaways• Manifesto is a preference, not a rule• Think in flows not components• Deliver most painful pieces first so when

priorities change, you’re not left half complete.• Process success >>> process name• Be realistic. DA provides flexibility and frequent

victories for morale and some value delivered sooner, but it won’t necessarily make a full architecture migration faster in calendar days.


Cory von Wallenstein@cvonwallenstein

Questions?


gluecon 2013 - dark architecture and how to forklift upgrade your system - dyn inc

Education

legacy system100

copies of system

isthe dark architecture

dark architecture manifesto1

dark architecture manifesto6

system engineering

system flows

system components8