continuous deployment: beyond continuous delivery

CONTINUOUS DEPLOYMENTBeyond Continuous Delivery

by Timothy Fitz

Timothy Fitz

Coined Continuous DeploymentTechnical Lead at IMVU

CTO at CanvasSoftware Consultant

TimothyFitz.comSystemsLive.com

@TimothyFitz

THIS TALK

• What is Continuous Deployment?

• Pitfalls of getting to CD

• The future of CD

CONTINUOUS DELIVERY

produce software in short cyclesensuring that the software

can be reliably released at any time

CONTINUOUS DELIVERY

deployableat the push of a button

CONTINUOUS DEPLOYMENT

Safe automatic deploymentof frequent small commits

CONTINUOUS DEPLOYMENT

SAFEThe automated process is

responsible for failures.Not just great test coverage.

Blameless post-mortems.

CONTINUOUS DEPLOYMENT

AUTOMATIC DEPLOYMENT

Code gets deployed without human intervention

CONTINUOUS DEPLOYMENT

FREQUENT SMALL

COMMITS Smaller commits are safer, faster, cheaper and simpler.

CONTINUOUS DELIVERY

push button

CONTINUOUS DEPLOYMENT

automatic

CONTINUOUS DELIVERY

CONTINUOUS DEPLOYMENT

Commit Build Test Deploy

Commit Build Test Deploy

IMVU was actuallyContinuous Delivery

IMVU was actuallyContinuous Delivery

Shhhhh

Don’t tell anyone!

BOTH

fully automated deploy

BOTH

significant automated test coverage

BOTH

deploy pipeline infrastructure

PUSH BUTTON

Human QAbefore deploy

Bug!

Bug!

Post-mortem

Bug!

Post-mortemMore Human QA

Bug!

Post-mortemMore Human QA

MOREBugs!

MOREBugs!

Reduced Confidence

MOREBugs!

Reduced ConfidenceLess frequent deploys

MOREBugs!

Reduced ConfidenceLess frequent deploys

AUTOMATIC

Bug!

AutomationIncreased Throughput

PUSH BUTTON

Encourages deploy = feature release

AUTOMATIC

Requires feature flippers

As simple as an if statement.As complex as your release process needs to be.

BUT…Can’t I just use Feature Flippers with push button deploy?

TOP DOWN

Please use feature flippers!

When? Where? How often?

Boss

BOTTUM UPA developer goes to commit some code

This feature isn’t ready, what can I do?

BossCheck out feature flippers!

PUSH VS PULL

• A methodology is “pushed” if adoption isoptional and driven by evangelism.

• A methodology is “pulled” if adoption ismandatory and driven by necessity.

PUSH BUTTONService Oriented Architecture

Commit Build Test Backend

Commit Build Test Frontend

PUSH BUTTONService Oriented Architecture

Commit Build Test Backend

Commit Build Test Frontend

PUSH BUTTONService Oriented Architecture

Commit Build Test Backend

Commit Build Test Frontend

PUSH BUTTONService Oriented Architecture

Commit Build Test Backend

Commit Build Test Frontend

PUSH BUTTONService Oriented Architecture

Commit Build Test Backend

Commit Build Test Frontend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONInter-service dependencies

Commit Build Test Frontend

Commit Build Test Backend

BROKEN!

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

PUSH BUTTONCorrect but SLOW

Commit Build Test Frontend

Commit Build Test Backend

REAL WORLD SOA :(

Frontend

Middleware

Backend #1 Backend #2 Backend #3

REAL WORLD SOA :(

Commit Build Test Frontend

Commit Build Test Frontend

Commit Build Test Backend #1

Commit Build Test Backend #2

Commit Build Test Backend #3

AUTOMATIC

Commit Build Test Frontend

Commit Build Test Frontend

Commit Build Test Backend #1

Commit Build Test Backend #2

Commit Build Test Backend #3

TEMPLATED

Commit Build Test Frontend

Commit Build Test Frontend

Commit Build Test Backend #1

Commit Build Test Backend #2

Commit Build Test Backend #3

UNIFIED

Commit Build Test

Frontend

Frontend

Backend #1

Backend #2

Backend #3

STAGING ENVIRONMENTS

STAGING ENVIRONMENTS

ARE BAD

STAGING ENVIRONMENTS• Shared mutable state

• Across multiple teams• Across multiple deploy pipelines

• No real users • No real risk• Deploying offers no real risk reduction

• Broken Regularly • 2nd-class compared to production

HOW TO NEVER CD TO PRODUCTION

Deploy to staging

Staging is broken!Less confidence in deploys

• Only tests should have access to environment• Must be able to recreate automatically• Ideally recreate on every test run• Every deploy pipeline gets its own environment

But where do I run integration tests?

Commit Build Test Production

GIT

GIT

IS BAD??

Branch-based workflows aren’t even Continuous Integration

Merge Build TestCode Review

Pull RequestCommit

Branch-based workflows aren’t even Continuous Integration

Merge Build TestCode Review

Pull RequestCommit

SOLUTION?

NEVER BRANCH

MISUSE OF CODE REVIEW

• Catch bugs(automate! test coverage)

• Teach junior engineers (pair programming, mentorship)

CODE REVIEW AFTER DEPLOYMENT

• Ensure readability

• Promote code reuse

• Promote knowledge sharing

• Minimize bus number

DEPLOYMENT IS STRESS FREE#1 most surprising result of adopting continuous deployment?

CONTINUOUS ANYTHING IS GREAT

HAVE CONTINUOUS DEPLYOMENT AS YOUR GOAL

THE FUTURECONTINUOUS DEPLOYMENT

THE FUTURESOFTWARE

THE PAST

2006 (BUILDBOT)

▸ twisted

▸ jinja

▸ simplejson

▸ pysqlite

THE PRESENT

2016 (5K LINE CURRENT PROJECT)

geoip2 - requests - maxminddb - ipaddr MySQL-python boto boto3 - botocore - jmespath - python-dateutil - six - docutils - futures celery - pytz - billiard - kombu - anyjson - amqp psycopg2 pyfakefs - mox3 - pbr - fixtures - testtools - extras - pyrsistent - python-mimeparse

- unittest2 - argparse - traceback2 - linecache2 airflow - alembic - SQLAlchemy - Mako - MarkupSafe - python-editor - chartkick - croniter - setuptools - dill - flask - Werkzeug - Jinja2 - itsdangerous - flask-admin - wtforms - flask-cache - flask-login - future - gunicorn - markdown - pandas - numpy - pygments - setproctitle

- thrift - Flask-WTF redis wsgiref repoze.lru datawarehouse user-agents - ua-parser nose

66!

THE FUTURE

PROJECTS HAVE MORE EVERYTHING

▸ More libraries

▸ More frameworks

▸ More API integrations

▸ More frequent deployment

THE FUTURE

CONTINUOUS DEPLOYMENT OF

▸ Libraries

▸ Frameworks

▸ Operating systems

THE FUTURE

CONTINUOUS DEPLOYMENT OF

▸ Libraries (Google Analytics)

▸ Operating systems (Linux via Ksplice Uptrack)

STOP WASTING TIME TRYING TO KEEP UP

THE FUTURE

CURRENT LANGUAGES AND FRAMEWORKS

Inherently tightly coupled Bottlenecked on client’s adoption of change

Perl 6?

Python 3?

AngularJS 2?

Ruby on Rails 2.x 3.x 4.x

THE FUTURE

NEXT BIG LANGUAGE/FRAMEWORK

Winner will bewhoever cansustain evolution!

THE FUTURE

NEXT BIG LANGUAGE/FRAMEWORK

▸ Built-in support for code migration

▸ Allow libraries to migrate your code when the library changes

THE FUTURE

REFACTORING DSL

▸ rename(OldClass, NewClass)

▸ change_signature(foo(x), foo(x,y=null))

▸ six.py

▸ go fmt

SOME HINTS OF THE FUTURE

THE FUTURE

NEXT BIG LANGUAGE

▸ Prioritize code changes over static code

THE FUTURE

NEXT BIG LANGUAGE

▸ Prioritize code changes over static code

THANKS!

ANY QUESTIONS?

@TimothyFitz