@PavelChunyayev

#nostagingGet rid of the false feeling of safety that staging environment gives

by Pavel Chunyayev, 2-9-2016Amsterdam, Pakhuis West

Software Circus

@PavelChunyayev

Inspiration for the talk• Dave Nolan - @davenolan• Edith Harbaugh - @edith_h • Myself :)

• https://speakerdeck.com/davenolan/number-nostaging• https://vimeo.com/channels/pipelineconf/162633462• http://blog.launchdarkly.com/staging-servers-are-dead-long-live-a-staging-server/• http://readwrite.com/2016/01/22/staging-servers/• http://

slideshare.net/PavelChunyayev/you-dont-need-dtap-backbase-implementation-amsterdam-17122015

@PavelChunyayev

The need for Staging environment

@PavelChunyayev

It all started with pushing code directly to productionBut it was painful, unstable, broke everyone’s work, created instability

@PavelChunyayev

Environment before production to ensure quality

@PavelChunyayev

Environment before production to ensure quality - Staging

@PavelChunyayev

More environments mapped into functions

@PavelChunyayev

DTAP street

@PavelChunyayev

Increase quality at the expense of timeFix must go through all the environmentsOr be pushed to production directly Data specific bugs are hard to reproduce before production

@PavelChunyayev

WaterfallTest things before productionGate changed to productionAggregate changes to productionApprove delivery to production

@PavelChunyayev

Waterfall deploymentsDeploy to production at 3 in the morning on SaturdayPrepare for weeks, execute for hoursIf something goes wrong, rollback

@PavelChunyayev

Get some sleep in another life

@PavelChunyayev

Main problem: long living environmentsConfiguration driftDirty environmentsTesting is complicated

@PavelChunyayev

Immutable infrastructure to the rescue!1. Create environment just in time for the purpose2. Use it3. Dispose it when no longer needed

@PavelChunyayev

Don’t give names to servers

@PavelChunyayev

Done?

@PavelChunyayev

Immutable DTAP is not enough

@PavelChunyayev

You can’t get clone of production3rd party integrations – should they be real or fake?Emails and notifications to usersInstance types and number

@PavelChunyayev

You can’t get clone of production

No real usersFake data

@PavelChunyayev

Testing on staging provides fake confidence

@PavelChunyayev

Testing in production

@PavelChunyayev

Testing in production???Isn’t it where everyone’s heading?What is needed to achieve this?

@PavelChunyayev

New style deploymentsShort-lived immutable infrastructureZero downtime deploymentsGood monitoringPossibility to quickly rollbackContinuous Delivery

@PavelChunyayev

Why would you need staging?Short-lived immutable infrastructureZero downtime deploymentsGood monitoringPossibility to quickly rollbackContinuous Delivery

@PavelChunyayev

Approaches• Feature flags• API Versioning• Circuit breakers• Dark launches• Incremental changes for schema migrations• Branch by abstraction• Preference for backwards-compatible changes

@PavelChunyayev

Why would you test on staging?You are testing synthetic scenariosThird of all the features are not required by usersThe rest are used differently than product manager expects

@PavelChunyayev

Why would you test on staging?We need to move as fast as possible from developer’s laptop to production to evaluate if feature is really needed

@PavelChunyayev

Perfomance testing on Staging?Not needed for every featureA lot of new features receive no load because users don’t need them

@PavelChunyayev

Perfomance testingProvision ephemeral server to run some load testsBetter do a rolling release and test with a real load, and add synthetic load on top

@PavelChunyayev

Staging microservices is painfulYou don’t have the systemYou have constantly changing system

@PavelChunyayev

When to stage microservices?At night?Every hour?At every change?

@PavelChunyayev

Why bother staging microservices?If deployments are independent Failures are identified quicklyAnd rollback are simple and automated

@PavelChunyayev

#nostaging in two easy steps:

@PavelChunyayev

Remove static DTAPAutomate environments provisioning and deploymentSo you can create any number of environments at any timeEstablish continuous delivery Heavily invest in automated testing Build quality in

@PavelChunyayev

Destroy Staging environmentInvest in monitoring, failure detection and recoveryPush small incremental pieces to production

@PavelChunyayev

Staging gives you a false sense of security that you don’t really have