releasing to production every week india

Releasing to Production Every Week

Owen RogersTwitter: @exortech

http://exortech.com/blog

1 year

49 releases

49 releases(excluding patch releases)

~ 1 release / week

summary

lessons learned

process

practices

tools

Poll:

how long is your release cycle?

Me

Vancouver

> 15 teams

Agile methods

Company

start up

real-time energy monitoring

building energy efficiency

save $

save kW

save

aggregate data

hosted solution

SaaS

Java

Team

• 9 developers

• 9 developers• 1 product manager

• 9 developers• 1 product manager• 1 graphic designer

• 9 developers• 1 product manager• 1 graphic designer• 1 tester

what’s missing?

operations?

support?

QA?

project manager?

DBA?

generalizing specialists

fungibility

rotating responsibility

daily support duty

few hand-offs

empowered

Process

Goal:• to surface risk as early

as possible to keep problems out of production

maximize feedback

shrink time

Continuous Integration

Continuous Integration

• 3-4 Commits/dev/day

Automated Build

Automated Build

• 3 minute build

Automated Build

• 3 minute build• 30-40 builds/day

Automated Build

• 3 minute build• 30-40 builds/day

Automated Test Deploy

Automated Test Deploy

• 4-5 times per day

Automated Test Deploy

• 4-5 times per day• Scheduled nightly at

11PM

Daily Standup

Daily Standup

• Weekly company standup

Daily Review

Daily Review

• Quick peer feature review

Weekly Demo

Weekly Demo

• 15 minute company-wide user-driven demo

Mon Tues Wed Thurs Fri

9:30AMPlanningMeeting

10AMUser-drivenDemo Release

Testing

5pmRelease

weekly cycle

11pmCut-off

Sustainable Pace

Sustainable Pace

Sustainable Pace

• regular release window

Sustainable Pace

• regular release window• quality of life and code

Sustainable Pace

• regular release window• quality of life and code• part of culture

Sustainable Pace

• regular release window• quality of life and code• part of culture• company heartbeat

“agile”

Scrum?

XP?

what’s missing?

user stories?

user stories?

user stories?

story boards?

story boards?

story boards?

estimation?

estimation?

estimation?fits or it doesn’t

“muda”

pairing?

pairing?

pairing?as needed

pairing?as needed

code reviews

pairing?as needed

code reviewsopen workspace

TDD?

TDD?

TDD?52% coverage

TDD?52% coverage+ Selenium

TDD?52% coverage+ Selenium

retrospectives?

retrospectives?

retrospectives?Release Review

retrospectives?Release Review

5 Whys

lightweight RCA

5 Whys

just-in-time

within 24 hours

1 Jira per question

Practices

Daily support rotation

5 Whys

Continuous Monitoring

Continuous Monitoring

App Server Socket Logger LogMaster

WARN | ERROR

(async)

Continuous Monitoring

Continuous Monitoring

• Proactive response to issues

Continuous Monitoring

• Proactive response to issues

• Clean logs

Continuous Monitoring

• Proactive response to issues

• Clean logs

• Message throttling

Continuous Monitoring

• Proactive response to issues

• Clean logs

• Message throttling

• Gmail

Continuous Monitoring

• Proactive response to issues

• Clean logs

• Message throttling

• Gmail

• Managing the signal-to-noise ratio

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

App Server

gmetad

web app

RRD

gmondUDP multicast

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

Continuous Monitoring

Zero-downtime Deployment

Zero-downtime Deploys

Load Balancer

App Server App Server App Server

Deploy

Zero-downtime Deploys

Zero-downtime Deploys

• Fully automated deployment

Zero-downtime Deploys

• Fully automated deployment

• One button deploy from TeamCity

Zero-downtime Deploys

• Fully automated deployment

• One button deploy from TeamCity

• Gracefully bring down servers

Zero-downtime Deploys

What about the DB?

Zero-downtime Deploys

Bering

Zero-downtime Deploys

012_rename_login_id_to_user_name

ALTER TABLE USER RENAME COLUMN login_id user_name

Zero-downtime Deploys

012_rename_login_id_to_user_name

012_add_user_name_column 02_remove_login_id_column

expansion contraction

Zero-downtime Deploys

Zero-downtime Deploys

• expansion

Zero-downtime Deploys

• expansion

• preserves backwards compatibility

Zero-downtime Deploys

• expansion

• preserves backwards compatibility

• contract

Zero-downtime Deploys

• expansion

• preserves backwards compatibility

• contract

• clean up

Zero-downtime Deploys

Zero-downtime Deploys

• Typically safe expansion operations:

Zero-downtime Deploys

• Typically safe expansion operations:

• add table, add column, insert data

Zero-downtime Deploys

• Typically safe expansion operations:

• add table, add column, insert data

• Typically unsafe expansion operations:

Zero-downtime Deploys

• Typically safe expansion operations:

• add table, add column, insert data

• Typically unsafe expansion operations:

• rename or remove columns or tables

Zero-downtime Deploys

• Typically safe expansion operations:

• add table, add column, insert data

• Typically unsafe expansion operations:

• rename or remove columns or tables

• add constraint

Zero-downtime Deploys

/expand/.../143001_create_login_table002_add_login_id_index003_create_user_table004_create_group_table

/.../contract/.../143001_add_inv_constraint002_drop_alias_column

/...

Zero-downtime Deploys

Zero-downtime Deploys

• Database migration decoupled from the release

Zero-downtime Deploys

• Database migration decoupled from the release

• Simplified rollback

Zero-downtime Deploys

• Database migration decoupled from the release

• Simplified rollback

• Some additional complexity in writing migrations

Release Options

Traditional Release

Traditional Release

Traditional Release

• high risk

Traditional Release

• high risk

• not scalable

Traditional Release

• high risk

• not scalable

• prolonged release: “Don’t release until it’s all ready”

Release Options

Release Options

Release Options

• release to user subset (private beta)

Release Options

• release to user subset (private beta)

• split test (experimentation)

Release Options

• release to user subset (private beta)

• split test (experimentation)

• incremental rollout

Release Options

• release to user subset (private beta)

• split test (experimentation)

• incremental rollout

• selective enabling/downgrading

Release Options

• release to user subset (private beta)

• split test (experimentation)

• incremental rollout

• selective enabling/downgrading

• disable feature (feature darkmode)

Release Options

• release to user subset (private beta)

• split test (experimentation)

• incremental rollout

• selective enabling/downgrading

• disable feature (feature darkmode)

• defer commit

Release Options

• release to user subset (private beta)

• split test (experimentation)

• incremental rollout

• selective enabling/downgrading

• disable feature (feature darkmode)

• defer commit

• defer release

Release Options

Release Options

• Features running in production for weeks before fully available

Release Options

• Features running in production for weeks before fully available

• lots of data

Release Options

• Features running in production for weeks before fully available

• lots of data

• “Release is a marketing term”

Release Options

• Features running in production for weeks before fully available

• lots of data

• “Release is a marketing term”

• Production levers

Release Options

• Features running in production for weeks before fully available

• lots of data

• “Release is a marketing term”

• Production levers

• WANGMI: the discipline to defer

Use Production Data

Use Production Data

Use Production Data

• Upload Nightly backups

Use Production Data

• Upload Nightly backups

• Automated Test DB refresh after every release

Use Production Data

• Upload Nightly backups

• Automated Test DB refresh after every release

• Local DB refresh on demand

Use Production Data

• Upload Nightly backups

• Automated Test DB refresh after every release

• Local DB refresh on demand

• Cleansed

Use Production Data

• Upload Nightly backups

• Automated Test DB refresh after every release

• Local DB refresh on demand

• Cleansed

• Test data is a liability

Single Feature Release

• Daily Support Rotation

• 5 Whys

• Continuous Monitoring

• Zero-downtime deployment

• Release Options

• Use Production Data

• Single Feature Release

Tools

Questions?

owen@pulseenergy.com