Continuous Delivery for DatabasesSQL Relay – Cardiff - 2014

Presented by: James SmithCo-Founder, DevOpsGuys

ABOUT ME

Co-founder @ DevOpsGuys

Two decades building & delivering enterprise web systems

Helped many, many companies implement Continuous Delivery practices

Found High Quality Belgian Beer in late nineties!

Who is this guy?

ABOUT DEVOPSGUYS

We provide

web application management

for customers, giving them access to the expertise necessary for

building, launching, maintaining & optimising applications

allowing them to

accelerate time-to-market

and to

focus on adding value to their business

Who are these guys?

WHY?Why is this important?

WHY?Why is this important?

BUSINESS HAS CHANGED

SPEED MATTERS

8

9

DEPLOYMENT RATES

300 Deployments / Year

50-60 Deployments / Day

10+ Deployments / Day

Every 11.6 seconds

10

HIGH PERFORMERS

HIGH PERFORMING IT ORGANISATIONS DEPLOY 30 TIMES

MORE FREQUENTLY, WITH 50% FEWER FAILURES WITH 8000X

FASTER LEAD TIMES THAN THEIR PEERS.

11

HIGH PERFORMERS

THEY ARE ALSO TWO TIMES MORE LIKELY TO EXCEED PROFITABILITY, MARKET SHARE &

PRODUCTIVITY GOALS

THEY EXPERIENCE 50% HIGHER MARKET CAPITALIZATION GROWTH OVER 3 YEARS

HOW?How have things changed?

12

2010

2012

2001 (1998)

2001 (and earlier)

DEV HAS CHANGED

WE KNOW AGILE WORKS

15

Plan

Code

Build

Test

Release

Deploy

Operate

CONTINUOUS X?

Continuous Integration

Continuous Delivery

Continuous Deployment

CONTINUOUS DELIVERYWhat is it?

REGULAR, RAPID, RELIABLE AND CONTROLLED DELIVERY OF WORKING SOFTWARE SYSTEMS INTO PRODUCTION

Matthew Skelton

REGULAR, RAPID, RELIABLE AND CONTROLLED DELIVERY OF WORKING SOFTWARE SYSTEMS INTO PRODUCTION

Matthew Skelton

Nhan Ngo, a QA engineer at Spotify

20

CONTINUOUS DELIVERY

1. Build Management & CI

2. Environments & Deployment

3. Release Management & Compliance

4. Testing

5. Data Management

22

BUT DATABASE CHANGE IS SCARY

Photo Credit: http://cache.lego.com/r/www/r/movie/-/media/franchises/the%20lego%20movie/explore/downloads/wallpapers/lego_wps_1600_emmet.jpg?l.r=-1410608028

NO, THIS IS SCARY

Photo Credit: ?

SO WE MINIMISE RISK

Photo Credit: https://ideas.lego.com/projects/62456

WE DEFEND THE THREAT

Photo Credit: "Rumrunner" via Compfight cc

26

WE STOP CHANGE

Photo Credit: s.kosoris via Compfight cc

27

CUSTOMERS GET SAD

Photo Credit: Kalexanderson via Compfight cc

28

THE BUSINESS GETS ANGRY

Photo Credit: powerpig builds via Compfight cc

29

AND IT GET THE BLAME!

Photo Credit: bobsfever via Compfight cc

30

YES BUT….

Dev are shipping application changes – frequently

Ops are creating servers - automatically

WHY IS THERE SO MUCH LEGO IN YOUR SLIDE DECK?

Perfectly normal question

LEGO = DEVOPS

Rapid proto-typing & experimentation

Building blocks – no right or wrong way

Promotes collaboration

Strong cultural appeal

Small batch sizes

Visibly measureable

We’ve even automated it! Manufacturing Zenon Mindstorms

Danish phrase leg godt, which means "play well".

WHY?Why is database automation so hard?

!= CTRL C, CTRL V

Database deployment is not copying and replacing.

It is the transformation from a previous version to the next version while preserving data integrity.

Deploying database change is hard

Deploying database change frequently is even harder

WHAT?What are the common challenges?

#1: VERSION CONTROL

Photo Credit: ntr23 via Compfight cc

VERSION CONTROL

Database changes not under VCS

Worse – Changes not “always” committed to VCS

Communication of change

Living in “the sea of ‘branch/merge’ filth”

38

#2: AGAINST THE FLOW

Photo Credit: slang589 via Compfight cc

Photo Credit: slang589 via Compfight cc

39

AGAINST THE FLOW

DEV PRODUCTION

DEV PRODUCTION

CODE

DATA

40

#3: COMPLEXITY

Photo Credit: L@go via Compfight cc

COMPLEXITY

Dependence

Centralisation

#4: DATA VOLUME

Photo Credit: http://www.calgaryherald.com/news/calgary/Gallery+LEGO+KidsFest/9848406/story.html

DATA VOLUME

Transit time

Historical data – no archiving

One of the biggest impacts on Cycle time

#5: CONWAY'S LAW

Photo Credit: kirk_arts via Compfight cc

45

CONWAY'S LAW

"Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design

whose structure is a copy of the organization's communication structure.“

- Conway, 1968

CONWAY'S LAW

Is (was) a database really needed? 3 Tiers anyone?

Centralised vs Decentralised?

Formalised [change] control

HOW?So what are some of the solutions?

#1: GET UNDER CONTROL

Photo Credit: RHiNO NEAL via Compfight cc

VERSION CONTROL

It’s code – it should be in VCS!

It’s code – it should be in VCS!

It’s code – it should be in VCS!

Schema & Static/Reference Data

Reverse engineer existing schema & reference data

50

#2: INTEGRATE CONTINUOUSLY

Photo Credit: ....Tim via Compfight cc

51

CONTINUOUS INTEGRATION

Automate your build steps (Full vs Incremental)

Build a library of manual tests

Automate your tests

52

#3: AUTOMATE TESTING

Photo Credit: s3aphotography via Compfight cc

53

TEST, TEST, TEST

Select the right tests for each stage;Unit testing Integration TestingDeployment ValidationBehaviour Validation

Determine the right data for testingDo you need it all?

54

#4: AUTOMATION

Photo Credit: pasukaru76 via Compfight cc

55

AUTOMATE ALL THE THINGS

Testing

Deployment

Back-up and more importantly restore

Archiving

Rollback

56

#5: MICROSERVICES

57

AS A SERVICE

Treat your database as a service

Contracts – that you do not break (lightly)

58

RE-ARCHITECT

Split data along sensible partitions

Polyglot persistence

59

POLYGLOT PERSISTENCE

60

#6: CHANGE YOUR WAYS

Photo Credit: Stéfan via Compfight cc

61

IT’S CULTURE

Technology is only half of the story

Dev’s must work with DBA’s (no silo’s)

Management must think of operations as part of development

Deployment is part of development

Data retention is part of development

Fail faster, but fail safely

TOOLINGCan tools help?

62

THE PIPELINESource

Control

Integration

TestDeploy

64

IN REALITY

Development

Plan Code Build Test Release Deploy Operate

Operations

SOURCE CONTROL

Source control is the foundation of Continuous Integration

Plus many, many more

66

CONTINUOUS INTEGRATION

Continuous Integration is the common orchestration point

67

DEPLOYMENT

Repeatable and consistent deployments every time.

MONITORING

Continuous delivery elevates the need for monitoring in production

THE TRUTH

Continuous Delivery relies on having 2 basic things;

1. Version Control

2. Automation

70

AND FINALLY…

Customers see results and new features more quickly.

Shorter feedback cycles increases our ability to learn.

Improve the whole system.

Reduce firefighting.

Everyone wins!

71

EVERYBODY WINS!

Photo Credit: Kalexanderson via Compfight cc

72

THAT’S ALL FOLKS

Photo Credit: Walter Benson via Compfight cc