virtual data : eliminating the data constraint in application development

Virtual Data Eliminating the data constraint in Application

Development

Kyle Hailey, Technical Evangelist at Delphix

Technology Disruption

“Software is eating the world.”- Marc Andreessen

Increasing Commoditization

Competitive Pressures

Consumerization of Software

New software required for success

PCs

2010

Mob

ile

• Problem : Data Constraint• Solution : Virtual Data• Use Cases : Development, Security, Cloud

In this presentation :

DevOps :

DevOps : Process• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast

DevOps : Process• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast

Tools:• Continuous Delivery• Cloud • Agile • Kanban• Kata

DevOps : Process• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast

Tools:• Continuous Delivery• Cloud • Agile • Kanban• Kata

The Phoenix Project

What is the constraint

in IT ?

Put your energy into the constraint Top 5 constraints in IT

1. Dev environments setup2. QA setup3. Code Architecture4. Development5. Product management

- Gene KimSurveyed • 14000 companies• 100s of CIOs

Flow of Features

Product Management

Development

QAIntegration

testing

Deployment

Testing

Customer

DevOps is a Goal

Fast flow of features from development to IT operations to the customers

- Gene Kim

Flow of Features

14

Product Management

Development

QAIntegration

testing

Deployment

Testing

Customer

1

DevelopmentEnvironments

2

QA & Testing Environments

Product ManagementFeatures

2 2

Code Architecture 3Code Speed

4 5

Data

Development Pipeline for QA

SQL

Build Deploy

Environment

Database

16

PRODDEV Test UAT

DBA

Sys Admin

Storage Admin

Legacy Data Movement: Slow & expensive

?

Slow environment builds: delays

17

Development Pipeline for QA

0 2 4 6 8 10 12 14 16 18 20 22 24

ResetTest ResetTest ResetTest

Physical Data

Wait Time

Hours

Refresh( > 80%)

Testing (< 20%)

18

Data Management not Agile

• 20% SDLC time lost waiting for data

• 60% dev/QA time consumed by data tasks

Conclusion:

Data management does not scale to Agile

- Infosys

Data is the Constraint

19

Application Development Constraints

1. Not enough resources2. Bad test data leading to bugs3. Slow environment builds

1. Not Enough Resources: shared bottlenecks

Frustration Waiting

1. Not Enough Resources : bugs because of old data

Old Unrepresentative Data

1. Not enough resources: limited environments

2. Bad data leads to bugs: subsets

24

Production

2. Bad data leads to bugs: Production Wall

2. Bad data leads to bugs: late stage bugs

Dev QA UAT Production

2. Bad data leads to bugs: late stage bugs

Dev QA UAT Production

# bugsFound

Dev Testing UAT Production

2. Bad data leads to bugs: late stage bugs

1 2 3 4 5 6 70

10203040506070

Cost ToCorrect

Software Engineering Economics – Barry Boehm (1981)

Developer Asks for DB

Get Access

Manager approves

DBA Request system

Setup DB

System Admin

Requeststorage

Setup machine

Storage Admin

Allocate storage (take snapshot)

3. Slow environment builds: delays

Why are hand offs so expensive?

1hour1 day

9 days

3. Slow environment builds: delays

Companies unaware

Could I have a copy of the production DB ?

Developer, tester or AnalystBoss, Storage Admin, DBA

• Data Constraint• Solution• Use Cases

In this presentation :

Development UATQA

99% of blocks are identical

Solution

Development QA UAT

Thin Clone

Three Technologies

Production

DevelopmentStorage

Provision

Synchronize (copy)

Clone (snapshot)

Three Technologies

Production

DevelopmentStorage

Provision

Synchronize (copy)

Clone (snapshot)

Virtual Copy Data Management+ masking & self service

Install Delphix on Intel hardware

• .• .• .• .• .• Data• .• Binaries• Application Stacks• EBS • SAP• Flat files

Allocate Any Storage to Delphix

Any Storage

Pure Storage + DelphixBetter Performance for 1/10 the cost

40© 2015 Delphix. All Rights Reserved. Private & Confidential.

One time backup of source database

Production

3 TB1 TB

41© 2015 Delphix. All Rights Reserved. Private & Confidential.

One time backup of source database

Production

3 TB1 TB

Provision

Synchronize (copy)

Clone (snapshot)

42© 2015 Delphix. All Rights Reserved. Private & Confidential.

Three Physical Copies Three Virtual Copies

Data Virtualization Appliance

43

PROD DEV DEV Test Test UAT

Data as a Service : fast, elastic, secure

Self Service

• Problem in the Industry• Solution• Use Cases

1. Development 2. Security3. Cloud Migration

Use Cases

Development: Virtual Data

Development

Virtual Data: Parallelize

gif by Steve Karam

Virtual Data: Full size

Production

Virtual Data: Self Service

Environments: increase the limit

Physical Data : late stage bugs

Dev QA UAT Production

Dev Testing UAT Production0

50

100

150

200

250

300

350

400

450

500

Bugs Discovered Legacy

1 2 3 4 5 6 70

10203040506070

Cost ToCorrect

Cost ToCorrect

Physical Data : find bugs fast

Dev QA UAT Production

Dev Testing UAT Production

1 2 3 4 5 6 70

10203040506070

Cost ToCorrect

53

RefreshTest RefreshTest RefreshTest

Virtual Data : Fast Refresh

0 2 4 6 8 10 12 14 16 18 20 22 24Hours

Virtual Data

Physical Data

Bookmark, Reset

99% Less Downtime Data FederationVersion ControlBookmark and BranchQuickly Refresh Sync across data sources

Virtual Data: Version Control

54

Dev Dev

2.1 2.2

Production Time Flow

Live Archive data for years• Archive EBS R11 before upgrade to R12• Sarbanes-Oxley• Dodd-Frank• Financial Stress tests

Production

Production Time Flow

QA

• Fast• Full Size• Run Parallel QA

Virtual Data : Parallel

Production

Virtual Data: Rewind

QA

Production Time Flow

Production

Virtual Data: A/B

Index 1

Index 2

Production Time Flow

Production

Modernization: Federated

Production Time Flow 1 Production Time Flow 2

Production 1

Production 2

Physical Data: Federated

“I looked like a hero”Tony Young, CIO Informatica

Virtual Data: Federated

1. Development & QA2. Security3. Cloud Migration

Use Cases

Tradition Protection: Network & Perimeter

EndpointsPerimeter DefenseProtect the Interior

Encryption

Network Intrusion Detection

Endpoint Defense

“Organizations should use data Masking to protect sensitive data at rest and in transit from insiders' and outsiders' attacks.”

- Gartner Magic Quadrant for Data Masking Technology

Insider Threats Are Costly

Botnets

Viruses, worms, trojans

Malware

Stolen devices

Malicious code

Phishing & social engineering

Web-based attacks

Denial of services

Malicious insiders

$1,075

$1,900

$7,378

$33,565

$81,500

$85,959

$96,424

$126,545

$144,542

Average Annualized Cyber Crime Cost Weighted by Attack Frequency

Consolidated view, n = 252 separate companies

2015 Global Cost of Cyber Crime Study, Ponemon Institute

Costs moreQuality is lower

Hard to mask consistently

Moving data from prod to non-prod takes a long time

Ease of UseInstant data Consistent

Virtual Data Masking

• Automates discovery • Provides different masking algorithms for different data types• Mask once clone many with thin cloning

Mask Data

6 hours Clone 18 Hours

Clone15 min

Mask Data

Mask4

hours

Mask Data

Production Dev, QA, UAT Reporting BackupSecurity problem

Production Dev, QA, UAT Reporting SandboxSecurity management improvement

ProductionDev, QA, UAT Reporting Sandbox

Security Solution

1. Development & QA2. Security3. Cloud Migration

Use Cases

70

Migration to Cloud

Three Clones=Moving 3 x the Source

71

Migration to Cloud with Delphix

Three Clones=Moving 1/3 of Source Size

72

Cloud OptimizationsON PREMISE /

PRIVATE CLOUD

Replication

Encrypted

Compressed

Masked

73

Cloud OptimizationsON PREMISE /

PRIVATE CLOUD

74

Cloud Optimizations

$$$

ON PREMISE / PRIVATE CLOUD

75

Cloud OptimizationsON PREMISE /

PRIVATE CLOUD

76

Cloud OptimizationsON PREMISE /

PRIVATE CLOUD

77

Cloud OptimizationsON PREMISE /

PRIVATE CLOUD

78

Cloud OptimizationsON PREMISE /

PRIVATE CLOUD

• Recovery• Forensics• Migration

Bonus : Production Support

9TB database 1TB change day : 30 days

week 1

week 2

week 3

week 4

0

10

20

30

40

50

60

70

originalOracleDelphix

StorageRequired(TB)

Days

81

RPO & RTO• RPO

– Any time in last 30 days– Down to the second

• RTO– Minutes– Push button

0

2

4

6

8

10

12

14

originalDelphix

Virtual Data: Recovery

Instance

Recover VDB

Drop

Production Time Flow

Production

Virtual Data: Forensics

Development

Production Time Flow

Production

Virtual Data: Development recovery

Development

Development

Prod & VDB Time Flow

Production

1. Development & QA– Dev throughput increase by 2x

2. Secure– Mask once, clone many

3. Cloud Enablement– Compressed, encrypted replication– active/active replication

Summary

• Problem: Data constraint • Solution: Virtual Data

Summary

• Projects “12 months to 6 months.”– New York Life

• Insurance product “about 50 days ... to about 23 days”– Presbyterian Health

• “Can't imagine working without it”– State of California

Virtual Data Quotes

Thank you!• Kyle Hailey - Technical Evangelist (Oracle Ace Director, Oaktable)

– Kyle@delphix.com– kylehailey.com– slideshare.net/khailey– @virtdata

89

ProductionDEV Test UAT

A  database refresh in 15 minutes?That is mind blowing!Delphix nailed it for us. - Matt Lawrence , Sr Director Wind River (Intel) Took 3 weeks to build a dev

envnow with Delphix takes less than a daythe db part is less than 15 minutes- Marty Boos , Stubhub (Ebay)Delphix goes beyond

storage Delphix so much more than We thought it was-Michael Brow State of Colorado

Worth investing on this productthe technology is strong and value prop is high- Deloitte

I'm convinced about Delphix'stechnology Delphix can reallyincrease the quality of Dev / QA - Oaktable Member

Delphix allows us to move fast and setup database copies in secondsDelphix is powerful and allowed us to scale from 2 projects to 11We need Delphix to scale our agile environment – Tim Campos, CIO, Facebook

The Goal : eliminate the constraint

Improvement not made at the constraint is an illusion

Theory of Constraints

Factory floor

ResinMolding

TrimmerLeak detection

Labeling

Palletizing

Shipping

Factory floor

ResinMolding

TrimmerLeak detection

Labeling

Pallet - izing

Shipping

constraint

Factory floor

ResinMolding

TrimmerLeak detection

Labeling

Pallet - izing

Shipping

constraint

Tuning here

Stock piling

Factory floor

ResinMolding

TrimmerLeak detection

Labeling

Pallet - izing

Shipping

constraint

Tuning here

Starvation

Factory floor

ResinMolding

TrimmerLeak detection

Labeling

Pallet - izing

Shipping

constraint

Goal: • find constraint • optimize it