virtual data : eliminating the data constraint in application development
TRANSCRIPT
Virtual Data Eliminating the data constraint in Application
Development
Kyle Hailey, Technical Evangelist at Delphix
Technology Disruption
“Software is eating the world.”- Marc Andreessen
Increasing Commoditization
Competitive Pressures
Consumerization of Software
New software required for success
PCs
2010
Mob
ile
• Problem : Data Constraint• Solution : Virtual Data• Use Cases : Development, Security, Cloud
In this presentation :
DevOps :
DevOps : Process• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast
DevOps : Process• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast
Tools:• Continuous Delivery• Cloud • Agile • Kanban• Kata
DevOps : Process• Goals Clarify • Metrics Define • Constraints Identify • Priorities Set • Iterations Fast
Tools:• Continuous Delivery• Cloud • Agile • Kanban• Kata
The Phoenix Project
What is the constraint
in IT ?
Put your energy into the constraint Top 5 constraints in IT
1. Dev environments setup2. QA setup3. Code Architecture4. Development5. Product management
- Gene KimSurveyed • 14000 companies• 100s of CIOs
Flow of Features
Product Management
Development
QAIntegration
testing
Deployment
Testing
Customer
DevOps is a Goal
Fast flow of features from development to IT operations to the customers
- Gene Kim
Flow of Features
14
Product Management
Development
QAIntegration
testing
Deployment
Testing
Customer
1
DevelopmentEnvironments
2
QA & Testing Environments
Product ManagementFeatures
2 2
Code Architecture 3Code Speed
4 5
Data
Development Pipeline for QA
SQL
Build Deploy
Environment
Database
16
PRODDEV Test UAT
DBA
Sys Admin
Storage Admin
Legacy Data Movement: Slow & expensive
?
Slow environment builds: delays
17
Development Pipeline for QA
0 2 4 6 8 10 12 14 16 18 20 22 24
ResetTest ResetTest ResetTest
Physical Data
Wait Time
Hours
Refresh( > 80%)
Testing (< 20%)
18
Data Management not Agile
• 20% SDLC time lost waiting for data
• 60% dev/QA time consumed by data tasks
Conclusion:
Data management does not scale to Agile
- Infosys
Data is the Constraint
19
Application Development Constraints
1. Not enough resources2. Bad test data leading to bugs3. Slow environment builds
1. Not Enough Resources: shared bottlenecks
Frustration Waiting
1. Not Enough Resources : bugs because of old data
Old Unrepresentative Data
1. Not enough resources: limited environments
2. Bad data leads to bugs: subsets
24
Production
2. Bad data leads to bugs: Production Wall
2. Bad data leads to bugs: late stage bugs
Dev QA UAT Production
2. Bad data leads to bugs: late stage bugs
Dev QA UAT Production
# bugsFound
Dev Testing UAT Production
2. Bad data leads to bugs: late stage bugs
1 2 3 4 5 6 70
10203040506070
Cost ToCorrect
Software Engineering Economics – Barry Boehm (1981)
Developer Asks for DB
Get Access
Manager approves
DBA Request system
Setup DB
System Admin
Requeststorage
Setup machine
Storage Admin
Allocate storage (take snapshot)
3. Slow environment builds: delays
Why are hand offs so expensive?
1hour1 day
9 days
3. Slow environment builds: delays
Companies unaware
Could I have a copy of the production DB ?
Developer, tester or AnalystBoss, Storage Admin, DBA
• Data Constraint• Solution• Use Cases
In this presentation :
Development UATQA
99% of blocks are identical
Solution
Development QA UAT
Thin Clone
Three Technologies
Production
DevelopmentStorage
Provision
Synchronize (copy)
Clone (snapshot)
Three Technologies
Production
DevelopmentStorage
Provision
Synchronize (copy)
Clone (snapshot)
Virtual Copy Data Management+ masking & self service
Install Delphix on Intel hardware
• .• .• .• .• .• Data• .• Binaries• Application Stacks• EBS • SAP• Flat files
Allocate Any Storage to Delphix
Any Storage
Pure Storage + DelphixBetter Performance for 1/10 the cost
40© 2015 Delphix. All Rights Reserved. Private & Confidential.
One time backup of source database
Production
3 TB1 TB
41© 2015 Delphix. All Rights Reserved. Private & Confidential.
One time backup of source database
Production
3 TB1 TB
Provision
Synchronize (copy)
Clone (snapshot)
42© 2015 Delphix. All Rights Reserved. Private & Confidential.
Three Physical Copies Three Virtual Copies
Data Virtualization Appliance
43
PROD DEV DEV Test Test UAT
Data as a Service : fast, elastic, secure
Self Service
• Problem in the Industry• Solution• Use Cases
1. Development 2. Security3. Cloud Migration
Use Cases
Development: Virtual Data
Development
Virtual Data: Parallelize
gif by Steve Karam
Virtual Data: Full size
Production
Virtual Data: Self Service
Environments: increase the limit
Physical Data : late stage bugs
Dev QA UAT Production
Dev Testing UAT Production0
50
100
150
200
250
300
350
400
450
500
Bugs Discovered Legacy
1 2 3 4 5 6 70
10203040506070
Cost ToCorrect
Cost ToCorrect
Physical Data : find bugs fast
Dev QA UAT Production
Dev Testing UAT Production
1 2 3 4 5 6 70
10203040506070
Cost ToCorrect
53
RefreshTest RefreshTest RefreshTest
Virtual Data : Fast Refresh
0 2 4 6 8 10 12 14 16 18 20 22 24Hours
Virtual Data
Physical Data
Bookmark, Reset
99% Less Downtime Data FederationVersion ControlBookmark and BranchQuickly Refresh Sync across data sources
Virtual Data: Version Control
54
Dev Dev
2.1 2.2
Production Time Flow
Live Archive data for years• Archive EBS R11 before upgrade to R12• Sarbanes-Oxley• Dodd-Frank• Financial Stress tests
Production
Production Time Flow
QA
• Fast• Full Size• Run Parallel QA
Virtual Data : Parallel
Production
Virtual Data: Rewind
QA
Production Time Flow
Production
Virtual Data: A/B
Index 1
Index 2
Production Time Flow
Production
Modernization: Federated
Production Time Flow 1 Production Time Flow 2
Production 1
Production 2
Physical Data: Federated
“I looked like a hero”Tony Young, CIO Informatica
Virtual Data: Federated
1. Development & QA2. Security3. Cloud Migration
Use Cases
Tradition Protection: Network & Perimeter
EndpointsPerimeter DefenseProtect the Interior
Encryption
Network Intrusion Detection
Endpoint Defense
“Organizations should use data Masking to protect sensitive data at rest and in transit from insiders' and outsiders' attacks.”
- Gartner Magic Quadrant for Data Masking Technology
Insider Threats Are Costly
Botnets
Viruses, worms, trojans
Malware
Stolen devices
Malicious code
Phishing & social engineering
Web-based attacks
Denial of services
Malicious insiders
$1,075
$1,900
$7,378
$33,565
$81,500
$85,959
$96,424
$126,545
$144,542
Average Annualized Cyber Crime Cost Weighted by Attack Frequency
Consolidated view, n = 252 separate companies
2015 Global Cost of Cyber Crime Study, Ponemon Institute
Costs moreQuality is lower
Hard to mask consistently
Moving data from prod to non-prod takes a long time
Ease of UseInstant data Consistent
Virtual Data Masking
• Automates discovery • Provides different masking algorithms for different data types• Mask once clone many with thin cloning
Mask Data
6 hours Clone 18 Hours
Clone15 min
Mask Data
Mask4
hours
Mask Data
Production Dev, QA, UAT Reporting BackupSecurity problem
Production Dev, QA, UAT Reporting SandboxSecurity management improvement
ProductionDev, QA, UAT Reporting Sandbox
Security Solution
1. Development & QA2. Security3. Cloud Migration
Use Cases
70
Migration to Cloud
Three Clones=Moving 3 x the Source
71
Migration to Cloud with Delphix
Three Clones=Moving 1/3 of Source Size
72
Cloud OptimizationsON PREMISE /
PRIVATE CLOUD
Replication
Encrypted
Compressed
Masked
73
Cloud OptimizationsON PREMISE /
PRIVATE CLOUD
74
Cloud Optimizations
$$$
ON PREMISE / PRIVATE CLOUD
75
Cloud OptimizationsON PREMISE /
PRIVATE CLOUD
76
Cloud OptimizationsON PREMISE /
PRIVATE CLOUD
77
Cloud OptimizationsON PREMISE /
PRIVATE CLOUD
78
Cloud OptimizationsON PREMISE /
PRIVATE CLOUD
• Recovery• Forensics• Migration
Bonus : Production Support
9TB database 1TB change day : 30 days
week 1
week 2
week 3
week 4
0
10
20
30
40
50
60
70
originalOracleDelphix
StorageRequired(TB)
Days
81
RPO & RTO• RPO
– Any time in last 30 days– Down to the second
• RTO– Minutes– Push button
0
2
4
6
8
10
12
14
originalDelphix
Virtual Data: Recovery
Instance
Recover VDB
Drop
Production Time Flow
Production
Virtual Data: Forensics
Development
Production Time Flow
Production
Virtual Data: Development recovery
Development
Development
Prod & VDB Time Flow
Production
1. Development & QA– Dev throughput increase by 2x
2. Secure– Mask once, clone many
3. Cloud Enablement– Compressed, encrypted replication– active/active replication
Summary
• Problem: Data constraint • Solution: Virtual Data
Summary
• Projects “12 months to 6 months.”– New York Life
• Insurance product “about 50 days ... to about 23 days”– Presbyterian Health
• “Can't imagine working without it”– State of California
Virtual Data Quotes
Thank you!• Kyle Hailey - Technical Evangelist (Oracle Ace Director, Oaktable)
– [email protected]– kylehailey.com– slideshare.net/khailey– @virtdata
89
ProductionDEV Test UAT
A database refresh in 15 minutes?That is mind blowing!Delphix nailed it for us. - Matt Lawrence , Sr Director Wind River (Intel) Took 3 weeks to build a dev
envnow with Delphix takes less than a daythe db part is less than 15 minutes- Marty Boos , Stubhub (Ebay)Delphix goes beyond
storage Delphix so much more than We thought it was-Michael Brow State of Colorado
Worth investing on this productthe technology is strong and value prop is high- Deloitte
I'm convinced about Delphix'stechnology Delphix can reallyincrease the quality of Dev / QA - Oaktable Member
Delphix allows us to move fast and setup database copies in secondsDelphix is powerful and allowed us to scale from 2 projects to 11We need Delphix to scale our agile environment – Tim Campos, CIO, Facebook
The Goal : eliminate the constraint
Improvement not made at the constraint is an illusion
Theory of Constraints
Factory floor
ResinMolding
TrimmerLeak detection
Labeling
Palletizing
Shipping
Factory floor
ResinMolding
TrimmerLeak detection
Labeling
Pallet - izing
Shipping
constraint
Factory floor
ResinMolding
TrimmerLeak detection
Labeling
Pallet - izing
Shipping
constraint
Tuning here
Stock piling
Factory floor
ResinMolding
TrimmerLeak detection
Labeling
Pallet - izing
Shipping
constraint
Tuning here
Starvation
Factory floor
ResinMolding
TrimmerLeak detection
Labeling
Pallet - izing
Shipping
constraint
Goal: • find constraint • optimize it