version control meets database control
DESCRIPTION
This joint webinar for DBmaestro (www.dbmaestro.com)and Delphix discuss the synergy between Delphix’s Database Virtualiztion and DBmaestro’s Database Enforced Change Management solutions. The session discuss the challenges in database development and show in practice how Database Enforced Change Management and Database Virtualization work together to create a version control, branching and merging method that addresses these challenges.TRANSCRIPT
![Page 1: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/1.jpg)
Webinar: Version Control Meets Database Control
DBmaestro Introductions
![Page 2: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/2.jpg)
Presenters
Kyle Hailey @kylehhailey• Technical Evangelist at Delphix• Oracle ACE, member of the OakTable Network
Uri Margalit @UriMargalit • Director, Product Management• Presenter at world-wide conferences: ODTUG,
ilOUG, etc…
![Page 3: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/3.jpg)
Before we start
• You will be on mute for the duration of the event
• We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)
• There will be a Q+A session at the end but please feel free to type your questions in the Questions box in the Control Panel in advance
• A recording of the full webinar will be put up online
![Page 4: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/4.jpg)
About Delphix
• Founded in 2008, launched in 2010• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))• Based in Silicon Valley, Global Operations• 10% of Fortune 500
![Page 5: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/5.jpg)
About DBmaestro
• Founded in 2008• Headed by Yariv Tabac and Yaniv Yehuda• Headquartered in Israel
![Page 6: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/6.jpg)
Version Control meets Data Control
Kyle Hailey & Uri Margalit
![Page 7: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/7.jpg)
The Business Need
Copyright@2008, Juniper Networks, Inc.
80% of unplanned downtime is due to Change
50% More than
of this, half is due to human errors
40% of changes FAIL
![Page 9: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/9.jpg)
Dealing with Risk
Smaller and more focused changes are easier to manage (Agile…) Automation of repeating tasks lowers risk of (human) error Development and Operations should work in synergy (DevOps)
![Page 10: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/10.jpg)
Source Control – Standard De Facto
Common version control tools:GitHubSVNPerforceTFSRTCVSS
![Page 11: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/11.jpg)
The Database Challenge
• The Database is a crucial part of the Application— Schema Structure— PL/SQL Code— Lookup Content
• The Database is a central resource• Business Data Must be preserved
• The Database is not native to traditional version control
• Objects are not files on a file system• How can we manage Content?• How can we branch a Database?
![Page 12: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/12.jpg)
Tradeoff: Speed, Quality, Cost
![Page 13: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/13.jpg)
Good, Cheap, Fast : choose two
Fast
Good Cheap
![Page 14: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/14.jpg)
What We’ve Seen
1. Inefficient QA: Higher costs of QA2. QA Delays : Greater re-work of code3. Sharing DB Environments : Bottlenecks4. Using DB Subsets: More bugs in Prod5. Slow Environment Builds: Delays
![Page 15: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/15.jpg)
1. Inefficient QA: Long Build times
Build Time
QA Test
96% of QA time was building environment$.04/$1.00 actual testing vs. setup
Build
![Page 16: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/16.jpg)
2. QA Delays: bugs found late require more code re-work
Build QA Env QA Build QA Env QA
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
1 2 3 4 5 6 70
10203040506070
Delay in Fixing the bug
Cost To
Correct
Software Engineering Economics – Barry Boehm (1981)
![Page 17: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/17.jpg)
3. Full Copy Shared : Bottlenecks
Frustration Waiting
Old Unrepresentative Data
![Page 18: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/18.jpg)
4. Subsets : cause bugs
![Page 19: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/19.jpg)
Production
4. Subsets : cause bugs
Classic problem is that queries that run fast on subsets hit the wall in production.
Developers are unable to test against all data
The Production ‘Wall’
![Page 20: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/20.jpg)
5. Slow Environment Builds: 3-6 Months to Deliver Data
Management
DBA
System Admin
Storage Admin
Developers Submit Request
Disk Capacity?
Approve Request $$ (2 Weeks)
Approve Request $$
(1 Week)
RequestAdditional
Storage?ProvisionCapacity
File SystemConfigured?
Configure LUNS & Build File System
Coordinate Replication w/ Infrastructure
Re-Parameterize & Configure DB
Mount Recovery DB to
Specific PIT
Begin Work
Approve Request $$ (2 Weeks)
(3 Days)
(3 Days)
(2 Days)
(3 Days)
(3 Days)
.……1-2 Weeks of Approvals, Delays, and Provisioning……
20
5. Slow Environment Builds: 3-6 Months to Deliver Data
![Page 21: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/21.jpg)
5. Slow Environment Builds: culture of no
![Page 22: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/22.jpg)
What We’ve Seen
1. Inefficient QA: Higher costs2. QA Delays : Increased re-work3. Sharing DB : Bottlenecks4. Subset DB : Bugs5. Slow Environment Builds: Delays
![Page 23: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/23.jpg)
Poll
Which of the following have you run into at your organization?1. Inefficient QA driving up costs2. QA Delays causing increased re-work of code3. Sharing DB causing development bottlenecks4. Subset DB database in development and QA
leading to bugs in production5. Slow Environment Builds causing project delays
![Page 24: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/24.jpg)
60% Projects Over Schedule and Budget
CIO Magazine Survey:
Data is the problem
Solve the data problem.
TODAY.
![Page 25: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/25.jpg)
UNLOCK YOUR DATA
![Page 26: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/26.jpg)
Clone 1 Clone 3Clone 2
99% of blocks are identical
![Page 27: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/27.jpg)
Clone 1 Clone 2 Clone 3
Thin Clone
![Page 28: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/28.jpg)
Virtualization Layer
Virtualization
![Page 29: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/29.jpg)
Three Physical Copies Three Virtual Copies
![Page 30: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/30.jpg)
![Page 31: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/31.jpg)
Install Delphix on x86 hardware
Intel hardware
![Page 32: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/32.jpg)
Allocate Any Storage to Delphix
Allocate StorageAny type
![Page 33: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/33.jpg)
One time backup of source database
Database
Production
Instance
File system
File system
![Page 34: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/34.jpg)
DxFS (Delphix) Compress Data
Database
Production
Instance
Data is compressed typically 1/3
size
File system
![Page 35: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/35.jpg)
Incremental forever change collection
Database
Production
Instance
File system
Changes
• Collected incrementally forever• Old data purged
Time Window
File system
![Page 36: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/36.jpg)
Typical Architecture
Production
Instance
Development
Instance
QA
Instance
UAT
Instance
Database
File systemFile system
Database
File systemFile system
Database
File systemFile system
Database
File systemFile system
![Page 37: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/37.jpg)
With Delphix
Database
File system
Production
Instance
Database
Development
Instance
Database
QA
Instance
Database
UAT
Instance
![Page 38: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/38.jpg)
![Page 39: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/39.jpg)
Three Core Parts
Production
Instance
Time Window
Instance
Virtual Database
1
2
3
1. Source Syncing
2. Storage (DxFS)
3. Self Service
Development
![Page 40: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/40.jpg)
Fast, Fresh, Full
Database
Production
Instance
File system Time Window
Instance
Virtual Database
![Page 41: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/41.jpg)
Free
Database
Production
Instance
File system Time Window
Instance
Virtual Database
Instance
Instance
Virtual Database
Virtual Database
![Page 42: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/42.jpg)
Branching to QA
Database
Production
Instance
File system Time Window
Instance
Virtual Database
Instance
Virtual Database Dev
QA
![Page 43: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/43.jpg)
Self Service
![Page 44: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/44.jpg)
What We’ve Seen With Delphix
1. Efficient QA: Low cost, high utilization2. Quick QA : Fast Bug Fix3. Every Dev gets DB: Parallelized Dev4. Full DB : Less Bugs5. Fast Builds: Fast Dev, Culture of Yes
![Page 45: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/45.jpg)
1. Efficient QA: Lower cost
Build
Ti
me
QA Test
1% of QA time was building environment$.99/$1.00 actual testing vs. setup
Build Time
QA TestBuild
![Page 46: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/46.jpg)
2. QA Immediate: bugs found fast and fixed
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
QA QA
Build QA Env QA Build QA Env QA
Sprint 1 Sprint 2 Sprint 3
Bug CodeX
![Page 47: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/47.jpg)
3. Private Copies: Parallelize
![Page 48: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/48.jpg)
4. Full Size DB : Eliminate bugs
Production
![Page 49: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/49.jpg)
5. Self Service: Fast, Efficient. Culture of Yes!
Management
DBA
System Admin
Storage Admin
Developers Submit Request
Disk Capacity?
Approve Request $$ (2 Weeks)
Approve Request $$
(1 Week)
RequestAdditional
Storage?ProvisionCapacity
File SystemConfigured?
Configure LUNS & Build File System
Coordinate Replication w/ Infrastructure
Re-Parameterize & Configure DB
Mount Recovery DB to
Specific PIT
Begin Work
Approve Request $$ (2 Weeks)
(3 Days)
(3 Days)
(2 Days)
(3 Days)
(3 Days)
.……1-2 Weeks of Approvals, Delays, and Provisioning……
![Page 50: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/50.jpg)
What We’ve Seen With Delphix
1. Efficient QA: Low cost, high utilization2. Quick QA : Fast Bug Fix3. Every Dev gets DB: Parallelized Dev4. Full DB : Less Bugs5. Fast Builds: Fast Dev, Culture of Yes
![Page 51: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/51.jpg)
![Page 52: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/52.jpg)
Challenges of Development & Release to Operation
Organizing the development of changes• Code• Database • Configuration • Metadata• Work Items
Development Test/ Staging/ UAT
Enabling safe migration into production• Moving the right
components• Enabling
Rollback & Recovery
Production
Development Operations
Moving just the right components needed for the release
• Release Approved Items
Agility dictates frequent changes & new tools are needed to streamline the process
ChangeManagement
ReleaseManagement
![Page 53: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/53.jpg)
What is DBmaestro TeamWork
• Database Enforced Change Management+ Database version control+ Plugs into the ALM (change request, tickets &
work items)
+ Database change impact analysis+ Database deployment automation
• DevOps Solution for databases + Deployment, rollback & recovery+ Plugs into release management
![Page 54: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/54.jpg)
The Challenges that DBmaestro Addresses
• Development delays• Silos in development, DBA and operations• Delays in deployment (internally and to operations)• Errors in production
![Page 55: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/55.jpg)
Development Delays
• Different methodologies for the application & database
• Code overrides• Lack of history of changes (who did what, where, when and
why)
• Manual writing of delta scripts• Lack of automation
![Page 56: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/56.jpg)
Silos in Development, DBA and Operations
• No sharing between the team• No visibility• Always looking for errors made by others
![Page 57: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/57.jpg)
Delays in Deployment (Internally and to Operations)
• Deployment automation does not really include the database tier
• Database scripts generated out of the scope of automation
• Lack of confidence in automation
![Page 58: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/58.jpg)
Errors in Production
• Missing changes• Deploying the wrong version of objects• What about the reference data?
![Page 59: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/59.jpg)
Poll
Which challenges have you experienced?
1. Development delays
2. Silos in development, DBA and operations
3. Delays in deployment (internally and to operations)
4. Errors in production
![Page 60: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/60.jpg)
How?
• Database version control – Enforced Check Out/In – Labels– Rollback/Undo – Audit trail reports
• Database impact analysis – Utilizes version control repository information – 3 way analysis
• Database deployment automation– API – Baselines – Conflict resolution – Customized business logic
![Page 61: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/61.jpg)
Without DCM - Two isolated Processes
Check-Out Script
Modify Script
Get updated Script from DB
Check-In Script
Compile Scriptin DB
Debug Scriptin DB
Version Control Process Development Process
?
??
?
![Page 62: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/62.jpg)
Development & Version Control Process
With DCM - One Enforced Process
Check-Out Object
Modify Object in DB
Run Applications’
Tests
Check-In Object
![Page 63: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/63.jpg)
Safety Net For Automation of Deployment
Source vs. Target
Action
= No Action
≠ ?
Source vs. Baseline
Target vs. Baseline
Action
= = No Action
≠ = Override
= ≠ Ignore
≠ ≠ Merge
You do not have all of the information
With Baselines and 3 way analysis the unknown is now known
Simple Compare & Sync Baseline aware Deployment
![Page 64: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/64.jpg)
Benefits - Development
• Database change repository• Follow SCM best practices (Check-Out/Check-
In) • All changes are documented• Manage who can do what, where, when & why
![Page 65: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/65.jpg)
Benefits - Operations
• Integrated deployment engine• Business level audit• Roles & responsibilities enforcement
![Page 66: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/66.jpg)
Benefits - Management
• Complete visibility into changes in progress• Management reports• No silos
![Page 67: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/67.jpg)
Live Demo
• Clone 2 virtual copies of the Trunk1. Dev1
2. Dev2
• Make changes & merge them into the Trunk: Developer1 modifies Dev1 Developer1 merges changes into the Trunk Developer2 modifies Dev2 Developer2 merges changes into the Trunk
• Rely on enforced changes & automation
![Page 68: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/68.jpg)
Time Window
Instance
Instance
Instance
Virtual DatabaseDev1
Dev2
Trunk
Virtual Database
Virtual Database
Developer 1 modify
Developer 2 modify
DBVC
![Page 69: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/69.jpg)
Merge Dev1 & TrunkMerge to dev2
Dev2
Dev1Merge to dev1
Merge Dev2 & Trunk
Trunk
Merge Dev1 & Trunk
Merge Dev2 & Trunk
DBVC
![Page 70: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/70.jpg)
Merge Dev1 to ForkMerge to dev2
Dev2
Dev1Merge to dev1
Merge Dev2 to Fork
Trunk
Merge Dev1 to Fork
Merge Dev2 to Fork
DBVC
Fork
Fork
Fork
Fork
![Page 71: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/71.jpg)
Q&AKyle Hailey @kylehhaileyDelphix: delphix.com
Uri Margalit @UriMargalit DBmaestro: dbmaestro.com
![Page 72: Version Control meets Database Control](https://reader035.vdocuments.net/reader035/viewer/2022062420/554f9021b4c905d25b8b5175/html5/thumbnails/72.jpg)
Thanks