evolving database design and architecture...database code like code refactoring (fowler 1999)...
TRANSCRIPT
Evolving Database Design and Architecture
Patterns and Practices
Pramod SadalageThoughtWorks Inc.
@pramodsadalage
1Thursday, November 17, 2011
Patterns of Database Changes
2Thursday, November 17, 2011
Patterns of Database Changes
•Architecture
2Thursday, November 17, 2011
Patterns of Database Changes
•Architecture
•Structure
2Thursday, November 17, 2011
Patterns of Database Changes
•Architecture
•Structure
•Data Quality
2Thursday, November 17, 2011
Patterns of Database Changes
•Architecture
•Structure
•Data Quality
•Referential Integrity
2Thursday, November 17, 2011
Patterns of Database Changes
•Architecture
•Structure
•Data Quality
•Referential Integrity
•Database Code
2Thursday, November 17, 2011
Timeline of Change
Original Transition Result
Implement the refactoring
Transition Period(old and new)
Refactoring completed
{ { {
Remove old schema, scaffolding code
Deploy new changes, migrate data, put in
scaffolding code
3Thursday, November 17, 2011
Architecture Patterns
Changes that improve the over all manner in which external programs interact with the database
4Thursday, November 17, 2011
Add Read Method
5Thursday, November 17, 2011
Introduce Read Only Table
6Thursday, November 17, 2011
Encapsulate Table with View
7Thursday, November 17, 2011
Migrate Method from Database
8Thursday, November 17, 2011
Introduce Calculation Method
9Thursday, November 17, 2011
Migrate Method to Database
10Thursday, November 17, 2011
more athttp://databaserefactoring.com
11Thursday, November 17, 2011
Structural Change Patterns
Change the structure of the database schema, for better database design
12Thursday, November 17, 2011
Split Column
13Thursday, November 17, 2011
Replace One-To-Many with Associative Table
14Thursday, November 17, 2011
Rename Table
15Thursday, November 17, 2011
Merge Columns
16Thursday, November 17, 2011
Replace Column
17Thursday, November 17, 2011
Split Table
18Thursday, November 17, 2011
more athttp://databaserefactoring.com
19Thursday, November 17, 2011
Data Quality Patterns
Changes that improve the quality of the information with in a database or ensure the consistency and usage of data
20Thursday, November 17, 2011
Make Column Non-Nullable
21Thursday, November 17, 2011
Introduce Column Constraint
22Thursday, November 17, 2011
Add Lookup Table
23Thursday, November 17, 2011
more athttp://databaserefactoring.com
24Thursday, November 17, 2011
Referential Integrity Patterns
Changes ensure Referential Data is maintained making sure Data Quality is improved
25Thursday, November 17, 2011
Add Foreign Key Constraint
26Thursday, November 17, 2011
Introduce Cascading Delete
27Thursday, November 17, 2011
more athttp://databaserefactoring.com
28Thursday, November 17, 2011
Database Code
Like code refactoring (Fowler 1999) refactor database code to improve the design of database code i.e Stored Procs and Triggers
29Thursday, November 17, 2011
Practices
Without good development Practices, using and implementing the Patterns is going to be difficult.
30Thursday, November 17, 2011
Configuration Management
31Thursday, November 17, 2011
Configuration Management
•Allow for common code ownership
31Thursday, November 17, 2011
Configuration Management
•Allow for common code ownership
•All database artifacts belong in source control repository
31Thursday, November 17, 2011
Configuration Management
•Allow for common code ownership
•All database artifacts belong in source control repository
•Include setup/config data
31Thursday, November 17, 2011
Configuration Management
•Allow for common code ownership
•All database artifacts belong in source control repository
•Include setup/config data
•Publish database artifacts with Continuous Integration
31Thursday, November 17, 2011
Give everyone a database Sandbox
32Thursday, November 17, 2011
Give everyone a database Sandbox
•Reduce waste and waiting time
32Thursday, November 17, 2011
Give everyone a database Sandbox
•Reduce waste and waiting time
•Automate data tasks
32Thursday, November 17, 2011
Give everyone a database Sandbox
•Reduce waste and waiting time
•Automate data tasks
•Allows local changes and experimenting
32Thursday, November 17, 2011
Give everyone a database Sandbox
•Reduce waste and waiting time
•Automate data tasks
•Allows local changes and experimenting
•Improves productivity
32Thursday, November 17, 2011
Give everyone a database Sandbox
•Reduce waste and waiting time
•Automate data tasks
•Allows local changes and experimenting
•Improves productivity
•Spinning up new environments is easy
32Thursday, November 17, 2011
√×
33Thursday, November 17, 2011
Behavior of Database
34Thursday, November 17, 2011
Behavior of Database
•Like objects, database has behavior
34Thursday, November 17, 2011
Behavior of Database
•Like objects, database has behavior
•Develop database objects using BDD style tests
34Thursday, November 17, 2011
Behavior of Database
•Like objects, database has behavior
•Develop database objects using BDD style tests
•Allows easy changes to the database
34Thursday, November 17, 2011
Behavior of Database
•Like objects, database has behavior
•Develop database objects using BDD style tests
•Allows easy changes to the database
•Protects against changes that affect dependent functionality
34Thursday, November 17, 2011
35Thursday, November 17, 2011
Requirements: Store a “Vehicle” with
•Unique VIN Number•Model Year 2005 and above•Model Year Not Null•Model Name Not Null•Make Not Null•Miles not above 10000
35Thursday, November 17, 2011
Requirements: Store a “Vehicle” with
•Unique VIN Number•Model Year 2005 and above•Model Year Not Null•Model Name Not Null•Make Not Null•Miles not above 10000
35Thursday, November 17, 2011
Requirements: Store a “Vehicle” with
•Unique VIN Number•Model Year 2005 and above•Model Year Not Null•Model Name Not Null•Make Not Null•Miles not above 10000
Behavior Tests:
•shouldNotSaveDuplicateVIN()•shouldSaveModelYear2010()•shouldNotSaveModelYear2004()•shouldNotSaveNullModelName()•shouldNotSaveNullMake()•shouldSaveMiles5000()•shouldNotSaveMiles12000()
35Thursday, November 17, 2011
CREATE TABLE vehicle( !id NUMBER(18) NOT NULL,
! vin VARCHAR2(32) NOT NULL,! name VARCHAR2(32) NOT NULL,! make VARCHAR2(32) NOT NULL,! year NUMBER(4) NOT NULL,! miles NUMBER(10) NULL,CONSTRAINT chk_vehicle_year_gt_2005 ! ! ! CHECK (year> 2004));CONSTRAINT chk_vehicle_miles_lt_10001 ! ! ! CHECK (miles< 10001));CREATE UNIQUE INDEX uidx_vehicle_vin ! ! ! ON vehicle(vin);!ALTER TABLE VEHICLE ADD CONSTRAINT ! ! ! pk_vehicle PRIMARY KEY (id);
36Thursday, November 17, 2011
Tracking Changes
37Thursday, November 17, 2011
Tracking Changes
•Write each change as a delta(migration) script
37Thursday, November 17, 2011
Tracking Changes
•Write each change as a delta(migration) script
•Migration scripts are development time activity not deployment time project
37Thursday, November 17, 2011
Tracking Changes
•Write each change as a delta(migration) script
•Migration scripts are development time activity not deployment time project
•Package delta scripts for automated deployment
37Thursday, November 17, 2011
Tracking Changes
•Write each change as a delta(migration) script
•Migration scripts are development time activity not deployment time project
•Package delta scripts for automated deployment
•Same scripts for: developers, QA, UAT and Production
37Thursday, November 17, 2011
38Thursday, November 17, 2011
ALTER TABLE customer ADD customeridentifier VARCHAR2(12);
UPDATE customer SET customeridentifier = customernumber;
--If No Transistion Period
ALTER TABLE customer DROP COLUMN customerNumber;
--//@UNDO
ALTER TABLE customer ADD customernumber NUMBER(10);
UPDATE customer SET customernumber = customeridentifier;
ALTER TABLE customer DROP COLUMN customeridentifier;
38Thursday, November 17, 2011
Deployed Database Version
39Thursday, November 17, 2011
Continuous Integration
40Thursday, November 17, 2011
Continuous Integration
•Test application code and database at one place
40Thursday, November 17, 2011
Continuous Integration
•Test application code and database at one place
•Generate code and database artifacts
40Thursday, November 17, 2011
Continuous Integration
•Test application code and database at one place
•Generate code and database artifacts
•Integrate application and database changes in an independent environment
40Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
DevDatabase
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
41Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
Migration scripts
41Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
Migration scripts
Migration scripts
IntegrationDatabase
41Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
Migration scripts
Migration scripts
IntegrationDatabase
41Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
Migration scripts
Applymigration scripts
Migration scripts
IntegrationDatabase
41Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
Migration scripts
Applymigration scripts
Migration scripts
Artifacts
Migration scripts
WarJar
IntegrationDatabase
41Thursday, November 17, 2011
Central integration environmentLocal dev environment
ANTMavenMakeRake
Continuous Integration Engine
SourceControl
PRODCTQA
DevDB
PRODUATQA
Environment
Check in application code and database migration scripts
Updateand build
Migration scripts
Applymigration scripts
Migration scripts
Artifacts
Migration scripts
WarJar
Migration scripts
WarJar
IntegrationDatabase
41Thursday, November 17, 2011
Deployment
42Thursday, November 17, 2011
Deployment
•Database migration/upgrade should be a development time task not deployment time task
42Thursday, November 17, 2011
Deployment
•Database migration/upgrade should be a development time task not deployment time task
•Package all the migration scripts, during Continuous Integration cycle
42Thursday, November 17, 2011
Deployment
•Database migration/upgrade should be a development time task not deployment time task
•Package all the migration scripts, during Continuous Integration cycle
•Apply these migration scripts
42Thursday, November 17, 2011
43Thursday, November 17, 2011
Pair with the Data Team
44Thursday, November 17, 2011
Pair with the Data Team
•Break down the silos
44Thursday, November 17, 2011
Pair with the Data Team
•Break down the silos
•Allows continuous reviews
44Thursday, November 17, 2011
Pair with the Data Team
•Break down the silos
•Allows continuous reviews
•Understand performance implications early
44Thursday, November 17, 2011
Pair with the Data Team
•Break down the silos
•Allows continuous reviews
•Understand performance implications early
•Put database code and application code in same repository
44Thursday, November 17, 2011
Resources
bit.ly/qconbdddbit.ly/evolvedb
45Thursday, November 17, 2011
Thanks@pramodsadalage
www.sadalage.comwww.databaserefactoring.com
46Thursday, November 17, 2011