sunz2010 development version control and automated deployment of source code in a data warehouse...
TRANSCRIPT
SUNZ 2010
Version Control and Automated Deployment of Source Code in a Data Warehouse Environment
Mike O’NeilMSD Data Warehouse Team
Content
• Background and Context• Motivation• Change Management• Automation of Deployment• Subversion and TortoiseSVN• Standards and Practices• SAS Features and Functionality• Where to from here?
Background and Context
• IAP Data Warehouse (SAS based)• Established 1994• 30+ Source Systems• 10,000 Information Consumers• UNIX and Windows SAS platforms• 600+ Control-M scheduled jobs• Huge SAS Code Base• Frequency of Change
Motivation
• Quality improvement• Reduction in Failure• Productivity and Efficiency• Cost• Physical Environment Separation• Maturing process• Removal of emotion• Increasing accountability and ownership
Change Management
• Change Management– Policy and process
• Change Control– Recording and approval
• Version Control– Asset Management
Change Management
Survey Results• Not all changes logged 95%• Changes not thoroughly tested 90%• Lack of process enforcement 85%• Poor communication and dissemination 65%• Lack of centralized process ownership 60%• Lack of change approval policy 50%• Frequent change notification after the fact 40%
Reference: Edward Stickel quoting Harris Kern in article athttp://www.itsmwatch.com/itil/article.php/11700_3367151_1
Automation of Deployment
• Common feature of commercial software development
• Continuous integration• Regression testing• Configuration driven (what goes where)• Version Control
IAP Automation of Deployment
• Subversion for Version Control• TortoiseSVN for Developer Interface• Configuration driven• Deployment Application written in SAS
(and UNIX shell script)
Subversion & TortoiseSVN
• Subversion – Database, Server, CLI• TortoiseSVN – Windows Explorer plug-in• Basic developer workflow:
– Check Out to create a Working Copy– Branch to manage your own stuff– Commit changes– Update when new release hits trunk– Tag when ready for release
Subversion
IAP Automation of Deployment
• Configuration “database”– Configuration_Objects – names all objects – Configuration_Paths – says where they go
• Owned and maintained by developers
IAP Automation of Deployment
• Reconciliation• Export from repository• Pre-deploy steps• Create target directory structure• Deploy software objects to target• Post-deploy steps
Standards and Practice
Typical IAP approach• Driver Shell
– Invoked by Control-M– Creates environment variables– Calls Driver SAS program
• Driver SAS– Retrieves environment variables (%sysget)– %includes SAS code to do work
Standards and Practice
• Driver Shell ScriptExport ENVAR=valueexport SASLIB=/lev1_3/app/sasdataexport SASCODE=/r02/app/sasprogs
• Driver SAS Program%let saslib = %sysget(SASLIB);libname applib “&saslib”;%let sascode = %sysget(SASCODE);filename appsrc “&sascode”;%include appsrc(worker1.sas);
SAS Features and Functionality
• Configuration “database” in CSV files– SAS Proc Import
• SVN List command –xml– SAS xml libname engine and xml maps
• Programs that write programsPut “cp a.sh /r02/test/ControlM/shells”; Put “chgrp develop /r02/test/ControlM/shells/a.sh”; Put “chmod 2771 /r02/test/ControlM/shells/a.sh”;
• Standards and practice (using %sysget)
Where to from here?
• Acceptance• Training• Complete repository set up• Complete required refactoring• Integrate with JIRA and Remedy• Run under Control-M