sunz2010 development version control and automated deployment of source code in a data warehouse...
TRANSCRIPT
![Page 1: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/1.jpg)
SUNZ 2010
Version Control and Automated Deployment of Source Code in a Data Warehouse Environment
Mike O’NeilMSD Data Warehouse Team
![Page 2: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/2.jpg)
Content
• Background and Context• Motivation• Change Management• Automation of Deployment• Subversion and TortoiseSVN• Standards and Practices• SAS Features and Functionality• Where to from here?
![Page 3: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/3.jpg)
Background and Context
• IAP Data Warehouse (SAS based)• Established 1994• 30+ Source Systems• 10,000 Information Consumers• UNIX and Windows SAS platforms• 600+ Control-M scheduled jobs• Huge SAS Code Base• Frequency of Change
![Page 4: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/4.jpg)
Motivation
• Quality improvement• Reduction in Failure• Productivity and Efficiency• Cost• Physical Environment Separation• Maturing process• Removal of emotion• Increasing accountability and ownership
![Page 5: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/5.jpg)
![Page 6: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/6.jpg)
Change Management
• Change Management– Policy and process
• Change Control– Recording and approval
• Version Control– Asset Management
![Page 7: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/7.jpg)
Change Management
Survey Results• Not all changes logged 95%• Changes not thoroughly tested 90%• Lack of process enforcement 85%• Poor communication and dissemination 65%• Lack of centralized process ownership 60%• Lack of change approval policy 50%• Frequent change notification after the fact 40%
Reference: Edward Stickel quoting Harris Kern in article athttp://www.itsmwatch.com/itil/article.php/11700_3367151_1
![Page 8: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/8.jpg)
Automation of Deployment
• Common feature of commercial software development
• Continuous integration• Regression testing• Configuration driven (what goes where)• Version Control
![Page 9: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/9.jpg)
IAP Automation of Deployment
• Subversion for Version Control• TortoiseSVN for Developer Interface• Configuration driven• Deployment Application written in SAS
(and UNIX shell script)
![Page 10: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/10.jpg)
Subversion & TortoiseSVN
• Subversion – Database, Server, CLI• TortoiseSVN – Windows Explorer plug-in• Basic developer workflow:
– Check Out to create a Working Copy– Branch to manage your own stuff– Commit changes– Update when new release hits trunk– Tag when ready for release
![Page 11: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/11.jpg)
Subversion
![Page 12: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/12.jpg)
![Page 13: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/13.jpg)
IAP Automation of Deployment
• Configuration “database”– Configuration_Objects – names all objects – Configuration_Paths – says where they go
• Owned and maintained by developers
![Page 14: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/14.jpg)
![Page 15: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/15.jpg)
IAP Automation of Deployment
• Reconciliation• Export from repository• Pre-deploy steps• Create target directory structure• Deploy software objects to target• Post-deploy steps
![Page 16: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/16.jpg)
Standards and Practice
Typical IAP approach• Driver Shell
– Invoked by Control-M– Creates environment variables– Calls Driver SAS program
• Driver SAS– Retrieves environment variables (%sysget)– %includes SAS code to do work
![Page 17: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/17.jpg)
Standards and Practice
• Driver Shell ScriptExport ENVAR=valueexport SASLIB=/lev1_3/app/sasdataexport SASCODE=/r02/app/sasprogs
• Driver SAS Program%let saslib = %sysget(SASLIB);libname applib “&saslib”;%let sascode = %sysget(SASCODE);filename appsrc “&sascode”;%include appsrc(worker1.sas);
![Page 18: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/18.jpg)
SAS Features and Functionality
• Configuration “database” in CSV files– SAS Proc Import
• SVN List command –xml– SAS xml libname engine and xml maps
• Programs that write programsPut “cp a.sh /r02/test/ControlM/shells”; Put “chgrp develop /r02/test/ControlM/shells/a.sh”; Put “chmod 2771 /r02/test/ControlM/shells/a.sh”;
• Standards and practice (using %sysget)
![Page 19: Sunz2010 Development Version Control And Automated Deployment Of Source Code In A Data Warehouse Environment](https://reader037.vdocuments.net/reader037/viewer/2022100519/55a93f831a28ab684e8b4572/html5/thumbnails/19.jpg)
Where to from here?
• Acceptance• Training• Complete repository set up• Complete required refactoring• Integrate with JIRA and Remedy• Run under Control-M