1. WebMD Adopts Automated Deployment in Support of Continuous Integration to Transform their SDLC Teresa Dietrich, Vice President Technical Operations Derek Chang, Senior Director

2. We were here 2

3. So we assembled a team 3

4. To fix our SDLC 4

5. And now everyone is happy. 5

6. Technology @ WebMD WebMD, Medscape, MedicineNet, eMedicine, UK cobrand Serving nearly 1 Billion Pageviews a month, 132 million unique Running over 200 separate applications, vast majority in-house developed Environments: Dev/Devint, QA01/02, QA00, Production/DR Two main data centers, geographically diverse OS: mix of Linux and Windows Datastores: Sql Server, Oracle, Mongo, Vertica and mysql Web: mix of Apache and IIS App: mix of Tomcat, ASP, .Net 3.x and 4.x Service: ActiveMQ, Memcache 6

7. Before - State of Deployments Build repository a free for all NAS share; manual upload Package management HP proprietary package library; manual upload/download Configuration management HP proprietary CML Deployment module HP software policy Deployment initiation SBM ticketing system Pre/post deployment tasks custom scripts and manual steps Dev Sends Email QA Opens Ticket Lead Assign Ticket Engineer Opens SubTickets DBA Runs Script Engineer deploys build QA Smoke Tests Ticket Closed 7

8. Before Deployment Metrics year total # of work days average daily max date 2011 2650 251 10.5 34 7/1/2011 2012 3174 251 12.6 50 9/20/2012 Average time to deploy: 55 minutes 8% of tickets rejected thus required clarification Average build deployment per day: 11.6 Daily Max: 50 8

9. Before - Pain Points Require multiple systems Relies heavily on email for communication Inconsistent environment setup Only Ops did deployments in QA and Prod Error prone Wrong information provided in ticketing system Tons of customized scripts Complexity of deployment procedures Manual steps involved in pre and post deployment High maintenance cost Configuration management Host inventory Deployment scripts 9

10. Becoming Agile Hired an Agile consulting group Signed up for Software to manage Agile Dev Trained Scrum Masters Learned to write user stories, plan sprints and used colored Post-Its Scheduled daily Stand-ups. Woohoo, now we are officially Agile! 10

11. Agile Implications for Deployments 11

12. Transform our SDLC 12

13. Waterfall vs Agile 13

14. Transform our SDLC 14

15. Promote Once vs Promote Continuously 15

16. Transform our SDLC 16

17. Static vs Elastic 17

18. Continuous Integration Team Formed a CI Team w/ representatives from: Dev WebOps QA DBA Platform Ops Each team member had to have day to day hands-on responsibilities. Each team member had to have the authority to make decisions on behalf of the organization they represented. Each team member had to be able to commit to at least 4 hours a week . Team reported back to Sr Mgmt at specified milestones. 18

19. Measuring Success Automated Unit Testing Build Ready to Build Deployed (by env) Successful Build Deployment (by env) Successful Automated Smoke test (by env) Build Deployment Execution time (by env) Build from Dev to Production time 19

20. Automated Deployment Sr Management created a document with 1 page on assumptions and 1 page on milestones to kick off the team on their first assignment. Assumptions included: Standards developed for: build, packaging, config, naming and monitoring Roles and Responsibilities by Environment Deployment success and Environment Exit criteria 20

21. Evaluation Process # Procedure Working group Management P1 principles and assumptions P2 define evaluation criteria X P3 define baseline functionalities and features collect list of products to be evaluated finalize product evaluation form and list to be evaluated X research products based on criteria review/rank products X X X P9 P10 review and finalize list of products for POC vendor demo In house POC X X X P11 review POC results X Description X P4 P5 P6 P7 P8 X X X referrals, forums X 5-7 products vendor documentation, forums and user community X POC should include must-have functionalities defined in F11 below 2-3 vendors 1 product at a time 21

22. Most Important Features Efficient configuration management Host inventory Capability of easy code promotion, backup and rollback Comparison artifact/environment/configuration Orchestration engine reusability and visibility Capability to integrate with other systems Support 22

23. Developing the Requirements Guidelines from Sr Management Review and understand the tools we were using. In-depth analysis of current deployment process Understand our current pain points. Learn from what products are available on the market. Learn from what other people are using and doing right. 23

24. Functional Requirements # Functionality Category importance F1 F2 start application stop application operation operation must-have must-have F3 application configuration management - administration operation must-have F4 F5 F6 F7 F8 F9 F10 application configuration management - versioning application configuration management - comparison application build deployment application build rollback fail-safe protection customizable post-deployment operations single interface for deployment-related operations administration administration operation operation usability operation usability must-have must-have must-have must-have must-have must-have must-have F11 log/report for troubleshooting and auditing administration must-have F12 F13 F14 F15 F16 integration administration administration Operation integration must-have must-have must-have must-have must-have integration with build server Visibility into deployment operation Event and hand-off Notification Scheduled Deployment Integration with configuration management solution 24

25. Functional Requirements # Functionality Category importance F17 integration with ticketing system F18 integration with revision control system integration integration nice-to-have nice-to-have F19 integration with software builder integration must-have F20 F21 F22 F23 F24 F25 F26 integration integration integration operation administration administration usability nice-to-have nice-to-have nice-to-have nice-to-have must-have nice-to-have must-have F27 deploy unbundled artifacts from multiple sources usability must-have F28 F29 F30 F31 usability integration administration usability nice-to-have must-have must-have must-have Integration with testing automation solution Integration with defect tracking solution Integration with availability monitoring database build deployment security and access control mechanism agentless architecture native support for scripting language mobile app deployment clustering database connectivity awareness resource inventory Support both windows/Linux platform 25

26. Evaluation Score Card Danny 3 Carrie 4 Matt 4 SRE 3 Ab Raj Yong total average Vendor support Vito 3 3 3 5 28 3.5 User community 5 4 4 5 4 4 3 5 34 4.25 Documentation 4 5 5 5 5 4 4 5 37 4.625 Cost for license* 1 2 2 1 1 1 2 2 12 1.5? Cost for infrastructure 2 2 3 3 2 2 1 2 17 2.125 Effort for maintenance 3 4 5 5 3 3 1 1 25 3.125 Cost for development and implementation 3 4 3 3 4 3 4 3 27 3.375 Interfaces such as API or web services 5 5 5 3 37 **4.625 Functionality Refer to table 1 5 4 5 4 5 5 5 5 38 4.75 Usability Ease of use 4 4 4 4 4 4 5 4 33 4.125 Learnability 4 4 3 3 4 3 5 4 30 3.75 Product support Cost (lower) Integrability 5 5 5 4 * including derived maintenance charge and yearly support ** scale will be adjusted prior to vendor evaluation phase 26

27. Necessary Standards and Adjustments Change of roles and responsibilities All configurations go into uDeploy and standards across apps. Standardize environment setup and rebuild if necessary. Standardize naming Application/component/environment naming Versioning and release number Configuration key/value pairs Agent/resource naming Automatic smoke testing 27

28. Identifying All Possibilities Source: referrals, communities, and conferences Up to 5 were proposed by each member 28 were identified as potential candidates 3 were given to each team member for initial screening 18 were eliminated for not meeting critical requirements 10 possible candidates moved to the second round 28

29. Choosing the Contenders Evaluation criteria: function checklist and vendor documents Priority ranked 10 remaining products Top 5 for on-site demo and discussion Each vendor was given criteria and use cases Use scorecard to rate and major pros/cons comparison Top 2 to participate on-site POC 29

30. Proof of Concept POC designed by WebMD Selected 2 most representative applications Success criteria use cases and functional requirements Timeline 5 days on-site and 2 weeks afterward Participated in installation, design and implementation Daily working session 30

31. Final Selection Use case verification and documentation In-depth review and discussion Pros and cons comparison The team voted unanimously and chose uDeploy! Evaluation result was communicated to Sr. Mgmt for final review On-site training for the entire Technology organization Demonstration along with highlights of the selection process 31

32. Demo 32

33. Key Benefits of uDeploy Usability, everything accessible in one tool Integration capabilities to existing SDLC tools Application Configuration Management Ease of promoting builds through environments Orchestration of 3rd party tools and reuse of templates Visibility process, troubleshooting, reporting, approval and notification 33

34. After Deployment Metrics Duration March/4/2013-June/3/2013 75 components developed around 30% implemented 419 agents in use 1070 build deployments 65 work days 15.1 per day Daily max: 43 on May/29/2013 Average - End to end time 3 minutes 11 seconds 34

35. Lessons Learned Change is hard - redefining roles and responsibilities Bottleneck just shifted forward, on to the next challenge. DevOps culture - collaboration between Dev, Ops and QA. Ease of use, less technological expertise required When a piece of software is the centerpiece of your SDLC, its worth the work to find the right solution. Cross functional teams work well when they have clear direction and authority from management. Integration with other tools is key. Defining the process, policy and standards are as important as selecting the right tool. 35

36. And now how here we are. 36

37. Appendix 1 roles and responsibilities # SDLC environment Devint QA01/QA02 QA00 Production t1 t2 Environment Prep initiation (ticket or email to identify need) capacity evaluation/deployment target designation for new application Dev Dev/Ops QA Dev/Ops/QA-PERF QA Dev/Ops/QA-PERF Ops Dev/Ops/QA-PERF t3-1 t3-2 Dev Dev ProdOps ProdOps ProdOps ProdOps ProdOps ProdOps t3-3 t3-4 t3-5 t4-1 t4-2 t4-3 t4-4 t4-5 Server provisioning request database request mssql storage request DNS/VServer provisioning request middleware installation setup new uDeploy component template for applications update existing uDeploy component template for applications application related config variables/definitions app related config values***** middleware related config variables/values***** Dev Dev HPSA - ProdOps and dev self service Dev; signed off by Ops/QA Dev; signed off by Ops/QA Dev; signed off by Ops/QA Dev Dev ProdOps ProdOps ProdOps n/a n/a n/a QA ProdOps ProdOps ProdOps ProdOps n/a n/a n/a ProdOps ProdOps ProdOps ProdOps ProdOps n/a n/a n/a ProdOps ProdOps t4-6 t5 t6 t6-1 t6-2 t6-3 t7 snapshot approval for deployment people who push Deployment button Troubleshoot if deployment fails Automatic smoke testing Troubleshoot if smoke testing fails Testing and Sign-off snapshot implementation will be based on discussion with professional service Dev Dev Dev uDeploy/Selenium n/a Dev QA QA QA uDeploy/Selenium QA QA QA ProdOps ProdOps**** uDeploy/Selenium ProdOps**** QA Ops management ProdOps ProdOps**** uDeploy/Selenium ProdOps**** QA *template includes component and process templates, config key/value pairs **e.g. functional testing for QA01/02; UAT for QA00 ***based on promotion approach; will revisit during on-site training as well as professional service engagement ****party initiates communication and troubleshooting, which will involve Dev/Ops/QA ***** values will be contributed by all parties 37

38. Appendix 2 provisioning process # 1 Setup uDeploy access** and perform uDeploy upgrade Step initiator CI 2 Setup devint based on Ops standard Dev 2.1 Provision new VMs - devint Deploy infrastructure software and servers 2.2 (IIS/Tomcat/Memcached/ActiveMQ) -devint 2.3 Provision DNS and Netscaler rules - devint owner PlatformOps ticket firstaid ticket#0 platformOps uDeploy ticket Dev SiteOps firstaid ticket#1 sysops host provisioning Dev ProdOps firstaid ticket#2 ProdOps general Dev ProdOps firstaid ticket#3 ProdOps general 2.4 Provision Storage - devint Dev StorageOps firstaid ticket#4 sysops storage provisioning 2.5 Provision DB - devint Dev DBOps firstaid ticket#5 DBOps general Dev Dev/ProdOps 3.1 naming standardization Dev/ProdOps Dev/ProdOps 3.2 Component mapping Dev/ProdOps Dev/ProdOps 3.3 Server inventory Dev ProdOps 3.4 Review/confirm server inventory Dev/ProdOps Dev/ProdOps 3.5 Install uDeploy agents Dev/ProdOps Sysadmin 3.6 Component/environment mapping Dev/ProdOps Dev/ProdOps 3 Application/component/environment population Dev/ProdOps Dev/ProdOps 4.1 Collect/Review/consolidate application config 4 Application config Dev/ProdOps Dev/ProdOps 4.2 Designate Config namespace Dev/ProdOps Dev/ProdOps 4.3 Import config Dev/ProdOps Dev/ProdOps 4.4 Build server configuration Dev Dev Dev Dev/ProdOps Dev Dev/ProdOps Dev Dev/ProdOps/DBOps Dev/QA/DBOps/ProdOps Dev/QA/DBOps/ProdOps QA firstaid ticket#6 sysops general Dev/ProdOps 5 Process Development 5.1 Develop Application Process if necessary Develop or Reuse component template (IIS/Tomcat/Windows 5.2 service/DB deployment,,etc) Designate roles for each environment (configuration, deployment, 6 approval)*** 7 Set up additional notification if necessary; with product email alias ** provision user accounts and assign members to each privileged groups as shown in table 2 below *** admin for each environment will be responsible to designate roles 38

39. Appendix 3 - guidelines Creation of automation templates should be done at the rollout of the tool and not for each product or application. Types of templates may include Applications, Configurations and DB changes. Before a build is automated for the first time through the tool in DevInt, an existing template must be identified or a new template must be developed. If a new template is needed, there should an agreement that none of the existing templates are appropriate and involvement from a cross functional team to develop a new one. The Automated Deployment Process begins in the DevInt environment. The exit criteria from each environment should be at least one successful automated deployment using the prescribed tool. Success is defined as a successful unit test in DevInt or a successful smoke test in QA. Until the tool is identified, assumption is that the configuration lives with the build. When QA is referenced below, the assumption is that it is a member of the new Deployment Management team not a general QA analyst. We need to define, review and agree to a set of standards with regards to builds, build packaging, configurations, naming and monitoring to ensure the success of the automation deployment process. The groups listed below each environment have the responsibility of creating and executing the deployment and first level troubleshooting within that environment. 39

40. Appendix 4 demo summary Application: https://deploy.webmd.net/#application/73d7afd9-ca16-41d8-b67c-5ec00857d0c2 Quick summary of state of component/environment and configuration Component: https://deploy.webmd.net/#component/ac0dbc11-ce86-4b87-9623-661575815d2f History view of who did what where. Environment: https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b9-94c5df514ee4 Where you see version history and request details Host inventory https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b9-94c5df514ee4/inventory Application property https://deploy.webmd.net/#application/d7de8f89-d861-458d-81c5-1778f7360dcb/properties Component property https://deploy.webmd.net/#component/ac0dbc11-ce86-4b87-9623-661575815d2f/properties Environment and environment-component property https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b994c5df514ee4/properties Global/system property: https://deploy.webmd.net/#system/properties More to come process, resource, version - properties/configuration management Application process https://deploy.webmd.net/#applicationProcess/222e0a13-874d-4df1-b5ac-87e69af6f360/13 Manage component operations during deployment sequence, lock, decision point and approval Component process https://deploy.webmd.net/#componentProcess/9b7208f5-3190-486c-994e-0922075f7d28/43 Where deployment actions take place stop iis, check application pool, download artifact, search and replacement, start iis Component process template Environment specific process https://deploy.webmd.net/#environment/08dd45de-a528-46ee-b07e5bbf527616c4/approvals Plugins Rest API Everything is versioned! Snapshot https://deploy.webmd.net/#application/d7de8f89-d861-458d-81c5-1778f7360dcb/snapshots 40

41. Appendix 5-1: application view 41

42. Appendix 5-2: process template 42

43. Appendix 5-3: component view 43

44. Appendix 5-4: component process 44

45. Appendix 5-5: Component property 45

46. Appendix 5-6: component-environment view 46

47. Appendix 5-7: environment property 47

48. Appendix 5-8: environment inventory 48

49. Appendix 5-9: artifact view 49