devops: sprinkle dev, sprinkle ops, let's make cake, not mud pies
TRANSCRIPT
Sprinkle Dev, Sprinkle Ops, lets make Cake not Mud Pies
Brian Paulsmeyer
Agile Gravy: September 29, 2016
Agile Development Pitfalls
… that warn us of mistakes in our agile implementation
05/03/2023www.centricconsulting.com 2
05/03/2023www.centricconsulting.com 3
Waterfall in 2 Week Iterations
•No ordering of stories•Gold plating of features•“Agile Waterfall” Assumptions• Learning from usage of system won’t change features• “All features will come together” in last day of last sprint• “Pause the world for 3+ years for us to finish”• “Value Proposition: All or Nothing”• Architecture can be “known” and not change• Deployments are still hard
•Unfortunate ending for “Agile Waterfall”• “We ran out of money, and have nothing in production.”
Constant Redefining of “Definition of Done”
•Done should simply mean done•Burndown/Burnup no longer reliable•Feature inversion occurs when time spent on defects and maintenance overtakes feature work
05/03/2023www.centricconsulting.com 4
1 2 3 4 5 6 7 8 9 10
20 20 2018 17 17 16 16 15
13
0 0 02 3 3 4 4 5
7
Feature InversionFeatures Defects and Hidden Work
Only focusing on Unit Tests and Manual Testing
•Unit tests• Won’t ensure business workflows• Only allows minor refactoring• Failure: Mars Climate Orbiter (1999)
• Team A: Pounds for unit of force• Team B: Newtons for unit of force (1 lb = 4.45 N)• Disintegrated in Mars atmosphere!• Each “Unit”/Component worked in isolation
05/03/2023www.centricconsulting.com 5
Continuous Delivery and DevOps
… simple concept, requires diligent implementation
05/03/2023www.centricconsulting.com 6
No more “Throw It Over the Wall”
05/03/2023www.centricconsulting.com 7
DevOps enables Continuous Delivery
•Team performs activities across disciplines•Apply principles from iteration 0 through delivery
05/03/2023www.centricconsulting.com 8
Regression
UATIntegration
Testing
DevOps – Full Lifecycle Delivery Pipeline and Container Management
Local Environments
Builds
Unit TestsMerge
Procedures
Log Interpretation
Dependency Mgmt
Dev Deployments
Traditional DEV activities Traditional OPS activities Traditional QA activities
Prod Deployments
Performance Mgmt
Monitoring
QA Deployment
s
Branch Mgmt
Change Mgmt
Log Management
Configurations
Integration Deployment
s
Performance testing
What It Takes
Whether it be security, regulatory, software limitations, etc., there are many reasons why organizations say CD is impossible
05/03/2023www.centricconsulting.com 9
Tooling
Process
Culture
Architecture
Basic Definitions and Levels of Maturity
Continuous…
05/03/2023www.centricconsulting.com 10
Automated Deployment to ProductionDeployment
Manual Compile/Build, Manual Testing, Manual DeploymentPain
Automated Deployment to Integration/Staging/Pre-Prod
Deployment Toolset Constant across all Environments
Delivery Automated Integration/Workflow Tests
Automated Component Builds
Automated Component Tests*
Integration Automated Unit Tests
* Many teams mistakenly consider Continuous Integration (CI) complete with only automated unit tests
Pipeline
•Recipe for building, deploying, and testing the system•Grouping of discrete stages for a complete workflow•Coordinates disparate technologies•Provides feedback of system state across environments
05/03/2023www.centricconsulting.com 11
Business & Customer
Develop
Customer
Deployto
Production
Deployto
Staging
Deployto QA
Buildand Unit Test
Automated Integration /
Workflow Testing
Detailed Pipeline
05/03/2023www.centricconsulting.com 12
Business & Customer
Backlog
Consolidate Features and Hotfixes
Iteration Planning
Build Server“X”
Stories
Development
1 Active Story Per Developer
1Commit
Acceptance Test
Customer
BuildArtifact
Acceptance Test Environment
QA Testing
QA Environment
Auto-Trigger Pipeline:Installs To AIT Environment
Executes Smoke and Acceptance Tests (<10 min)
Manual Trigger Pipeline:Installs To
QA EnvironmentExecutes Smoke Tests
UAT Testing
Manual TriggerPipeline:
UAT EnvironmentUAT Smoke &
UAT Tests
Production
Manual TriggerPipeline:
Prod EnvironmentProd Smoke Tests
UATEnvironment
Product Pipeline
05/03/2023www.centricconsulting.com 13
Pipeline Workflow Tools
… to speed implementation
05/03/2023www.centricconsulting.com 14
Pipeline Stage Implementation
•Pipeline workflow steps trigger external tools
05/03/2023www.centricconsulting.com 15
Business & Customer
Develop
Customer
Deployto
Production
Deployto
Staging
Deployto QA
Buildand Unit Test
Java.Net
TypeScriptPHPetc.
AntMaven
MSBuildMakeetc.
JUnitNunitetc.
BashPowerShellChef/Puppet
VMsDocker/CoreOS
etc.
Automated Integration /
Workflow Testing
SeleniumCucumber JBehaveSpecFlow
Codeceptionetc.
Same Toolsas Deploy to QA
Only ConfigurationDifferences
Same Toolsas Deploy to QA
Only ConfigurationDifferences
Build Tool Jobs (ex. Jenkins)
Key Pipeline Coordination Tool Features
•Pipeline Triggers• Automatic on code check-ins to source control• Manual• Programmatic API
•Workflow• Parallel steps• Sequential steps• Failure-only steps• Graphical view of steps
•Artifact Tracking• Traceability from check-in to final deployment• View of deployed version in each environment• History of deployments• Ability to trigger updates to agile tracking tools (ex. JIRA)
•Environment Support• Same deploy scripts, allows configuration changes per environment
05/03/2023www.centricconsulting.com 16
Pipeline Workflow Tool : Jenkins
•On-Site•Version 2.0 Provides Workflow View•Software Language Agnostic
05/03/2023www.centricconsulting.com 17
Pipeline Workflow Tool : ThoughtWorks Go
•On-Site•Software Language Agnostic
05/03/2023www.centricconsulting.com 18
Pipeline Workflow Tool : Octopus Deploy
• .Net
05/03/2023www.centricconsulting.com 19
Automated Testing
… so every day can be deployment day
05/03/2023www.centricconsulting.com 20
Automated Component/Integration/Workflow Testing
•Opinions of why automated testing isn’t needed• “Manual testing is sufficient”• “Business only wants to spend money on features”• “Developers only want to code features”• “Our <insert technology> is too hard to test with automation”• “Automated tests run too slow”
05/03/2023www.centricconsulting.com 21
Best Case: Missed DeadlineWorst Case: Catastrophic Failure in Production
Result of skipping automated tests
Selenium WebDriver
Direct Test using Selenium WebDriver
WebDriver driver = new FirefoxDriver();// Navigate to webpagedriver.get("http://www.google.com");// Find the Search Field (UI Locator)WebElement element = driver.findElement(By.name(“Search"));// Fill in the dataelement.sendKeys(“something interesting");// Now submit the form. WebDriver will find the form that contains the elementelement.submit();// Assert the test passed
05/03/2023www.centricconsulting.com 22
Selenium WebDriver Web Page
Test
Test
Test
Test
Page Object Model
05/03/2023www.centricconsulting.com 23
• Separate Test Code and UI Locators
• Supports Don’t Repeat Yourself (DRY) for Element Locators
• Less Technical Test Case
SearchPage page = new SearchPage(selenium);// Enter Search informationpage.search(“something interesting");// Now submit the formpage.submit();// Assert the test passed
Page Object Model(s) Web Page
Test
Test
Test
Test
Selenium WebDriver
ServerAPI
BDD Testing
• Gherkin (English) Text File
• Step Definitions Map Between English Test and System
• Business Readable Test Case
Given user searches for “something interesting”When user performs the SearchThen assert something is true
05/03/2023www.centricconsulting.com 24
Page Object Model(s) Web Page
Step Definition
Step Definition
Step Definition
Selenium WebDriver
ServerAPITest
Test
Test
Test
Capabilities
… we need to be successful
05/03/2023www.centricconsulting.com 25
Rapid Response to Changing Business
•Need to be able to react to disruptive technology and business models•Stagnant architecture limits new technology adoption• Changes blocked by fear of breaking system• Inability to deploy new systems to production• Unable to adopt new security upgrades• Unable to adopt new software versions
05/03/2023www.centricconsulting.com 26
Drive Defects From Most Used Features
“You can’t test quality in”
At a minimum make sure defects aren’t in the standard user workflows
05/03/2023www.centricconsulting.com 27
05/03/2023www.centricconsulting.com 28
Adjustable/Changeable Architecture
•Magic Quadrant Architecture• “Best” choice today ≠ Best choice tomorrow• Failed technologies and companies in all parts of magic quadrant
•Natural product End-of-Life requires new architecture•Continuous Delivery and DevOps allows fluid architecture changes
OODB(Object-Oriented Database)
EJB 1.0
Cadence
•What if everyday was a normal day•What if those normal days were release days•What if those release days were multiple releases throughout the day
•Low Risk On-Demand Releases
05/03/2023www.centricconsulting.com 29
Deployment Styles
… for production deployment
05/03/2023www.centricconsulting.com 30
Standard Deployment
•Deployment Steps• Disable user access• Upgrade servers from V1 to V2• Test deployment• Enable user access• Upgrade destabilizes running environment
•Problem Mitigation• Re-Enable backup if possible• Developer troubleshooting requires
customer downtime
05/03/2023www.centricconsulting.com 31
05/03/2023www.centricconsulting.com 32
Blue Green
Router / Load Balancer
Blue-Green Deployment
•Two Identical Production Environments• Blue• Green
•One Active: Router Controlled•Deployment Steps (Green Active)• Deploy to Blue environment – No users
affected• Perform smoke tests on Blue environment• Optional: Switch subset of users to Blue
(similar to Canary releases)• Switch all users to Blue environment
•Problem Mitigation• Don’t switch to 2nd environment• Developer troubleshooting has no customer
impact
Canary
Router /Load Balancer
Canary Deployment
•One Production Environment•Deployment Steps• Deploy to 1 server (Canary)
Optional: Provision and build new server from scratch
• Router sends portion of users to Canary• Random• Internal users only• User-type based (ex. Test users)
• Gradually upgrade servers to V2• Gradually switch users to V2 servers
•Problem Mitigation• Don’t switch additional users to Canary
environment• Developer troubleshooting has no customer
impact
05/03/2023www.centricconsulting.com 33
Case Studies
… that show it can be done
Before Build Automation
•23 Teams• 150 Team Members (Developers, Quality Engineers, Scrum Masters)• 100 Developers
•Service Oriented Architecture• Managers, Engines, Data Access
•Baseline End of PI6 (SAFe)• PI6 demo system setup
• Two weeks to setup• Manual Installations – not repeatable• Individual Teams created “magic instructions” in isolation to install and run solution
• Some teams unable to run full solution• Custom data loaders used to skip workflow steps• Developers unable to test full workflow• Movement of data through workflow steps untested till end of PI
• Update of component versions caused “Day of Downtime” for developers to fix unexpected issues
05/03/2023www.centricconsulting.com 35
After Build Automation: Jenkins Easy Button Builds
• Starting Point: a machine with Windows — that’s it!• The rest is automated:
−Provisioning: verify/install tools takes ~50 minutes− Puppet Agent, PowerShell, 7-Zip, VNC− SQL Server, .Net, Visual Studio− Noticia Repono Client
−Install Custom Software takes 5-10 minutes− Clean system from any previous runs− Install/Configure Base Components− Install System
− Install and Run Smoke Tests
At this point the system is ready to be used to do further testing, demos, etc.
• Deploy cycle repeats
Chun
ky M
onke
yVa
nilla
Definition of Done
… that makes our customers happy
05/03/2023www.centricconsulting.com 37
Continuous Delivery Definition of Done
•Automated Deployments• Fully Automatic or Push-Button Deploys• Same Executable Scripts for all Environments• No Deployment Hero
•Automated Testing• Unit• Integration• Full System
•Always Shippable• No Temporary Code• No Untested Code• Feature Toggles
•One Backlog• Production Bug Fixes• New Features
05/03/2023www.centricconsulting.com 38
About Me
05/03/2023www.centricconsulting.com 39
• Our highest priority is building lifelong relationships with clients based on trust, respect, and collaboration.
• We invest in our talented team and support their well being by keeping them challenged and inspired.
• Our localized company structure allows us to play very active roles in the lives of our families and community.
This relationship-centric focus keeps us passionate, committed and motivated in all facets of our lives. That’s why our team is here to stay.
Centric Consulting
• 17 years as a software developer and architect• Full stack developer in Java, .Net, TypeScript (Angular 2), and PHP• DevOps tooling experience including VMs, Docker, Chef, and Puppet• Continuous Delivery and Automated Testing implementations• Experience across industries including FDA and SEC regulated
Continue the Journey • http://centricconsulting.com/centric-st-louis-scaled-agile-workshop/
Thank You for Attending!