AW15 Agile Development Concurrent Session 11/12/2014 4:15 PM
"Putting Quality in the Driver’s Seat with DevOps and ATDD"
Presented by:
Adam Auerbach Captial One
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Adam Auerbach is the Technology Director for Advanced Testing and Release services for Capital One Financial Corporation, a diversified bank with 65-million customer accounts worldwide and more than 900 branch locations. Adam is responsible for Capital One’s enterprise performance and automated testing departments as well as enterprise release management. Since joining Capital One, he has provided leadership for the agile transformation of their quality assurance group and led the enterprise adoption of DevOps and ATDD. Before joining Capital One, Adam was with Chase and other financial and insurance companies, in various leadership positions focusing on quality and agile practices.
About Me
• Technology Director at Capital One – @Bugman31 | [email protected]
• Responsible for Capital One’s Advanced Testing and
Release Services Teams, which include: – Performance Testing
– Automated Testing
– Release Management
• Provides leadership for the enterprise adoption of DevOps and ATDD.
• Before joining Capital One in 2013 Mr. Auerbach was with Chase and other Financial and Insurance companies, in various leadership positions, with a focus on Quality and Agile practices.
As we adopted Agile, we recognized that we still had opportunities for improvements
Development Sprints
Testing Sprints
Integration and Hardening
Key Observations: • Working features delivered every couple months
• Integration of code happened in later sprints
• Environment availability was top impediment
• Automated Tests built and maintained by dedicated team
• Stories took several sprints to complete
To drive adoption, we created a team to own the implementation
LOB System Teams
LOB Accountable
Execs
Enterprise SWAT Team
Some lessons learned from creating these new teams
1. Treat as a new Agile Team – All team members are properly trained – Team members are fully dedicated – Ensure access to all tools and systems needed
2. Leverage a sprint 0, to focus on grooming backlog and understanding
of platform supporting
1. Focus on early outcomes to develop credibility and gain momentum
With System Teams in place, we still needed more…
Develop Build Test
Develop Build Test
Hardening
Key Opportunities: • Teams still working in silos (PO, Dev, Testers)
• Stories too large to get done in single sprints
• Automation occurs outside of feature teams, technical debt was piling up
• No improvements on multiple monthly lead time
• Industry practice in which whole team collaborates on Acceptance criteria and the definition of done
• Automate acceptance tests while production code is being developed – Automation completes before development
• Developer focuses on making acceptance tests pass • Tests become part of build pipeline and are run throughout the sprint
Acceptance Test Driven Development (ATDD)
• Acceptance Tests are customer tests that demonstrate that the business intent of the system are being met • Users point of view • External view of the system
• Examine externally visible effects • Inputs and outputs • State changes, Business rules • External Interfaces
• Implementation independent • Guide development to design cleaner code
What are Acceptance Tests?
The overall ATDD process incorporates BDD and TDD principles
Product Owner/Business Analyst
Product Owner/Developer/Tester
Developer
Tester
Tester/Developer
User Story
Expand Scenarios/
Steps Cucumber
Gherkin
Pair “Show Me”
AUT
Acceptance Test
Feature Files Cucumber
Gherkin(BDD)
Develop
User Acceptance
Automate Tests
Step Definition Ruby Code Libraries
1. Author the tests • Customer (PO/BSA) with tester, developer together • Tester and developer can create more and have it reviewed by
customer
2. Connect tests to the system • Developer or technical testers by writing short bits of code
3. Run the tests
• Developer, then tester • Manual or Automated (however Automated acceptance test allows
them to be run as regression tests to ensure new features do not interfere with previously developed features)
Who does what?
• Tighter cross-functional Team integration – Because the PO, BSA, Dev, Testers are involved in Acceptance Test creation there is tighter cross-functional integration
• Workflows working first time the system was brought up, Getting business rules right prevents rework
• Crisp Visible Story completion criteria – visibly demonstrate that the story is complete
• Not a line a code is written that does not provide business value
• Acceptance tests can be used as Regression tests at the end of the sprint
• Run suite daily, CI/CD practices (find out what's broken faster)
Here are some benefits we observed from teams using ATDD
Success Criteria for implementing ATDD
Criteria Success Condition
Training All Training for BSA's, Developers and testers is complete including installation, hands on practice, development, execution and refactoring
Gherkin Approved Gherkin format to be used by all BSA's Train all BSA's and commitment from BSA Review and approved actual Gherkins/stories
Competency BSA are able to write Gherkin features to the specified standard Functional Testers able to develop and execute overall 80% of scripts by End of XX sprints or release/PSI
Software Availability of all required Software in EAPL, Ruby 2.x, Cucumber, Ruby Mine, Selenium etc..
Environment/Infrastructure Stable Environment for continous testing, Infrastructure for parallel and distributed execution using GRID including support for multi browser and mainframe/Terminal Emulator (windows environment)
End-End flow Should be able to automate an end-end flow, including all frontend, backend and middleware validations with E2E integrations
ATDD Workflow
BSA, PO - write stories and give basic Gherkin features Tester/Developer - Elaborate scenarios/Acceptance tests Tester/Developer - Step definitions 3 Amigo's Tester - Ruby Automation scripts Exeucte Automation Script Developer - Buld/TDD-refactor until Acceptance test passes Refactor Acceptance tests Integration Test
TDM - Test Data TDM team should be able to read gherkin and provide test data aligned with the gherkin CI/CD Integration Invoke/Execute/Report after every code drop
Governance Model Established governance model for code reviews, peer reviews, guidelines, executed and signed off
Release Management
Quality Assurance, Test Automation &
Performance Validation
Quality Driven Delivery puts Quality in the driver’s seat
After implemented Quality Driven Delivery, we were able to see significant benefits
Before After Features implemented every couple months
Working features delivered to production more frequently
Integration of code happened several sprints after initial development
Integration happens multiple times a day
Environment availability was top impediment
Environment issues have limited impact on our teams
Automated Tests built and maintained by dedicated team
Automation is owned by teams
Stories took several sprints to complete Stories are small and completed each sprint.