agile test automation strategy, v2

MJ Half-‐day Tutorials 10/3/16 13:00

Test Automation Strategies for the Agile World

Presented by:

Bob Galen

Velocity Partners

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/

Bob Galen Velocity Partners An agile methodologist, practitioner, and coach, Bob Galen ([email protected]) helps guide companies in their adoption of Scrum and other agile methodologies and practices. Bob is a principal agile evangelist at Velocity Partners; president of RGCG; and frequent speaker on software development, project management, software testing, and team leadership. He is a Certified Scrum Coach, Certified Scrum Product Owner, and an active member of the Agile and Scrum Alliances. Bob published Scrum Product Ownership–Balancing Value from the Inside Out.

1

Test Automation Strategies for the Agile World

Bob Galen President & Principal Consultant

RGCG, LLC [email protected]

Copyright © 2016 RGCG, LLC 2

Introduction Bob Galen n  Independent Agile Coach (CEC) at RGCG, LLC n  Director, Agile Practices at

n  Somewhere ‘north’ of 30 years overall experience J n  Wide variety of technical stacks and business domains n  Developer first, then Project Management / Leadership, then

Testing n  Senior/Executive software development leadership for 20+ years n  Practicing formal agility since 2000 n  XP, Lean, Scrum, and Kanban experience n  From Cary, North Carolina

Bias Disclaimer:

Agile is THE BEST Methodology for Software Development…

However, NOT a Silver Bullet!

2

Outline

n  Traditional Automation – Business Case & ROI n  3-Pillars n  Agile Test Automation Pyramid n  Agile Automation – Business Case & ROI n  Implementation Strategy n  Communication n  Wrap-up

Copyright © 2016 RGCG, LLC 3 3

Let’s start with… Traditional Automation Strategy n  What are your current strategies towards:

q  Test Automation q  Frameworks q  Tooling q  Maintenance q  ROI q  Team structure

n  Get together in “pairs” and chat about this for 20 minutes. List your approaches and tactics.

n  Then leverage SWOT or FFA analysis to capture your current position. Then we’ll gather your results…


3

Let’s start with… Traditional Automation Strategy n  SWOT – Often used to analyze strategic direction or

positions. List the “position”, then identify aspects of: q  Strengths q  Weaknesses q  Opportunities q  Threats

Often in Quadrants

n  Force Field Analysis – again, list a position q  Forces (FOR) è q  Forces (AGAINST) ç


Traditional Automation Business Case & ROI


4

It’s simple really…

n  Manual testing is: q  Slow q  Labor intensive q  Not intellectually stimulating q  Error prone q  Iterative q  Did I say slow?

n  And we’re testing software…right? q  So we ought to be able to automate the testing.


It’s simple really…

n  Automated tests are: q  Fast q  Low/no errors q  Free after initial development q  Run continuously q  Did I say fast? q  Did I say ‘free’?

n  And we’re testing software…right? q  So we ought to be able to automate ALL of the testing.


5

9 Copyright © 2016 RGCG, LLC 9

ü  Capture size of the team ü  Cost per tester – usually hourly ü  # of Test cases to be run ü  Time to run them ü  Cyclical nature of the testing cycles (regression, integration,

system, etc.) ü  Configurations ü  Number of tests run per release ü  Number of releases per year

n  Do the math to come up with iterative testing run-time investments…and costs

Test Automation ROI Sample Points – Manual Testing


n  Then contrast that against the cost for automation: ü  Time / cost to establish automation infrastructure ü  Time / cost to establish automated tests ü  Time / cost for ongoing maintenance and results analysis ü  Cost for automation tooling – depends on ‘type’, ex: Keyword

Driven ü  Cost per automation developer vs. manual tester ü  Automation replacement target for manual test cases

n  Do the simple, discrete math to come up with cost & time

to replace manual testing run-time and ROI savings

Test Automation ROI Sample Points – Automation Development

6


n  Run more tests, faster and with fewer people (lower costs)

n  VALUE being driven by: Running tests! n  Did I say with fewer people (testers)?

q  We can hire more developers with the cost savings! And get MORE done…

q  We can even reduce the size of the bathrooms J

Traditional Business Case


http://www.aspiresys.com/testautomationroi/index.php#ROI

7


http://www.aspiresys.com/testautomationroi/index.php#ROI

Sample of Test Automation ROI calculators n  Aspire Systems seems to have pulled down their link,

here are other samples…

q  http://sourceforge.net/projects/ta-roi/

q  http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=58:roi-calculator&catid=37:calculators&Itemid=65

q  http://www-01.ibm.com/software/rational/offerings/testing/roi/tool/ROI_Rational.html


8

But often there is underestimation of risks… So the ROI was always a ‘fuzzy’ estimate §  Underestimating the nature of a software project

§  Architecture & design, complexity, risks, unknowns, ambiguity, etc.

§  Underestimating parallel software projects (product + automation) complexity §  Integration, dependencies, coordination

§  Underestimating ongoing support & maintenance §  Repairs & updates §  Designing and adding new tests §  Impact of new product technologies

§  Underestimating skill levels for automation development


It’s also very dangerous! Will Robinson… §  It’s commoditizing your testing activity. Could you imagine –

§  Measuring developer value by LOC…and doing ROI? §  Measuring architectural integrity by complexity analysis…and doing

ROI? §  Measuring product owner value by number of backlog elements…and

doing ROI?

Based on some sort of automation introduction? Of course not. It’s insane.

§  Then why do we do it with tests…and testers?


9


n  We’re all using them, so they must be good and the approaches must be ‘correct’…right?

n  However, there are challenges— q  Training q  Size q  UI-centric q  COST q  Can’t test everything (Back-end database code) q  One-size fits all strategy q  Limited developer / whole team usage

Traditional Test Automation Tools Just a quick stop…

Typical Deployment Level of effort


Initial design & Architecture

Initial Test Case Automation

Technology Disruption

Ongoing Maintenance

MORE

Typical Time & Budget Investment

LESS

é Level of Effort -

Investment ê

10

Now it may sound…

n  Like I’m opposed to test automation. Truly, I’m not. I think automation rocks! Or like I’m opposed to gaining a ‘return’, absolutely not. q  Automation is clearly a foundation

element for increased quality and going faster – releasing more often

n  But, this traditional model or approach of focusing on ROI…is sort of insane


3-Pillars of Agile Quality & Testing


11


3-Pillars Genesis

n  I’ve learned that “Balance” is important n  A sad tale of:

q  Thousands of ATDD testing; Gherkin run amok q  All of them are working; continuously testing; increasing

“coverage’ and life is Good!

n  BUT q  These same teams couldn’t write a cohesive User Story to save

their life q  So, where were the Acceptance Tests coming from?

3-Pillars of Agile Quality


Development & Test Automation

•  Pyramid-based Strategy:

(Unit + Cucumber + Selenium)

•  Continuous Integration

•  Attack technical infrastructure in the Backlog

•  Visual Feedback – Dashboards

•  Actively practice ATDD and BDD

Software Testing

•  Risk-based testing: Functional & Non-Functional

•  Test planning @ Release & Sprint levels

•  Exploratory Testing

•  Standards – checklists, templates, repositories

•  Balance across manual, exploratory & automation

Cross-Functional Team Practices

•  Team-based Pairing

•  Stop-the-Line Mindset

•  Code Reviews & Standards

•  Active Done-Ness

•  Aggressive Refactoring of Technical Debt

•  User Stories, “3 Amigo” based Conversations

•  Whole Team Ownership of “Quality” •  Knowing the Right Thing to Build; And Building it Right

•  Healthy – Agile Centric Metrics •  Steering via: Center of Excellence or Community of Practice

•  Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement

12

Foundation of the 3-Pillars


•  Whole Team Ownership of “Quality”

•  Knowing the “Right” thing to Build AND Building it “Right”

•  Healthy – Agile Centric Metrics

•  Steering Required – CoE or CoP

•  Strategic balance across 3 Pillars; Assessment,

Recalibration, and Continuous Improvement

•  Whole team view includes building it right, everyone tests, everyone demo’s, etc.

•  Focus on features/stories, confirmation, conversation, and getting them staged

properly OVER testing

•  4-tier metrics: Quality, Value, Prediction, Team

•  Agile strategies need light-handed “steering”; establish a CoE (heavier weight) or a CoP

(lightweight)

•  Consider finding an assessment framework and then tying it to your strategy

measurement, recalibration, and continuous improvement.

•  Make the foundation visible thru information radiators and metrics



Development & Test Automation

•  Pyramid-based

Strategy: (Unit + Cucumber + Selenium)

•  Continuous Integration

•  Attack technical infrastructure in the

Backlog

•  Visual Feedback – Dashboards

•  Actively practice ATDD and BDD

A central part of agile adoption is focusing on CI, 3-tiered Automation development, and Dashboards to

begin incrementally building coverage for faster feedback on changes.

100% automation is NOT the Goal!

In the interim, Hardening or Stabilization Sprints and

having a risk-based Release Train concept help

It’s important that Test or QA not ‘own’ the tooling or all of the automation efforts. The strategy can come from QA, but the tactical automation development is

best left to the team.

Mature teams invest in Automation, Tooling, and Technical Debt reduction as part of Done-ness and

continually add it to their backlogs

13



Software Testing

•  Risk-based testing: Functional & Non-

Functional

•  Test planning @ Release & Sprint levels

•  Exploratory Testing

•  Standards – checklists, templates, repositories

•  Balance across manual, exploratory &

automation

Exploratory Testing (SBET with pairing) can be an incredibly effective way to establish a whole-team,

collaborative view towards quality and testing. It also emerges new tests.

Leverage ‘plans’ as a whole-team collaboration-conversation mechanism; at Sprint and Release

levels.

Do not measure testing or tester progress; instead, measure throughput, output, sprint outcomes, and

done-ness escapes at a team level.

You need a balanced test team; not everyone needs to be able to program. But everyone needs to be

passionately skilled testers with curiosity.

Agile testing is a Risk-Based play in every Sprint and across a release sequence.



Cross-Functional Team Practices

•  Team-based Pairing

•  Stop-the-Line Mindset

•  Code Reviews & Standards

•  Active Done-Ness

•  Aggressive Refactoring of Technical Debt

•  User Stories – 3 Amigo based Conversations

One of the hardest areas to get ‘right’ culturally. It needs leadership alignment from Quality/Testing to Product to Development and a consistent voice of

whole-team approaches.

This is where LEAN Thinking lives, where whole-team collaboration happens, where professionalism

and craftsmanship are held dear.

I like the view of testers becoming the VOC, champions of quality, and consistent questioners of

what is being build. Are we solving the right problems…as simply as possible. Notions of Minimal

Viable Product / Feature help with focus.

And yes Virginia, there ARE standards, templates, and a focus on x-team consistency!

14

Agile Test Automation Pyramid



Agile Test Automation Pyramid n  Coined by Mike Cohn, ~ 2005 n  Establishes the validity of turning Traditional Automation

– upside down

n  Invests: q  Most effort (%) in Unit Tests q  Moderate effort (%) in Middle-tier tests (business logic, API,

component, feature) q  Least effort (%) in UI-centric, top down tests

n  Granularity of the tests is part of the strategy

15


Agile Test Automation Pyramid Often: n  Tied to Definition-of-Done from a maintenance

perspective q  You break it, you fix it

n  Percentages vary; intent does not n  Whole-team ownership of automation

q  Although it skews at either end of the pyramid n  Run as much as possible

q  Check-in, Overnight, Cyclically q  Prime Directive = Real-time Feedback

Agile Test Automation Pyramid


+80%, UI Centric Automation

0-10%, Service or Middle Tier Automation

0-10%, Unit level Automation

0-10%, UI Centric Automation

20-40%, Service or Middle Tier Automation

50-60%, Unit level Automation

Traditional Agile

16

Agile Test Automation Pyramid Mike Cohn; Lisa Crispin & Janet Gregory http://behaviordrivendevelopment.wikispaces.com/Testing


Definition of Done iContact example

n  As part of story design, consider 3-levels of automation required for solid implementation: q  Unit q  Cucumber q  Selenium

n  If the team warrants the automation has value (ROI) then automate it

n  Implement those automated test cases WITH the story n  Demonstrate them in the Sprint Demo


17

Definition of Done iContact example

n  Previous sprint automation needs to continue to work; if you break it…then Fix It!

n  Previous CI/CD – automation wiring needs to continue to work; if you break it…then Fix It!

n  All appropriate (RBT) Acceptance Tests should be regressed so that we haven’t “lost value”

n  And in Readiness Criteria q  Automation research spikes and architectural look-ahead


Brainstorm… Agile, Multi-tiered Automation n  Get together in small groups of 4-6 to discuss

n  Take a few minutes and think about your current automation approaches: q  Tooling, approaches & strategies, strengths, weaknesses,

opportunities, maintenance challenges, future technology, etc.

n  What sorts of adjustments would you need to make to take this approach?

n  What would be the largest challenges in taking this approach? How might you overcome them?

n  Do you “buy” the whole-team view to automation?


18

Agile Test Automation ROI


What agile has exposed to me… wrt/Automation & Testing n  Testers vs. a Whole Team view

q  Build teams in ratios – developers vs. testers q  Instead invest in cross-functional teams composed of the skills to design,

construct, test, and deliver excellent products

q  Only testers write test automation q  The entire team writes test automation and, oh by the way, can test as

well

q  Only testers test the application q  The whole team participates in writing / running automation, functional

testing, exploratory testing, regression testing, etc.


19

What agile has exposed to me… wrt/Automation & Testing n  Traditional vs. Open Source Tools

q  Instead of leveraging singular, UI-centric tools q  Leverage multiple tools at all-tiers of your application stack

q  Higher cost q  Lower cost and a sense of “Community”

q  Developers won’t use them; costs q  Everyone is leveraging the same tool-sets; AND integrating them in

Continuous Integration and Continuous Deployment


What agile has exposed to me… wrt/Automation & Testing n  People are your primary value proposition

q  Instead of looking at testers as a commodity q  Consider good testers to be as valuable as any other team member

q  Instead of thinking that testing is a by-rote exercise with little intellectual pursuit

q  Testing is a profession. Testers are valuable craftspeople. Testing is an intellectual exercise.

q  Instead of thinking the value is in the test cases q  The value is in the testers (people) continuously reevaluating the right

things to test at the right time…for valuable feedback


20


http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html

The NEW Test Automation Business Case n  So, don’t

q  Count pennies q  Count heads q  Count cycles

n  Work as part of a TEAM q  Focus in on testing continuously – just the right things at just the

right time q  Providing feedback, transparency, and reaction q  Allowing your teams time to learn and improve q  Solving your customers’ problems; delivering value


21


n  Run more tests, faster, unattended, continuous feedback, learning & adjustments q  Yes, saving time. Now what to do with it???

n  Drive your new VALUE by: q  Early feedback for adjustments—risk avoidance, learning, solving

real problems q  Having more time to continuously refine your test cases

(automated and not) q  Having more time to focus on building Quality into your products q  Creatively testing the application (Exploratory Testing, Test Idea

Generation, Test Design)

The NEW Test Automation Business Case

So, with the additional time… INVEST in your testers & quality ü  In building In Quality ü  In attacking the customers REAL problems ü  In better testing ü  In more balanced automation ü  In team training; in slack time ü  In better collaboration ü  In doing more than thought possible ü  In creative solutions ü  In adapting to new technologies ü  In having FUN


22

How do we “Sell” this change? n  You don’t sell it from a testing perspective. Nor from a bean-counting perspective.

n  You allow the results to begin speaking for themselves as we begin focusing on and talking about— q  How one builds truly innovative and creative software teams q  That understand and solve customer problems q  With simple solutions that fundamentally work q  And do so iteratively & quickly, while nimbly adjusting to

customer feedback


But Bob…

n  That doesn't work in the “Real World”!

n  My boss would never go for it without ROI justification and hard measures

n  Testing IS a commodity

n  We’ve done automation perfectly – seeing none of the issues/risks you’ve pointed out


23

But Bob…

n  Think about what I’m saying; taking the time and…

q  Invest in people q  Invest in better quality & testing q  Invest in better product solutions to customer problems q  Invest in improved workflow and delivering high customer value

WOW – how crazy is that? n  Meet in small groups of 2-3 (pro / con) and chat for 10

minutes about the “sides” and the possibility for change? n  Then we’ll debrief…


Agile Test Automation Implementation Strategy


Automation is one of the foundational investments of effective Agile Adoptions. It’s not a choice, it’s an imperative! It’s a cost of doing business.

24

Implementation Strategy

Typically a 5-Phased Strategy

1.  Strategy – This session: 3 Pillars + Pyramid + Team

2.  Team Formation 3.  Tools Selection 4.  Training & Infrastructure Development 5.  Automation Development 6.  Care & Feeding


Phase 2 Team Formation n  GOAL…whole team ownership n  Skill-set discussion; SDET vs. Test Professional? n  May influence

q  Development focus on Unit Testing q  Automation team to “gain momentum” q  Agile execution (Scrum, Kanban, transparency, demo, etc)

n  Connect the dots architecturally n  Bias: I like to lead automation with a Test Automation

Architect q  Rather than with someone from the development team


25

Phase 3 Tools Selection n  Open Source vs. Commercial n  Singular, all-purpose vs. Tool-kit

q  Cobble together or Integrate disparate tools

n  Always perform a “Bake off” within your teams n  If Open Source, consider support implications n  Connect to development tooling

q  Including CI and CD

n  Additional Considerations: Test data, virtualization, DevOps

n  Be prepared to iterate, learn, and possibly fail


Phase 4 Training & Infrastructure n  Design, coding, and documentation standards n  Templates n  Community of Practice

q  Example code, standards, forum, collaborative groups

n  Models: q  Keyword, Data-driven, Table-driven, DSL, Model-driven, others q  Test Suite; Hierarchy, Naming

n  Commercial: invest in training and/or consulting n  Open Source: hire and/or acquire direct expertise

q  Also can build the expertise


26

Phase 5 Automation Development n  Move it to your teams ASAP n  Make it a part of Definition-of-Done

q  It should become a natural outcome and not a choice or question

n  Complete it within each sprint q  SAFe as a goal model for completing work in the same sprint! q  Salesforce as well

n  It needs to be a negotiated part of your Product Backlog n  Have % targets; n  For every story:

q  The TEAM decides the requisite automated tests at each level of the pyramid?


Phase 5 Automation Development Attacking the Pyramid

n  Usually I attack the Unit-level first q  Development team engagement; sheer #’s q  Baseline quality improvement q  Integrated with CI/CD; fast feedback

n  Then select either middle or UI tier; context based q  Tools selection q  Training q  Framework development


27

Phase 5 Automation Development Attacking the Pyramid

n  Complete one layer nearly completely before moving to the next layer

n  Unit Test Strategy – Don’t dig the “hole” any deeper q  Legacy vs. New q  Encapsulate vs. Proper unit tests q  Refactoring implications


Phase 6 Care & Feeding n  Definition of Done & Product Backlogs n  Automation evolution roadmap

q  Merge with your architectural look-ahead q  Merge with your Product Roadmap q  Factor in ongoing automation investments

n  Refactoring & Repairs as part of your Technical Debt handling

n  Consider it at the same level of investment as your application code

n  Stop-the-Line n  Commitment


28

Communication

n  Leadership Vision q  High-level Plans, Strategic sharing with your teams q  Leadership collaboration (Development + QA) q  Intent to doggedly pursue automation

n  Information Radiators n  Roadmaps & Product Backlogs n  Sprint & Release Reviews

q  Even though it’s hard to show “working software” q  Let the “process” transparently communicate your progress,

impediments, plans, challenges


Let’s end with… Agile Automation Strategy n  Based on what you’ve heard, discussed, learned…

n  Get together in “pairs” or small groups and chat about this for 20 minutes. List your strategic approaches. Then leverage FFA to capture your driving forces FOR and AGAINST moving towards Agile Automation

n  Most importantly, focus on strategies to overcome your obstacles.

n  Then we’ll gather your results…


29


Copyright © 2016 RGCG, LLC

Wrap-up THANK YOU!

§  I hope I’ve added to your previous assumptions and understanding §  Inspired you to change your view towards Automation

ROI and investment §  Introduced you to models for attacking Agile Automation Strategy and ROI

§  Final questions or discussion?

58 58

30

Contact Info Bob Galen Principal Consultant, RGalen Consulting Group, L.L.C.

Experience-driven agile focused training, coaching & consulting

Cell: (919) 272-0719 [email protected] www.rgalen.com

@bobgalen https://www.linkedin.com/in/bobgalen

Podcast on all things ‘agile’ -

http://www.meta-cast.com/


References


31

Links

n  http://www.agileengineeringdesign.com/2012/01/7-deadly-sins-of-automated-software-testing/

n  Link to PDF of Pillar-1 chapter n  Links to “old” STP 3-part automation series n  Link to Mary Thorn’s – Agile Test Manager article n  + other materials, join the mailing list for download

password at - http://goo.gl/ORcxbE


Open Source Tools

n  Unit Level q  x-Unit q  Junit, nunit, phpunit,

n  ATDD q  FitNesse, Robot Framework

n  BDD q  Cucumber, JBehave

n  UI q  Selenium, Watir, HP-UTF


32

…DD much from the XP space n  TDD - Test Driven Development

q  Test-first; Unit Tests as a design aid q  Debate around creating the unit tests “first”

n  ATDD - Acceptance Test Driven Development q  User Story – Acceptance Tests q  Executable Requirements

n  BDD - Behavior Driven Development q  jBehave, Cucumber; Gherkin – Given, When, Then q  More useful phrasing

n  Tests as functional documentation n  Product Owners, Customer – writing tests


Agile Testing Quadrants Brian Marick; Lisa Crispin & Janet Gregory


Exploratory testing Scenarios

Usability testing UAT

Alpha / Beta

Unit tests Component tests

API tests

Functional tests Story tests Examples Prototypes Simulations

Performance testing Load testing

Security testing Non-functional requirements

Q1

Q2 Q3

Q4

Automated & Manual

Automated & Manual

Manual

Automation, Tools, and

Manual

Business Facing

Technology Facing

Sup

porti

ng th

e Te

am C

ritique the Product

agile test automation strategy, v2

Documents