Agile Testing Practices

Megan S. SumrellDirector of Transformation ServicesValtech

Introductions

About us…Now about you…

Your nameYour companyYour roleYour experience with Agile or Scrum?

Personal Expectations

Agenda

IntroductionsAgile OverviewTraditional QA TeamsTraditional Automation ApproachesRole of an Agile TesterTesting Activities

Refine Acceptance CriteriaTDDManual / Exploratory TestingDefect ManagementDocumentationPerformance TestingRegression Testing

Agenda Continued

Test Automation on Agile TeamsTesting on a Greenfield ProjectTesting on a Legacy ApplicationEstimation SessionsSprint Planning MeetingsRetrospectivesInfrastructureSkills and TitlesClosing

Agile Overview

Agile Manifesto

"We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more."

Scrum

Terms and Definitions

User Story: high level requirementsProduct Backlog: list of prioritized user storiesSprint : one cycle or iteration (usually 2 or 4 weeks in length)Daily Stand-up: 15 minute meeting every day to review statusScrum Master: owns the Scrum process and removes impedimentsProduct Owner: focused on ROI and owns priorities on the backlogPigs and Chickens

Traditional QA Teams

How are you organized?When do you get involved in the project?What does your “test phase” look like?What testing challenges do you have?

Traditional Test Automation

Automation Challenges

Cost of toolsHard to learnCan’t find timeMaintenanceUI dependentOnly a few people can run them

Traditional Test Pyramid

UNIT TESTS

Business Rules

GUI TESTS

Will these strategies work in an Agile environment?

Food for Thought…….

“Tests play several pivotal roles during the software development process. First, tests unambiguously communicate how things are supposed to work. Second, they provide feedback on whether the system actually works the way it is supposed to work. Third, tests provide the scaffolding that allows developers to make changes throughout the development process…”

Lean Software Development – Mary and Tom Poppendieck

Role of An Agile Tester

Role of “QA” on an Agile Team

Testing starts with the specification, not verificationFocus on acceptance criteriaFocus on automation, not manual testingUser Stories are not done until the automated acceptance tests are passingForces the team to think about testability at the beginning

Testing Activities

Testing Activities

Refine acceptance criteriaTest Driven DevelopmentManual / Exploratory TestingDefect ManagementDocumentationPerformance TestingRegression Testing

Refine Acceptance Criteria

Requirements vs. Tests

Refine Acceptance Tests

• Who?– Analysts, Testers,

Developers, PO• How?

– Collaborate!– Build executable

requirements if possible• Why?

– Avoids misinterpretation of the requirements

– Full transparency into the test cases

– Sets expectations

ATDD: Acceptance Test Driven Development

Development is driven by examplesAcceptance testing happens before development startsBusiness facing testsRepresent workflows and business logic rules

Some Questions to Ask…….

What else do the developers need to know about this user story?What can go wrong during the story?What am I assuming?Are there circumstances when this story may behave differently?

Workshop: Let’s write some acceptance tests….

User Story: As a mathematician, I want to be able to use the calculator to divide numbers and get the answer so that I can solve problems.

Acceptance Tests:Verify that the division is correctVerify that the answer is in ##.## format

Workshop - continued

User Story: As a user, I want to search for a name so that I can find that person’s phone number.Acceptance Tests:

Search by first or last name‘Like’ match searchResults are in first name, last name, phone number order

Test Driven Development

TDD is not unit testingTDD is not writing tests where it is easy to do itTDD is not testing after you have codedTDD is not manually testing your code

TDD IS ABOUT TESTING FIRST

TDD: Test Driven Development

TDD is not magicTo be effective, tests must be:• Accurate• Appropriate• Sufficient

TDD: Test Driven Development

TDD is a discipline that developers must adoptTesters can help educate developers on how to write good tests

Developers tend to test what worksTesters tend to look for what breaks

TDD: Test Driven Development

testDivideByZero extends TestCase {Calculator calc = new Calculator();try {

calc.divide(6, 0);fail(“No exception was thrown”);

} catch(MathException e) {assertEquals(“Cannot divide by zero”, e.message());

}}

Write the Test

Before:

public int divide(int divisor, intdividend) {

return divisor / dividend;}

Write the Code

Write the Code

After:

public int divide(int dividend, int divsor) {if (divisor == 0)

throw new MathException(“Cannot divide by zero”);

return divisor / dividend;}

Refactor

Remove duplicationImprove implementationSimplify

Exploratory Testing

Exploratory Testing

• Who?– Testers

• How?– Paired testing

• Why?– Ensure all negative or

unusual paths are covered

– Paired testing adds value

Session Based Exploratory Testing

James and Jonathan Bach – 2000Consists of the following:

CharterSessionSession ReportDebrief

“Exploratory testing is simultaneous learning, test design, and test execution.”

-James Bach

Workshop - Create Charters

Think about buying something on-line. What charters would you create for exploratory testing on an on-line checkout system

Defect Management

Defect Management

• Who?– Testers (sometimes

Analysts)• How?

– Verify that unit tests were written

– Add automated tests for the future

• Why?– Ensure that the defects

were fixed– Automate tests to ensure it

doesn’t break again

Documentation

Workshop: Documentation

What do you document on your agile teams?What are your test artifacts?

Eliminate Waste

Ensure all artifacts are valuableDon’t create things people won’t useDon’t Repeat Yourself (DRY)

If a test can take the replace of a requirements document, don’t write both

Don’t do more than is necessaryIf a wireframe sketch is sufficient to describe a GUI, don’t spend the time and money to create a photo-realistic mock-up

Performance Testing

Performance Testing

• Who?– Performance Testers

• How?– Map test cases to

performance requirements

• Why?– Uncover performance

issues early!!!

Regression Testing

How can you regression test every iteration?

Automation (lots and lots of it!)Should happen every dayImmediate feedback when code changes are made

Test Automation on Agile Teams

Agile Test Pyramid

GUI TESTS

Business Rules

UNIT TESTS

Automate Acceptance Tests

• Who?– Testers and Developers

• How?– Collaborate!– Work to automate defined

acceptance tests• Why?

– Allows everyone to track progress

– Ensures requirements are met

– Allows for fast, easy regression testing

More Reasons Why We Tackle Automation this Way….

Test FirstStart automation without the UIInvolves entire team in testingInvolves customer in testingVisibility of resultsAvoid gold platingLeads to good architecture and designTests are easy to readCollapse the V modelTesting code is minimalAnd more…….

Executable Requirements

Think back to our requirement vs. test conversation…..Let’s see this in action

FitNesse

Unit vs. GUI vs. FitNesse

Code oriented

GUI TESTING

FitNesse

UNIT TESTING

GUI specific

Customer Oriented

FIT TABLE FIXTURE SYSTEM

FitNesse has several fixture types…

Column - calculationsRow/Array – data setsDo/Action - workflowActionDatabaseAnd more….

Division

Numerator Denominator Quotient?

10 2 5

100 2.5 40

First Row Identifies the Test/Requirement. This is the name of the fixture (the class)

Anatomy of a Column Fixture

Division

Numerator Denominator Quotient?

10 2 5

100 2.5 40

Second Row is the inputs and outputs. Expected outputs have the ? Or () after them, inputs do not

Division

Numerator Denominator Quotient?

10 2 5

100 2.5 40

Each additional row is an instance of the test that explains the requirement we are testing

Workshop: Write a Column Fixture

User Story 1: As a mathematician, I want to be able to use the calculator to divide numbers and get the answer so that I can solve problems.Acceptance Tests:

Verify that the division is correctVerify that the answer is in ##.## format

Anatomy of a Row Fixture

ListofUsers Active

FirstName LastName UserName

John Smith Jsmith

Beth Jones Bjones

Suzy Plus splus

First Row Identifies the Test/Requirement. This is the name of the fixture (and the name of the class)

Other cells on the first row represent inputs passed into the fixture

ListofUsers Active

FirstName LastName UserName

John Smith Jsmith

Beth Jones Bjones

Suzy Plus splus

The second row lists the data elements in the expected output

ListofUsers Active

FirstName LastName UserName

John Smith Jsmith

Beth Jones Bjones

Suzy Plus splus

The 3rd through nth rows list the data you expect the test to return

Workshop: Write a Row Fixture

User Story 3: As a user, I want to search for a name so that I can find their phone number.Acceptance Tests:

Search by first or last name“Like” match searchResults are in first name, last name, phone number

Do Fixture

Do fixtures should read like a storyUsed to test workflowsOften tie to epicsEach step in the do fixture likely has detailed tests elsewhereLet’s look at an example….

Example of a Do Fixture

As a roulette player, I want to place multiple bets and have my winnings automatically calculated so that I can become a millionaire.To do this, a series of steps need to happen (that could each be a separate story)

PLACE BETS

SPIN WHEEL

CALCUALTE WINNINGS

FIXTURE NAME: defines the class

SIMPLE ACTIONS: method names that execute them

�

�

�

�

ROW FIXTURES: method returns a row fixture

�

�

Some Additional Considerations

Your automation should control your dataYour automation should be quickYour automation should be machine agnosticYour automation should let you sleep at nightYour test should run in any order

Some Additional Considerations

Who will write the tests?Fitnesse is designed to permit Customers to write testsHowever, the Customer will probably need help from QA• Fitnesse Wiki syntax• Corner cases and unhappy paths

Some Additional Considerations

Impact on DevelopmentThe fixtures used in Fitnesse tests need internal implementations written in the same language as the codeThese internal fixtures are the “glue” that binds the test tables to the application’s implementation that is being testedThis code must be written and maintained by developers

Testing on a Greenfield Application

Define Acceptance Tests

Exploratory

Automate Acceptance Tests

TDD (Test Driven Development)

Manual Testing

Verify Bug Fixes

Full Regression

Performance

SPRINT START SPRINT END

Testing on a Legacy Application

Legacy Challenges

What challenges exist with Legacy Applications that make agile testing challenging?

Strategy 1: Not recommended

Define Acceptance Tests ExploratoryAutomate Acceptance Tests

TDD (Test Driven Development)

Manual Testing

Verify Bug Fixes

Full Regression

Performance

SPRINT START SPRINT END

Strategy 2: Hardening Sprint

Exploratory

SPRINT ONE SPRINT N(Hardening Sprint)

SPRINT TWO

Full RegressionAutomate Acceptance Tests

TDD

Manual Testing

Verify Bug Fixes

Exploratory

Automate Acceptance Tests

TDD

Manual Testing

Verify Bug Fixes

Exploratory

Performance

Estimation Sessions

Estimation Techniques

• Planning Poker• Rock, Paper, Scissors• Others?

Detail an Accuracy

• Some believe that better estimates require more detailed analysis

– More detailed analysis takes additional time and resources

• Others believe that the initial “gut” reaction is sufficient for Agile planning purposes

– Time required for additional analysis does not result in sufficiently better estimates to justify the added costs

Detail and Accuracy

• Estimates can be like statistics– Out of context, an individual estimate may

have little meaning– In the overall iteration, though, some

estimates will be low and others will be high• In the end, they cancel each other out

What are things testers often think about when estimating?

Sprint Planning Meetings

What are their Purpose?

Define the sprint backlogDefine “DONE”Commit to the sprint

Workshop: Let’s Create Tasks

User Story 1: As a mathematician, I want to be able to use the calculator to divide numbers and get the answer so that I can solve problems.

Task Breakout

• Who creates tasks?

• Who estimates tasks?

• Who owns tasks?

Definition of Done

Retrospectives

Retrospectives

Opportunity to review what worked and what didn’tMust be conducted in a non-threatening environmentTo be effective

All participants must feel comfortable and willing to shareShould be held at each iteration boundaryShould result in tangible action items that are put into place immediately

Workshop: Run a Retrospective

Infrastructure

Continuous Integration

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily -leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.”– Martin Fowler

Check-in Code

Run Tests

Build Entire Project

PublishResults

How Does CI Work?

Single Source Code RepositoryBuild is AutomatedBuild is self-testingEveryone commits every dayThe build is fastVisible resultsAutomated deployment

Test Environments?

Do you have an environment you have complete control over?Are builds pushed or pulled to it?How often do you get builds?Does it look like your production environment?Do you “own” it?

Test Data

• Where do you get your test data?

• How well do you know your test data?

• Do you control it?• Can you re-set it?• Can you run the same

script over and over again?

Data Needs

Different testing activities require different data – one database will not suit all your needs.

Manual / ExploratoryAutomationPerformanceAny others?

Skills and Titles

Titles….

QA Analyst

QA Engineer

Software

Tester

QA Architect

Developer

Senior DeveloperTest Engineer

Workshop: Hiring an Agile Tester

What skills would you want from your agile testers?

Closing

Agile Foundations

© 2009 Valtech, Inc. All rights reserved.

For more information,

Visit us on the web at http://www.valtech.us

Email us directly at vicki.pate@valtech.com

Call us at (972) 789-1200

Fax us at (972) 789-1340

Write us (or just drop by) atValtech19111 North Dallas Pkwy, Suite 300Dallas, TX 75287

Sources

Mountain Goat Software (www.mountaingoatsoftware.com)“Testing Extreme Programming” by Lisa Crispin and Tip House“Test Driven .NET Development with FitNesse” by Gojko AdzicFitNesse.org“Agile Estimating and Planning” by Mike Cohn“User Stories Applied” by Mike Cohn“Lean Software Development” by Mary and Tom Poppendieck“Collaboration Explained” by Jean Tabaka