et workshop v. 1.20 - how to?©2002 amland consulting3-1 these slides are distributed under the...

ET Workshop v. 1.20 - How to?

©2002 Amland Consulting 3-1

These slides are distributed under the Creative Commons License.

In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one.

For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.



Planning, Documentation and Execution Outline

Introduction

Test Management and Techniques

ET Planning, Exec. and Documentation

ET Styles

ET Management

• How much planning is done with ET?

• How do you execute the tests with ET?

• What kind of documentation is produced with ET?

• How to use Pair Testing with ET?

• How do you use ET with eXtreme Programming (XP)?

1.

2.

3.

4.

5.



3. Exploratory Test Planning, Execution and Documentation.

3.1 Planning, Tasks and Documentation

3.2 Exploratory Testing in Pairs

3.3 Exploratory Testing with eXtreme Programming

1.

2.

3.

4.

5.



ET Planning“We plan as much as we can (based on the knowledge available), when we can (based on the time available), but not before.”

“When we apply Exploratory Planning to Testing, we create Exploratory Testing.”

Introduction


ET Planning and Documentation

ET Styles

ET Management

Lee Copeland, Exploratory Planning, 2001



1.

2.

3.

4.

5.

Exploratory Testing Tasks

ExploreExecute

TestsDesignTests

Product(coverage)

Techniques

Quality(evaluation)

Discover the elements

of the product.

Discover howthe product

should work.

Discover testdesign techniquesthat can be used.

Decide whichelements to test.

Observe product behavior.

Speculate aboutpossible quality

problems.

Evaluatebehavior against

expectations.

Configure & operate the

product.

Select & applytest design techniques.

Testingnotes

TestsProblems

FoundFrom Rapid Software Testing, copyright © 1996-2002 James Bach



1.

2.

3.

4.

5. PreparationDecompose the Product into elements

Areas / Groups of functions or features you expect to be able to test within a day or two

Identify the Quality Criteria

Ref. Satisfice Testing Model

Check your toolbox

Test techniques

ApplicationTo be tested

Decomposed Into elements

Element Element Element

Each element is expected to be tested within a day or two.



1.

2.

3.

4.

5.

ET Planning – Architecting Charters

Pick an element to testDecompose further – into smaller elements

Each new element should be tested within an hour or twoPrepare a charter for each element

Element

Charter

ElementTo be tested

Decomposed Into smaller

elements /Charters

Each Charter is expected to be tested within an hour or two.

Charter Charter



1.

2.

3.

4.

5. Doing Exploratory TestingKeep your mission clearly in mind.Keep notes that help you report what you did, why you did it, and support your assessment of product quality.Keep track of questions and issues raised in your exploration (logging and debriefing).To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.

From Rapid Software Testing, copyright © 1996-2002 James Bach

Charter/Mission

Notes, Risks, Issues,Questions and Errors

Execution: Pairs & Sessions

DebriefingDebriefing



1.

2.

3.

4.

5.

Charter: A clear mission for the session

A charter may suggest what should be tested, how it should be tested, and what problems to look for.A charter is not meant to be a detailed plan.General charters may be necessary at first:

“Analyze the Insert Picture function”

Specific charters provide better focus, but take more effort to design:

“Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”


Charter/Mission






1.

2.

3.

4.

5. Lateral Thinking

but periodically take stockof your status against your mission

Let yourself be distracted…

‘cause you never knowwhat you’ll find…




1.

2.

3.

4.

5. Charter – summary…“Architecting the Charters” i.e. Test PlanningBrief information / guidelines on:

Mission: Why do we test this?What should be tested?How to test (approach)?What problems to look for?

Might include guidelines on:Tools to useSpecific Test Techniques or tactics to useWhat risks are involvedDocuments to examineDesired output from the testing

Charter/Mission






1.

2.

3.

4.

5. Taking NotesTest Coverage Outline/MatrixEvaluation NotesRisk/Strategy ListTest Execution LogIssues, Questions & Anomalies

It would be easier to test if you changed/added…How does … work?Is this important to test? How should I test it?I saw something strange…


Charter/Mission






1.

2.

3.

4.

5. ET Execution - SessionsSession

Basic testing work unitUninterrupted

e-mail, meetings, telephone calls etc.

ReviewableA report should be produced

CharteredMission associated with this session; What are we testing? What are we looking for?

Charter/Mission






1.

2.

3.

4.

5. Exercise 1a

Prepare a test charter for testing one selected area of StarOffice 5.2

Bullet and Numbering orTables

Work in pairs.



1.

2.

3.

4.

5. ET DocumentationPlanning:

Charters – overview of what to test (plan)Migh be a flip chart on the wall

Missions – What are we looking for?

ExecutionNotes – what happened during testing?

What did I do? Why did I do it? Used to assess product quality after test.

Data files – input data used for testingBug reports – enough details to recreate the test / bugTrack questions and IssuesDebriefing

Charter/Mission






1.

2.

3.

4.

5.

Key Ideas of Exploratory Testing

The results from the tests you design and execute influence the next test you will choose to design and execute.You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave.You test what you know about, and you are alert for clues about behaviors and aspects of the product that you don’t yet know about.




1.

2.

3.

4.

5.

Challenges ofHigh Accountability Exploratory

Testing

Architecting the system of charters (test planning)Making time for debriefingsGetting the metrics rightCreating good test notesKeeping the technique from dominating the testingMaintaining commitment to the approachFor example session sheets and metrics see

http://www.satisfice.com/sbtm




1.

2.

3.

4.

5. Problems to be Wary ofHabituation may cause you to miss problems.Lack of information may impair explorationExpensive or difficult product set-up may increase the cost of exploringExploratory feedback loop may be too slowOld problems may pop up again and againHigh MTBF may not be achievable without well defined test cases and procedures, in addition to exploratory approach.




1.

2.

3.

4.

5. Exercise 1b

Prepare a test charter for testing one selected area of MiniTest

Work in pairs.







1.

2.

3.

4.

5.



ET and Pair Testing

Observations made at Satisfice, several independent companies and Florida Tec – spectacular results:

Testing workgroups:High productivityHigh creativity

Introduction



ET Styles

ET Management

Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)



What is Pair Testing?Two testers and (typically) one machineTypically (as in XP):

Pairs work together voluntarily. One person might pair with several others during a dayA given testing task is the responsibility of one person, who recruits one or more partners (one at a time) to help out.

Some stable pairs have worked together for yearsOne tester strokes the keys (keyboard might pass back and forth in a session) while the other suggests ideas or tests, pay attention and takes notes, listen, asks questions, grabs reference material etc.

Introduction



ET Styles

ET Management




A Paired Testing SessionStart with a charter

Testers might operate from a detailed project outline, pick a task that will take a day or lessMight (instead or also) create a flipchart page that outlines this session’s work or the work for the next few sessions

An exploratory testing session lasts about 60-90 minutes

The charter for a session might include what to test, what tools to use, what testing tactics to use, what risks are involved, what bugs to look for, what documents to examine, what outputs are desired etc.

Introduction



ET Styles

ET Management




Benefits of Pair TestingGenerate more ideas.More fun.Helps the tester stay on task.

Also, the fact that two are working together limits the willingness of others to interrupt them…

Better Bug Reporting.Great Training.Additional technical benefit.

Introduction



ET Styles

ET Management




Risks and SuggestionsSenior / junior pairs: The pairs are partners; the junior tester is often the one at the keyboard, and he / she is always allowed to try out his / her own ideas.Accountability must always belong to one person.Some people are introverts. They need time to work alone and recharge themselves for group interaction.Some people have strong opinions and don’t work well with others. Coaching may be essential.Learn from XP experiences, Laurie Williams of NCSU (collaboration.csc.ncsu.edu/laurie/)

Introduction



ET Styles

ET Management




Introduction



ET Styles

ET Management

Pair Programming vs. Testing

Pair Testing is developed independently of paired programming, but many of the same problems and benefits apply.The eXtreme Programming community has a great deal of experience with paired work and offers many lessons:

Kent Beck, Extreme Programming ExplainedRon Jeffries, Ann Anderson & Chet Hendrickson, Extreme Programming Installed

Laurie Williams of NCSU does research in pair programming. For her publications, see http://collaboration.csc.ncsu.edu/laurie/




Introduction



ET Styles

ET Management Pair Programming – Problems?

Paper by Gerold Keefer, AVOCA GmbH:“So far there is no evidence about the usefulness of this practice [pair programming] in general,….In addition there are several reports that indicate that pairing leads to faster exhaustion… On the other hand no one will deny that pairing in certain situations, such as educational settings or debugging sessions, is good common sense.”Alternative to pair programming:

Mutual Programming: A pair of developers does mutual QA on each others work products.Mutual Testing?



1.

2.

3.

4.

5. Exercise 1c

Prepare a test charter for testing the WEB-page: www.amland.no/et_test

Work in pairs.







1.

2.

3.

4.

5.



Introduction



ET Styles

ET Management ET and XP analogy…

Exploratory Testingis to Scripted Testing

whatXP is to

traditional “requirements and specifications” basedsystem development methods



Introduction



ET Styles

ET Management

What is eXtreme Programming? (1)

Agile methodologyWorking in small development groupsTwo roles:

CustomerDeveloper

Everybody participate in design, programming and testIterative development

Iterations not more than 2 – 3 weeksCreate executable code every night



Introduction



ET Styles

ET Management


Key areas:Communication

in the team, I.e. with the customerSimplicity

Keep everything simple, should be possible to implement within hours (days)Reduce risk

FeedbackFrom customers and othersContinuous system improvement

Accept ChangesLow threshold to changesContinuously specify, design, code and test



Introduction



ET Styles

ET Management


TestingAutomated testingCreate test before coding

Test Driven Design

The test will document requirementsDaily Build and Test

Unit Test, Integration Test and System Test



Introduction



ET Styles

ET Management


Note, please, that in the XP context,

only the Customer Test can /validate/

that the program does the right thing.

The Programmer Tests merely /verify/

that the program does what the programmers

intended.

Ron Jeffries, (e-mail on agile-testing list, April 29, 2002)



Introduction



ET Styles

ET Management

XP and Exploratory Testing (1)

“You develop the application test-first, where automated customer tests specify what the system is to do.

At some points, you invite James Bach in to do

exploratory testing. Perhaps at the ends of iterations.

If you’re so inclined, you say that James is part of the Customer Role.”

Brian Marick (e-mail on agile-testing list, March 7, 2002)



Introduction



ET Styles

ET Management

XP and Exploratory Testing (2)

“I’ve found that exploratory testing helps me to think of

additional acceptance tests to propose to the customer.

Lots of times you don’t think of everything about a system until

you use it.”

Lisa Crispin (e-mail on agile-testing list, March 8, 2002, writing book on XP and

Testing)



Introduction



ET Styles

ET Management XP and ET – Problems?Paper by Gerold Keefer, AVOCA GmbH:

Extreme Programming Considered Harmful for Reliable Software Development (2002):

Discussing pros and cons of XP (including Pair Programming)

“The most recent study … suggests that … pair programming is rather expensive technology.. and ...XP-like pair programming appears less efficient than it is reported” (with references)

Reviewing XP’s most well known reference project, C3 at DaimlerChrysle Company

Launched in 1995, Kent Beck (XP) took over 1996Supposed to be used by 87.000 employees, never used by more then 10.000The project was cancelled in 2000 after 4 years in XP mode!

How should this effect Exploratory Testing?



Introduction



ET Styles

ET Management ET – The VanguardThe object of your mission is to explore

the Missouri river, & such principal streams of it, as, by its course and

communication with the waters of the Pacific ocean...may offer the most direct

& practicable water communication across this continent for the purposes of

commerce.

- Thomas Jefferson's letter to Meriwether Lewis, June 1803Robinson, H., Microsoft, Exploratory Modeling



1.

2.

3.

4.

5.

Planning, Documentation and Execution Summary

Introduction


ET Planning, Exec. and Documentation

ET Styles

ET Management

• How much planning is done with ET?

• How do you execute the tests with ET?

• What kind of documentation is produced with ET?

• How to use Pair Testing with ET?



1.

2.

3.

4.

5. Exercise 2

Based on previously produced charter and available information about StarOffice:

Select one (or more) testing technique(s) and test the application. Explain your mission / reasoningExplain technique(s) used and errors found.

et workshop v. 1.20 - how to?©2002 amland consulting3-1 these slides are distributed under the...

Documents

exploratory planning

pair testing

testing modelcheck

exploratory test planning

amland consultingplanning

amland consulting3

techniqueset planning

product behavior