copyright © 1996-2002, satisfice, inc. v1.4.3 i grant permission to make digital or hard copies of...
TRANSCRIPT
![Page 1: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/1.jpg)
Copyright © 1996-2002, Satisfice, Inc.
v1.4.3 I grant permission to make digital or hard copies of this work for personal or
classroom use, provided that (a) Copies are not made or distributed for profit or commercial advantage, (b) Copies bear this notice and full citation on the first page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion. Abstracting with credit is permitted. The proper citation for this work is Rapid Software Testing (course notes, Fall 2002), www.testing-education.org, (c) Each page that you use from this work must bear the notice "Copyright (c) James Bach, [email protected]” or, if you modify the page, "Modified slide, originally from James Bach", and (d) If a substantial portion of a course that you teach is derived from these notes, advertisements of that course must include the statement, "Partially based on materials provided by James Bach." To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from James Bach, [email protected].
![Page 2: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/2.jpg)
Acknowledgements
Some of this material was developed in collaboration with Dr. Cem Kaner, of the Florida Institute of Technology.
Many of the ideas in this presentation were reviewed and extended by my colleagues at the 7th Los Altos Workshop on Software Testing. I appreciate the assistance of the LAWST 7 attendees: Cem Kaner, Brian Lawrence, III, Jack Falk, Drew Pritsker, Jim Bampos, Bob Johnson, Doug Hoffman, Chris Agruss, Dave Gelperin, Melora Svoboda, Jeff Payne, James Tierney, Hung Nguyen, Harry Robinson, Elisabeth Hendrickson, Noel Nyman, Bret Pettichord, & Rodney Wilson.
![Page 3: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/3.jpg)
Assumptions
You test software. You have at least some control over the design of
your tests and some time to create new tests. One of your goals is to find important bugs fast. You test things under conditions of uncertainty
and time pressure. You have control over how you think and what
you think about. You want to get very good at software testing.
![Page 4: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/4.jpg)
Goal of this Class
This class is about how to test a productwhen you have to test it right now,under conditions of uncertainty,
in a way that stands up to scrutiny.
(This skill comes in handy even whenyou have more time, more certainty,
and are subject only to your own scrutiny.)
![Page 5: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/5.jpg)
Internet time!
Less time than you’d like. Faster is better. Your plans will be interrupted. You are not fully in control of your time.
What is Internet time like?
![Page 6: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/6.jpg)
The Challenge
When the ball comes to you…Do you know you have the ball?
Can you receive the pass?
Do you know what yourrole and mission is?
Do you know whereyour teammates are?
Are you ready to act, right now?
Can you let your teammates help you?
Do you know your options?
Is your equipment ready?
Can you read the situation on the field?
Are you aware of the criticality of the situation?
![Page 7: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/7.jpg)
Your Moves:Rapid Testing Cycle
Make sense ofyour status
Compare statusagainst mission
Focus on whatneeds doing
START STOPDo a burst oftesting
Report
![Page 8: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/8.jpg)
What About “Slow” Testing?
All Testing
Rapid
automationextensive
preparationsuper
testabilitysuperskill
Rigorousor Thorough
Management likesto talk about this…but they don’tfund it.
You can do this,no matter what.
![Page 9: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/9.jpg)
KEY IDEA
![Page 10: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/10.jpg)
TechnicalKnowledge
Specifications
Product
Ah! Problem!
Problem
Testing is in Your Head
Coverage
ProblemReport
Communication
The important parts of testing don’ttake place in the computer or on your desk.
CriticalThinking
DomainKnowledge
Experience
![Page 11: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/11.jpg)
Welcome to Epistemologythe study of knowledge
![Page 12: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/12.jpg)
Epistemology
Epistemology is the study of how we knowwhat we know. The philosophy of science
belongs to Epistemology.
All good testers practice Epistemology.
![Page 13: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/13.jpg)
Basic Skills of Epistemology
Ability to pose useful questions. Ability to observe what’s going on. Ability to describe what you perceive. Ability to think critically about what you know. Ability to recognize and manage bias. Ability to form and test conjectures. Ability to keep thinking despite already knowing. Ability to analyze someone else’s thinking.
![Page 14: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/14.jpg)
Epistemology Lesson:Abductive Inference
1. Collect data.2. Find several explanations that account for the data.3. Find more data that is either consistent or
inconsistent with explanations.4. Choose the best explanation that accounts for the
important data, or keep searching.
Abductive inference means finding the best explanation for a set of data.
![Page 15: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/15.jpg)
Epistemology Lesson:Conjecture and Refutation
We never know product quality for certain. We conjecture about quality. To conjecture is to
explore plausible realities. A good conjecture is falsifiable: we can imagine facts
that would refute the conjecture. A conjecture can never be proven, only corroborated.
Corroborating evidence is most interesting when gained as the result of a genuine attempt to refute.
Good testing is a serious attempt to refute and corroborate conjectures about product quality.
![Page 16: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/16.jpg)
Epistemology of Magic
Our thinking is limited We misunderstand probabilities We use the wrong heuristics We lack specialized knowledge We forget details We don’t pay attention to the right things
The world is hidden states sequences processes attributes variables identities
Magic tricks workfor the same reasons
that bugs exist
Studying magic canhelp you developthe imagination
to find better bugs.
Sufficiently advancedtechnology is
indistinguishablefrom magic.
![Page 17: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/17.jpg)
Tester vs. Tourist:Testers Ask Critical Questions
How well does the product work? How do you know how well it works? What evidence do you have about how it works? Is that evidence reliable and up to date? What does it mean for the product to “work”? What facts would cause you to believe that it doesn’t work? In what ways could it not work, yet seem to you that it does? In what ways could it work, yet seem to you that it doesn’t? What might cause the product not to work well (or at all)? What would cause you to suspect that it will soon stop working? Do other people use the product? How does it work for them? How important is it for this product to work? Are you qualified to answer these questions? Is anyone else qualified?
![Page 18: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/18.jpg)
Heuristics:Generating Solutions Quickly
adjective:
“serving to discover.”
noun:
“a useful method that doesn’t always work.”
“Heuristic reasoning is not regarded as final and strictbut as provisional and plausible only, whose purpose is to discover the solution to the present problem.”
- George Polya, How to Solve It
![Page 19: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/19.jpg)
Some Everyday Heuristics
It’s dangerous to drink and drive. A bird in hand is worth two in the bush. Nothing ventured, nothing gained. Sometimes people stash their passwords near their
computers. Try looking there. Stores are open later during the Holidays. If your computer is behaving strangely, try
rebooting. If it’s very strange, reinstall Windows. If it’s a genuinely important task, your boss will
follow-up, otherwise, you can ignore it.
![Page 20: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/20.jpg)
How Heuristics Differ fromOther Procedures or Methods
Heuristics are known to be wrong, at least some of the time.
No one can say for sure when a heuristic will work.
Heuristics aid or focus an open-ended problem-solving or solution-searching effort.
Heuristics can substitute for complete or rigorous analysis.
![Page 21: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/21.jpg)
Tunnel-Vision is OurGreat Occupational Hazard
Problems you
can find with your
biases…
invisible
problems
invisible
problems
![Page 22: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/22.jpg)
A Tester’s Attitude
Cautious Jump to conjectures, not conclusions. Practice admitting “I don’t know.” Have someone check your work.
Curious What would happen if…? How does that work? Why did that happen?
Critical Proceed by conjecture and refutation. Actively seek counter-evidence.
Good testersare hard to fool.
(and Courageous)
![Page 23: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/23.jpg)
Important Vocabulary
Abductive inference is a way to make respected guesses. Conjectures (a.k.a. hypotheses) help us remember that we
may be wrong about our beliefs. Refutation means checking how you could be wrong,
before trusting a conjecture. Evidence is either consistent with (corroborates) or
inconsistent with a given conjecture. Heuristics help us get the right ideas at the right times. Raising questions and issues is the heart of testing.
![Page 24: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/24.jpg)
The Triangle Program
This program takes three numbers as input. The numbers represent the dimensions of a
triangle. When you click on the check button, the program
tells you what kind of triangle the sides represent: scalene (no side equal to any other) isosceles (two sides are equal) equilateral (all sides are equal)
I want you to test this program.Any questions?
![Page 25: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/25.jpg)
KEY IDEA
![Page 26: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/26.jpg)
The Universal Test Procedure
“Try it and see if it works.”
Learn about it Model it Speculate about it Configure it Operate it
Know whatto look for
See what’sthere
Understand the requirements
Identify problems Distinguish bad
problems from not-so-bad problems
Models EvaluationCoverage
![Page 27: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/27.jpg)
All Product Testing is Something Like This
ProjectEnvironment
ProductElements
QualityCriteria
TestTechniques
PerceivedQuality
![Page 28: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/28.jpg)
The Five Problems of Testing
ProjectEnvironment
ProductElements
QualityCriteria
TestTechniques
PerceivedQuality
Logistics Problem
Coverage ProblemEvaluation
Problem
ReportingProblem
Stopping Problem
![Page 29: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/29.jpg)
Models
A Model is… A map of a territory A simplified perspective A relationship of ideas An incomplete representation of reality A diagram, list, outline, matrix…
No good test design has ever been done without models.
The trick is to become aware of how you modelthe product, and learn different ways of modeling.
![Page 30: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/30.jpg)
Coverage
There are as many kinds of coverage as there are ways to model the product. Structural Functional Data Platform Operations
Product coverage is the proportion of theproduct that has been tested.
![Page 31: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/31.jpg)
An evaluation strategy ishow you know the product works.
“...it works.”
really means
“...it appeared to meet some requirement to some degree.”
Which requirement? Appeared how?To what degree? Perfectly? Just barely?
![Page 32: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/32.jpg)
Rapid Evaluation:“HICCUPP”
Consistent with History: Present function behavior is consistent with past behavior.
Consistent with our Image: Function behavior is consistent with an image that the organization wants to project.
Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products.
Consistent with Claims: Function behavior is consistent with what people say it’s supposed to be.
Consistent with User’s Expectations: Function behavior is consistent with what we think users want.
Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product.
Consistent with Purpose: Function behavior is consistent with apparent purpose.
![Page 33: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/33.jpg)
Exercise
Explain what you covered and what your evaluation strategies were for triangle.
![Page 34: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/34.jpg)
KEY IDEA
![Page 35: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/35.jpg)
Test Cycle:What You Do With a Build
1.Receive the product. Formal builds Informal builds Save old builds.
2. Clean your system. Completely uninstall earlier builds.
3.Verify testability. Smoke testing Suspend test cycle if the product is untestable.
![Page 36: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/36.jpg)
Test Cycle:What You Do With a Build
4.Determine what is new or changed. Change log
5.Determine what has been fixed. Bug tracking system
6.Test fixes. Many fixes fail! Also test nearby functionality.
7.Test new or changed areas. Exploratory testing.
![Page 37: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/37.jpg)
Test Cycle:What You Do With a Build
8.Perform regression testing. Not performed for an incremental cycle. Automated vs. manual Important tests first!
9.Report results. Coverage Observations Bug status (new, existing, reopened, closed) Assessment of quality Assessment of testability
![Page 38: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/38.jpg)
The Test Cycle
Time
Qu
alit
y
Typical Quality Growth
Ship it!
![Page 39: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/39.jpg)
Risk Focus:Common and Critical Cases
Core functions: the critical and the popular. Capabilities: can the functions work at all? Common situations: popular data and pathways. Common threats: likely stress and error situations. User impact: failures that would do a lot of damage. Most wanted: problems of special interest to
someone else on the team.
![Page 40: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/40.jpg)
Test Cycle Convergence
Time
Qu
ality
Good Enough Quality
Build
Cycle EndCycle Start
Tests failed: known bad
Tests passed: known good
Tests not run:Unknown}
Ideal Test Cycle Convergence
Untestable Quality
![Page 41: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/41.jpg)
Rapid Bug Investigation
Identification Notice a problem. Recall what you were doing just prior to the problem. Examine symptoms of the problem w/o disturbing system state. Consider possibility of tester error.
Investigation How can the problem be reproduced? What are the symptoms of the problem? How severe could the problem be? What might be causing the problem?
Reality Check Do we know enough about the problem to report it? Is it important to investigate this problem right now? Is this problem, or any variant of it, already known? How do we know this is really a problem? Is there someone else who can help us?
Identify
Investigate Check
![Page 42: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/42.jpg)
Exercise
“I’ve modified the Triangle program so that it now renders an image of the triangle. I’m also worried about the limitations of the input field. Please do another test cycle on it.”
-- Your Friendly Programmer
![Page 43: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/43.jpg)
Exercise
Provide a clear report on how completelyyou tested Triangle
![Page 44: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/44.jpg)
An Innocent Question from Management…
“On a scale of 0 to 100, how completely did you test the Triangle program?”
completelyuntested
completelytested
0 10050
1. Answer with a number.
2. You can also offer a comment, if you have one.
3. Any questions?
![Page 45: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/45.jpg)
What Could We Report?
Tasks: Things we are doing. Example: “We are trying to reinstall the server so we can do the smoke tests. We have 28 fix verifications to get through.”
Product Coverage: What we have examined. Example: “We have tested printing, performance, and compatibility.”
Product Risk: Problems or potential for problems in the product. Example: “We have found 53 bugs.”
Agreement: What we specifically contracted to do. Example: “We have implemented 90% of the tests on our test plan. As we agreed, we have started testing for printer compatibility.”
![Page 46: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/46.jpg)
What Could We Report?
Project: Schedule, documents, resources, people, or anything else that makes it possible for us to test. Example: “Testing is on schedule. But Jaeline goes on vacation next week.”
Mission: The ultimate goal, which may be to ship a product that meets quality criteria, satisfy customers, or some other goal that can be a guiding principle for the testing. Example: “We don’t yet know enough about feature X of the product.”
Client Satisfaction: What our clients think of our work. Example: “The project manager is happy with our work.”
![Page 47: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/47.jpg)
Reporting Considerations
Reporter safety: What will they think if I made no progress? Client: Who am I reporting to and how do I relate to them? Rules: What rules and traditions are there for reporting, here? Significance of report: How will my report influence events? Subject of report: On what am I reporting? Other agents reporting: How do other reports affect mine? Medium: How will my report be seen, heard, and touched? Precision and confidence levels: What distinctions make a
difference?
Take responsibility for the communication.
![Page 48: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/48.jpg)
KEY IDEA
![Page 49: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/49.jpg)
Test Strategy
“How we plan to cover the product so as to develop an adequate assessment of quality.”
A good test strategy is: Product-Specific Risk-focused Diversified Practical
![Page 50: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/50.jpg)
Test Strategy
Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test strategy.
Example of a poorly stated (and probably poorly conceived) test strategy: “We will use black box testing, cause-effect graphing,
boundary testing, and white box testing to test this product against its specification.”
![Page 51: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/51.jpg)
Test Strategy
Makes use of test techniques. May be expressed by test procedures and cases. Not to be confused with test logistics, which
involve the details of bringing resources to bear on the test strategy at the right time and place.
You don’t have to know the entire strategy in advance. The strategy should change as you learn more about the product and its problems.
![Page 52: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/52.jpg)
Test Cases/Procedures
Test cases and procedures should manifest the test strategy.
If your strategy is to “execute the test suite I got from Joe Third-Party”, how does that answer the prime strategic questions: How will you cover the product and assess quality? How is that practical and justified with respect to the specifics of this
project and product?
If you don’t know, then your real strategy is that you’re trusting things to work out.
![Page 53: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/53.jpg)
Exercise
Produce a test strategy for DecideRight
![Page 54: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/54.jpg)
Test Strategy Heuristic:Diverse Half-Measures
There is no single technique that finds all bugs. We can’t do any technique perfectly. We can’t do all conceivable techniques.
Use “diverse half-measures”-- lots of differentpoints of view, approaches, techniques, evenif no one strategy is performed completely.
![Page 55: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/55.jpg)
Strategy Heuristic:Function/Data Square
Data
Functions
risk testing
Function testing
reliabilitytesting
smoketesting
![Page 56: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/56.jpg)
The Fallacy of Repeated Tests:Clearing Mines
mines
![Page 57: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/57.jpg)
Totally Repeatable TestsWon’t Clear the Minefield
mines fixes
![Page 58: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/58.jpg)
Variable Tests areTherefore More Effective
mines fixes
![Page 59: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/59.jpg)
Exercise
Find at least two situations where the heuristic:
“Variable tests are better than repeated tests”
is WRONG.
![Page 60: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/60.jpg)
Test Techniques
A test techniqueis a recipe
for performingthese tasks that
will reveal somethingworth reporting
Analyze the situation. Model the test space. Select what to cover. Select evaluation methods. Configure the test system. Operate the test system. Observe the test system. Evaluate the test results.
![Page 61: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/61.jpg)
General Test Techniques
Function testing Domain testing Stress testing Flow testing Scenario testing User testing Regression testing Risk testing Claims testing Random Testing
For any one of these techniques,there’s somebody, somewhere,
who believes that is theonly way to test.
![Page 62: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/62.jpg)
Function Testing
“March through the functions”
- A function is something the product can do.
- Identify each function and sub-function.
- Determine how you would know if they worked.
- Test each function, one at a time.
Key Idea:
Summary:
Good for: assessing capability rather than reliability
![Page 63: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/63.jpg)
Domain Testing
“Divide the data, and conquer”
- A domain is a set of test data.
- Identify each domain of input and output.
- Analyze limits and properties of each domain.
- Identify combinations of domains to test.
- Select coverage strategy:
e.g., exhaustive, boundaries, best representative
Key Idea:
Summary:
Good for: all purposes
![Page 64: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/64.jpg)
Stress Testing
“Overwhelm the product”
- Select test items and functions to stress.
- Identify data and platform elements that relate to them.
- Select or generate challenging data and platform configurations to test with:
e.g., large or complex data structures,
high loads, long test runs, many test cases
Key Idea:
Summary:
Good for: performance, reliability, and efficiency assessment
![Page 65: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/65.jpg)
Flow Testing
“Do one thing after another”
- Define test procedures or high level cases that
incorporate many events and states.
- The events may be in series, parallel, or some combination thereof.- Don’t reset the system between events.
Key Idea:
Summary:
Good for: finding problems fast(however, bug analysis is more difficult)
![Page 66: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/66.jpg)
Scenario Testing
“Test to a compelling story”
- Design tests that involve meaningful and complex interactions with the product.- A good scenario test is a plausible story of how someone who matters might do something that matters with the product.- Incorporate multiple elements of the product, and data that is realistically complex.
Key Idea:
Summary:
Good for: Finding problems that seem important
![Page 67: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/67.jpg)
User Testing
“Involve the users”
- Identify categories of users.- Understand favored and disfavored users.-Find these users and have them do testing or help you design tests.-User testing is powerful when you involve a variety of users.
Key Idea:
Summary:
Good for: all purposes
![Page 68: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/68.jpg)
Regression Testing
“Test the changes”
- Identify what product elements changed.
- Identify what elements could have been
impacted by the changes.
- Coverage strategy:
e.g., recent bug fixes, past bug fixes,
likely elements, all elements
Key Idea:
Summary:
Good for: managing risks related to product enhancement
![Page 69: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/69.jpg)
Claims Testing
“Verify every claim”
- Identify specifications (implicit or explicit).
- Analyze individual claims about the product.
- Ask customer to clarify vague claims.
- Verify each claim.
- Expect the specification and product to be
brought into alignment.
Key Idea:
Summary:
Good for: simultaneously testing the product and specification, while refining expectations
![Page 70: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/70.jpg)
Risk Testing
“Imagine a problem, then look for it.”
- What kinds of problems could the product have?- How would you detect them if they were there?- Make a list of interesting problems and design tests specifically to reveal them.- It may help to consult experts, design documentation, past bug reports, or apply risk heuristics.
Key Idea:
Summary:
Good for: making best use of testing resources;leveraging experience
![Page 71: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/71.jpg)
Random Testing
“Run a million different tests”
- Look for an opportunity to automatically generate thousands of slightly different tests.
- Create an automated, high speed evaluation strategy.
- Write a program to generate, execute, and evaluate all the tests.
Key Idea:
Summary:
Good for: Assessing reliability across input and time.
![Page 72: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/72.jpg)
KEY IDEA
![Page 73: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/73.jpg)
Dynamic Quality Paradigm
Perfect
Awful
unnecessary quality
unacceptable quality
Item A
Item B
It’s more important to work on Item B.
Further improvement would not be a good use of resources.
Further improvement is necessary.
Good enoughquality bar
floating line
![Page 74: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/74.jpg)
A Heuristic for Good Enough
1. X has sufficient benefits.
2. X has no critical problems.
3. Benefits of X sufficiently outweigh problems.
4. In the present situation, and all things considered, improving X would be more harmful than helpful.
BenefitsProblems
All conditionsmust apply.
![Page 75: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/75.jpg)
Good Enough...
…with what level of confidence? …to meet ethical obligations? …in what time frame? …compared to what? …for what purpose? …or else what? …for whom?
Perspectiveis
Everything
![Page 76: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/76.jpg)
Test Project Dynamics:Context Model
Test TeamExpertiseLoadingCohesionMotivationLeadershipProject Integration
Test LabTest PlatformsTest ToolsTest LibraryProblem Tracking SystemOffice Facilities
Product MissionStakeholdersQuality CriteriaReference Material
RequirementsProductProject LifecycleProject ManagementConfiguration ManagementDefect PreventionDevelopment Team
Development
MissionFind Important ProblemsAssess QualityCertify to StandardFulfill Process MandatesSatisfy StakeholdersAssure Accountability
Advise about QAAdvise about TestingAdvise about QualityMaximize EfficiencyMinimize CostMinimize Time
TestProcess
StrategyLogisticsWork-products
![Page 77: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/77.jpg)
Test Project Dynamics:Context Model
T es t T eam T est L ab
RequirementsDevelopment
M iss ion
T estP roc ess
![Page 78: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/78.jpg)
Test Project Dynamics:Context Model
GIV ENS GIV ENS
GIV ENSGIV ENS
M iss ion
CHOICES
![Page 79: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/79.jpg)
CHOI
VENSMotivation
Capability
Test Project Dynamics:Givens vs. Choices
Motivation: What testing does the situation require? Capability: Can we perform that testing in this situation?
![Page 80: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/80.jpg)
Tailoring the Test Process
Are GIVENS good enough?
Do CHOICES about process exploit the GIVENS and address the MISSION well enough?
Is MISSION achieved well enough?
How do you know?
![Page 81: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/81.jpg)
MISSION:The most important part
Find important problems Assess quality Certify to standard Fulfill process mandates Satisfy stakeholders Assure accountability
Advise about QA Advise about testing Advise about quality Maximize efficiency Minimize time Minimize cost
The quality of testing depends on which of thesepossible missions matter and how they relate.
Many debates about the goodness of testingare really debates over missions and givens.
![Page 82: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/82.jpg)
Slow down project toimprove testability
Driving analogy When on the freeway, go 65mph. When on a suburban artery, go 40mph. When in a neighborhood, go 25mph. When in a parking lot, go 5mph. Park very slowly.
Change Control Funnel The closer you are to shipping, the more reluctantly you
should change code. To minimize retesting, carefully review proposed
changes. “You can change what you like, but change may
invalidate everything we thought we knewabout quality.”
More risks,lower speeds.
time
![Page 83: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/83.jpg)
“Why didn’t you find that bug?”
Highway Patrol analogy: Do you realizehow hard it would be to patrol the highways sothat all speeders are always caught?
Risk-based improvement argument: Our goal is to find important problems fast. Was that bug important? Could we economically have anticipated it? If so, we’ll modify our approach.
Testability reframe: We didn’t find it because the developers didn’t make the bug obvious enough for us to notice. The developers should put better bugs into the code. Let’s make a more testable product.
![Page 84: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/84.jpg)
Exercise
We may want to purchase DiskMapper.
Analyze it and tell us what the testing issues might be.
![Page 85: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/85.jpg)
KEY IDEA
![Page 86: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/86.jpg)
Contrasting Approaches
In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.
In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.
Product
Tests
Tests
(yes, this is a simplified view)
![Page 87: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/87.jpg)
Forward-Backward Thinking
search for evidenceto corroborate or
refute the conjecture
discover evidenceand make conjectures
Evidence Conjectures
![Page 88: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/88.jpg)
Lateral Thinking
but periodically take stockof your status against your mission
Let yourself be distracted…
‘cause you never knowwhat you’ll find…
![Page 89: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/89.jpg)
Exploratory Testing Tasks
ExploreExecute
TestsDesignTests
Product(coverage)
Techniques
Quality(evaluation)
Discover the elements
of the product.
Discover howthe product
should work.
Discover testdesign techniquesthat can be used.
Decide whichelements to test.
Observe product behavior.
Speculate aboutpossible quality
problems.
Evaluatebehavior against
expectations.
Configure & operate the
product.
Select & applytest design techniques.
Testingnotes
TestsProblems
Found
![Page 90: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/90.jpg)
Taking Notes
Test Coverage Outline/Matrix Evaluation Notes Risk/Strategy List Test Execution Log Issues, Questions & Anomalies
It would be easier to test if you changed/added… How does … work? Is this important to test? How should I test it? I saw something strange…
![Page 91: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/91.jpg)
The Plunge in and QuitHeuristic
This is a tool for overcoming fear of complexity. Often, a testing problem isn’t as hard as it looks. Sometimes it is as hard as it looks, and you need to quit
for a while and consider how tackle the problem. It may take several plunge & quit cycles to do the job.
Whenever you are called upon to testsomething very complex or frightening, plunge in!
After a little while, if you are very confusedor find yourself stuck, quit!
![Page 92: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/92.jpg)
Exploration Trigger Heuristic:No Questions
If you don’t have any issues or concernsabout product testability or test environment,
that itself may be a critical issue.
When you are uncautious, uncritical, or uncurious,that’s when you are most likely to let important
problems slip by. Don’t fall asleep!
If you find yourself without any questions,ask yourself “Why don’t I have any questions?”
![Page 93: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/93.jpg)
Key Ideas ofExploratory Testing
The results from the tests you design and execute influence the next test you will choose to design and execute.
You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave.
You test what you know about, and you are alert for clues about behaviors and aspects of the product that you don’t yet know about.
![Page 94: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/94.jpg)
Doing Exploratory Testing
Keep your mission clearly in mind. Keep notes that help you report what you did,
why you did it, and support your assessment of product quality.
Keep track of questions and issues raised in your exploration.
To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.
![Page 95: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/95.jpg)
Exercise
Write an explicit test procedure.
Make it at least 10 steps long.
Include coverage and evaluation instructions.
![Page 96: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/96.jpg)
KEY IDEA
![Page 97: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/97.jpg)
It Boils Down To…
YOU: Skills, equipment, experience, attitude
THE BALL: The product, testing tasks, bugs
YOUR TEAM: Coordination, roles, support
THE GAME: Risks, rewards, project environment, corporate environment, your mission as a tester
YOUR MOVES: How you spend your attention and energy to help your team win the game.
![Page 98: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/98.jpg)
Rapid Testing
Develop your scientific mind. Know your coverage and evaluation strategy. Run crisp test cycles that focus first on areas of risk. Use a diversified test strategy that serves the mission. Assure that your testing fits the logistics of the project. Let your tests evolve as new information comes to light.
![Page 99: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/99.jpg)
Copyright © 1996-2001, Satisfice, Inc.
v1.0
![Page 100: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/100.jpg)
Rapid Test Management
Establish a strong and supportive testing role. Re-visit your strategy and mission every day. Advocate for bugs and testability. Maintain three lists: risks, issues, coverage. Test with your team. Continuously report the status and dynamics of
testing.
![Page 101: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/101.jpg)
KEY IDEA
![Page 102: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/102.jpg)
This is our role.We make informed decisions about quality possible.
We can do this under a variety of conditions.
Testers light the way.
![Page 103: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/103.jpg)
This is the formula…
You must demand… complete specs and quantifiable criteria. protected schedule and early involvement. zero defect philosophy and control over release. resources to achieve complete test coverage.
…for hypochondria.
![Page 104: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/104.jpg)
Look, testing is a difficult job.
“The perfect cure for hypochondria, 100 percent effective, is to contract a potentially fatal disease. It cures you instantly. It happened to me.”
Gene WeingartenThe Hypochondriac's Guide to Life and Death
That’s why they hire smart people, like us, to do it for them.
![Page 105: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/105.jpg)
Too many textbooks seem to thinktesters are wind-up toys.
“This model identifies a standards-based life cycle testing process that concentrates on developing formal test documentation to implement repeatable structured testing on a software or hardware/software system. The general intent is that the test documentation be developed based on a formal requirements specification document...Once the documentation is developed, the test is executed.”
-- from a real article about testing. (I added the boldfacing to emphasize instructions)
Where’s the tester in this picture?
![Page 106: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/106.jpg)
Testing skills compensate for a difficult project environment.
complete specs quantifiable criteria protected schedule early involvement zero defect philosophy control over release complete test coverage
– implicit specs & inference– meaningful criteria– risk-driven schedule– good working relationship– good enough quality– don’t be the gatekeeper– enough information
Instead of this… consider this.
![Page 107: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/107.jpg)
Goal setting Risk identification
Task planning Risk planning
Task completion Risk reduction
failure
failure
success
success
We think about failure,and that can be “negative”.
![Page 108: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/108.jpg)
Eight Commitmentsworth making to developers
We’ll test your code as soon as we can after it’s built. We’ll test important things first, and focus on important problems. We’ll write clear, thoughtful, and respectful problem reports. We’ll try not to be a bottleneck for development. We’ll tell you how we’re testing, and consider your suggestions. We’ll look for ways to test better and faster. We’ll try to accommodate how you like to work. We will not waste your time.
![Page 109: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/109.jpg)
Our job is information.If they don’t want information…
Stop fretting and just be as helpful as you can. Let events play out. Carefully record what happens. Detach yourself from the outcome and refocus
on improving the next project, instead. Help the team use the experience as a
reference point for improvement.
use the Reality Steamroller technique:
![Page 110: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/110.jpg)
KEY IDEA
![Page 111: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/111.jpg)
Public Relations Problem
“What’s the status of testing?”
“What are you doing today?”
“When will you be finished?”
“Why is it taking so long?”
“Have you tested _______, yet?”
![Page 112: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/112.jpg)
The Problem
Management has little patience for detailed test status reports.
Management doesn’t understand testing. Testing is confused with improving. Testing is considered a linear, independent task. Testing is assumed to be exhaustive. Testing is assumed to be continuous. Test results are assumed to stay valid over time. Impact of regression testing is not appreciated. Test metrics are hard to interpret.
![Page 113: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/113.jpg)
A Solution
Report test cycle progress in a simple, structured way...
...that shows progress toward a goal... ...manages expectations... ...and inspires support... ...for an effective test process.
No process?
No Problem!Easy setup
![Page 114: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/114.jpg)
The Dashboard Concept
Project conference roomLarge dedicated whiteboard
“Do Not Erase”
Project status meeting
![Page 115: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/115.jpg)
Test Cycle Report
Test Effort
Test Coverage
Quality Assessment
Time
Product Areas
vs.
vs.
![Page 116: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/116.jpg)
Areafile/editviewinsertformattoolsslideshowonline helpclipartconvertersinstallcompatibilitygeneral GUI
Efforthighlowlowlowblockedlowblockednonenonestart 3/17start 3/17low
C.11+22+12011003
Q. Comments
1345, 1363, 1401
automation broken
crashes: 1406, 1407
animation memory leak
new files not delivered
need help to test...
need help to test...
lab time is scheduled
Testing Dashboard Updated: Build:
2/21 38
![Page 117: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/117.jpg)
Product Area
15-30 areas (keep it simple) Avoid sub-areas: they’re confusing. Areas should have roughly equal value. Areas together should be inclusive of
everything reasonably testable. “Product areas” can include tasks or risks-
but put them at the end. Minimize overlap between areas. Areas must "make sense" to your clients,
or they won’t use the board.
Areafile/editviewinsertformattoolsslideshowonline helpclipartconvertersinstallcompatibilitygeneral GUI
![Page 118: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/118.jpg)
Test Effort
None
Start
Low
High
Pause
Blocked
Ship
Not testing; not planning to test.
Regression or spot testing only; maintaining coverage.
Focused testing effort; increasing coverage.
No testing yet, but expect to start soon.
Temporarily ceased testing, though area is testable.
Can’t effectively test, due to blocking problem.
Going through final tests and signoff procedure.
![Page 119: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/119.jpg)
Test Effort
Use red to denote significant problems or stoppages, as in blocked, none, or pause.
Color ship green once the final tests are complete and everything else on that row is green.
Use neutral color (such as black or blue, but pick only one) for others, as in start, low, or high.
![Page 120: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/120.jpg)
Test Coverage
0
1
1+
2
2+
3
We don’t have good information about this area.
More than sanity, but many functions not tested.
Common & Critical:
Sanity Check:
Some data, state, or error coverage beyond level 2.
Complex Cases:
all functions touched; common & critical tests executed.
strong data, state, error, or stress testing.
major functions & simple data.
![Page 121: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/121.jpg)
Test Coverage
Color green if coverage level is acceptable for ship, otherwise color black.
Level 1 and 2 focus on functional requirements and capabilities: can this product work at all?
Level 2 may span 50%-90% code coverage. Level 2+ and 3 focus on information to judge
performance, reliability, compatibility, and other “ilities”: will this product work under realistic usage?
Level 3 or 3+ implies “if there were a bad bug in this area, we would probably know about it.”
![Page 122: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/122.jpg)
Quality Assessment
“We know of no problems in this area that threaten to stop ship or interrupt testing, nor do we have any definite suspicions about any.”
“We know of problems that are possible showstoppers, or we suspect that there are important problems not yet discovered.”
“We know of problems in this area that definitely stop ship or interrupt testing.”
![Page 123: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/123.jpg)
Use the comment field to explainanything colored red, or any non-green
quality indicator.
Comments
Problem ID numbers. Reasons for pausing, or delayed start. Nature of blocking problems. Why area is unstaffed.
![Page 124: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/124.jpg)
Using the Dashboard
Updates: 2-5/week, or at each build, or prior to each project meeting.
Progress: Set expectation about the duration of the “Testing Clock” and how new builds reset it.
Justification: Be ready to justify the contents of any cell in the dashboard. The authority of the board depends upon meaningful, actionable content.
Going High Tech: Sure, you can put this on the web, but will anyone actually look at it???
![Page 125: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/125.jpg)
KEY IDEA
![Page 126: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/126.jpg)
Exploratory testing relies on tester intuition. It is unscripted and improvisational.
How do I, as test manager, understand what’s happening, so I can direct the work and defend it to my clients?
![Page 127: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/127.jpg)
SKILLthere’s no shortcut
No one can read your mind.
You must gain the skill to explain your testing…so that you can be accountable for it.
That requires a lot of practice.In our experience: several months of daily practice.
This is a black box…Just like your mind.
![Page 128: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/128.jpg)
Introducing the Test Session
1) Charter
2) Time Box
3) Reviewable Result
4) Debriefing vs.
![Page 129: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/129.jpg)
Charter:A clear mission for the session
A charter may suggest what should be tested, how it should be tested, and what problems to look for.
A charter is not meant to be a detailed plan. General charters may be necessary at first:
“Analyze the Insert Picture function”
Specific charters provide better focus, but take more effort to design: “Test clip art insertion. Focus on stress and flow techniques, and
make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”
![Page 130: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/130.jpg)
Time Box:Focused test effort of fixed duration
Brief enough for accurate reporting. Brief enough to allow flexible scheduling. Brief enough to allow course correction. Long enough to get solid testing done. Long enough for efficient debriefings. Beware of overly precise timing.
Short: 60 minutes (+-15)Normal: 90 minutes (+-
15)Long: 120 minutes (+-15)
![Page 131: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/131.jpg)
Debriefing:Measurement begins with observation
The manager reviews session sheet to assure that he understands it and that it follows the protocol.
The tester answers any questions. Session metrics are checked. Charter may be adjusted. Session may be extended. New sessions may be chartered. Coaching happens.
![Page 132: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/132.jpg)
Reviewable Result:A scannable session sheet
Charter #AREAS
Start Time Tester Name(s) Breakdown
#DURATION #TEST DESIGN AND EXECUTION #BUG INVESTIGATION AND REPORTING #SESSION SETUP #CHARTER/OPPORTUNITY
Data Files
Test Notes Bugs
#BUG
Issues #ISSUE
CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | ViewStrategy | Function TestingStrategy | Functional Analysis START-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20
![Page 133: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/133.jpg)
The Breakdown MetricsTesting is like looking for worms
Test Design and Execution
Bug Investigation and Reporting
Session Setup
![Page 134: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/134.jpg)
Reporting the TBS BreakdownA guess is okay, but follow the protocol
Test, Bug, and Setup are orthogonal categories. Estimate the percentage of charter work that fell into
each category. Nearest 5% or 10% is good enough. If activities are done simultaneously, report the
highest precedence activity. Precedence goes in order: T, B, then S. All we really want is to track interruptions to testing. Don’t include Opportunity Testing in the estimate.
![Page 135: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/135.jpg)
Activity HierarchyAll test work fits here, somewhere
all work
non-session
session
opportunity on charter
test bug setup
inferred
![Page 136: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/136.jpg)
Non-Session61%
Test28%
Bug4%
Opportunity1%
Setup6%
Work Breakdown:Diagnosing the productivity
Do these proportions make sense? How do they change over time? Is the reporting protocol being
followed?
0.0
50.0
100.0
150.0
200.0
250.0
300.0
5/26 6/9 6/23 7/7 7/21 8/4 8/18
![Page 137: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/137.jpg)
Coverage:Specifying coverage areas
These are text labels listed in the Charter section of the session sheet. (e.g. “insert picture”)
Coverage areas can include anything areas of the product test configuration test strategies system configuration parameters
Use the debriefings to check the validity of the specified coverage areas.
![Page 138: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/138.jpg)
Coverage:Are we testing the right stuff?
Is it a lop-sided set of coverage areas?
Is it distorted reporting?
Distribution of On Charter TestingAcross Areas
0
20
40
60
80
100
120
• Is this a risk-based test strategy?
![Page 139: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/139.jpg)
Using the Data to Estimate a Test Cycle
1. How many perfect sessions (100% on-charter testing) does it take to do a cycle? (let’s say 40)
2. How many sessions can the team (of 4 testers) do per day? (let’s say 3 per day, per tester = 12)
3. How productive are the sessions? (let’s say 66% is on-charter test design and execution)
4. Estimate: 40 / (12 * .66) = 5 days5. We base the estimate on the data we’ve collected.
When any conditions or assumptions behind this estimate change, we will update the estimate.
![Page 140: Copyright © 1996-2002, Satisfice, Inc. v1.4.3 I grant permission to make digital or hard copies of this work for personal or classroom use, provided that](https://reader036.vdocuments.net/reader036/viewer/2022062517/56649f175503460f94c2e0aa/html5/thumbnails/140.jpg)
Challenges ofHigh Accountability Exploratory Testing
Architecting the system of charters (test planning) Making time for debriefings Getting the metrics right Creating good test notes Keeping the technique from dominating the
testing Maintaining commitment to the approach
For example session sheets and metrics seehttp://www.satisfice.com/sbtm