et workshop v. 1.20 - test management©2002 amland consulting5-1 exploratory test management outline...

ET Workshop v. 1.20 - Test Management

©2002 Amland Consulting 5-1

Exploratory Test Management Outline

Introduction

Test Management and Techniques

ET Planning, Exec. and Documentation

ET Styles

ET Management

• Exploratory Test Team Management

• Risk-Based Test Management

• Session Based Test Management

1.

2.

3.

4.

5.



Exploratory Test Management

These slides are distributed under the Creative Commons License.

In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one.

For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.



5. Exploratory Test Management

5.1 Exploratory Test Team Management

5.2 Risk-Based Test Management5.3 Session Based Test Management

1.

2.

3.

4.

5.



Introduction


ET Planning and Documentation

ET Styles

ET Management Traditional Test Teams… …are built on…

Available knowledge and skillsBusiness needsEmployee’s ambitions

…provide a career path through…Execution coordination planningDefect reporting analysis Strategy

This might not work for Exploratory Testing



How to build and lead an effective test team

Slides and Questionnaire from Lloyd Roden, Grove Consultants,

[email protected]

Copyright © 2002 Lloyd Roden, Grove Consultants ([email protected])

1.

2.

3.

4.

5.



Introduction



ET Styles

ET Management The questionnaire

Things to noteThings to noteNo right or wrong answerTry not to think too muchHelps us assess our strengthsCan be used with other management questionnaires (Belbin, 16PF etc)Plot values on the graph

Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)



Introduction



ET Styles

ET Management

The questionnaire - how to complete

Friendly __ Approachable __ Casual __Open __Unstructured __Social __Intuitive __Random __Warm __Perceptive __ __

Formal __Retiring __Business Like __Guarded __Organised __Introvert __Logical __Focused __Cool __Insensitive __

X - Axis

X

X

XXX

X6

XX

X

X




Introduction



ET Styles

ET Management

The questionnaire - how to complete

To the point __ Challenging __ Quick __Insistent __Lively __Impatient __Adventurous __Confronting __Competitive __Strong Minded __ __

Indirect __Accepting __Leisurely __Thoughtful __Relaxed __Patient __Cautious __Receptive __Co-operative __Analytical __

Y - Axis

X

X

2

XXX

XX

XXX




Introduction



ET Styles

ET Management The Grid

X (6,2)


1 2 3 4 5 6 7 8 9 10

10

9

8

7

6

4

3

2

1

0

0



Introduction



ET Styles

ET Management The Model

The The

PragmatistPragmatistThe The

PioneerPioneer

The The AnalystAnalyst

The The FacilitatorFacilitator




Introduction



ET Styles

ET Management

Key words for the ‘Pragmatic’ Style Tester

LikesLikes

strategic / goalspositiveresults / briefpracticalefficiencytasks

DislikesDislikes

indecisionvaguenesstime-wastingunproductive




Introduction



ET Styles

ET Management

The ‘Pragmatic’ style tester will…

be good for setting and monitoring short/long term goals for the teambe good at documenting factual ‘test reports’remain positive through pressurebe keen to adopt ‘Most Important Tests’ first principlebe a strong driving force - ensure a task is donewant to implement efficiency into the teambe self-motivated and task orientedwill make quick decisionsenjoy challenging testing tasks




Introduction



ET Styles

ET Management

Key words for the ‘Pioneer’ Style Tester

LikesLikes

new / ideaschangeopennessresults/efficiencyinvolving othersrisks

DislikesDislikes

standardsdetail‘norm’paper-work




Introduction



ET Styles

ET Management

The ‘Pioneer’ style tester will…

be good at ‘ad-hoc’ testing / bug hunting / error-guessing/ exploratory testingbe good at challenging and improving things to make more efficient and effectiveenjoy “GUI” type testing/lateral testerhave good ideasbe good at brainstorming Test Conditionsshare ideas about different ways to approach testingidentify and take necessary risks when requiredhave creative test ideas - how to find more faults




Introduction



ET Styles

ET Management

Key words for the ‘Analysing’ Style Tester

LikesLikes

accuracyattention to detailproofstandardsreliableall alternatives

DislikesDislikes

new / changeuntested / risksbrief / speedletting go




Introduction



ET Styles

ET Management

The ‘Analysing’ style tester will…

be good at defining and documenting test casesbe good at producing test standards and proceduresanalyse problems and finding root causeproduce work which is accurate and completeenjoy logical tests scenariosprovide proof when faults are founddocument thorough test reports complete work regardless of what it takeschallenge requirements




Introduction



ET Styles

ET Management

Key words for the ‘Facilitating’ Style Tester

LikesLikes

networkingpositiveteam orientedconsensus / sharingbuilding bridgesstatus quo

DislikesDislikes

pressure / deadlinesconfrontationisolation dictated




Introduction



ET Styles

ET Management

The ‘Facilitating’ style tester will…

be good in a RAD environment or a ‘buddy’ test teamoften ask opinion before raising issuesbe good at documentation co-operate well with other departmentsoften see the ‘other side’be good at defusing ‘us’ v ‘them’ syndromebe popularmake things happen - eventually!will provide support in testing to other team members




Introduction



ET Styles

ET Management Tester Style - patternsusually operate within a certain boundarywhat if you are on the line/centre

flexible between stylescan be difficult to manage

opposites repel maybe reasons for team tension!

analysts & pragmatists tend towards ‘Tasks’facilitators & pioneers tend towards ‘People’




Introduction



ET Styles

ET Management The Ideal Test Team…

A MIXTURE IS THE BEST

BUT IT DEPENDS...Copyright © 2002 Lloyd Roden, Grove Consultants (www.grove.co.uk)






1.

2.

3.

4.

5.



1.

2.

3.

4.

5.

”Ordinary” Project Plan including Testing

Specification Design Coding Test

Plan:

Specification Design Coding Test

In the Real World:



1.

2.

3.

4.

5.

”Incremental” Project Plan with Testing

Spec. Design Coding Test



Plan:

In the Real World:Spec. Design Coding Test?

Spec. Design Coding Test?

Spec. Design Coding Test......



1.

2.

3.

4.

5. Why Risk Based Testing?It’s about controlling:

TimeResourcesNew TechnologyLack of knowledgeLack of experienceQuality DemandsScope???



1.

2.

3.

4.

5. What is Risk?

“A risk is an unwanted event that has negative consequences.”

Shari Lawrence Pfleeger

In other words, a risk is a problem waiting to happen.

Shari Lawrence Pfleeger (2000)



1.

2.

3.

4.

5.

What is Risk Management?

Plans to avoid these unwanted

events or, if they are inevitable,

minimize their negative consequences.

Shari Lawrence PfleegerShari Lawrence Pfleeger (2000)



1.

2.

3.

4.

5. Problems vs. Risk

Problem (Issue):Something that has or will happen

RiskSomething that might happen (in the future)



1.

2.

3.

4.

5. Types of RiskProject / Process Risk

ResourcingPlanningContractsEtc.

Business / Product RiskStabilityPerformanceQuality / Errors (Quality Risk = Potential Errors)Etc.



1.

2.

3.

4.

5.

Business Risk – Why is it so difficult? (1)

Flaw in the Pentium chip (1994)6 million PCs sold, potential USD300 loss per chipIntel’s risk impact USD 1.8 billionIntel:

”Average” computer user: wrong answer every 27.000 years”Heavy user”: every 270 yearsConclusion: The flaw is not meaningful to most users

IBM:”Average” computer user: a problem every 24 daysA large company (500 PCs): 20 problems per day!Conclusion: stopped selling Pentium PCs

IBM’s assessment 400.000 times worse than Intel’s!

Pfleeger (2000)



1.

2.

3.

4.

5.

Business Risk – Why is it so difficult? (2)

Evaluating the risk of an accident at a small ammonia storage plant (1988 to 1990)11 EU countries (national teams) + several private firms

The national teams’ assessment varied by a factor of 25.000, reaching wildly different conclusions.

Pfleeger (2000)



1.

2.

3.

4.

5. How to identify risk?

1. A loss associated with the event

2. The likelihood that the event will occur

3. The degree to which we can change the outcome



1.

2.

3.

4.

5.

What is Risk-Based Testing?

”Risk-based testing carries at least two major meanings:”

Risk-Based Test ManagementTo determine what things to test next (prioritisation)

Doing Risk analysis for the Purpose of Finding Errors

Lessons Learned in Software Testing, Cem Kaner et al 2001b



1.

2.

3.

4.

5.

RBT: The purpose of finding errors

Make a prioritised list of risks.

Perform testing that exploreseach risk.

As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop.

James Bach 1999a



1.

2.

3.

4.

5.

Risk-Based Test Management

1. Define all requirements to be tested 2. Based on risk assessment –

prioritise the requirements3. Plan and define tests according to

requirement prioritisation (coverage to be defined in test plan)

4. Execute test according to prioritisation

and acceptance criteria (as defined in the test plan)



1.

2.

3.

4.

5.

Risk-Based Test Management

Statistical Risk Analysis (Hans Schaefer)Failure Mode and Effect Analysis (FMEA)Software Reliability engineering, John D. MusaRisk based test strategy (TMap, Iquip.nl))Decision Theory (Bayesian Belief Nets)Rational Unified Process (RUP)...and many, many more



1.

2.

3.

4.

5.

Statistical Risk Analysis – Theory

Equation:

Re(f) - Risk Exposure of function fP(f) - Probability of a fault in function fC(f) - Cost related to a fault in function f

P(f)*C(f)Re(f)

RBT Management - Statistic Risk Analysis – Hans Schaefer

Schaefer 1998, Amland 1999



Introduction



ET Styles

ET Management

Statistical Risk Analysis – Matrix

RBT Management - Statistic Risk Analysis - Hans Schaefer


CostWeight (1-5) Weight (1-5) Risk

Function 1 3 Wieghted Exposure(1 - 3) Indicator 1 Indicator 2 Average

A 2 1 1 2 4

B 1 3 2 4,5 4,5

C 3 2 2 4 12

Probability

Weighted



Introduction



ET Styles

ET Management

Statistical Risk Based Testing

Plan: Identify Elements to be TestedLogical or physical Functions, Modules etc.

Identify Risk IndicatorsWhat is important to predict the probability of faults?

Identify Cost (consequence) of faults

Identify Critical ElementsI.e. functions, tasks, activities etc. based on Risk Analysis (Indicators and Cost)

ExecuteImprove the Test Process and Organization: Schedule and Track




Introduction



ET Styles

ET Management

Example – Statistical Risk Analysis Matrix

RBT Management - Statistic Risk Analysis

InterestCalc.

Close Account

CustomerProfitablty

C(s) C(c) Avrg.NewFunc.

DesgnQual. Size

Com-plexity

Weigh.Sum

RiskExposure

5 5 1 3

3 3 3

1 3 2

2 1 1,5

2 3 3 3 37

2 2 2 3 31

3 3 2 3 41

111

62

61,5

Other Probability Factors might include: Function Points, Frequency of Use etc.

Cost * Probability = Re




Introduction



ET Styles

ET Management

Risk Based Testing - Reporting

510

440

10 1

TECHNICAL

INTERFACE

RISK

439

370

11 2 2

369

302

5

Low Medium High

BUSINESS RISK

Low Medium High

Low

M

ediu

m

Hig

h

Consequence

Pro

bab

ilit

y






1.

2.

3.

4.

5.



1.

2.

3.

4.

5.

Introducing the Test Session

1) Charter2) Time Box3) Reviewable Result4) Debriefing vs.

From Rapid Software Testing, copyright © 1996-2002 James Bach

Charter/Mission

Notes, Risks, Issues,Questions and Errors

Execution: Pairs & Sessions

DebriefingDebriefing



1.

2.

3.

4.

5. Charter – summary…“Architecting the Charters” i.e. Test PlanningBrief information / guidelines on:

Mission: Why do we test this?What should be tested?How to test (approach)?What problems to look for?

Might include guidelines on:Tools to useSpecific Test Techniques or tactics to useWhat risks are involvedDocuments to examineDesired output from the testing

Charter/Mission






Introduction



ET Styles

ET Management

Time Box:Focused test effort of fixed duration

Brief enough for accurate reporting.Brief enough to allow flexible scheduling.Brief enough to allow course correction.Long enough to get solid testing done.Long enough for efficient debriefings.Beware of overly precise timing.

Short: 60 minutes (+-15)Normal: 90 minutes (+-

15)Long: 120 minutes (+-15)


Charter/Mission






Introduction



ET Styles

ET Management

Reviewable Result: The session sheet

Charter#AREAS

Start TimeTester Name(s)Breakdown

#DURATION#TEST DESIGN AND EXECUTION#BUG INVESTIGATION AND REPORTING#SESSION SETUP#CHARTER/OPPORTUNITY

Data Files

Test NotesBugs

#BUG

Issues#ISSUE

CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | ViewStrategy | Function TestingStrategy | Functional Analysis START-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20


Charter/Mission






Introduction



ET Styles

ET Management

Debriefing:Measurement begins with

observationThe manager reviews session sheet to assure that he understands it and that it follows the protocol.The tester answers any questions.Session metrics are checked.Charter may be adjusted.Session may be extended.New sessions may be chartered.Coaching / Mentoring happens.


Charter/Mission






1.

2.

3.

4.

5.

Exploratory Test Management Summary

Introduction



ET Styles

ET Management

• Exploratory Test Team Management

• Risk-Based Test Management

• Session Based Test Management



Exercise 5Based on available information;

Discuss how you would manage and track a test team working on testing StarOffice. What would your ideal test team look like?What management strategy would you use? Why?Outline a risk matrix

Select a few areas or functionsIdentify probability indicatorsHow to define Consequence of an error?



Summary (1)Strengths:

Customer-focused, risk-focusedTakes advantages of each tester’s strengthsResponsive to changing circumstancesWell managed, it avoids duplicative analysis and testingHigh Bug find rates

Blind Spots:The less we know, the more we risk missingLimited by each tester’s weaknesses (can be mitigated with careful management)This is skilled work, juniors aren’t very good at it

Copyright © 1996 – 2002 Cem Kaner



Summary (2)Exploratory testing in pairs, is a very effective test approach

Use it as a complementary testing approach (it depends…)

Skilled testers can become very good exploratory testers.

Novice testers need mentoring to do exploratory testing.



Learn more…Test Training

www.testingeducation.org

Exploratory Testing:Rapid Software Testing, by James Bach, www.satisfice.com [email protected] Box Software Testing, by Cem Kaner, www.kaner.com [email protected] Consulting, www.amland.no, [email protected]

People Issues, test techniques, inspections:Grove Consultants, www.grove.co.uk+ many, many more



Presentation - Summary

Introduction



ET Styles

ET Management

Introduction: Exploratory Testing and Risk,Why and when to use Exploratory Testing.

Test Management, Test Techniques and Test Process

ET Planning, Documentation and Execution.How to do Exploratory Testing

Exploratory Test StylesHow to do Exploratory Testing, continued

Exploratory Test Management How to Manage Exploratory Testing and Teams

1.

2.

3.

4.

5.



References/Bibliography (1)

Amland, S., 1999. Risk Based Testing and Metrics, EuroSTAR '99 Preceding, Barcelona, Spain, http://www.amland.no/articlesAmland, S., 2000. ”Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study”, Journal of Systems and Software, Volume 53, Issue 3, 15.9.2000, Elsevier publishing, UK.Argus, C. & Johnson, B., Ad Hoc Software Testing; Exploring the Controversy of Unstructured Testing, http://www.testingcraft.com/ad_hoc_testing.pdf Bach, J., 1999a. Risk-Based Testing. How to conduct heuristic risk analysis, Software Testing & Quality Engineering Magazine, November/December 1999, vol. 1, issue 6, http://www.stqemagazine.com. See also satisfice.com Bach, J. , 1999b. What is Exploratory Testing? www.satisfice.com Bach, Jonathan, 2000, Session-Based Test Management, STQE Magazine, 11/00, www.satisfice.comBeizer, Boris, Software Testing Techniques, Van Nostrand Reinhold. NY. 1990.Beizer, Boris, 1995, Black Box Testing, Wiley




Bereza-Jarocinski, B., 2000, Test and Decision Theory, Enea Data AB, [email protected], +46 50 709 714 293 (www.bbj.com.pl)Black, Rex, 1999. Managing the Testing Process, Microsoft Press.Buwalda, Hans, Janssen, Dennis, and Pinkster, Iris, 2001. Integrated Test Design & Automation Using The TestFrame Method, Addison Wesley, Copeland, Lee, Exploratory Planning, Sep. 3, 2001, StickyMinds.comCollard, Ross, 2002, to be published, [email protected] .Fenton, N.E. & Pfleeger, S.L., 1997. Software Metrics, a rigorous & practical approach, 2nd edition, International Thomson Computer Press.Fewster, M., Graham, D., 1999, Software Test Automation: Effective Use of Text Execution Tools, Addison-WesleyGerrard, P., 1999, Risk-Based Testing, TEST Congress, London, UK, www.evolutif.co.uk Gilb, T., Principles of Software Engineering Management, Addison-Wesley, Wokingham




Kaner, C., Falk, Nguyen, 1999, Testing Computer Software, John Wiley & SonsKaner, C., 2000. Rethinking Software Metrics, Evaluating measurement schemes, Software Testing & Quality Engineering Magazine, March/April 2000, vol. 2, issue 2.Kaner, C., 2001a, Black-Box Software Testing, training at Satisfice July 2001.Kaner, C., Bach, J., Pettichord, B., 2001b, Lessons Learned in Software Testing, John Wiley & Sons; http://www.testinglessons.com/, ISBN: 0471081124 Kaner, C., Bach, J. 2001c, Exploratory Testing in Pairs, presentation at STAREast, Orlando, FL, www.kaner.com Karolak, Dale Walter, “Software Engineering Risk Management”, IEEE Computer Society Press, 1996.Keefer, G., 2002. Extreme Programming Considered Harmful for Reliable Software Development, www.avoca-vsm.com Keith, Geordie, 2002, “All Hands on Deck: How we used our whole company to test”, STQE Magazine, vol. 4, issue 4, July / August 2002.Lyndsay, James and Eeden, Niel van, 2002, “Adventures in Session-Based Testing”, paper at StickyMinds.com presented at EuroSTAR 2002




Marick, Brian, 1995, The Craft of Software Testing, Prentice Hall PTR, ISBN 0-13-177411-5Myers, Glenford, 1979. The Art of Software Testing, John Wiley & SonsNeumann, P.G., Computer-Related Risks, Addison-Wesley/ACM Press, ISBN 0-201-55805-X, 1995, http://www.csl.sri.com/users/neumann/Nguyen, 2000, Testing Applications on the WEB, John Wiley & SonsOttevanger, Dr. I.B., 1999. A Risk-Based Test Strategy, Presented at STARWest 1999, http://www.sqe.com/startwest. Part of TMAP, IQUIP Informatica B.V. PO Box 263, 1110 AG Diemen, The Netherlands (www.iquip.nl).Pfleeger, S.L., 2000. Risky Business: what we have yet to learn about software risk management, Journal of Systems and Software, issue 11, 2000, Elsevier Publishing, UK.Robinson, H., Microsoft, Exploratory Modeling, http://www.testingcraft.com/exploratory-robinson.html, home page: http://www.geocities.com/model_based_testing/Roden, Lloyd, 2001. How to build and lead an effective test team, tutorial EuroSTAR 2001, Grove Consultants, www.grove.co.uk, [email protected]




Schaefer, H., 1998. Surviving under time and budget pressure, keynote STAR West '98, [email protected], http://www.sqe.com/starwest, http://home.c2i.net/schaefer/Våga, J., Amland, S., 2002. High Speed Web Testing, contribution to the book ” Software Quality and Testing in Internet Times”, Editor: Dirk Meyerhoff et al, Springer-Verlag, Berlin Heidelberg, Germany 2002, ISBN:3-540-42632-9Whittaker, J. and Jorgensen, A., 1999. Why software fails. ACM Software Engineering Notes, July. http://se.fit.edu/papers/ and www.stickyminds.com Whittaker, J. and Jorgensen A., 2002, “How to Break Software: A Practical Guide to Testing”, Addison-Wesley, ISBN: 0-201-79619-8, presented at EuroSTAR 2000 Øvstedal, E. Ø. and Stålhane, T., 1992. A goal oriented approach to software testing, Reliability Engineering and System Safety. Elsevier Science Publishers Ltd., UK.



Linkswww.stickyminds.comwww.satisfice.comwww.kaner.comwww.testingeducation.orgwww.pettichord.comwww.amland.nohome.c2i.net/schaefer/www.bbj.com.plhttp://www.stqemagazine.comhttp://www.testinglessons.com/http://www.context-driven-testing.com/ www.grove.co.uk



Ståle Amland,Amland Consulting

Hulda Garborgsv. 2, N-4020 STAVANGER

NorwayPhone: +47 905 28 930Fax: +47 51 58 55 24

e-mail: [email protected]: www.amland.no

Contact Details

1.

2.

3.

4.

5.

et workshop v. 1.20 - test management©2002 amland consulting5-1 exploratory test management outline...

Documents

et workshop

techniques et planning

effective test team

exploratory testing

lloyd roden

grove consultants

questionnaire things

graph copyright