risk-based testing for agile projects
DESCRIPTION
Many projects implicitly use some kind of risk-based approach for prioritizing testing activities. However, critical testing decisions should be based on a product risk assessment process using key business drivers as its foundation. For agile projects, this assessment should be both thorough and lightweight. PRISMA (PRoduct RISk MAnagement) is a highly practical method for performing systematic product risk assessments. Learn how to employ PRISMA techniques in agile projects using risk-poker. Carry out risk identification and analysis, see how to use the outcome to select the best test approach, and learn how to transform the result into an agile one page sprint test plan. Practical experiences are shared and results achieved employing product risk assessments. Learn how to optimize your test effort by including product risk assessment in your agile testing practices.TRANSCRIPT
W10
Agile Testing
5/7/2014 1:45:00 PM
Risk-Based Testing for Agile
Projects
Presented by:
Erik van Veenendaal
Improve Quality IT Services BV
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Erik van Veenendaal
Improve Quality IT Services BV
A leading international consultant, trainer, and recognized expert in software testing, Erik van Veenendaal (erikvanveenendaal.nl) is the founder of Improve Quality Services BV, a company that specializes in testing, requirements engineering, and quality management. Erik is the author of a number of books and papers, a core developer of the TMap testing methodology and the TMMi improvement model, a participant in the International Requirements Engineering Board working parties, currently on the TMMi Foundation board, and a frequent speaker at international testing conferences. For his major contribution to the field of testing, Erik received the 2007 European Testing Excellence Award.
© 2011 Improve Quality Services BV PRISMA tutorial- 1
1
Practical Risk-Based Testing
also for
Agile projects
PRISMA®
Erik van Veenendaal
www.erikvanveendaal.nl
Improve Quality Services BV 2
Erik van VeenendaalErik van Veenendaal
� Founder and major shareholder ImproveQS
� In testing since 1989 working for many
different clients and in many different roles
� Author “TMap”, “TMMi model” and many
other books and papers
� Former Vice-President International Software
Testing Qualifications Board (ISTQB)
� Supporting member IREB board
� Keynote speaker, e.g. EuroSTAR, STAR
� Winner of the European Testing Excellence
Award
www. erikvanveenendaal.nl
© 2011 Improve Quality Services BV PRISMA tutorial- 2
Improve Quality Services B.V. 3
Testing ChallengesTesting Challenges
� Increasing importance and size of
software in society as a whole− e.g., amount of software in consumer product
doubles every 24 months,
− number of defects is almost constant
� High Competition & Outsourcing− Time-To-Market, Product Quality, Price Levels are essential for
business success
� Testing often takes 30 - 40% of project costs
�Risk based testing deals with both effectiveness
and efficiency
Improve Quality Services B.V. 4
What is Risk?What is Risk?
� “A factor that could result in a future negative
consequence; usually expressed as impact and
likelihood” (ISTQB Glossary)
�Testers ‘only’ have the responsibility to identify the
risks and provide information on their status
� “to dare to undertake”
− management attitude and styleCC..
© 2011 Improve Quality Services BV PRISMA tutorial- 3
Testing = Risk ManagementTesting = Risk Management
� Objective: most feasible coverage
− effective usage of limited resources
− optimize test effort
� Limited resources
�staffing
�infrastructure
�time !
�..
� The right level and type of coverage on the right
parts at the right time
� Always applicable, whether it is V-model, Agile or ....
Improve Quality Services B.V. 5
Our Challenge!.Our Challenge!.
Improve Quality Services B.V. 6
if if only we only we
knew !!knew !!
© 2011 Improve Quality Services BV PRISMA tutorial- 4
RiskRisk--Based TestingBased Testing
�Risk identification looks at ways of
establishing what the risks are and where
they are
�Risk analysis looks into the critical, complex
and potential error prone areas
�Then we build tests to mitigate the risk
�Subsequently we monitor and report
regarding the risks
Improve Quality Services B.V. 7
Based on Practical ExperiencesBased on Practical Experiences
Improve Quality Services B.V. 8
8
© 2011 Improve Quality Services BV PRISMA tutorial- 5
Risk IdentificationRisk Identification
� Split up test basis in functional or technical items
� V-model: based on requirements
� Agile: backlog items / user stories for next sprint
� Can also be based on a brainstorm session
� Maximum number of appr. 35 risk items
Improve Quality Services B.V. 9
Risk item 1 Functionality
Risk item 2 Security
Risk item 3 Functionality
Risk item 4 Interoperability
Risk Risk AnalysisAnalysis
� Risk = impact x likelihood
− What is the impact for the business ?
− What is the likelihood that there are defects ?
� Determine factors based on previous projects, e.g.,
defect patterns
Improve Quality Services B.V. 10
Impact – business risk
Lik
eli
ho
od
tec
hn
ica
l ri
sk
You already know this !
Exercise: Risk Factors
© 2011 Improve Quality Services BV PRISMA tutorial- 6
Improve Quality Services B.V. 11
Factors From PracticeFactors From Practice
� Likelihood
− complexity
− new development
(level of re-uses)
− interfacing
− size
− technology
− geographical spread
− inexperience
(of development team)
� Impact
− business importance
(“selling item”)
− financial (or other)
damage (e.g. safety)
− usage intensity
− external visibility
− cost of rework
Customization
required
defect patterns / history
Risk Assessment in a NutshellRisk Assessment in a Nutshell
1) Stakeholder analysis: Assigning Factors
2) Individual scoring
3) Consensus meeting
4) Position the risk items in a Product Risk Matrix
5) Define a differentiated risk-based test approach
Improve Quality Services B.V. 12
Business
importance
Usage
intensity
Safety
Item 1
Item 2
Item 3
Item 4
5
5
4
5
5
4
3
2
they shall
make
choices
© 2011 Improve Quality Services BV PRISMA tutorial- 7
Product Risk MatrixProduct Risk Matrix
Improve Quality Services B.V. 13
IIII
IVIV
II
IIIIII
L
H
M
L HM
xx
x
x
x
Likelihood
Impact
Must Test
Should Test
Could Test
“Won’t Test”
focus of
unit testing
focus of
acceptance
testing
MoSCoW priorities
AgileAgile -- Play the Card Game: Play the Card Game: RiskRisk--PokerPoker
� Poker Planning / Wide Band Delphi based
Improve Quality Services BV 14
© 2011 Improve Quality Services BV PRISMA tutorial- 8
Improve Quality Services B.V. 15
Wide Band Delphi / Poker Planning (1)Wide Band Delphi / Poker Planning (1)
� Bottom-up estimation techniques- wide variance in estimates
- commitment of people involved
- accurate estimation we use the collective wisdom of the team
� Each estimator has cards with valid estimates
� A user story (product backlog item) is discussed
� Each estimator selects an estimate
� Cards are turned over
� Discuss differences (especially outliers)
� Repeat until consensus
Improve Quality Services B.V. 16
User Story
Explain
Estimates
Collect
Share info
OK?
Final
estimate
Moderator Estimator
OK?
Wide Band Delphi / Poker Planning (2)Wide Band Delphi / Poker Planning (2)
© 2011 Improve Quality Services BV PRISMA tutorial- 9
Improve Quality Services B.V. 17
Risk Poker the same, but!Risk Poker the same, but!
� Scoring separately for likelihood and impact
� Discussion shall be factor based
� Limited number of cards for each color (value)
� Agreement, Product owner (impact) and lead-
developer (likelihood) act as tie breaker
Improve Quality Services B.V. 18
Example User Story
As as bank account holder, I want a yearly financial
overview of my accounts, so that I can submit my taxes
Impact ?
Likelihood ?
Let’s PlayLet’s Play
Impact:
Likelihood:
© 2011 Improve Quality Services BV PRISMA tutorial- 10
Improve Quality Services B.V. 19
AgileAgile PRISMA PRISMA RiskRisk MatrixMatrix
See any “problems” with this?
Example
User
Story
Improve Quality Services BV 20
Differentiated Test Approach !!Differentiated Test Approach !!
� Test design
techniques
� Reviews & inspection
� Test design start-up
meetings
� Reviews of test design
� Level of detail of test
cases
� Exit criteria
� Level of independence
� More time & effort
� Most experienced
person
� Priority setting
� Regression testing
� Re-testing
without this risk assessment
doesn’t make much sense !!
© 2011 Improve Quality Services BV PRISMA tutorial- 11
Improve Quality Services BV 21
RealReal--LifeLife AgileAgile ExampleExample
IIII
IVIV
II
IIIIII
LL
HH
MM
LL HMM
xx
x
x
x
Likelihood
Impact
Must Test
Should Test
Could Test
“Won’t Test”
Test design techniquesReview designSupport module testsReview module tests
Test design techniquesReview design
Exploratory TestingSupport module testsReview module tests
Exploratory Testing
AgileAgile ““OneOne Page Test Plan”Page Test Plan”
Improve Quality Services B.V. 22
Must Test
C.. Test Approach C..
Should test
CC Test Approach C..
Could Test
C.. Test Approach C..
Would Test
C.. Test Approach C..
Shall include
a definition of “done”
A 1-2 hour session
to achieve a one page
Sprint Test Plan
© 2011 Improve Quality Services BV PRISMA tutorial- 12
CommunicationCommunication
Improve Quality Services B.V. 23
Product Risk Matrix Product Risk Matrix –– InitialInitial
US
58
US
31US
49
US
27
Impact - Business risk
Likelihood-TechnicalRisk
IVIII
III
© 2011 Improve Quality Services BV PRISMA tutorial- 13
Product Risk Matrix Product Risk Matrix –– ProgressProgress
US
58
US
31US
49
US
27
Impact - Business risk
Likelihood-TechnicalRisk
IVIII
III
Improve Quality Services BV 26
Key learning pointsKey learning points
��A structured and A structured and practical approachpractical approach for for
riskrisk--based based testing is testing is availableavailable
��Make it Make it AgileAgile, but use the formal rules, but use the formal rules
��Efficient, Effective and AttractiveEfficient, Effective and Attractive
��Define Define a a riskrisk--based based differentiated test differentiated test
approachapproach
��Provide Provide riskrisk--based based trackingtracking & & reportingreporting
© 2011 Improve Quality Services BV PRISMA tutorial- 14
DownloadsDownloads!!
On www.erikvanveenendaal.nl
� A full PRISMA white paper (e-book)
� Paper: 10 Recommendations on
Risk-Based Testing
� Paper: Product Risk Assessments in Agile Projects
(Risk-Poker)
� To acquire the PRISMA freeware tool sent a
request to [email protected]
� Book “Practical Risk-Based Testing; The PRISMA
Approach”, ISBN9-78490-986070 (www.utn.nl)Improve Quality Services B.V. 27
!!
Improve Quality Services B.V. 28