Test Process Improvementwith TPI Next

What the model does NOT tell you BUT you should know

Tapani Aaltio Sogeti Finland Oy

Lvov, Ukraine Nov 7, 2013

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Using a Maturity Model as a Reference Helps in Structuring and Communicating the Improvement Actions

…How can you tell if you do the right things (right)…

TMM(i)

CMM(i)

SPICE

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Improving the Test Process is a Continuous Change Towards Better Maturity

Evaluate and Redirect

Implement Actions

Make a Plan of Action

Define Improvements

Create Awareness

Determine Goal, Scope

and Approach

Assess Current Situation

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The TPI NEXT Model Consists of 16 Key Areas, Four Maturity Levels and 157 Checkpoints

Test maturity matrix

Key areas16

Maturity levels4

Checkpoints157

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

In the TPI NEXT Model, the Field of Testingis Divided into 16 Key Areas in Three Groups

Stakeholder Relations

1 Stakeholder commitment

2 Degree of involvement

3 Test Strategy

4 Test Organization

5 Communication

6 Reporting

Test Management

7 Test Process Management

8 Estimating and Planning

9 Metrics

10 Defect Management

11 Testware Management

Test Profession

12 Methodology Practice

13 Tester Professionalism

14 Test Case Design

15 Test Tools

16 Test Environments

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The TPI Model has Four Maturity Levels, with Requirements for Each Key Area on Each Maturity Level

• Adhoc Activities

Initial

• Doing the right Things

Controlled

• Doing it the right way

Efficient

• Continuously adapting

Optimizing

The test strategy enables the distribution of the test efforts and resources among the test levels

and test activities.

The test strategy is used to achieve adequate balance between analyzed product risks, test coverage and available efforts and resources,

involving all stakeholders.

The method of the test strategy is maintained well to ensure easy and valid application.

Description of Key Area ”Test Strategy” on different maturity levels

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The TPI Model has Four Maturity Levels, with 2-4 Checkpoints for Each Key Area on Each Maturity Level

• Adhoc Activities

Initial

• Doing the right Things

Controlled

• Doing it the right way

Efficient

• Continuously adapting

Optimizing

Examples of check points of Key Area ”Test Strategy” on different maturity levels

The principal stakeholder agrees with the documented test strategy.The test strategy is based on product risk analysis.

All relevant stakeholders agree with the documented test strategy.The overlap and gaps in test coverage between test levels or test types are well considered.

The process of creating the test strategy is evaluated regularly.The test strategy is evaluated against metrics on indicents that occur in production.

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The Results of an Assessment are Presented Visually in a Testing Maturity Matrix

16 Key Areas • Doing the right Things

Controlled

• Doing it the right way

Efficient

• Continuously adapting

Optimizing

Total 157 Check Points

63% of checkpoints

met

12% of checkpoints

met

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

To Understand the Whole and the Details, Use Multiple Channels to Gather Information

Interview the right people with different

points of view

Investigate documents to find

evidence

“Hang aroundin the lobby”

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Defining the Improvements is a Crucial Step – And it is Fully Supported by the TPI NEXT Model

Evaluate and Redirect

Implement Actions

Make a Plan of Action

Define Improvements

Create Awareness

Determine Goal, Scope

and Approach

Assess Current Situation

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The TPI Model Offers Concrete Improvement Suggestions for Each Key Area on Each Maturity Level

• Adhoc Activities

Initial

• Doing the right Things

Controlled

• Doing it the right way

Efficient

• Continuously adapting

Optimizing

Include various stakeholders in Product Risk Analysis and Test Strategy.

Investigate possible overlaps between different test levels. Consider combining test levels, if possible.

Define the role of Test and Evaluation Coordinator, to co-ordinate and monitor all test and evaluation

activities.

Improvement Suggestions of Key Area ”Test Strategy” on different maturity levels

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The Clusters (A, B, C, … ) Help in Finding the Balanced Improvement Paths to a Higher Maturity Level

16 Key Areas • Doing the right Things

Controlled

• Doing it the right way

Efficient

• Continuously adapting

Optimizing

Prioritize Key Areas to create Business Driven

Improvement Paths

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The Complete TPI® NEXT Model

Improvement suggestions

Key areas

Maturity levels

Checkpoints

Enablers

Clu

sters

Test Maturity Matrix

e.g. Requirements Management andProject management

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Making a Realistic Plan of Action Requires Detailed Planning – Don’t be Greedy and Try to Do Too Much!

Evaluate and Redirect

Implement Actions

Make a Plan of Action

Define Improvements

Create Awareness

Determine Goal, Scope

and Approach

Assess Current Situation

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

To Sell your Ideas to Management, Remember Use Both Facts and Feelings

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

If the Planned Improvement Actions cannot be Implemented, the Whole Investment has Gone to Waste

Evaluate and Redirect

Implement Actions

Make a Plan of Action

Define Improvements

Create Awareness

Determine Goal, Scope

and Approach

Assess Current Situation

The fun part

The tough part

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Typical Impediments for Improvement – as Seen by Customers

Resource availability for the Improvement Project Resource competencies in testing and quality

assurance Cultural change – not easy to implement Improving feels like “an additional effort” No clear benefits seen Unclear scope and focus No owner for the improvement

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

The Organization must be Ready to Accept Declining Productivity in the Early Phases of Implementation

Work systematically to reach a higher

level

Current level

Goal

Don’t give up here!

Productivity

Time

Currentlevel

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

In a Successful Improvement Project, Assessing the Current Situation Represents only 10-30% of Total Work

# Task Sogeti(days)

Customer (days)

% of total

1 Assessing the Current Situation

20 5 12,5%

2 Organizing and training the Customer Test Improvement Team

20 20 20,0%

3 Define General Test Agreements and Test Processes

40 10 25,0%

4 Support for Implementation of the new process in projects

60 25 42,5%

Total 140 60 100,0%19

Example Improvement Project Work Estimates

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

TPI NEXT Provides Tangible Profit to Business

Improved quality (60%) Prevented damage by finding a majority of critical & major

defects in testing phase Shorten time for testing (24%)

Reduction in test duration due to a uniform controlled test strategy and risk based testing approach

Reduce cost (35%) Less test cases & higher coverage = less test effort Shorter fix cycle

Implement Test Fix Test Fix Test Etc

Deadline?

Why not do it right the first time – and minimize the fixing cycle?

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Quality Gates Help in Catching Defects in the Same Phase Where They are Injected – No Faults Forward!

Project without quality gates

ST FAT SIT E2E UAT Aftercare

47

74

21

6

109

167

Defects per Test Level

ST FAT SIT E2E UAT Aftercare

47

74

21

6

109

167

519 21

54

34 29

Defects per Test Level

Waste, customer dissatisfaction

Project with quality gates

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

ROI calculation for No Faults Forward! -without Quality Gates vs. with Quality Gates

Baseline # defects (without quality gates) 276

Pilot # defects (with quality gates) 63

Reduction # defects 213

Test cost per defect ≈€ 1.500,00

Rework cost per defect ≈€ 2.000,00

Total cost per defect ≈€ 3.500,00

# Defects * cost per defect € 745.500,00

Investment QA-consultant € 30.400,00

Cost reduction € 715.100,00

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Summary

TPI NEXT is an easy-to-use, practice based assessment model for your test process

The model provides balanced, business-driven improvement paths to reach a higher level of maturity

The implementation of the improvements is hard work. The following ideas might help you achieve good results Engage the organisation from day one Use facts and feelings to sell your improvements to key people and

management Find an executive sponsor – and keep him/her engaged Proceed in small steps Demonstrate added value early Don’t give up before you have reached a higher level of productivity

For more information on TPI NEXT, navigate to http://www.tmap.net/en/tpi-next

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Information on TPI NEXT is Easily Available in Different Formats

Check out online resources at

http://www.tmap.net/en/tpi-next, including the Excel-based Test Maturity Matrix ToolBuy the book at

http://www.utn.nl/and read it!

Download Android Appat Google Play

Pass the EXIN TPI FoundationCertificate exam

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Now it’s your Turn – Questions, Answers and Discussion...

Thank you for your attention, this was

“Test Process Improvement with TPI Next – What the model

Does NOT tell you BUT you should know”

Tapani AaltioSogeti Finlandtapani.aaltio@sogeti.com

Lvov, Ukraine Nov 7, 2013

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Back-up Slides

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

TPI Next Can be Applied together with CMMi, Using CMMi-driven Clusters

Download white paper on TPI Next and CMMi at http://www.tmap.net/en/tpi-next/publications

Tapani Aaltio: What the TPI Model does not tell you – but you should knowSQA Days Lvov, Nov 7,

2013

Prep Spec Exec Comp

Infra

Ctrl

Plan

The Structured Test Process According to TMap®

Preparation Collection of the test basis, Testability assessment

Specification Creating test specifications, Defining central starting points, Specifying test object intake

CompletionEvaluating the test process, Preserving testware

ExecutionIntake of test object, Preparing the starting points, Executing the (re)tests, Checking and assessing the test results

Planning (main tasks)Analysing the product risk, Determining the test strategy, Estimating the effort, Defining test products, organisation and infrastructure

ControlManagement, Monitoring, Reporting, Adjusting

Infrastructure managementSpecifying the infrastructure, Realising the infrastructure, Specifying the infrastructure intake, Intake of the infrastructure, Maintaining the infrastructure, Preserving the infrastructure