TEST METRICS IN AGILE TEST MANAGEMENT

A POWERFUL TOOL TO SUPPORT CHANGES

Yulia Zavertailo, Senior Test manager

03.05.2023

Agenda

− About me.

− Introduction to the client’s case.

− What needs to be improved?

− How do we visualize our results?

− A closer look at the KPIs and how to gather them.

− Conclusions.

/ 3

03.05.2023

About me

− 10 years of work experience in the field of Testing and Quality Assurance, 7.5 years in Itera.

− Moved to Oslo in November 2014. − Key competences:

− test management, − test advisory on process establishment

in a project and entire organization. − Very passionate about the ISO

9000/ISO29119 standards.− Mentor and coach for junior testers. − Love running, skiing and oil painting.

/ 4

03.05.2023

IF Skadeforsikring – general

/ 5

− Waypoint is the largest digital solution at IF built on modern technology platform with lots of interfaces for clients and internal users (1000+).

− Methodology – Agile (Scrum, Kanban).

− 13 Agile teams.

− Each team has both business and IT competence.

− IT development is done in headquarters in Oslo.

− Large Scandinavian insurance company present in the Nordic countries and Baltics with 3,6 millions of clients and 6 800 employees.

03.05.2023

– System is not covered by tests.

– No professional IT testing is done in the project.

– Business users are involved to do the functional acceptance and

regression testing.

– Long production cycle: releases are 3-4 times a year.

/ 6

IF Skadeforsikring - as a test challenge

− Critical issues found in production after release.

03.05.2023

IF Skadeforsikring - in a result

− IT test team has grown from 1 to 17 IT testers in Riga, Latvia.

− Frequency of releases is every 5 weeks.

− Response on the entire system quality and stability within a few hours.

− Project discovers and fixes 136 bugs in average every release .

− There are still functional modules which do not have enough test

coverage = facing high risk of defects.

/ 7

03.05.2023

Specifically… what are we going to improve?

1. Increase frequency of releases so that IT delivers functionality to the

business every 5 weeks by:

– Automating the regression test coverage.

– Discovering and fixing bugs earlier in the cycle.

– Improving development teams efficiency by doing continuous IT testing.

/ 8

03.05.2023

Specifically… what are we going to improve?

2. Decrease amount of bugs in the production by:

– Running automated regression suits regularly.

– Improving the quality of testing itself.

– Introducing continuous manual testing while developing.

– Focus business resources doing acceptance testing only.

/ 9

03.05.2023

Hmmm…. but how to prove the succeeded result?

Present test results our client is interested in:

– Test coverage.

– # of defects found in production.

– # of defects found in sprints.

– Time spent by end users.

– Time for feedback about system’s quality.

/ 10

03.05.2023

KPI #1 – Test Coverage

What do we measure?

− Percentage of requirements covered by at least one test.

Why do we gather it in our Agile projects?

– Visualize “black spots” of our application.

– Visualize that the larger test coverage is, the less risk of missing

defects is.

– Visualize the functionality that needs more attention since it is less

covered by tests.

/ 11

03.05.2023

Application test coverage - example

/ 12

Module 1 Module 2 Module 3 Module 4 Module 5 Module 60

200

400

600

800

1000

12001075

739

430

800

230

419

310244

130

465

70141

5413 8

694 10

Total user stories US covered manual US covered automated

03.05.2023

How to gather Test Coverage KPI

1. Configure your Agile TM tool in order to build a traceability matrix in

TM tool (TFS, Jira, QC).

2. Use the graphs module of your TM tool or export to Pivot tables in

Excel.

3. Go for the Excel-option if you do not have any TM tool.

/ 13

03.05.2023

KPI #2 - Defect open and close rates

What Do we measure?

Defect Open and Close rate = Defects found during testing in a release *100 (Defects found during testing in a release + Defects found after release)

/ 14

Why do we gather it in our Agile projects?− Categorizing defects as either open or closed.

− Certain defects could slip through the cracks and show up in the finalized

release.

− Testers and developers to work together to identify and address software

issues.

03.05.2023

Sprint vs production defects - trend example

/ 15

Releas

e 1

Releas

e 2

Releas

e 3

Releas

e 4

Releas

e 5

Releas

e 6

Releas

e 7

Releas

e 8

Releas

e 9

Releas

e 10

Releas

e 11

Releas

e 12

Releas

e 13

Releas

e 14

Releas

e 15

Releas

e 16

Releas

e 17

Releas

e 18

Releas

e 19

Releas

e 20

Releas

e 21

0

50

100

150

200

250

170

131

85 85

150

85105

129141

112

182 174152

231

175

229

139

226

175

222

124

267 13 7 10 17 10

29 20 17

47

2033

21 21 25 2413 15 18

0

144124

72 78

140

68

95 100121

95

135154

119

210

154

204

115

213

160

204

124

Total Production Defects Sprint Defects

03.05.2023

How to gather defect open and close rates KPI

To get a rate:

− Get the number of defects found during a release/sprint.

− Get the number of bugs reported by the business after release to

production.

− Calculate rate based on a formula.

To build a trend graph:

− Jira + Zephyr dashboards.

− TFS + pivot tables in Excel.

/ 16

03.05.2023

KPI #3 - Issues reported by customers

What Do we measure?

% of customer reported issues = Total # of issues reported by customers *100 total number of issues reported

Why do we gather it in our Agile projects?− Does the product meet the needs of a customer/end user?

− Effectiveness of test team.

− Types and number of defects lingering in released products.

− Critical issues with current QA processes?

/ 17

03.05.2023

Customer issues reported – example of trend

/ 18

Release 1

Release 2

Release 3

Release

4

Release

5

Releas

e 6

Releas

e 7

Release 8

Release 9

Release 10

Release 11

Release 12

Release 13

Release 14

Releas

e 15

Releas

e 16

Release 17

Release 18

Release 19

Release 20

Release 21

0

50

100

150

200

250

112 7 3 6 4 2 3 2 1 4 4 3 1 2 9 2

14 8 11 5

144124

72 78

140

68

95100

121

95

135

154

119

210

154

204

115

213

160

204

124

Customer reported isues Bugs all

03.05.2023

Issues reported by customers per functional area – example of trend

/ 19

Release1

Release2

Release3

Release4

Release5

Release6

Release7

Release8

Release9

Release10

Release11

Release12

0 5 10 15 20 25 30 35 40 45 50

10

3

10

1

6

3

9

3

2

5

15

5

6

1

3

1

1

3

2

1

1

12

2

3

1

4

4

1

5

1

3

4

3

7

2

2

1

2

1

4

6

5

5

2

5

1

2

2

2

2

1

3

9

5

5

5

4

6

4

7

6

2

2

2

1

2

4

Module1 Module2 Module3 Module4 Module5 Module6 Module7

03.05.2023

How to gather issues reported by customers KPI

To get a rate:

− Get the number of defects found by customers.

− Get the total number of defects found in a sprint.

− Calculate the rate based on a formula.

To build a trend graph:

− TFS + pivot tables in Excel.

/ 20

03.05.2023

KPI #4 – Efficiency of end users’ testing

/ 21

What do we measure?

− Time spent by end users.

− # of bugs found by end users during acceptance testing of release vs

total.

Why do we gather it in our Agile projects?

− Feedback about the quality of requirement/acceptance criteria.

− Feedback about the quality of the functional and regression testing.

− To see the cost (time spent vs amount of bugs found).

03.05.2023

Time spent by end users – example of trend

/ 22

Releas

e 1

Releas

e 2

Releas

e 3

Releas

e 4

Releas

e 5

Releas

e 6

Releas

e 7

Releas

e 8

Releas

e 9

Releas

e 10

Releas

e 11

Releas

e 12

Releas

e 13

Releas

e 14

Releas

e 15

Releas

e 16

0

50

100

150

200

250

300

350

9783

135.5

161.5

124 127 123

237

152.5135.5

288.5

112

131.5147.5

159

110

30 30 29 28.5 30 31 31 2615.5 16.5 21.5 15 14.5 16 12 13.5

80 7562

54

3447 48

40 40

12 10 7.5 6.5 8 9 5 7 6 5 8 5 10 9.5 5 6.5

30 29

79 8569 69 69

96

3948

193

54 54 54

77

42

TotalCountry Domain 1Country Domain 2Country Domain 3Country Domain 4Country Domain 5

03.05.2023

KPI #5 - Regression test suite duration

What do we measure?

− Time for feedback about system’s stability and quality – TA.

− Time for feedback about system’s stability and quality – Manual.

Why do we gather it in our Agile projects?

− Get a clue about time needed to regression-test a hot fix or feature.

− Decision-making input for further planning.

− Visualized benefit and efficiency of test automation.

/ 23

03.05.2023

Regression test suite duration - example

/ 24

Test suite # of test cases in a suite

Time to run test suite with TA, hrs

Time to run suite manually, hrs

Smoke test suite 6 0,2 1,5

Regression suite 1 191 5,5 47,75

Regression suite 2 204 5,5 51

Regression suite 3 194 5,5 48,5

Total 595 16,7 149

03.05.2023

KPIs framework – in a few simple steps

1. Decide which KPIs are important for your client (focus on KPIs 1-3).

2. Configure your test management tool and visualize.

3. Continuously use KPIs at all levels of communication.

/ 25

03.05.2023

Summary

− KPIs framework is a powerful tool to be an Agile test leader!

− Control the status of quality in a measurable way.

− Control the general healthiness of the test solution and strategy in your

project.

− Lead and make decisions in your project.

− Empower yourself to prove that IT- testing is a must in your project!

/ 26

03.05.2023

QUESTIONS?

/ 27

03.05.2023

Contacts

• LinkedIn: https://no.linkedin.com/in/yuliazavertailo • Email: iuliia.zavertailo@itera.no

/ 28

03.05.2023

THANK YOU!

/ 29