the business value of sw quality

The Business Value

of SW QualityMinks, November 2015

G. Bazzana – ISTQB® President

ISTQB® 2015 2

• Quantifying the impacts on sw engineering practices on Sw Project productivity– Approach– Quantitative Data

• The importance of testing– Good practices– Certification of sw testing competencies– ISTQB®

Contents

ISTQB® 2015 3

Drivers

COST QUALITY RISK AGILITY• Quicker

time to market

• Consumption based pricing

• Increasing dependency of business from SW

• Multiple platforms

• SW Intensive systems

• Knowledge retention

• Competition• Tightening

margins

ISTQB® 2015 4

APPLICABLE MARKETS

ITEngineeringTelco

ISTQB® 2015 5

• We need to provide evidence of quantitative benefits

• In God we trust, all others must bring data

Identifying «Best Practices»

Metrics

Benchmarks

ROI Analysis

ISTQB® 2015 6

BASIC DEFINITIONS OF SOFTWARE QUALITY

• Functional Software QualitySoftware that combines low defect rates and high levelsOf user satisfaction. The software should also meet alluser requirements and adhere to international standards.

• Structural Software QualitySoftware that exhibits a robust architecture and can

operateIn a multi-tier environment without failures or degraded performance. Software has low cyclomatic complexitylevels.

• Aesthetic Software QualitySoftware with elegant and easy to use commands andInterfaces, attractive screens, and well formatted outputs.

ISTQB® 2015 7

ECONOMIC DEFINITIONS OF SOFTWARE QUALITY

• “Technical debt”The assertion (by Ward Cunningham in 1992) thatquick and careless development with poor quality leadsto many years of expensive maintenance and enhancements.

• Cost of Quality (COQ)The overall costs of prevention, appraisal, internal failures, and external failures. For software these mean defect

prevention,pre-test defect removal, testing, and post-release defect

repairs.(Consequential damages are usually not counted.)

• Total Cost of Ownership (TCO)The sum of development + enhancement + maintenance +support from day 1 until application is retired.(Recalculation at 5 year intervals is recommended.)

ISTQB® 2015 8

World Quality Report Findings

ISTQB® 2015 9

• Budgets dedicated to QA&Testing are growing at warp speed, with a growth pace much higher than anticipated, reaching in average 35% of the IT budgets

• Test automation is increasing, jumping in one year from 28% to 45% of test cases

• Digital transformation is putting more and more emphasis on customer value and end-user experience as targets for testing

• Security is the most pressing concern• Agile and DevOps adoption has become widespread and call for extreme

level of speed and integration in testing practices; a lack of professionals with Agile Testing expertise is among the top-three challenges

• Mobile testing is maturing, being adopted by 92% of the organizations and consuming 35% of the QA&Testing Budgets for new projects

• Testing Centers of Excellence are widely adopted and are turning into more agile and hybridized forms

• Test environments are being enhanced with virtualization and cloud solutions

HIGHLIGHTS FROM THE 2015 WORLD SW QUALITY REPORT

ISTQB® 2015 10

• Software engineering and software project management are complex activities. Both software development and software management have dozens of methodologies and scores of tools available that are beneficial. In addition, there are quite a few methods and practices that have been shown to be harmful.

• In order to evaluate the effectiveness or harm of various methods and practices Capers Jones has developed a scoring method.

• Data collected from 1984 through 2015– About 725 companies (150 clients in Fortune 500)– About 40 government/military groups– About 25,000 total projects– New data = monthly benchmark studies– Data collected from 27 countries– Observations during more than 17 lawsuits

• The analysis is based on the author’s book Software Engineering Best Practices published by McGraw Hill in 2010. Some new data is taken from The Economics of Software Quality published by Addison Wesley in 2012.

QUANTITATIVE BENCHMARKS(Based on work of Capers Jones)

ISTQB® 2015 11

Certified reusable components80%

Experienced development teams 65%

Effective methodologies for specific project types 40%

High-level programming languages 30%

Use of inspections for complex systems 27%

Experienced managers25%

Moderate unpaid overtime by teams 20%

Impact of Critical Factors on Software Productivity . POSITIVE (1)

ISTQB® 2015 12

Low requirements creep20%

Logical, planned architecture for large systems 20%

Model-based development20%

Due diligence on COTS acquisitions 20%

Use of static analysis before testing 18%

High CMMI levels15%

Low cyclomatic complexity (< 10) 15%

Effective project status tracking15%

Effective defect prevention15%

Experienced test teams12%


ISTQB® 2015 13

Experienced clients10%

SCRUM < 1000 function points 10%

24-hour continuous development 7%

Effective parametric estimating tools 7%

Testing by certified test personnel 7%

Annual staff training > 5 days7%

Formal mathematical test case design 6%

Co-located teams5%


ISTQB® 2015 14

High requirements creep: poor change control -60%

Inexperienced managers -50%

Truncating testing to "meet schedule" -45%

Concealing problems in status reports -45%

Inexperienced clients-40%

Chaotic, unplanned architecture for large systems -37%

Inexperienced development teams -35%

Inaccurate manual estimates -33%

Concurrent maintenance and development tasks -30%

False claims by outsource vendors -30%

Impact of Critical Factors on Software Productivity – NEGATIVE (1)

ISTQB® 2015 15

Adding personnel to late projects -25%

Low-level programming languages -25%

Excessive unpaid overtime by team -23%

Manual estimates > 1000 function points -23%

Poor status tracking -20%

Unverified, buggy COTS acquisitions -20%

High cyclomatic complexity (> 25) -18%

Ineffective methodologies -15%

Inexperienced test teams -15%

Distributed teams: poor communications -15%


ISTQB® 2015 16

Waterfall > 5000 function points -12%

Low CMMI levels -10%

Agile > 5000 function points - 9%

Informal test case design - 8%

No annual staff training - 7%

Testing by developers only - 6%


ISTQB® 2015 17

CONCLUSIONS ON SOFTWARE QUALITY

• No single quality method is adequate by itself.

• Inspections + static analysis + formal testing > 99% efficient.

• Defect prevention + pre-test removal + formal test best overall

• Higher CMMI levels, TSP, RUP, Agile, XP are effective

• Quality excellence has ROI > $15 for each $1 spent

• High quality benefits schedules, productivity, users!

• Poor quality leads to cost and schedule overruns!

ISTQB® 2015 18

USEFUL AND HARMFUL QUALITY METRICS

• Useful quality metrics– Defect potentials using function points– Defect detection efficiency (DDE)– Defect removal efficiency (DRE)– Delivered defects per function point– Defect removal cost per function point– Cost of quality (COQ)– Cyclomatic complexity– Test coverage

ISTQB® 2015 19

US AVERAGE FOR SW QUALITY AS OF 2015

Defect Removal DeliveredDefect Origins Potential Efficiency Defects

Requirements 1.11 88.00% 0.13Architecture 0.25 94.00% 0.02Design 1.20 93.00% 0.08Coding 1.30 97.00% 0.03Documents 0.50 93.40% 0.04Bad Fixes 0.35 85.00% 0.07

TOTAL 4.71 92.57% 0.35

(Data expressed in terms of defects per function point)

(Function points show all defect sources - not just coding defects)(Code defects = 35% of total defects)

ISTQB® 2015 20

RANGES OF DEFECT REMOVAL EFFICIENCY

Worst Median Best Defect Potentials 1,100 1,000 900 Defect Prevention 40% 50% 60%

Pre-test inspections 70% 85% 90%

Pre-test static analysis 40% 65% 75%

Unit tests 20% 25% 35%

New Function tests 25% 35% 45%

Regression tests 15% 25% 35%

System test 35% 45% 55%

Acceptance/Beta tests 20% 30% 40%

DELIVERED DEFECTS 53 18 2

REMOVAL EFFICIENCY 95% 98% 99.7%

ISTQB® 2015 21

ROI FROM STRUCTURED TESTING

Requirement Analysis Developme

nt Test Production

5 5 % 20 % 40 % 30 % < 5 %

4 3 % 12 % 30 % 50 % 5 %

3 0 % 2 % 20 % 70 % 8 %

2 0 % 0 % 3 % 80 % 17 %

1 0 % 0 % 2 % 50 % 48%

FAULT DISTRIBUTION

1 5 20 50 100

TEST

MAT

URI

TY

ISTQB® 2015 22


With no structured test process, 50% of the overall issues are found during the test phase.

With a structured test it is reasonable to expect to find 80% of the issues during the test phase.

Bug fixing cost if the problem is found during the test phase: 2 md

Bug fixing cost if the problem is found after release: 4 md A medium sized application generates overall 500 issues (both

test and production). Suppose to reserve a testing team for an effort of 180 md, then…

ISTQB® 2015 23


RESULTS: Issues in production (after release) are reduced by 60%The overall effort (test + bug fixing) is reduced by 8%

Without Structured

Test

With Structured

TestNumber of faults found during test 250 400Effort to fix during test 500 800Number of faults found in production 250 100Effort to fix faults found in production 1000 400Effort for structured testing 0 180

Overall effort 1500 1380

ISTQB® 2015 24

• In the following some factors related to testing are presented, with their impact in terms of reduction or increase in project work hours wrt to projects consisting of traditional application development methods such as “waterfall” development performed by organizations that do not have apply systematically sound software engineering practices

INCREASING PRODUCTIVITY OF SW PROJECTSGOOD TESTING PRACTICES CAN HELP

Factor Impact on productivity

Experienced test teams 12% ImprovementTesting by certified test personnel

7% Improvement

Testing by developers only 6% WorseningInformal test case design 8% WorseningInexperienced test teams 15% WorseningTruncating testing to "meet schedule"

45% Worsening(Capers Jones, Scoring and evaluating software methods, practices, and results, version 15.0,

September 6, 2014)

ISTQB® 2015 25

• ISTQB®: International Software Testing Qualifications Board (www.istqb.org):

• - Non-profit association

– Founded in 2002– Has its own constitution, rules and regulations– Composed of volunteer international Testing Experts– Responsible for the “ISTQB® Certified Tester” scheme worldwide

• ISTQB® is the world’s leading organization for Software Testing Certification

WHAT IS ISTQB®?

Advancing the software testing profession

http://www.istqb.org/

ISTQB® 2015 26

ISTQB® - Current Product Architecture

ISTQB® 2015 27

“To continually improve and advance the software testing profession by:

Defining and maintaining a Body of Knowledge which allows testers to be certified based on best practices,

connecting the international software testing community, and encouraging research.”

ISTQB® VISION

ISTQB® 2015 28

ISTQB® VIDEOS

• ISTQB® Videos give you insights into the ISTQB® Certified Tester scheme

http://www.istqb.org/introduction-to-istqb.html





ISTQB® 2015 29

WORLD-WIDE COVERAGE

Countries covered by Member Boards (49 Member Boards covering 72 countries, representing over 90% of the world-wide GDP) and Global Exam Providers

Countries covered by Global Exam Providers The list of Member Boards and Global Exam Providers is available on the ISTQB® Web Site

http://www.istqb.org/istqb-where-you-are.html



ISTQB® 2015 30

More than

560.000 exams world wide!

Figures as of 2015Q2Executive Summary

In 2015Q2, 18993 exams and 14036 certificates

Trend YOY is OKWrt FL+AL+EL + 8,5%

Adding-up Agile + 14,5%

Close to

410.000 certificates world wide!

ISTQB® 2015 31

Agile uptake VERY Good:

- Almost 2000 exams - 1600 certificates

- By 33 Boards- Forecast for 2015: over

4000 exams

Figures as of 2015Q2 Executive Summary (2)

Certificates issued in 108 countries

ISTQB® 2015 32

ISTQB® ECO-SYSTEM

ISTQB® 2015 33

• International recognition of acquired competencies and skills

• Authorized to use the “Certified Tester” logo (specifying the level of certification)

• Whole career path support, from Foundation to Expert level

• Higher appeal in the labor market

BENEFITS FOR PROFESSIONALS

ISTQB® 2015 34

BENEFITS FOR PROFESSIONALS - SURVEY

• Would you recommend the ISTQB® Foundation Level (CTFL) certification to your colleagues?

ISTQB® 2014 35

BENEFITS FOR COMPANIES

• ISTQB® certification can provide a competitive advantage for companies, promising a higher level of reliability of the applications being developed due to efficient and cost effective testing practices derived from the ISTQB® competencies

• Consulting companies with certified staff can offer higher-level services to customers, increasing revenues and brand value

• ISTQB® has defined a “Partner Program” for companies that engage a large number of certified testers

ISTQB® 2014 36

BENEFITS FOR ATPs(Accredited Training Providers)

• Educational institutions and consulting companies may become an ISTQB® Accredited Trainer Provider (ATP) according to processes and rules defined at the international level

• Accredited Training Providers ensure a high standing of training is delivered through having:• certified trainers• the content, quality and syllabus coverage of

training materials checked by ISTQB® Boards• advance notice of changes to the ISTQB®

Glossary and syllabi • Accredited Training Providers are entitled to use

relevant logos and are listed in the ISTQB® Website

ISTQB® 2015 37

• The exam is based on multiple-choice questions for Foundation and Advanced Levels• For Expert Level, an essay is also required• Exams can be attempted also without having attended a training course (e.g. through

self-study)

EXAMS

ISTQB® 2015 38

• Questions are classified according to the cognitive level, the K-level (also known as level of knowledge):

• K1 = Remember (recognize a term or concept)• K2 = Understand (able to explain a concept)• K3 = Apply (select correct application of concept or technique)• K4 = Analyze (can distinguish between facts and inferences for deeper

understanding)• K5 = Evaluate (can make judgments based on criteria and standards)• K6 = Create (can put elements together to form a coherent or functional whole)

• The number of questions for each topic is related to the length of the topic in the syllabus

• For more details, see the FAQ section in the ISTQB® Website

EXAM QUESTIONS – K LEVELS

ISTQB® 2014 39

Provides recognition to Companies

that are investing in theISTQB® scheme

ISTQB® PARTNER PROGRAM

ISTQB® 2015 40

ISTQB® Planned evolution

ISTQB® is evaluating an evolution of its product portfolio architecture and contents in order to: Maintain the mission and vision of ISTQB® and keep the

high quality of deliverables that has marked the success of the scheme world-wide

Make the scheme more modular Make it easier for professionals to obtain the certification

they are interested in Provide an overall framework in which all future potential

modules may fit in a way which is coherent and understandable to our stakeholders

Maintain the validity of certifications already obtained

ISTQB® 2015 41

The customer perspective

Some people like to follow a breadth-first generalist approach (going from Foundation to the “classic” Advanced)

Whereas others prefer to follow a depth-first approach (going from Foundation to Specialist)

We have so many different professionals interested in our programme that we cannot think we can propose a “one size fits all” approach …

Indeed, it is exactly the goal of a modular approach to allow people to design their own training/ certification path and allow them to broaden their knowledge base

ISTQB® 2015 42

Evolution Axes

The review of the product architecture and product portfolio embraces three aspects

1. The overall framework in which the various certification modules have to fit

2. The modules that populate the framework: fitting the current modules (the existing ones and those under development) and identifying/ positioning the potential new ones

3. The preconditions/ pre-requisites/ entry criteria

ISTQB® 2015 43

Overall Framework/ architectureLevels and categories

The future ISTQB® Portfolio will be a Matrix characterized by: Levels (FL, AL, EL): they identify progressively

increasing LOs Categories: they identify differing target groups of

certification modules: Core

they cover a topic in a broad, horizontal way, valid for any technology/ methodology/

application domain Allowing for a common understanding

Agile So important in new developments

Specialist they cover a specific topic/ domain in a vertical

way, using a drill down / deep-dive approach

Let’s Make a Difference…Let’s support our

Customers…

THANK YOU!