Better Outsourced TestingCreating and managing agreements with offshore vendors

Introduction

This presentation will highlight three key failure points to outsourced testing:

1.misunderstanding of the cost of quality;

2.misalignment of contractual arrangements to the required testing/delivery outcomes; and

3.lack of effective metrics.

Traditional outsourcing agreements..

..focus on the test execution statistics and not quality as an objective.

Is there a solution?

It is important to understand these issues can be mitigated and rectified by examining current testing capability and improving the testing processes, not by just policing the fine print in the outsourcing contract.

Defining the problem

Symptoms

Along with the typical less than optimal outcomes (e.g. poor quality, cost overruns and late delivery), other potential symptoms of poorly managed outsourced testing include:

● poor or missing test planning documentation;

● minimal test results;

● deployment issues to test environments;

● reactive focus on testing;

● high severity issues upon deployment;

● increases over time to the length of the test phase; and

● little to none build or test automation.

Organisations should “Understand that testing is required to verify and validate that a business objective is being realised... throughout the life cycle, not only at a user acceptance phase.

Therefore, clear functional and non-functional acceptance criteria should be defined for each phase of the life cycle.”

Gartner: ID:G00225707, ID:G00209445

Misunderstanding Cost of QualityDifferent outcomes from a unit cost agreement and one where total cost of quality is an objective

Unit Cost vs. Total Cost of Quality

Misaligned contractual arrangementsOften the commercial agreement is the only governance provided to measure, manage, report and maintain the output of the outsourced testing provider.

Different Outsourced Engagement Models

The true value of an outsource provider is meeting the client’s delivery need with the capability (and ability) to deliver

Lack of Effective Metrics

If an outsource testing provider is bound by unit cost measures like total defects found, the onus is on finding the maximum number of defects (regardless of quality of defects, analysis, triage, allocation, etc.) in the allotted period of time.

Common Results

Typically, the outcome will be an unquantified list of unqualified defects, test results with a high failure or blockage rate, and no working solution.The outsource provider may then be rewarded with more resources to come up with another outcome (reduce the number of defects or pass tests), giving the provider more revenue and leaving the client with cost and time overruns.

Organisations “have limited internal baseline metrics regarding quality and efficiency of their testing capabilities.”

Gartner: ID:G00225707, ID:G00209445

What’s the solution?

1: A Capability Review

Assessing the current testing capability may be as simple as reviewing past test results against estimated and planned outcomes. It could also involve a professional external organisation performing a capability review and auditing policies, processes, procedures and skills.

What is a Capability Review

This review can be used to recommend improvements to specific areas. These could include a new or updated test strategy (or master test plan), to ensure the alignment to desired outcomes, the introduction of metrics to align to KPIs in the outsource contract, and defining effective (light touch) governance around roles and responsibilities to reach the desired outcomes.

Capability Review

How and What to Measure?

relative test effort;

test coverage of requirements;

percentage of tests automated;

defects remaining upon completion, and

defect analysis.

These metrics should be holistically called the cost of quality.

The cost of defects increases throughout the testing lifecycle and the longer a defect stays within the process, the more costly it is to fix.

Cost of DefectsTo avoid cost overruns, consider using defect classification metrics to more easily identify trends in your test process

Cost of Fixing Software Defects

To avoid cost overruns, consider using defect classification metrics to more easily identify trends in your test process.

Defect reporting vs. defect analysis

Clients may already have effective defect reporting which can be invaluable for providing a clear view of the current status of quality. The move to defect analysis begins with reporting as an input, combined with additional quantitative information to allow analysis of what is called phase effectiveness.

There are two key metrics that assist with understanding trends in quality:

(PCE) Phase Containment Effectiveness

PCE is the ratio of faults captured in a phase and signifies how effective the testing process is at preventing these defects in the first place.

(PCE) Phase Containment Effectiveness

In Figure 4, the PCE is calculated as follows:

Requirements captured 10 of 22 defects = 45%

Design contained seven of 22 defects = 32%

Build contained three of 22 defects = 14%

Test contained two of 22 defects = 9%

Phase Containment Effectiveness

(PSE) Phase Screening Effectiveness

PSE is the ratio of previously missed (not contained) defects being captured in each phase.

For the same example, the PSE is calculated as:

Requirements missed 12 of 22 defects

Design screened seven of these 12 defects = 58%Build screened three of these 5 defects = 60%

Test screened two of the remaining two defects = 100%

Create or update a testing framework, which will fully align these metrics to the total cost of quality.

Organisations failing to fully understand their outsourced contracts and not including effective metrics and role accountabilities to monitor these often see cost overruns and project delays.

KPIs drive behaviour

Organisations will be left with statistics to prove the KPI was met, but a less than optimal quality outcome

3 Simple Steps to Rectify

Outsourced testing can be successful.

It can speed up the test execution, improve quality and reduce costs.

It is important to ensure the total cost of quality is understood, the quality outcomes are defined and aligned to the commercial agreement.

1: Capability Review

Involve a professional, external organisation to perform a capability review

2: Outsourced Agreement Review

Recommend improvements or enhancements to the organisation’s governance of the commercial arrangement and the expected testing outcomes

3: Improved Metrics

Together with changes to the metrics for both the outsourced provider and the client organisation should be used to report against the testing outcomes

Nathan Shearer - Want to work with me? rapidexecutive.com/coaching

Like This?Feel Free to Share it with someone who could benefit → Connect with me on LinkedInConnect with me on Twitter