managed testing services

MANAGED TESTING SERVICES

Focus on your core competencies by

test outsourcing

DR DANIEL SIMON

Senior Research Manager

[email protected]

Daniel Simon studied Computer Science and has been working for SQS since

2005. His main responsibilities include conducting research projects for the

Group, evolving and innovating SQS service portfolio, and demonstrating its

thought leadership. His core competencies relate to outsourcing, technical

quality, KPI systems, and quality management. His range of experience covers

all sectors of the IT industry, and he is Programme Chair for the iqnite

conferences 2012 at Vienna and Geneva.

DR FRANK SIMON

Head of SQS Research

[email protected]

Frank Simon studied Computer Science and has been with SQS since 2001.

He is Head of SQS Research, Head of the BITKOM working stream Software

Development Processes and Tools, and a Member of the German Testing

Board. His responsibilities include thinking ahead, innovating SQS service

portfolio, evaluating hypes and trends, conducting research projects, devel-

oping services, and presenting SQS research in lectures.

WHITEPAPER

Managed Testing Services Page 2

tion has achieved a reasonable level of maturity

and is demanding changes towards process and

organisational modularisation, standardisation,

automation, and focusing on core business com-

petencies. If the prerequisites are not yet met,

MTS in themselves will be worthwhile to develop

an organisation towards the higher maturity level.

Since MTS are a specic instantiation of the

outsourcing concept, the current status of this

market is analysed and an outlook is given on

expected future developments for MTS: Managed

Testing Services will mainly be delivered by inde-

pendent testing companies, blurring the differ-

ence to the large full-service providers.

Based on these high expectations, this paper de-

scribes the steps towards setting up and operating

MTS, focusing on costs and benets, opportun-

ities and risks in the context of these services.

Since MTS will often be applied to already existing

testing projects, a separate section explains the

transition step which is necessary to apply MTS.

As Adam Smith outlined as early as 1776 in his

famous An Inquiry into the Nature and Causes

of the Wealth of Nations (1), progress and a

qualitative increase in productivity in society

lie in specialisation and industrialism, essen-

tially represented by the concept of division of

labour. In Smiths work, the different approaches

to manufacture pins were investigated and com-

pared, but the same principles apply to the IT in-

dustry today and reect the evolution seen over

the past decades.

This whitepaper outlines the concept of Managed

Testing Services (MTS), an outsourcing technique

for testing-related tasks across one or more pro-

jects delivering one or more applications and

systems, spanning the life cycle of software and

system development and system operation.

The paper argues the idea of IT industrialisation

as the underlying motivation for using MTS, and

shows that MTS are an important step on the indus-

trialisation road. It explains the different param-

eters that need to be taken into account when de-

ciding on IT outsourcing, and gives an overview of

important prerequisites that have to be checked

before MTS benets can be leveraged.

MTS are a logical consequence of IT industrial-

isation in modern IT environments. MTS promise

signicant savings and quality improvements due

to the effects of division of labour and specialisa-

tion, as well as exploiting location-independent

delivery of well-dened pieces of work. Neverthe-

less, MTS are most efcient when an organisa-

This section gives a short overview of the funda-

mental basics of so-called Managed Testing Ser vices

(MTS). Before a clear denition is provided, it will

be shown how both IT and innovations in IT have

evolved over the last decades, explaining that Man-

aged Testing Services are a manifestation of a more

fundamental concept. This overall idea leads to

specialisation (i.e. organisations focus on their core

competencies), division of labour (i.e. outsourcing

in different variations) and (cross-country / cross-

cultural) supply chains (i.e. global delivery).

1 MANAGEMENT SUMMARY

2 THE CONCEPT OF MANAGED TESTING SERVICES


evolve according to the maturity of the busi-

ness partners, offering increased predictability

of costs as well as exibility and improved over-

all resource utilisation. A more detailed view on

these dimensions can be obtained by means of

the global delivery cube explained in the previ-

ous whitepaper.

Finally, Managed Testing Services also can be

directly linked to IT industrialisation since they

help customers focus on their core competency,

which rarely is in the testing area.

2.2 THE TERM MANAGED TESTING

SERVICES

Managed Testing Services are a combination of

managed services and testing activities. Both of

these are rened in terms of the innovation

YkX[(i[[


Figure ': Global delivery cube with its dimensions

BUSINESS PROCESS OUTSOURCING

ONSI

TE

ONSH

ORE

NEAR

SHOR

E

OFFS

HORE

APPLICATION OUTSOURCING

BUSINESS VALUE-BASED

INFRASTRUCTURE OUTSOURCING

TRANSACTION-BASED

OUTTASKING

FIXED PRICEBODY LEASING

COST PLUSDIY

TIME & MATERIAL


with regard to its corresponding specication.

The output will be used to remove errors from

the test asset and to assess the overall comple-

tion of development work. Before the go-live of

systems, test results can be used to evaluate the

systems readiness for use in productive envir-

onments and the risks associated with the go-

live. Testing nowadays is widely recognised as a

mandatory activity in the software development

life cycle and should be closely linked to enter-

prise risk management since testing identies

defects impeding specic risks if rolled out.

In software and systems development, testing

activities can start as early as requirements

elicitation and can span the entire life cycle

down to deployment and maintenance phases

of the software and systems. Recent surveys

conrm that testing activities are deemed most

effective when they are performed by an organi-

sational unit that is independent of the develop-

ment units. ())

There are international standards for testing

(e.g. those provided by the ISTQB), including

capability and qualication schemes. Testing is

a prime candidate for service outsourcing for

customers, not only in order to focus on core

competencies but also to exploit the advantage

of utilising independent resources. Moreover,

many organisations are already used to collabor-

ating with testing teams and will have identied

and set up their testing capacities in a proper

organisational framework. Depending on the

maturity of the organisation, tests are executed

independently from the development tasks and

report to different lines to ensure their inde-

pendence. This set-up is widely used and makes

it possible to integrate a specialised service pro-

vider with reasonable effort.

Managed services can be pinpointed by the fol-

lowing dimensions within the framework of the IT

service innovation cube:

Engagement styles: Managed services can be

applied for infrastructure outsourcing (e.g.

managed services for delivering databases),

application outsourcing (e.g. managed services

for delivering ofce suites), and even business

process outsourcing (see below, Managed Test-

ing Services).

Geographical locations: There is no need

for a particular geographical favourite since

managed services focus on output. However,

in many cases nearshore and / or offshore

resources are utilised to be able to offer

competitive pricing, if legal and regulatory

constraints do not hinder it.

Compensation models: In view of the aims

of competitive pricing and the realisation of

scaling effects, and in order to account for

the investment into the set-up of a managed

service, the overall volume of the services

purchased must be reasonably sized. In most

cases, managed services focus on repetitive

tasks suggesting a transaction-based compen-

sation model. If there is a linkage between the

delivered service and the business value, even

business value-based pricing might be possible.

2.2.2 TESTING ACTIVITIES

In the IT industry, testing refers to the process

consisting of all life cycle activities, both static

and dynamic, concerned with planning, prepara-

tion and evaluation of software products and re-

lated work products to determine that they sat-

isfy specied requirements, to demonstrate that

they are t for purpose and to detect defects. (() Typically, the input to testing is the asset under

test and the corresponding specication of the

asset under test. The output of testing is an

analysis of deviations of the asset under test


Output- or transaction-based compensation

model: This makes it possible to overcome

the time & material-based models and shift

enterprise resource risks to the provider.

Thus, it simplies budget planning and

generates more budget exibility.

Leveraging scaling effects on the MTS pro-

viders side: An MTS provider specialises in

testing and he processes many similar tasks

around testing which permit setting up the

best possible t-for-purpose infrastructure

that is supported by the best-tting tools inte-

grated into an overall smooth toolchain. These

scaling effects allow for very quick testing

cycles (because usually the provider uses an

optimised tool suite for automation) and short

response times (because an MTS provider has

an established and matured process landscape

in place, suggesting dedicated processes for

incident management or reporting).

High quality of service: For the provider of

MTS, testing is a core competency. He has ac-

cess to specialists, best practices, tools and in-

frastructures to deliver high-quality services.

In addition, many service providers are able

to use the advantage of being truly independ-

ent of the development disciplines, as the

provider is not integrated into the customers

organisation (cf. Section 3.3).

Innovation improvement: Using MTS allows

customers to refocus on their core competen-

cies again. Thus, the services free up custom-

er employees capacities, improve motivation,

and generate new innovation potential.

Increased range of the service-oriented

approach: The idea of thinking in service-

oriented structures and organisations is not

limited to testing but can be expanded to HR,

marketing, sales, etc. Using MTS supports this

general approach.

2.2.3 THE DEFINITION OF MANAGED

TESTING SERVICES

In view of the preceding two subsections, a def-

in ition of Managed Testing Services might read

as follows:

Managed Testing Services (MTS) are man-

aged services for testing-related tasks across

one or more projects delivering one or more

applications and systems, spanning the life

cycle of software and system development

and system operation. The resources for test-

ing (staff, testing infrastructure, system under

test) are managed by and under the respon-

sibility of the service provider, to support the

customers business processes. Based on

transaction- or business value-based prices,

the service provider ensures scalability of the

MTS and takes care of the resource manage-

ment according to the utilisation scheme re-

quired to support the tasks.

Denition 1: Managed Testing Services (MTS)

Managed Testing Services according to this def-

inition have the following objectives (cf. (*)): Complexity reduction: Since MTS mean

outsourcing on a high level within the process

chain, they reduce the overall process and

organisation complexity on the customers

side by providing a black-box view of it.

Risk reduction: MTS providers usually take on

a high degree of risk to deliver MTS. This is

underpinned with corresponding Service Level

Agreements (SLAs) to shift testing risks to the

provider altogether.

Competitive pricing: This is typically realised

by a low onshore rate and by moving the

geographical focus of the service mainly to

nearshore and offshore locations.


3 Automation: For each activity within each

process step or even for complete process

steps, the possible level of automation has

to be considered. There will be activities

which require a lot of manual interac-

tion, while others might be automated

completely.

4 Focus on core competencies: At this level,

it can be decided if MTS are possible or not.

The fundamental decision whether it makes

sense to utilise MTS can be taken if and

only if management have a conscious view

on what the core competencies of the busi-

ness are. This decision can only be taken

for identied tasks (modularisation) using

standards (e.g. by the ISTQB) and automa-

tion to a certain degree.

The decision for or against MTS can only be

made if Steps 1 to 3 are successfully completed.

Otherwise, it is not clear (i) what testing means,

(ii) how testing works, and (iii) to what degree

testing can be automated. It is important to

point out that a company on its way to Level

4 can already realise signicant benets along

the way by improving its industrialisation level.

For example, standardisation has values of its

own long before MTS can nally be used. And

these values may be utilised even without tak-

ing the last step of using MTS.

MTS volume: Due to scaling effects and ramp-up

efforts, the degree to which MTS pay off depends

on the size of the testing effort and the number

of applications subject to Managed Testing Ser-

vices. Typically, the benets of MTS are realised

in three- to ve-year engagements and are not

achieved in a single project-based liaison be-

tween customer and provider. Therefore, the cus-

tomer should be prepared to make the strategic

investment with the respective support by senior

management. For MTS, a customer needs to be in

a position to commit to a three- to ve-year plan

(often known as mid-term planning). Otherwise,

the increased risks for the service provider will

2.3 PREREQUISITES AND BENEFITS

MTS have benets for both service providers

and customers. The benets for the service

providers are transferred to the consumer, e. g.

by higher cost efciency and more competi-

tive prices compared to time & material-based

projects.

Even though the monetary and non-monetary

benets are widely recognised, it should be

noted that the exploitation of these benets

comes with prerequisites. If these prerequi-

sites are not met on either the customers or

the service providers side, the risk of false

promises and failure to achieve the objectives

is high. However, a well-dened transition pro-

cess like the one presented in Section 4 will

identify those gaps as early as possible and will

bridge them efciently.

2.3.1 PREREQUISITES

There are three different types of prerequisites

that should be fullled before leveraging MTS

benets:

Industrialised process chain: First of all, the

organisation and processes on the customers

side must be mature enough to support the

division of labour across organisational units.

This typical industrialisation state has at least

four prerequisites:

1 Modularisation: The overall business process

chain must be rened to a level where a

detailed process landscape can be identied.

This process landscape denes activities,

responsibilities, dependencies, and results.

2 Standardisation: Based on the modularisa-

tion, the process landscape as well as the

interfaces must utilise standards. Applying

those standards guarantees a specic level

of quality and makes establishment easier

due to an abundance of existing literature

and knowledge.


tion, the best possible organisational set-up for

running independent tests has been chosen. Both

of these aspects will have a positive impact on

the overall effectiveness of the testing process,

and will ultimately result in a higher quality of

the applications and systems being tested (cf.

Section 3.3).

In many cases, the volume of the MTS is high

enough to justify investments to leverage cross-

project and system synergies in terms of test au-

tomation and systematic regression testing. Not

only does this increase the effectiveness of the

testing service but it also reduces response times

from the testing functions to the development or-

ganisation.

If the MTS provider can make use of offshore

testing in different time zones and the delivery

processes are set up properly, a follow-the-sun

approach of MTS delivery can be achieved which

will dramatically reduce the turnaround times of

testing. This means that the development and the

testing capabilities in a project can be used in a

24-hour schedule by following the time zones. On

the other hand, this approach requires mature

processes and a signicant amount of trust be-

tween the different parties to be successful.

From the perspective of the customer, a signi-

cant risk and management effort concerning the

resources is delegated to the service providers.

Since the latter will have different customers with

different schedules, the scalability of resources in

the project life cycle is no longer an issue for the

customer and will be managed by the service pro-

vider. Therefore, the customers internal resour-

ces can be released from non-business-related

tasks and can focus on his business core activi-

ties. He will still need to contribute and provide in-

put, but this can happen in a more systematically

structured way and thus will result in both higher

quality and less effort.

have to be compensated for by higher prices.

Openness for change: Setting up MTS in an

organisation implies the shift of labour from

internal resources (including body leasing or

outtasking) to external resources having their

own responsibilities. In pure MTS, the external

resources are even no longer visible to the

customers staff. Thus, for most customers

changes will occur that need to be managed

appropriately, and management must be

fully aware of the consequences for internal

resources and address concerns proactively.

Ideally, the customer will already have experi-

ence in the eld of supplier management and

can build upon this experience to collaborate

with the service provider.

2.3.2 BENEFITS FOR CUSTOMERS

Key benets for service consumers can be real-

ised over the run-time of the MTS agreement,

most prominently the cost savings resulting from

transparent pricing models such as output-based

pricing, as well as delegated procurement of test

infrastructure and tooling. If set up properly, cus-

tomers will see an increase in transparency of cost

with regard to their application life cycles. Note

that this does not necessarily imply lower pricing

but prices are made explicit rather than being hid-

den in various budgets. One additional benet is

provided by cost attening: usually, the ramp-up

phase needs more effort than the exe cution of

MTS. However, this peak effort is not directly in-

voiced to the customer, who instead pays a at fee

over the whole life cycle. This is a key enabler for

management as an engineering discipline.

As MTS will be delivered by service providers

having expertise in the eld of quality assurance

and testing, it can be assumed that these service

providers have highly skilled resources and mod-

ern technical environments. If the MTS providers

are independent of the development organisa-


on stafng needs. The signicant sizing of MTS

allows the service provider to set up core teams

that facilitate an internal handover in case of staff

uctuations or peak workload. He also has the op-

tion to build up internal core expert teams which

will be cross-pollinated from testing expertise and

customer / domain expertise. At the same time,

resource demands from different projects (even

across different customers) can be managed on a

more exible and scalable basis.

2.3.3 BENEFITS FOR PROVIDERS

Providing MTS to a customer has a number of

advantages over time & material projects for

the service providers. Typically, MTS are aligned

with an overall strategic sourcing decision of the

service customer, and a strategic partnership is

established for delivering MTS. Strategic partner-

ship decisions usually cover a timeline of three

to ve years commitment and accordingly give

the service provider a security of revenue for his

own planning, along with an enhanced forecast

3.1 OUTSOURCING AND CLOUD COMPUTING

Outsourcing as the overall approach behind MTS

is already a well-known and established concept.

However, in the IT industry, maximum coverage

has not yet been achieved. Typical trends in out-

sourcing and how they have evolved over the last

few years as a percentage of total IT budgets are

given in Figure (, based on a study published by Computer Economics. (+)

3 MARKET CURRENT STATUS AND OUTLOOK

12 %0 % 4 % 8 %

2008

2009

2010 7.1 %

6.1 %

3.8 %

Figure (: Outsourcing ratios as a percentage of total IT budgets over the last three years

These numbers clearly demonstrate the overall

trend of IT industrialisation: a growing number of

companies use third parties to deliver IT services.

And this trend is set to increase over the next

few years. A study conducted by market research

rm PAC expects an outsourcing ratio of about

25 % by 2020 this means that one out of four

euros in the IT market will be spent on outsour-

cing. (,)


Some studies refer 40 % to 50 % of the cost of

software development to testing (.), others are measuring a ratio of 1:1 for development to test

effort (/). In a recent survey conducted on the occasion of the iqnite 2011 conference, 65 % of

the participants reported spending between 20 %

and 40 % of their overall project budget on qual-

ity management and testing activities. ())

However, most of the testing budget is spent on

system integrators which contradicts the best-

practice approach of separating product develop-

ment from system testing. In a PAC study, 78 %

of the overall quality assurance budget was spent

on system integrators and only 22 % was spent

on dedicated and independent testers. ('&) This is expected to change signicantly over the next

few years, as currently 57 % of companies still do

testing on the DIY level and will need to mature in

that respect.

3.3 FULL-SERVICE PROVIDERS VS. TEST

SERVICE SPECIALISTS

In general, Managed Testing Services can be pro-

vided by two different types of providers:

Test service specialists: Specialist providers

typically offer services in the subject mat-

ter testing only. They do not offer forward

development (implementation services), they

do not design software architectures, nor do

they have any particular interest in (or benet

from) any proprietary solution compromising

their independent approach. The advantage

of those players is that they are independent

of all systems under test. According to most

known testing literature, being independent is

one of the most important prerequisites for ef-

cient testing. Usually, test service specialists

full this requirement by being an independent

test organisation (see Denition 3).

Today, outsourcing is strongly associated with the

usage of cloud services and cloud computing in

general. For the following, we will use Gartners

denition of Cloud Computing:

Public cloud computing [is a] style of com-

puting in which scalable and elastic IT-en-

abled capabilities are delivered as a service to

external customers using Internet technolo-

gies. Private cloud computing is dened as

a style of computing in which scalable and

elastic IT-enabled capabilities are delivered

as a service to internal customers using In-

ternet technologies.

Denition 2: Cloud computing (-)

Viewed from the outsourcing perspective, cloud

services are a particular type of outsourcing with

some additional constraints. Nevertheless, cloud

computings part of the outsourcing market will

increase from less than 20 % today to 70 % in

2020. (,) A direct comparison between the def-inition of MTS and the denition of cloud services

reveals a high similarity between MTS and cloud

computing because MTS are scalable and elastic

as well, most of the delivery of MTS is IT-enabled,

and the results can be provided using Internet

technologies.

This similarity suggests that MTS will have a simi-

lar rate of growth and can be delivered by means

of cloud computing.

3.2 TESTING SERVICES

The general overview of the IT market provided

in the previous section can be rened to focus on

MTS. Both academic research and industry experi-

ence report a range of gures on how much test

effort is necessary and reasonable for a project.

direct comparison between the def-

others are

In a recent survey conducted on the


The main advantages of using a full-service pro-

vider are:

Company size: A broader portfolio of services

and tools allows for a bigger market enabling

a bigger company size. Typically, full-service

providers are known globally and have a large

pool of employees at their disposal.

All-inclusive single sourcing: Using a full-

service provider simplies both the process

of selecting third parties and the negotiation

step. Having a contract with a full-service pro-

vider means getting everything through one

contract, with one key-account person it is a

one-stop shop.

Technical knowledge: Introducing an independ-

ent test organisation to a new IT system costs

more time and money than using one of the

service providers developers. The full-service

provider has deep knowledge of the technolo-

gies he develops and provides, which enables

him to start working without having a clear

knowledge-transfer step.

Both independent test organisations and full-ser-

vice providers will try to move into the Managed

Testing Services market (see Section 2.3.3). How-

ever, the market will be further segmented: the

more mature an organisation is (e.g. in terms of

industrialisation), the more important attributes

like provider exchangeability, dependency reduc-

tion, and output-based pricing are going to be.

These considerations assume an explicit testing

awareness on the customers side. If it is given,

the application of MTS is a natural step, i.e. MTS

will successfully be provided by test service spe-

cialists.

Figure ) shows the total revenue generated by in-dependent test organisations in 2010.

An independent test organization is an or-

ganization, person, or company that tests

products, materials, software, etc. according

to agreed requirements. [] They are inde-

pendent because they are not afliated with

the producer nor the user of the item being

tested: no commercial bias is present.

Denition 3: Independent test organisation ('')

The key advantages of using an independent

test organisation are the following:

Independence: Independent people are more

likely to question everything, provide an

objective second opinion, identify implicit

requirements, and objectively report any nd-

ings. Different lines of reporting ensure that

no conicts of interest will interfere.

Scaling effects: Because independent test or-

ganisations are usually pure-play testers, they

test many different projects at the same time,

which allows them to set up highly sophisti-

cated toolchains and utilise a large number of

synergies between different projects.

Resource capabilities: The skills provided by a

test service specialist are clearly focused on

one single topic, i.e. testing. This guarantees

a high, coherent skill level for all employees

and simplies the exchange of resources.

Full-service providers: These providers usu-

ally offer any services relating to IT systems,

starting from collecting requirements, design-

ing system architectures, implementing the

entire system, testing the system, and nally

deploying the system into productive use.

Also administration of the systems and testing

services, including Managed Testing Services,

form part of the comprehensive service port-

folio. In many cases the full-service providers

even market their own tool platforms, comple-

menting and mixing their service ideas with

software license vending.


testing approach to MTS may prove difcult. In

other words, MTS are likely to remain a side prod-

uct for full-service providers, and consequently

only about 5 % of all testing activities provided

by them will be delivered as MTS by 2015. Using

the revenue of the full-service providers listed in

the NelsonHall Report, this amounts to an overall

business size of up to 150 million of revenue for

MTS (5 % of the total market size as listed in ('()).

Consequently, the independent test organisations

will reach the same market share for MTS as the

full-service providers.

Currently, conservative estimates anticipate about

20 % to 25 % of all testing activities provided by

test service specialists to be realised through MTS

by 2015. For the independent test organisations

listed, this amounts to an overall business size of

about 121 million of revenue for MTS (25 % of

the total market size as listed above).

On the other hand, full-service providers usually

blend testing activities and package them into an

all-inclusive bundle. If the client lacks awareness

of what really is involved in testing, testing effort

and testing outputs, motivating him to change the

INDEPENDENT TEST ORGANISATIONS

163

70

40

28

20

16

14

13

13

12

11

10

9

9

8

7

7

7

6

6

5

5

5

0 20 40 60 160

SQS

Applabs*

Tescom

DNV-IT

Tesnet

Plan IT

Thinksoft

Imbus

FHG / IESE

Amsphere

MTP

Acial

ps-testware

Quality House

Maverick

QA Infotech

RTTS

Genilogix

Squerist

Experior

Immune

Assurity

KJR

Figure ) : Revenue of independent test organisations ('()

* In

20

11, A

pp

labs

was

tak

en o

ver

by C

SC

Co

mpa

ny.

m


The Pre-Engagement step covers the strategic

activities a client needs to perform in order to ap-

ply MTS. The MTS Framework Design step covers

the tactical activities within each clients project

to prepare MTS application. And the MTS Opera-

tion step leverages the benets associated with

MTS. Each of these steps is rened in the follow-

ing subsections.

4.1.1 PRE-ENGAGEMENT PHASE

At the very beginning of the transition to MTS,

the customer needs to define and commit to

the vision of industrialising his IT environment

(see Figure +). On the one hand, management need to identify the strategic services and core

competencies they want to operate on their

own behalf, and on the other hand, they need

to specify the supporting non-core services to

be outsourced.

The vision to industrialise IT and make use of ser-

vice providers has to be supported by conscious

planning and budgeting decisions. A success-

ful migration to an outsourcing model demands

proper preparation of the underlying business

case. Before MTS can be deployed into the or-

ganisation, the strategic sourcing decision must

be taken and at least mid-term planning perspec-

tives have to account for handing over a com-

plete process to a service provider. The strategic

decision is a sine qua non because of the sizing

of the engagement. Typically, using a managed

service requires the investment of signicant ef-

forts into organisational maturity, and the corre-

sponding changes have to be communicated and

managed appropriately.

The nal step of the Pre-Engagement phase is to

set up the relationship to potential service pro-

viders and selecting the best supplier. For MTS,

criteria for supplier selection start with match-

ing the three dimensions from the IT innovation

cube (engagement style, geographical location,

and compensation model, cf. Section 2.1) to the

customers needs. Additional parameters to be

considered are the following:

Level of testing expertise

Independence from the development

organisation

Domain knowledge

Size of the provider

Commercial background

Track record and reputation of the service

provider

4.1 TRANSITION OVERVIEW

The MTS process is structured into three phases Pre-Engagement, MTS Framework Design, and

MTS Operation and laid out in Figure *.

Figure *: MTS process overview

1. PRE-ENGAGEMENT2. MTS FRAMEWORK

DESIGN3. MTS

OPERATION

4 MANAGED TESTING SERVICES SET-UP AND OPERATION

On the one hand, management


Heterogeneous views on core business com-

petencies and unclear strategic considerations

with regard to sourcing decisions

Depending on the results from the maturity as-

sessment, different initiatives to industrialise and

professionalise IT will be launched as business

processes are set up, streamlined, and optimised.

Preparing the organisation for MTS increases the

necessity to outline the strategic vision of future

IT set-ups, and makes the corporate strategy vis-

ible to all employees.

Accordingly, the typical set-up for an IT improve-

ment towards industrialisation consists of the

four steps mentioned in Section 2.3.1:

1 Modularise the entire value chain within the

organisation.

2 Standardise the process landscape, the pro-

cesses themselves, and their interfaces.

3 Automate single processes, process interac-

tions, or process chains, wherever possible.

4 Focus on core competencies and delegate per-

ipheral services to external service providers

as managed services.

The benets of preparing for MTS are tangible for

customers when MTS are being used as a driver

for maturing the organisation. As a side effect

of the MTS Framework Design phase, customers

need to reconsider their internal processes and

organisational set-up. The maturity assessment

of processes and organisation can be leveraged

to foster clarity and transparency of core com-

4.1.2 MTS FRAMEWORK DESIGN PHASE

One of the greatest challenges when setting up

MTS lies in the degree of IT industrialisation on

the consumers side. Clearly, without an explicit

organisational set-up and mature processes, the

realisation of benets from division of labour

and the consumption of specic processes /

services from the outside are rarely possible.

Therefore, the rst step in bringing MTS to life is

the assessment of the customers current organ-

isation and its processes to evaluate their tness

for purpose (see Figure ,). The assessment typic-ally unveils a number of necessary improvements

to close the gaps and achieve the adequate level

of maturity. However, these improvements do not

only affect MTS readiness but essentially generate

added value even without using MTS.

In our experience, customers aiming at the use

of MTS (or other managed services) often over-

estimate their readiness for the consumption of

services. Typical unrecognised shortcomings are

the following:

Insufciently structured and dened pro-

cesses, tasks, and supporting IT landscape as

well as undened or unclear ownership of the

process or IT components

Insufcient deployment of process and IT stand-

ards (e.g. non-existent or incomplete process

landscape, high number of variations in business

processes, unclear and undocumented inter-

faces between processes and departments)

High proportion of manually executed pro-

cesses and technological and organisational

ruptures in process chains as well as decien-

cies in process interactions

Figure +: Pre-Engagement activities

VISION TO INDUSTRIALISE IT

STRATEGIC PLANNINGCONSIDERATIONS

(MID-TERM PLANNING)

PROVIDERPRESELECTION

The assessment typic-


4.1.3 MTS OPERATION PHASE

After having set up the general MTS framework

for collaboration between the customer and ser-

vice provider, the testing activities for the indi-

vidual applications have to be transferred from

the customers responsibility to the aegis of the

MTS provider. For greeneld projects, this seems

a straightforward task whereas for existing pro-

jects there is potentially more substantial change

involved. In any case, the readiness for transfer

is ensured in the rst step, where potential gaps

are identied and closed accordingly (see Fig-

ure -). Secondly, the knowledge transfer (both from the development organisation to the MTS

provider, and from the test organisation to the

MTS provider) is initiated, and the rst iteration

of delivery is accomplished jointly. This collabora-

tive delivery is a one-off undertaking designed

(i) to ensure a successful transfer of the testing

activities to the third step (Service Operation) by

passing on comprehensive knowledge to the MTS

provider, and (ii) to sort out any issues that might

arise during stand-alone delivery to the best pos-

sible extent.

petencies and non-core supporting functions. It

is worth mentioning that through the course of

process improvement, the relationship with the

service provider can be established and the trans-

formation can be used to build mutual trust.

Finally, when readiness has been accomplished,

the customer needs to design the framework

agreements with the service provider of choice.

Utilising Managed Testing Services requires meth-

ods to monitor and control service delivery. The

challenge lies in a proper denition of the SLA:

due to the black-box view of MTS, it should focus

on the compensation model rather than on inter-

nal MTS details. In order to link the KPIs to the

compensation model and evaluate them accord-

ing to a reasonable schedule, a sound framework

needs to be dened. From the very beginning, the

denition of KPIs should take into account the

evolution of service level objectives over time to

establish appropriate productivity goals. For moni-

toring and controlling purposes, these KPIs are

utilised for an effective governance and commu-

nication structure and process, so as to be able to

react when the control limits are reached.

Figure -: MTS Operation processes

CONTINUOUS IMPROVEMENT

VERIFICATION OF TRANSFER

READINESS PERAPPLICATION

KNOWLEDGETRANSFER &

COLLABORATIVEDELIVERY

SERVICEOPERATION

SERVICECOMPLETION

Figure ,: MTS Framework Design activities

ASSESSMENT OF CURRENT

ORGANISATIONAND PROCESSES

PROCESS ANDRESOURCETUNING

FRAMEWORK AGREEMENT

WITH PROVIDER

the knowledge transfer (both


laborative Delivery step, it helps determine and

ne-tune the agreed KPIs of the framework agree-

ments and can be used to adjust the SLA. While

Service Operation is repeatedly executed, Con-

tinuous Improvement ensures that the KPIs are

properly monitored and adequate action is taken

to improve and control the delivery. The same

KPIs used to report back on service performance

should also be applied to manage and govern the

MTS Operation phase. Since the benets based

on improvements affect both consumer and MTS

provider, the sharing of the benets needs to be

clearly dened in the SLA.

Out of the four MTS phases, the MTS Operation

phase is the one covering the longest time span

it implies a shift from a project governance struc-

ture towards a programme governance structure.

A stereotypical organisational set-up is shown in

Figure ..

Subsequently, the Service Operation step may be

executed as often as necessary over the lifetime

of the application, until this lifetime comes to a

close and both the application and its MTS are

decommissioned in the Service Completion step.

Since the Service Completion step is the reverse

of the transition step, planning for this step helps

limit dependency on a specic MTS provider. The

Service Completion step takes care of gathering

all information (including internal details like test

scripts, test data, or test procedures) necessary

to deliver MTS for a particular project. This allows

the customer to hand over the collateral informa-

tion to any other MTS provider, or to insource the

testing activities at a later point in time, ensuring

a shorter introduction time.

The sequence of four processes is overarched by

the Continuous Improvement of the MTS Opera-

tion phase. During the Knowledge Transfer & Col-

Figure .: Governance structures for the MTS Operation phase

PROJECT GOVERNANCE MANAGED SERVICE GOVERNANCE

EXECUTIVE STEERING COMMITTEE

SERVICE DELIVERY COMMITTEE

INNOVATION & TECHNOLOGY COMMITTEE

Test FactoryMS Delivery

Management

Applications

TestingServices

Test Team

TestingServices

Test Team

TestingServices

Test TeamFunctionalSub-Projects

TestingServices

Technical Sub-Projects

QualityAssurance

ProjectManagement


proven governance structure consists of the fol-

lowing decision bodies in a strategic and an op-

erational layer:

STRATEGIC LAYER

Executive Steering Committee (ESC) Meet-

ings are held quarterly (or as described in the

framework agreement) as senior management

meetings to monitor overall MTS performance,

to govern MTS from a long-term perspective,

and to de-escalate issues arising from the

operational layer.

Innovation & Technology Committee (ITC)

The ITC meets quarterly to share the latest

industry, innovation and technical trends, in

order to decide on major innovation projects

and jointly agree on business cases and fund-

ing, if required. Its members propose changes

to the agreements to the ESC and also consult

the ESC on strategic decision-making.

OPERATIONAL LAYER

Service Delivery Committee (SDC) The SDC

performs weekly reviews of the current perform -

ance and progress, focusing on the domain,

project and service levels. It decides on prior-

ities, resolves escalation issues as best pos-

sible, and elaborates on how to mitigate risk

and issues within the scope of the services

operation. Issues outside the SDCs responsi-

bility are referred to the ESC.

Operational Status Meetings These meetings

are scheduled weekly and aligned with core

testing processes in order to review progress,

focusing on the project or application level,

and to decide on priorities and work on risks

and issues, seeking for resolution wherever

possible or referring them to the SDC.

Project Governance rests entirely with the Project

Management (which then reports to the executive

level). The Project Management function sup-

ported by the Quality Assurance function per-

forms Testing Services as per project by the same

hierarchical decomposition as the Functional and

Technical Sub-Projects. In the MTS setting, the

Testing Services liaise with the Quality Assurance

function and are managed as per application by

suppliers interfacing with the respective project

teams. The reporting lines to the executive level

are no longer purely bottom-up but complement

the internal reports with an independent view on

projects and applications from the suppliers per-

spective. This typically gives customer manage-

ment deeper insight into the project status and

increases trust in the reports.

As a general principle, the MTS Operation phase

must be supported by an open dialogue between

customer and service provider relating to the

provision of the services throughout the term. In

order to keep up with technological changes over

the run-time of the overall agreement, details of

the agreement may need to be amended, whether

as a result of market trends, new technology,

evolution of new services, or otherwise, and must

be kept under constant review, control, and sub-

ject to proactive search for improvements in the

MTS Operation phase. To facilitate the dialogue,

both partner organisations typically appoint a Re-

lationship Manager at senior management level

to cover all matters concerning the agreements.

The service provider appoints a Service Delivery

Manager as the point of contact for operational

matters affecting the services (e.g. adding new

applications to the portfolio; customer projects

ordering services).

Having dened the roles, a corresponding formal

and structured channel of escalation and com-

munication has to be set up and maintained. The


4.2.1 INEFFECTIVE TESTING

If testing is ineffective, the uncertainty about the

system to be rolled out increases because nobody

knows about possible hidden errors in produc-

tion and potential impacts. Since risks are based

on uncertainties, the overall risk of an IT system

increases as well. This does not necessarily im-

ply that the system fails in production but that

it lacks the necessary transparency and reliable

insights about production readiness. Therefore,

ineffective MTS for a business-critical application

might induce additional risk.

From an MTS perspective as described above, the

following countermeasures are to be taken:

Within the transition step, many prerequisites

for MTS are veried. The transition step will

identify most of the detailed hurdles for ef-

fective testing and eliminate the respective

reasons.

Before MTS enter the operational phase, the

collaborative delivery is an explicit mitigation

activity to ensure in a dry-run that testing can

be implemented as planned.

The MTS process incorporates the denition

of an overall control framework underpinned

by supporting KPIs to make the status of the

testing process transparent to all stakehold-

ers. Therefore, uncertainties about the test

progress should be minimised.

4.2 MTS RISK MANAGEMENT

As with all other business activities, there are

risks associated with MTS. For risk, we use the

following denition, based on the concept of un-

certainty:

Uncertainty is the lack of complete certainty,

that is, the existence of more than one possi-

bility. [] Risk is a state of uncertainty where

some of the possibilities involve a loss, catas-

trophe, or other undesirable outcome.

Denition 4: Uncertainty and risk ('))

So each risk is an uncertainty but not vice versa.

Uncertainties can have both positive or negative

impacts, risks only have negative impacts on busi-

ness. Please note that this understanding of risk

is more focused than the one used in ISO 31.000

since there even positive impacts are classied as

risk.

From a risk management perspective, the focus is

on negative risks because they require mitigation

actions to deal with their impacts.

For a customer, MTS involve two different types

of risks:

Ineffective Testing: Testing does not work

properly, i.e. the SLAs for the MTS or the stra-

tegic objectives to do MTS are not fullled.

Unfullled Collaterals: Testing works but other

collaterals are uncertain. This uncertainty can

cover the economic future of the service pro-

vider or might touch the cultural acceptance of

MTS within an organisation.

Both risks are rened in the following subsec-

tions.


Communication risks: In general, MTS need

more mature communication channels (no

longer gossiping) when communicating

from customer to testing. In addition, a more

mature defect management is necessary since

the organisational boundaries discourage

on-the-oor communication. And last but not

least, cultural and language issues are a major

source of misunderstanding, in particular

if cross-national and offshore locations are

involved. MTS might generate risks in this

eld, but the industrialisation steps, that are

checked on the customers side in advance, as

well as the systematic transition step minimise

them.

Dependency risks: Outsourcing is often associ-

ated with outsourcing responsibilities, i.e. test-

ing capabilities for MTS. The less knowledge is

retained on the customers side the more he

is dependent on MTS providers. Typ ical

examples of a high dependency are price

adjustments without any reason, disburdening

details of the negotiations, or low quality. In all

cases, MTS propose the well-dened Service

Completion step, ensuring that all custom-

ers can step back. Due to the detailed entry

criteria (industrialisation), the customer has a

very clear understanding of the dependency,

possible impact, and alternatives for delivery.

4.2.2 UNFULFILLED COLLATERALS

Outsourcing a well-dened service might stipu-

late uncertainty in some collateral areas. Most

of them are not solely related to MTS but to all

major changes in the context of service delivery.

In a study about risks of outsourcing (not limited

to MTS), a set of most common risks was identi-

ed by survey respondents. ('*) Most of them are valid for MTS as well:

Pricing risks: In many cases, using MTS should

have a positive effect on costs. However, the

specic costs of MTS are not ultimately known

before the rst Service Operation step. The

MTS framework provides general guidance but

only the Collaborative Delivery will determine

the nal price. From a customer perspec-

tive, a mitigation action would be to select an

experienced MTS provider whose rst effort

estimation does not need much adjustment. If

there are price adjustments while the service

is delivered, the Service Completion step en-

sures that it is possible to change the service

provider to a cheaper one.

Planning risks: If some other constraints from

MTS are not met (e.g. missed deadlines, un-

clear planning), this might generate additional

risks as well. The MTS process guarantees to

identify deviations as early as possible. In most

cases, any deviations are followed by contrac-

tual penalties compensating negative effects

by planning risks.


The market for MTS will increase dramatically

over the next few years. Even if both, full-service

providers and test service specialists, will try to

do business in that area, the test service special-

ists will dominate the market in a few years time

due to many advantages in their MTS delivery.

Their process to establish MTS on the consumers

side reects a well-dened current-state analysis

as well as a corresponding transition step for MTS

readiness. Based on a continuous improvement

loop, MTS execution itself can be managed ef-

fectively. A possible closing step at the end can

reect an applications end as well as the demand

to in-house all activities again.

Today, testing is established as an important dis-

cipline for risk management. For maximum ef-

ciency, the testing activities, the methods, the

practices, tools and skills should only be of inter-

est to those who execute testing. From a business

perspective, testing should be seen as a black box

focusing testings output. So there is a great op-

portunity to outsource testing activities at the

rate of output-based pricing: Managed Testing

Services.

Applying MTS is the next step of IT industrialisa-

tion and needs a mature process landscape on

both the consumers side and the producers side.

Even if there is no need to apply MTS as a last

step on the industrialisation roadmap, the road

itself generates a considerable amount of added

value for improving testing efciency and effect-

iveness.

5 CONCLUSION AND OUTLOOK

Bibliographical References Page 21

1 Adam Smith. An Inquiry into the Nature and Causes of the Wealth of Na-

tions. Oxford University Press, 2008.

* Dr Wolfgang Balze, Dr Wolfgang Rebel and Peter Schuck. Outsourcing und arbeitsrechtliche Restrukturierung von Unternehmen. Verlagsgruppe Hthig

Jehle Rehm, 2007.

+ Computer Economics. IT Outsourcing Statistics 2010 / 2011: Outsourcing and Offshoring Trends, Cost/Service Level Experiences, and Analysis for 11

Outsourced IT Functions. California, Irvine: Computer Economics, 2010.

, Karsten Leclerque. Outsourcing im Jahr 2020. www.cio.de. [Online] 09/09/2010. [Cited: 19/07/2011.] http://www.cio.de/knowledgecenter/out-

sourcing/2247905/index.html.

- Daryl C. Plummer, David Mitchell Smith, Thomas J. Bittman, David W. Cear -ley, David J. Cappuccio, Donna Scott, Rakesh Kumar and Bruce Robertson.

Five Rening Attributes of Public and Private Cloud Computing. Gartner,

May 2009.

. Praveen Ranjan Srivastava. Estimation of Software Testing Effor t: An intel-ligent approach. Birla Institute of Technology and Science, Pilani, Rajasthan,

India: Computer Science and Information System Group, 2009.

/ Michael A. Cusamano and Richard W. Selby. Microsoft Secrets. New York: Simon and Schuster, 1995.

'& PAC. Compliance and Software Testing. Compliance Magazine. [Online] 04/04/2007. [Cited: 19/07/2011.] http://www.compliancemagazin.de/markt/

studien/pac040407.html.

'' Wikipedia. Independent test organization. Wikipedia, the free encyclopedia. [Online] 27/07/2011. [Cited: 27/07/2011.] http://en.wikipedia.org/wiki/Independ-

ent_test_organization.

2 International Software Testing Qualications Board, Glossary Working Party.

Standard Glossary of Terms Used in Software Testing. Homepage: ISTQB,

December 2007.

) iqnite conference. iqnite interaktiv SMS voting. iqnite germany program. [Online] 01/07/2011. [Cited: 28/07/2011.] http://www.iqnite-conferences.com/

Trial%20Wiki/iqnite_interaktiv_sms-voting.aspx.

Dr Wolfgang Balze, Dr Wolfgang Rebel and Peter Schuck. Outsourcing und

Computer Economics. IT Outsourcing Statistics 2010 / 2011: Outsourcing

Karsten Leclerque. Outsourcing im Jahr 2020. www.cio.de. [Online]

Wikipedia. Independent test organization. Wikipedia, the free encyclopedia.

-

t: An intel-

Bibliographical References Page 22

'( NelsonHall Report. Independent Testing Providers vs. System Integrators. Cologne: SQS, April 2010.

'3 Douglas Hubbard. The Failure of Risk Management: Why its broken and how to x it. John Wiley & Sons, 2009.

'* Diamond Management & Technology Consultants. 2006 Global IT Outsour-cing Study. Chicago: Suite 3000 John Hancock Center, 2006.

Cologne: SQS, April 2010.

to x it. John Wiley & Sons, 2009.

cing Study. Chicago: Suite 3000 John Hancock Center, 2006.

SQS Software Quality Systems AGPhone: +49 (0)2203 9154-0Stollwerckstrasse 1151149 Cologne / Germanywww.sqs.com

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False

/CreateJDFFile false /Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice

managed testing services

Documents