Continuous Testing and Service Virtualization Chris Sloan – Global SSR Chris Haggan – Global CTP

2

Agenda

Intro

Continuous Testing – what it means to the key players

PM View

TM View

Testers View

How IBM facilitates continuous testing

What have we seen in the field?

Summary

3

Setting the scene

Business drivers for Continuous Testing and Service

Virtualisation

– Cost

– Quality

– Risk

– Time to market

4

$59B

+9.5%

Looking at the total cost of Quality

Prevention & appraisal costs

– Tools & labor to prevent (reviews, training, process, etc.) and identify

defects (testing)

Internal failure costs

– Debugging, scrap & rework, regression testing

External failure costs

– Downtime (e.g. $390M Knight Capital), recalls (e.g. $3B Toyota Prius

antilock system)

(1) IDC - Worldwide and U.S. Discrete Testing Services 2012–2016

(2) Ovum worldwide market for software and systems testing services

(3) Capers Jones – Software Quality in 2012: A survey of the state of the art

(4) NIST - The Economic Impacts of Inadequate Infrastructure for Software Testing

$307B

+0.9%

$132B

+0%

Client challenges with addressing the market shifts

Systems of Interaction

Continuous client experience

Partner value chain

Cloud-based Services

Systems of Engagement Systems of Record

CRM HR

DB ERP

of applications rolled back

>80%

of outsourced projects fail

to meet objectives

>50%

of resources devoted to maintaining existing systems and products

>70%

to deliver application changes to customers

4-6 Weeks

Line-of-business

Takes too long to introduce or make

changes to mobile apps and services

Operations

Rapid app releases impacts system

stability and compliance

Development/Test

Speed mismatch between faster moving front office and slower

moving back office systems, delaying time to get feedback

Suppliers

Delivery in the context

of agile

Change & Configuration

Management Dashboards

Jazz, OSLC and Open Standards Platform

DevOps Solution: Adoption paths, open platform and key capabilities

Deployment

Provisioning

Release /

Deploy

Develop /

Test

Monitor /

Optimize

Monitoring

Customer Feedback

Code

Test

Portfolio Management

Requirements

Plan /

Measure

7

Solving the Continuous Testing dilemma

Traditional test environments are both underutilized and insufficient

– Environments are expensive: hardware, software, and labor

– Tester and developer time wasted waiting for appropriate environment

Agile test environments require four enabling capabilities

Dynamic Infrastructure

Application Release

Automation

Service

Virtualization

Test Data

Management

Test Environments

DevOps Foundation

What facilitates Continuous Testing? Green Hat does

RTCP – Rational

Test Control Panel

Rational Test

Workbench

RTVS – Rational Test

Virtualization Server • Tests and virtualized

components are designed

in Rational Test Workbench

• They are published to the

Rational Test Control Panel

• Designed for small projects

up to enterprise use

• Allows non-technical users

to configure and run virtual

components

• Simple client/server set-up

The Project Manager

Project Management and the Critical Path

Project A – 100 days

Project B - 20 days

Project C – 30 days

1

2

• The critical path is 100 days – any delays from Projects B and C will delay the program

• The impact of late delivery from Project C is much greater as the burn rate from the program is

greater at that point

• Failure at a technical integration level between the projects is the greatest risk

Burn rate

$20k per day

Burn rate

$75k per day

1. Project B is at risk of running late. Test environments for integration between A and B are not

ready. Every day it runs late, $20k will be burned, and the critical path is pushed out.

2. Project C is running late. They are only half-way through their testing and are not able to

integrate yet. Every day it runs late, $75k will be burned.

The Test Manager

The Test Manager’s Strategy

Schedule Driven: Deliver on time… as long as it works… more or less…

Features are more important than quality and reliability

Effectively turning your customer into a tester…

Quality Driven: Deliver with Zero defects

No way to determine how many defects still exist

Not economical… and the marketplace won’t wait for perfection

Time

Defects

Schedule-driven release

Quality-driven release

Risk based strategy for Systems/Software Delivery

Risk Exposure = Probability (Loss) x Size (Loss)

“Loss” – financial; reputation; future prospects, etc.

Actual risk exposure is the sum across multiple sources of loss

Time

Risk

Exposure Many critical defects

Few minor

defects Probability of defects loss due to defects

Low opportunity cost; Weak competition

High opportunity cost; Strong competition

Probability of competitors size of loss to competition

Time

Risk

Exposure Many critical defects

Few minor

defects Probability of defects loss due to defects

Risk based strategy for Systems/Software Delivery

Risk Exposure = Probability (Loss) x Size (Loss)

“Loss” – financial; reputation; future prospects, etc.

Actual risk exposure is the sum across multiple sources of loss

Low opportunity cost; Weak competition

High opportunity cost; Strong competition

Probability of competitors size of loss to competition

Total Risk = Σ All-Risks

Time

Risk

Exposure Many critical defects

Few minor

defects Probability of defects loss due to defects

Risk based strategy for Systems/Software Delivery

Risk Exposure = Probability (Loss) x Size (Loss)

“Loss” – financial; reputation; future prospects, etc.

Actual risk exposure is the sum across multiple sources of loss

Low opportunity cost; Weak competition

High opportunity cost; Strong competition

Probability of competitors size of loss to competition

Release

when

Overall Risk

Exposure is

Minimal!

Competitive risk and opportunity cost are getting bigger than reduction in risk associated with further quality improvement

Total Risk = Σ All-Risks

The Role of a Tester

Red curve is

Green curve is

Risk

Exposure

Time

Time to Release

Market dynamics

Risk management To deliver earlier, without

hurting quality…

you need to lower your Risk!

The Role of a Tester -- As Is

Risk

Exposure

Time

Time to Release

Running existing test cases…

just to get them out of the way…

Worst case: Test Monkeys

Better: Define test cases, then measure execution against these test cases

Slightly better: Define test cases, automate execution, then measure pass/fail

– Speeding up velocity, but no cost reduction – Automation comes with setup and maintenance cost

Even better: Enhance scripted test execution with Exploratory Testing

Running more test cases,

more often

Improving Quality

The Test Manager’s New Planning Perspective

Development team

uses virtual

services to unit test

code on their

workstation – no

elaborate

environment

required

Integration Operability User Acceptance System Integration System Unit Integration Operability User Acceptance System Integration System Unit

Performance

Technical testers

can test and

performance test in

an iterative and

incremental

manner. Using CI

Testing team removed

downstream dependencies

As testing becomes more

formal. test scheduling and

sharing of assets becomes

more important – invocation

via RQM.

Later in the cycle, final pre-

production versions of services

are swapped in as they

become available. Final testing

is done against real services

The old world without Service Virtualization

Performance

Requirements

Unit Tests

UAT

OT

Integration Tests Sys Tests Sys Tests

• Accelerates testing

• Reduces costs

• Lowers risk Big Bang

Shifting testing to the left!

Shifting focus from Unit Testing to Integration Testing

A new world with Service Virtualization

Test Management - RQM and RTVS

RQM RTCP /

RTVS RTW / RIT

Associate Test with Virtual Service

Publish Virtual Service

Start Virtual Service

Run

Test Stop Virtual Service

Output linked

to Test Case Result

RQM: Rational Quality Manager RTCP: Rational Test Control Panel

RTVS: Rational Test Virtualization Server

RTW: Rational Test Workbench

RIT: Rational Integration Tester

Create Test Case Create Virtual Test Service

The Tester

Testing is hard!

Application complexity is exploding:

Mainframe (one interface)

Client Server (a few interfaces)

SOA/Web/Cloud apps (100’s of

interfaces)

Development and Test teams are

getting larger, often outsourced,

geographically dispersed.

Costs are escalating and quality is suffering.

Waterfall models and serial projects are out, agile and continuous

testing are in.

The Godzilla Tester

Integration testers require:

• Technical knowledge

• Test tools knowledge

• Business domain knowledge

• Testing knowledge

Accelerate the learning process with a

collaborative approach.

Testing

Technical Test tools

Business domain

knowledge

The Emerging Role of a Tester: More Technical, More Business Oriented

24

Technical

Risk

Business

Risk

Maturity around protocols

and technologies enable

automation of

service virtualization

Testers to work in sync

with Development to

reduce Technical Risk

Systems of engagement

are a major disruptive

force that cannot be

ignored.

Testers to refocus on

reducing Business Risk

• Automated Deployment

and Continuous Testing

to enable Continuous

Delivery

• Defect Removal

Efficiency

25

Automated Integration Testing

26

Service Virtualization

Continuous Testing and Continuous Deployment

DevOps

Continuous Deployment Continuous Testing

uDeploy Automated

Integration

Testing

Service

Virtualization

1 2

3

= RTVS

Toolset

= URBAN

CODE

1. uDeploy enables continuous deployment of available environments into the targeted

infrastructure

2. RTVS can automatically test the deployment and integration of that environment - do the

components communicate correctly?

3. Service Virtualization allows the testers to:

a. Stand up Service Virtualization in place of components that are unavailable (not built yet,

too expensive, resource contention, 3rd parties etc.)

b. Plan for test conditions, such as negative testing that may not be possible with the real

components.

28

Service Virtualization and Automated Deployment Technology acquired from UrbanCode and Green Hat provide a winning combination

for continuous testing and rapid software deployments

Deploy what is ready, virtualize the rest Perform continuous integration testing and progressive deployments throughout the

software delivery lifecycle

28

Automate the creation of virtual test environments as part of the

end-to-end DevOps process

Establish a virtual system pattern with automated deployment to standardize and

share test environments quickly and easily across projects and teams

Enable an end-to-end Agile software delivery process Increase the frequency of releases by testing earlier in the development cycle and

deploying reliably to pre-production and production environments..

Additions To The Solution

MQ Queue

Manager

Queue

Zero Environment Configuration

Real

System

MQ

Exit

Client that

initiates

transaction (or

RIT)

MQ Exit is installed on the MQ

Queue Manager. Directs

inbound and outbound traffic for

execution purposes.

Virtualized

Component

MQ Exit

Intercepts

message

before it hits

the inbound

queue.

If stub is down,

then MQ Exit

directs

message to

Live system.

MQ Queue

Manager

Queue

Real Time Dynamic Switching (Sift and Pass Through)

Real

System

MQ

Exit

Vacation Booking

UI and Rules

Engine

MQ Exit is installed on the MQ

Queue Manager. Directs

inbound and outbound traffic for

execution purposes.

Virtualized

Component

MQ Exit

Intercepts

message

before it hits

the inbound

queue.

If stub is down,

then MQ Exit

directs

message to

Live system.

Pass through

route.

Database Virtualization

32

To allow testers to virtualize the database an application is using to control the data that is returned from queries and stored procedures with minimal configuration

GOAL

Record interactions between an

application and a database

Create a database stub whilst monitoring

an application containing data that is just

sufficient to re-run the scenarios against

the stub

Allow a user to modify the data in the

stub to create specific test scenarios

Use a real database to implement the

stub so vendor-specific calls work as

expected and state is managed

Allow a user to run new scenarios

against a stub and have it respond

correctly

USE CASES

Z - CICS Testing Solution

33

CICS

Transaction

Gateway

WebSphere

MQ Queues

CICS

Mainframe

Applications

SOAP

Gateway

• Test mainframe applications via a

number of input channels. Tests can

be created by hand or by observing

(recording) real interactions that take

place

Rational Test Workbench

Developers & Testers

Z - CICS Virtualization Solution

34

Virtual

Mainframe

Apps &

Databases

Mainframe

or

Distributed

App Cobol Copybook messages

Virtual

CICS

Programs

Distributed

Java App

Messages via CTG

Virtual

Web

Services

CICS

Mainframe

App EXEC calls to distributed

Virtual

CICS

Mainframe

Programs

CICS

Mainframe

App DPL calls

MQ Transport / HTTP

• Inter-Mainframe: Test

multiplatform apps by

virtualizing dependent

mainframe apps in multiple

z/OS or distributed

deployments

• Intra-mainframe : Test

mainframe apps by

virtualizing dependent

mainframe apps inside the

same z/OS instance

Z - IMS Testing and Virtualization Solution

35

IMS Connect

WebSphere

MQ Queues

IMS

Mainframe

Applications

SOAP

Gateway • Test and virtualize mainframe

applications via a number of input

channels. Assets can be created by

hand or by observing (recording) real

interactions that take place

Rational Test Workbench

Developers & Testers

Rational Test Workbench – Breakdown

Rational

Integration

Tester

RTW - Breakdown 1. Rational Integration Tester (RIT) –

design virtualized components and

integration tests. Focuses on technology

below the UI

2. Rational Functional Tester (RFT) –

traditional test automation at the UI

Layer. Functional testing and regression

testing tool. This software provides

automated testing capabilities for

functional, regression, GUI, and data-

driven testing.

3. Rational Performance Tester (RPT) -

validates the scalability of web and

server applications. Identifies the

presence and cause of system

performance bottlenecks

4. Mobile Tester – automate the creation,

execution, and analysis of functional

tests for mobile and web-based

applications. Includes Selenium

Rational

Functional

Tester

Rational

Performance

Tester

Mobile Test

Automation

1 2

3 4

In the Field

Accelerating product and service innovation

European lottery provider makes efficiency gains and reduces programme slippage

Need Reduce programme slippage to avoid contract penalties

Solution Automated integration testing

Implementation Integration testing of all non-GUI components and simulation of bespoke 3rd

party system to test middleware (service bus) handling of all error codes

Value Reduced regression testing time from 1 month to 1 day

Elimination of two month programme slippage enabling on-time system

go-live avoiding £millions of penalties

Accelerating product and service innovation

Leading US Healthcare Organization moves to continuous testing model with service

virtualization and automated integration testing

Need Move to DevOps model of testing with 100s of 3rd party interfaces

Solution Service virtualization and early integration testing. Create stubs for 3rd parties

to simulate interfaces. This enabled early and continuous testing

Implementation “On demand” model implemented for stub creation. Enterprise solution for

customer and 3rd parties (see “on demand” slide)

Value Reduced environment costs for 3rd party interfaces

Enabled switch to progressive development methodology (continuous

delivery)

Reduced start-up and maintenance costs for new projects through sharing of

SV and test assets

40

• Adopt continuous integration testing - Get to “Done Done” faster; test applications end to

end earlier. Everyone wants testing continuously. PMs, TMs and Testers

• Project and Test Managers think differently - they now plan to be flexible at project and

test level

• Employ Green Hat across the lifecycle – always be testing

• Share the responsibility of quality – everyone contributes (analysts, programmers, testers)

Continuous testing

Faster time to market, increased quality, reduced costs

• Isolate defects at the source – faster defect resolutions =

shorter, more efficient iterations

• Eliminate Agile’s testing bottlenecks – Service

Virtualization to the rescue