step-in summit 2008

STeP-IN SUMMIT 2008

5th International Conference On Software Testing

Delivering Defect Free High Customer Experience Solution through Agile driven Test Automation and

Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a European Telecom

Giant

by Amandweep Goel,

Subramanian Thangavelu & Venu Kozhummal

Infosys Technologies Limited

[email protected] [email protected]

[email protected]

Copyright: STeP-IN Forum and Quality Solutions for Information Technology Pvt. Ltd.

Published with permission for restricted use in STeP-IN SUMMIT 2008 in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without

the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during STeP-IN SUMMIT 2008 as per the applicable conditions.

Practices Experience Knowledge Automation

Produced By Hosted By

www.stepinforum.org www.qsitglobal.com

mailto:[email protected]



http://www.stepinforum.org/

http://www.qsitglobal.com/

STeP-IN SUMMIT 2008 Delivering Defect Free High Customer Experience Solution through Agile

Paper Publication

driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a

European Telecom Giant


http://www.qsitglobal.com

1. Abstract

Customer Experience has multiple dimensions to it, of which most important is the density of

delivered defects. High Customer Experience derives itself as a result of close partnership

between the participating organizations through and through, but is never complete without a

Defect Free Delivery.

While cross team discipline and strict process adherence might seem like a wholesome recipe to

achieve precise control over Defect Injection Rate [DIR] and a 100% Defect Removal Efficiency

[DRE] thus enabling Defect Free Delivery, it cannot ensure success when

1. The project requires very high effort and is expected to be executed in an agile manner

to enable real-time re-alignment with the constantly changing scope, not leaving any

waste for the client in the whole churn.

2. The system delivered interfaces not just with another software system, but also a real,

large, next generation nation-wide telecom network that itself is evolving hand-in-hand

with the system.

3. The delivering organization has to interact not only with its client but also needs to

manage, for its delivery, alignment with other suppliers to that client, which themselves

are some of the largest Consulting Firms, System Integrators, Network Equipment

Manufacturing Giants and Software Product Vendors across the globe.

4. Every defect gets attention of likes of the CIO’s in the client organization, given that the

initiative, to which the project contributes, is actively followed by analysts and hence

affects the shareholder confidence.

The authors here, attempt to present their experiments and success with continuous innovation in

management of one such project, resulting in exceeding the promised measures of customer

experience and 100% defect free delivery.


Paper Publication





2. Target Audience

This experience share would prove useful to IT Project Managers who have committed to respond

actively to change, but are facing issues in controlling the quality of their deliverables. Also this can

be a thought provoking excerpt to those who have implemented Agile Models in their projects and

are looking for ways to mature further in that unexplored area. Finally the whole IT community

would benefit by extrapolating best practices established through this example.

3. Application Area

The project to which these practices were applied, aims to deliver under extremely challenging

deadlines & multi-vendor environment, a defect free (functional, performance & user experience)

web-based OSS (Operating Support Systems) solution which can provide flow through provisioning

(Service Fulfillment) and activation for the next generation network of the client.

4. Achieving 100% Defect Free Delivery and High Customer Experience

4.1 Need

Competitiveness in the telecommunications space asks for bringing new services to customer and

bringing them first in the market! To keep such a pace, the systems enabling that telecom company

should always service the customer orders ‘right in the first try’ and that too with less ‘Order Cycle

Time’ of the competitor.

For a System Integrator, this translates into an objective to match the pace by delivering the

solution right in the first go (defect free, both from the functional, performance and user-

experience perspective) without compromising the resilience to multi-magnitude of scope change or

the delivery timelines. Constant innovation is the order of day in such circumstances, especially

when agile working patterns are followed to achieve resilience to change, which by their basic

nature induce flatulence to the project execution.

4.2 Execution Details

A combination of the following was employed:

1. Continuous Integration through 100% Test Automation & Iterative Implementation to ensure

resilience to constant scope changes and elimination of verification and validation

overheads that come coupled with the growth of the application

2. UAT during Requirement Analysis Phase which enabled the software to be delivered

without any functional, performance and user experience defects


Paper Publication





3. Performance Engineering and Test Driven Development that ensured the application to be

really ready for deployment anytime

4. Co-Location of a core system engineering team with E2E test and UAT teams that

reduced the communication wastage to nil and leveraged the collective knowledge of all

the teams to thrash out inappropriate defect candidates faster, resulting in reduced

delivery time.

5. WAR Room – Virtual combat of defects through constant analysis and optimization. Can also

be called as ‘Frequent Cyclic Retrospective’ [FCR] and ‘Active Defect Prevention’ [ADP].

4.2.1 Continuous Integration

Rapid realignment of the project scopes required breaking down the project scope into smallest

possible chunks and aligning the delivery in accordance, enabling the customer to have a fully

tested, working piece of software at any given point of time. Therefore, realigning the scope of the

project became feasible without generating any waste in the development and testing process. This

demanded for releases on much lesser intervals as small as a week.

Continuous Integration is a software development practice where members of a team integrate

their work frequently. Each time the integrated piece of functionality is verified by an automated

build-deploy-test process enabled by tools to detect software variances as quickly as possible and

get them fixed immediately.

A method of churning out weekly releases with fully tested working functionality was realized

through deployment of tools and methodologies that help automate the repetitive phases like build,

deploy, test, report and release.

Each tool that enabled CI was used to target a specific area of implementation:

# Tool Name Tool Description Used to 1 JUnit Open Source framework

for automation of unit

testing

Used to facilitate the development of automated

unit test cases for the Java classes employed in

the application

2 Apache ANT Freeware for automation

of build and deploy of

enterprise archives

Used for automated build and deploy of the

application. Removed person dependency and

zeroed the scope of manual errors in the build /

deploys


Paper Publication





# Tool Name Tool Description Used to 3 Compuware

Test Partner

Third Party tool to

automate GUI based

functional testing

Used to develop scripts for automating the

functional testing of the application from end-to-

end concentrating more on the GUI and

automated interface

4 AutoIt Freeware Windows

Automation Language

Used to invoke Test Partner Scripts on Windows

after deployment is completed on Unix platform

thus enabling E2E automated execution

5 Cruise Control Used to co-ordinate the execution of build, Unit

Tests and AutoIt for functional test execution and

report from E2E process of release automation.

Open Source framework

for a continuous build and

test process

These tools together facilitated the build, deploy, regression test (unit tests and functional tests),

report and release process at the offset of a single mouse click, thereby achieving 100% automation.

Automated Build-Deploy-Test-Report-Release Process – How does it work?

The Automated Build-Deploy-Test-Report-Release Process is realized in the following manner:

1. Developer checks in unit tested code and test classes into the CVS

2. Release Manager schedules a build in Cruise Control

3. At pre-scheduled time Cruise Control checks out source code, JUnit test classes, build and

install scripts from CVS


Paper Publication





4. Cruise Control runs JUnit test classes on the checked out code, and makes available for

viewing the report generated by JUnit in its dashboard

5. Cruise Control proceeds with the build if all JUnit test cases ran successfully; else halts

the build at this stage, which needs correction before proceeding further

6. Cruise Control builds and installs the Application using ANT build and install scripts

7. Cruise Control invokes AutoIt scripts on the Windows PC designated for functional testing

8. The AutoIt scripts perform post installation steps on the PC designated for functional

testing using the COTS Product Admin Client

9. The AutoIt scripts then initiate the Test Partner automated functional test scripts

10. Test Partner executes the functional tests on multiple environments in parallel to reduce

test time

11. Test Partner generates a report with the status of the functional test cases run, which can

be viewed from the Test Partner Admin Client

This completes the Build-Deploy-Test-Report-Release process without any manual intervention at all

and hence enables the project to have weekly releases without adding any overheads.


Paper Publication



4.2.2 UAT during Requirement Analysis Phase

To enable the software to be delivered without

any functional, performance and user experience

defects, three-fold virtual UAT comprising of

Virtual Integration Testing, Performance &

Process Review and Continuous Usability Check

was launched that started right from the

Requirement Analysis Phase.

Virtual Integration Testing – For the system in context here, integration tests with interfacing systems and

the network [vended by world’s largest OEMs] used to be performed by the client teams themselves as a

part of UAT. This testing phase typically used to take a quarter for each quarter’s delivered

functionalities, since the defects identified in that late stage caused a big impact to by then hardened

system and resulted in increased turnaround times.

To compress this testing phase and reduce the

resultant high rework, the requirement team

while identifying the requirements, worked

alongside requirements team of other

components [delivered by other international big

consulting firms and SI partners] to virtually test

the interfaces through exchanging hand-crafted

XMLs for interface design and development

rather than just agreeing on interface

specification documents. These XMLs were then

used with stubs by all the interfacing system

development teams to validate the interface.

http




Paper Publication



Performance & Process Review – With the new

evolving service and network design, the Solution

Design Teams [vended by a world renowned IT

consulting company] were concentrating on the

functional requirements to feed into component

delivery to meet the stated deadlines by the

business. This is a typical scenario for any new

initiative being rolled out. The common thing

that gets ignored is the non-functional

requirements and also the process change that

the transformation would cause. This void gets

detected late in the process post-deployment of

the solutions and causes a huge flux before it

gets settled thereby increasing the time to

actual-use-of-systems.

To tap these delays early in the implementation

lifecycle, the team involved itself with the users,

network, service and solution designers to arrive

at the expected real life loads that the

application would be subjected to and also other

non-functional aspects like security. In addition

to this, applicability of the assumed process was

checked for through close collaboration with the

users. The results were fed into the solution

design and major acceptance defects were

avoided.

Continuous Usability Check – Rather than

allowing the users to UAT the delivered system

and report back the deviations from the

expected, the requirements team and project

management involved users partly through the

above stated Process Review, but mostly also

organized active feedback sessions with users,

where the application was shown to users at

frequent intervals (almost once every month),

feedback was taken and preferences were

implemented after getting the solution design

tuned. This made users pretty comfortable with

the application and also avoided any UAT defects

that might have been reported even due to

incomplete solution design, something whose

delivery we did not even participate in.

http




Paper Publication



4.2.3 Performance Engineering and Test Driven Development

Running into issues with the performance of the application and solving them, developing the

application and then testing it are the typical ways projects are executed. This pattern was

reversed and applied to get better control and early detection of defects.

Performance Engineering – A small team of two was set up to find the memory leaks in the

application that caused a threat to its sustainability, to perform load testing on the application and

report back to the users to get agreement or expected figures, experiment concurrent access and

http




Paper Publication



object contentions through the available 100% automated test suite and proactively tune the

application making it ready for real live deployment. This made sure that not only the application

provides the functionality to the users; it can also sustain the live conditions and give extended up-

times that were required.

Test Driven Development - Test-Driven

Development (TDD) is a software development

technique consisting of short iterations where

new tests covering the desired improvement or

new functionality are written first, then the

production code necessary to pass the tests is

implemented, and finally the software is re-

factored to accommodate the changes. The

availability of tests before actual development

ensures rapid feedback after any change.

Practitioners emphasize that test-driven

development is a method of designing software,

not merely a method of testing.

This method of functioning achieves multiple

benefits

1. The tests are written before the

application is developed, this ensures

that the tests performed on the

application are user / requirement

centric rather that being design /

implementation specific. Thus the

typically unmet validation phase gets

pronounced as required.

2. Periodic tests performed help reducing the waste because there is always an option to

revert back to the previous fully tested version

http



http://en.wikipedia.org/wiki/Software_development

http://en.wikipedia.org/wiki/Test_case

http://en.wikipedia.org/wiki/Software_engineering


Paper Publication



http



4.2.4 Co-Location of a core system engineering team with E2E test and UAT teams

E2E Test and UAT Phase comprised of testing a stack of applications together end-to-end along with

the network downstream out of which the most critical application was the application in context

here. This was because the application was at the bottom of the aforesaid application stack

touching the network and translating a very high level order given somewhere near the top of the

application stack to complex inter-interlinked activations and also synching up the inventories of

the systems all along this system stack.

The complexity experienced therefore was to wait for the orders to start flowing through from the

topmost system in the stack and making its way through various systems, sorting out their

functionalities and interfaces in the process. In very obvious circumstances of these upstream

systems taking a lot more time to straighten themselves than what should have been

proportionately taken up, the system in context was just expected to work right every time the E2E

Test Team could achieve pushing an order through to it.

While the means discussed in the previous and the following sections ensured that the system in

context was robust and 100% defect free, any valid error even in the case of an upstream system

faltering its expected behavior was initially recognized as a defect in this system lying at the

bottom of the stack and touching the network. Since the timelines left for this last systems were

almost nil, escalations went to the likes of CIO’s of the client putting the credibility of the

delivering organization into question.

Rather than being reactive to this course of actions, and attempting to analyze the issues faster,

getting into face-saving discussions with senior managers at the client site, the project management

took a proactive step by co-locating a part of its engineering team with the E2E test team and the

UAT teams.

What resulted thereof was a combination of localized End to End tests between systems as a result

of recommendation of this engineering team thus giving equal breathing time to every system to

straighten itself with a clear understanding into the E2E and UAT test teams of the expected

behavior of various systems thus enhancing their accuracies in localizing the defected system.


Paper Publication



http



This also helped the UAT teams to ask for minor confidence-building tweaks in the systems. One

such example for providing for a gate in the system touching the network which the user needs to

open before the system actually starts activating the network. This gave an opportunity to the UAT

teams to check everything that was going to the network before it went, acting as a huge

confidence building measure. And well all the other procedures and measures discussed here

ensured that it was always right!

4.2.5 WAR [Weekly Active Retrospective] Room

FCR [Frequent Cyclic Retrospective] & ADP [Active Defect Prevention] are the two terms coined by

the management of this project. While Agile Practices, cross-team discipline and strict adherence

to processes was helping the project sailing through very low Defect Injection Rates [DIR] and very

high Defect Removal Efficiencies [DRE], the retrospectives and Defect Analysis Meetings identified

that root causes of defects that slipped into further stages of SDLC were different every time and

that the processes set up for the project needed constant fine-tuning. Comparing this to the vision

of Defect Free Delivery that the management had set for the project, there was always a gap that

realized itself and broadened in between any two delivery cycles, thus allowing some dusty corners

to getting carried over into further stages.

Lateral retrospection of this strange pattern evolved an understanding of the way client was

reacting to the business and changing its processes every quarter and hence its effect to the

relevance of the set processes of the project in context here. This change was not just a quarterly

change, but a continuous one, primarily because the new processes identified every quarter took a

whole quarter almost to settle down thus causing continuous ripples to the project processes.

As an experiment, a weekly active retrospective

was scheduled to check the relevance of the

current processes and suggest changes. Also a

mini Defect Prevention exercise was undertaken

to discuss just the defects raised within the

previous week, identifying the root cause of each

one of them and setting process changes and

initiatives to combat the same. This exercise

involved every participant from the team that

delivered the functionality in which the defect

was identified indirectly building high

conscience, sensitivity and discipline in the

team.


Paper Publication



The result was drastic reduction in the defects carried over to the further stages making it a defect-

free customer experience eventually [no defects in the E2E test and UAT phased and zero live

incidents].

5. Issues and Resolutions

1. Integration testing with other systems and network asked for tremendous manual regression

testing effort thus hitting the delivery time - Automated the tests even when testing with

the network and other systems!

2. Unavailability of test network and interfacing systems during requirements phase since

these were getting evolved in parallel - Worked with the respective co-vendors to get the

tests executed in their development labs and developed stubs to combat the absence of

interfacing systems!

3. Project Processes getting outdated with constantly changing Delivery Environment –

Implemented ‘Frequent Cyclic Retrospective’ and channeled the findings into ‘Continuous

Process Optimization’.

4. Developers and Testers mind-set as a blocker – Constant motivation of team members,

facilitation and encouragement of un-ending innovation opportunities.

5. In a multi-vendor scenario, aligning the deliveries with the program expectations was highly

difficult - Early identification of all dependencies, cross-alignment with co-vendor delivery

http




Paper Publication



http



roadmaps, continuous progress tracking and developing a congenial environment where each

vendor contributes to the early integration testing thus resolving issues to a great extent.

6. Benefits Achieved

Following benefits were realized by applying these innovative practices:

1. Time to deliver: Software delivery cycle time brought down by 55%

2. Defect reduction: 100% DRE – No Defects in E2E tests, UAT and production

3. Effort reduction: Regression test effort maintained at zero even with linear growth of the

application

4. Increased Return On Investment: Tests automation in all areas saved 12% of effort which

was used to deliver more features to business at the same cost

5. Rework effort: Reduced from 5% to 1% of the total effort

Glossary

ADP - Active Defect Prevention EMS -Element Management System

ANT - Another Neat Tool FCR - Frequent Cyclic Retrospective

CI - Continuous Integration IT - Information Technology

CIO -Chief Information Officer OSS - Operation Support System

CLI - Command Line Interface PC - Personal Computer

COTS -Commercial Off The Shelf SDLC - Software Development Life Cycle

CVS -Concurrent Versioning System SI - System Integrators

DIR - Defect Injection Rate TDD - Test Driven Development

E2E - End To End UAT - User Acceptance Testing

XML - Extensible Markup Language

ALL RIGHTS RESERVED Copyright in whole and in part of this document ‘Delivering Defect Free High Customer Experience Solution through Agile driven Test Automation and Innovative Defect Prevention Practices in Multi OEM Vendor/SI Environment for a European Telecom Giant’ belongs to the authors. This work may not be used, sold, transferred, adapted, abridged, copied or reproduced in whole or in part in any manner or form or in any media without the prior written consent of the authors.


Paper Publication



About the Author(s)

Amandweep Goel - The author is a Project Manager working with the ‘Communications Service Provider’ division of Infosys Technologies Limited (www.infosys.com), Bangalore - India.

He holds Master of Science, Birla Institute of Technology & Science, Pilani, India; Bachelor of Technology, National Institute of Technology, Punjab, India.

He has more than seven years of experience in implementing various software projects in Operational Support Systems (OSS) domains for major OEMs and Telco’s across the globe.

Currently he is responsible for delivering Service Provisioning and Assurance solutions to a major telecom giant in Europe using agile best practices and re-usable component development.

Subramanian Thangavelu, PMP - The author is a Senior Project Manager working with the ‘Communications Service Provider’ division of Infosys Technologies Limited (www.infosys.com), Bangalore - India.

He holds Bachelor of Engineering Degree from College of Engineering, Guindy, Chennai (Madras) - India.

He has more than nine years of experience in implementing various software projects in Network Management Systems (NMS) and Operational Support Systems (OSS) domains for OEM and Telco’s in America and Europe.

Currently he is responsible for delivering Service Provisioning and Assurance solutions to a telecom giant in Europe using agile best practices and re-usable component development.

Venu Kozhummal - The author is a Test Manager working with the ‘Independent Validation Solutions’ group of Infosys Technologies Limited (www.infosys.com), Bangalore - India.

He holds Master of Technology Degree from College of Engineering, Trivandrum - India.

He has more than eight years of experience in implementing various CAD-CAM Engineering, Banking and Telecom OEM software projects for clients across the globe.

Currently he is managing Integration Testing of multiple OSS Solutions

http



http://www.infosys.com/




Paper Publication



http



delivered by multiple System Integrators for planning, building and activating the next generation telecom network of a leading service provider in Europe.

step-in summit 2008

Documents