ife as a evel 5 test organization - software testing & qa ...€¦ · star ‘99 east -...

P R E S E N T A T I O N

International Conference On

Software Testing, Analysis & ReviewMAY 10-14, 1999 • ORLANDO, FL

PresentationPaperBio W11

Wednesday, May 12, 19993:15PM

LIFE AS A CMM LEVEL5 TEST ORGANIZATION

John RomanakTelcordia Technologies, Inc.

Software Quality AssuranceIndependent Testing

“Life as a CMM Level 5 Test Organization”May 12, 1999

John N. RomanakExecutive Director

Software Quality Assurance Independent Testing(732) 699-8444

[email protected]

Copyright © 1999 Telcordia Technologies, Inc. Used by permission only.

Life as a CMM Level 5 Test Organization - 2

STAR ‘99 EAST - Bellcore Independent Testing

Bellcore, with its long history of making innovative

telecommunications technology work, is now

Telcordia Technologies

Our name has changed, but our tradition of high

standards and exceptional performance continues.


“Quality, timeliness, cost-effectiveness”

. . . You cannot have all three


Bellcore has proven otherwise!



Corporate Facts• Fast Facts About Bellcore: A Wholly Owned Subsidiary of

Science Applications International Corporation (SAIC)

• Bellcore is a leading provider of Communications Software,

Engineering and Consulting Services based on World-Class

Research

Corporate Statistics• 1998 Revenues exceeded $1 billion

• Employs 6,000

Who is Bellcore?



Key Facts• Approximately 3500 employees are part of Software Quality System

• CMM Level 3 and ISO9000 Certified

• 482 Technical Quality Engineers

• 164 Major Releases Shipped

• 214 Million Lines of Code Tested

• “Best in Class” Cumulative Field Fault results

• “Best in Class” Test Cost Efficiency results

Software Quality System Profile


47.3

43.5

40.2

35.8

28.5

15.4

9.2 8.57.2

4.63.3

0

10

20

30

40

50

4Q93 2Q94 4Q94 2Q95 4Q95 2Q96 4Q96 2Q97 4Q97 2Q98 4Q98

Industry Best In Class

Fau

lts/

KF

P


QMO Compliance

ISO 9001 Certification (Phase I)

ISO 9001 Certification

(All Business Units)

CMM Level 3

Formation ofIndependent TestOrganization (5/93)

Field Fault Density- KFP

Source:Software Systems Metrics Operations Report


709

220

12795

0

100

200

300

400

500

600

700

800

Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec

1995

1996

1997

1998

Source: Customer Service Center Metrics Database


Data for 1995, 1996 and 1997 is displayed as provided at the end of each year and not retroactively adjusted

Sev1/2 Maintenance Requests


Source: Bellcore Corporate Survey

60%65%

80%85%

91%95%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1992 1993 1994 1995 1996 1997

Positive Trend in Customer Satisfaction

Customer SatisfactionSTAR ‘99 EAST - Bellcore Independent Testing

ISO 9001 Registered

CMM Level 3 Rating


STAR ‘99 EAST - Bellcore Independent Testing Start the Quality Journey

June, 1989: Why start:

“Mindset Change”

Independent Test - Clean Separation of System Test from Design & Development

Poor Relationship

With Development

Testing Provides

Low Value

Low Test Automation

Low Customer

Satisfaction

Product Vs.Solution Focus


STAR ‘99 EAST - Bellcore Independent Testing The Journey Continues

May, 1993: “Corporate Organizational Change”

Independent Test Launched - Software Systems Institutionalizes Clean Separation of System Test from Development.

Test Management Vision Launched

• What Would the World Class Testing Organization Look Like?

• How Would the World Class Testing Organization Operate?

• What Would it be Like to be Part of the World Class

Testing Organization?

• What Would the Results Look Like?


STAR ‘99 EAST - Bellcore Independent Testing The Journey Advances

April, 1995: “Cultural Realization”:

Management Change - Corporate Alignment

Strict Quality Discipline - QMO

• Testing is a Key Component Throughout the Process

• Early Testing is Critical to Success = “Prevention Oriented”


• Alignment with Business Unit Strategies

• Test Strategies Driven by Product Life Cycle

• Tactical Implementation of Business Model Goals

• Focus on Cost Reduction, Cost Avoidance, Time-to-Market


Know Your Products - Manage Them “Fundamentally” Different

SQA Testing Business Model

“Advance the Leadership”

Manage Test like a Business


STAR ‘99 EAST - Bellcore Independent Testing Strong Software Quality Assurance Structure

October, 1997: “Software Quality Assurance Alignment”

Corporate alignment of Quality Operation Center with new President & COO

SQA Web Site

• Internal Access by Corporate Officers

• Business Operations Council Review

• Overall Quality Assessment - “Quality Conscience”


The Software Systems Quality Assurance Testing Organizations,provide independent centralized test services for Bellcore’s

Software Systems Applications.


TelcordiaTechnologies

Shipped on Time

SQA Home

Business Models

Documentation

Meeting Info

Products

QA Automation Arch.

Solution Status

Product Status

Year 2000

SQA Web Site - Home Page

Qual. & Oper. Center

Metrics



10200 Software Quality Assurance TestingProduct Releases in Development - Product Status Summary

May 12, 1999(Corporate VP)

Department 10230 (J.N. Romanak - ED)

VP - GM

Business Unit

Rel # Ship Date

Y PRODUCT 1 1.1 6/30/99

Y PRODUCT 2 1.2.3 7/1/99

R PRODUCT 3 4.0 7/29/99

Y PRODUCT 4 2.2 6/30/99

Y PRODUCT 5 2.2.2 10/31/99

Y PRODUCT 6 2.3 6/30/99

G PRODUCT 7 2.4 10/31/99

G PRODUCT 8 3.1.2 9/30/99

G PRODUCT 9 4.1 10/31/99

SQA Web Site - Product Status


STAR ‘99 EAST - Bellcore Independent Testing SQA Web Site - Detailed Metrics

HISTORICAL IN PROGRESS

Estimate Current

Release # 1.0 2.0 3.0 4.0

Contractual Ship date 07/29/99

Planned Ship Date 07/29/99

Actual Ship Date

RELEASE PROFILE

Assessment

Key Points

PT PROFILE

FAULT PROFILE

Milestones Plan Forecast Actual



• Soar with your Strengths√ Technical Staff

• Advance the Leadership√ “The Management”

• Dare to Soar√ Rapid Growth

Reset the Test Management Vision - ‘98

Business Growth↑ 204% Increase in Products Tested↑ 267% Increase in Deployed Code↑ 145% Increase in Major Releases Shipped Quality Improvement↑ 306% Increase in Test Automation↑ 93% Improvement in Field Fault Density/KFP↑ 81% Improvement in Field Fault Density/MLOC

Cost Efficiency↓ 61% Decrease in Test Cost Per Line of Code

Tested

Impact to the Corporation


* Field Fault Density

* Sev 1/2 Maintenance Requests

• Test Automation

• PT/MUT Test Faults

• Removal Efficiency

• Total Lines of Code Supported

• Pre-Shipment Fault Density

• Major Releases Shipped

• Product Test Fault Density

• Test Cost Efficiency

Measures of Progress - Blue Chart

“The Yardstick”

STAR ‘99 EAST - Bellcore Independent TestingProven Quality Results

Future

• Automatic Test Case Generation

• Test Coverage

• Strong Partnership with Applied Research


• Industry Leader in Quality Process and Discipline

• Strong Quality Champion and Catalyst

• SQA Testing Business Model

• Solution/Integration Testing Focus

• Industry Leader in SQA Testing Measures of Progress Results

• State of the Art Test Tools

• “Best In Class” Independent Centralized Testing Operation

“Soar with your Strengths...”

STAR ‘99 EAST - Bellcore Independent TestingProven Strengths - Distinctive Competence Differentiators



Quality Is Our Differentiator

“You can have all three”

Quality, Timeliness, Cost Effectiveness

“Life as a CMM Level 5 Test Organization”

The Differentiator

Life as a CMM Level 5 Test OrganizationCopyright 1999 Telcordia Technologies, Inc. Used by permission only.

1

Bellcore, with its long history of making innovativetelecommunications technology work, is now

Telcordia Technologies

Our name has changed, but our tradition of high standardsand exceptional performance continues.

________________________________________________

STAR’99EAST Bellcore Paper

“Life as a CMM Level 5 Test Organization”

Introduction:

Quality, timeliness, cost-effectiveness – for years software professionals have

believed you cannot have all three. Bellcore has proven otherwise. Bellcore’s

approach to Software Quality has made us a leading provider of World-Class software.

Bellcore’s Software Development organization numbers about 3500 employees, with a

portfolio of 150 operations support systems and over 110 Million Lines Of Code

(MLOC). Positioning Bellcore for SEI CMM Level Five in 1999.

Testing is needed now more than ever and testing is now more complex than

ever. Testing is not a revenue-producing activity, but rather a revenue-preserving one.

However, testing cannot be an add-on activity, after software development is complete.

It must be integrated throughout the development lifecycle in order to be cost-effective.

A more effective testing capability leads directly to increased customer satisfaction, as

systems work and work right the first time. A more efficient testing capability leads

directly to increased Quality, as test cost savings increase so does customer

profitability and satisfaction.


2

Goals and Objectives:

• Journey through Development and Test Culture paradigm changes

• Understand how to Manage Test like a Business

• Experience what it’s like to be a Corporate Independent SQA Testing Centeraligning directly to the President of the Company

• Explore Bellcore’s sophisticated SQA Web interface for overall SoftwareQuality Assessment, which can be accessed by executives to view eachsystem’s overall quality “health” status based upon proven process anddiscipline

• Proven Measures of Progress historical metrics that serve as Bellcore’sdistinctive competence differentiation in being an Industry Leader inIndependent SQA Testing operations

Meeting the Challenge for Test Excellence:

The quality of software testing is normally inconsistent, with different organizations

performing testing in different ways. Although corporate quality models can exist, with

clearly defined milestones and deliverables, the processes used to create deliverables

are not always detailed and the model is not always used by all organizations. While

many excellent test processes and knowledgeable people exist within the industry, this

knowledge is not applied consistently to achieve predictable, consistent testing

outcomes.

Key issues encountered in the Industry with its current testing environment are:

• Unclear ownership of the testing discipline in the company

• General lack of esteem in which testers are generally held compared toother software development professionals

• Inconsistent levels of involvement of users in the test planning process, andtesters in the planning and software development efforts


3

• Ineffective integration of software testing throughout the softwaredevelopment life cycle

• Proliferation of locally controlled testing facilities and testing within theproduction environment

• Lack of a centralized program to define, develop and maintain test skills.

Bellcore’s testing challenge is for a robust, mature testing practice to ensure that

corporate software quality goals are achieved consistently and predictably. This will

require a corporate-wide testing methodology, tools, training and guidance, as well as

infrastructure support for the initiative. Key goals of this practice are:

• Clear demonstration of management support for testing

• Well-defined test processes used by trained individuals

• Testing efforts that are aligned with business goals

• Testing measurements to ensure adequate test quality, performance and

accountability

• High level of quality and overall customer satisfaction for testing.

Bellcore clearly understands testing vision and software quality assurance challenges

that lie ahead. There are many similarities between software industry’s current situation

and its challenges and Bellcore’s experiences with software development. Often,

testing of software products is an afterthought - a requirement for sign-off or at best an

opportunity to find the most obvious of the software “bugs” prior to production. The

focus is typically on defect removal instead of on defect prevention. It is much more

cost effective to avoid “bugs” before coding is completed than to test, identify and

correct software anomalies after coding is completed. Bellcore’s solution for software

quality assurance testing is based on the premise that the software testers are involved

throughout the Software Development Lifecycle. From the development of software

testing strategies during the software sales process to the production implementation at

a customer site, the Testers are always valued members of the software development

team. The Testers provide important insights on the testability of the requirements, the


4

usability of the designs, and the overall test strategy required to fully test the feature,

product or solution at hand. Since the testers typically have the most “hands-on”

experience (next to the actual customer), they bring highly valuable customer insights

to the team that will help to improve overall customer satisfaction. The involvement of

the Independent Test Staff throughout the Software Development Lifecycle will also

help to raise the overall technical expertise of the Test staff as they increase their

understanding of the product and the overall end-customer requirements and usage.

Experience Bellcore’s Journey for Test Excellence:

Bellcore has been an industry leader in testing techniques and quality assurance

applications. Bellcore has developed a unique testing program and methodology that

has led to a level of product and process interoperability that is unmatched in the

industry. Bellcore’s Software Systems Organization has received industry recognition

in two major quality arenas:

♦ In October 1996, Bellcore’s ISO 9001 Certification was expanded to include all of

Software Systems. The quality system was examined for conformance to the

twenty elements required by ISO through a random sampling across a wide

spectrum of projects. With no open non-conformities at the end of the audit, we

were recommended for immediate certification.

♦ In December 1996, Bellcore Software Systems successfully completed the

Software Engineering Institute’s Capability Maturity Model (CMM) assessment and

achieved a Level 3 rating with significant strengths noted in some key process

areas at higher levels. Only 11% of organizations assessed have been rated as

Level 3 or higher. In terms of organizational size and breadth of scope, Bellcore

Software Systems was the largest assessment ever performed.

Bellcore’s software quality journey for “Best in Class” Independent Testing began in the

1992/1993 timeframe. At that time, only 30% of our testing was automated. Testing

was viewed as providing low value and the relationship between the testing and

development organizations was poor at best.


5

Bellcore’s approach to Independent Software Quality Assurance Testing, a clean

separation of system test from design and development, has been a key component of

this success. Bellcore's testing methodology specifies the ownership of tasks and

responsibilities and also the process of obtaining agreement and buy-in across

organizations. In addition, goals are concise and quantitative and linked to Bellcore’s

appraisal process. Bellcore's testing methodology requires that quality is everyone's

responsibility across the organization. Bellcore's methodology specifies the roles of

each organization and of each participant. Deliverables by all organizations involved in

the system development are reviewed and signed off according to the criteria set by the

team. Each test activity has roles and participants clearly identified. Bellcore’s strategy

includes methods for determining the number of staff, level of expertise and particular

job functions required for a software project/product based on where in the software

lifecycle the project/product falls. In general, Bellcore’s methodology includes 4 primary

job functions: Test Project Manager (responsible for initiatives related to the Quality

Assurance Test function), Test Manager (responsible for the overall test activities for

the product/project that his/her staff supports and the management of this staff), Lead

Tester (responsible for the technical aspects of the testing and for mentoring the

testers) and Tester (responsible for detailed test planning and execution).

It is a common behavioral characteristic of software development organizations to staff

the software testing function with under-qualified personnel. Most often, talented

market hires are directed to the software development shop before consideration is

given to the software testing organization. It is not uncommon for a Test Organization

to report to the same management as the Requirements and Development staff. This

structure lends itself to the redirection of qualified technical staff to work on systems

engineering and/or development efforts instead of software testing. The first step in

the evolution of improving software quality is to improve the quality of the test staff.

The key here is what Bellcore has done with the formation of an independent test

organization, so there is a clean separation of system test from design and

development. Make it known within the company that only “the best” will work in the

Independent Testing Organization. Make sure this Test Organization has the best


6

equipment – not “hand me downs” from Development. Hire highly technical,

experienced people. Seed the Test

Organization with a handful of stars from inside and outside the company to increase

the credibility of the Test Organization and to act as role models, mentors, and trainers

for the less technical staff. Do not allow poor performers to transfer to the Independent

Testing Organization. Reward the test staff when they succeed. Make it clear that

Testing is a profession with a clear career path. Set up clear job descriptions for each

level of staff with titles that are comparable to those of the Development Organization.

Make sure the Training Organization provides clear training tracks and courses for

professional testers. Provide opportunities for the test staff to attend outside training

and Test conferences in order to raise awareness that Testing is a valuable career

choice for software professionals.

Bellcore’s experienced test managers tailor and implement comprehensive testing

solutions, while developing an overall test strategy and plan to support its Software

Systems’ Business. By following a well defined Test Strategy, we are able to

implement a corporate-wide testing methodology, tools, training and guidance. Testing

efforts will be aligned with business goals, and progress will be measured to ensure

adequate test quality, performance and accountability, with a focus on quality and

overall customer satisfaction. Bellcore’s business of software testing is based on the

maturity of the software product or solution being tested. For example, a product’s test

automation strategy will be very different for an emerging product than for an

established product. The focus of the automation strategy is cost reduction, cost

avoidance, and time to market. In addition, Bellcore’s Independent Testing operations

moves the decision-making and budget responsibilities for each product’s test function

to the line manager of the test group. The Test Manager is responsible for test cost

estimation, test budget tracking, and personnel. Also, the test manager and the

individual testers are responsible for signing the test readiness checklist and signoff

sheet for each product release. Bellcore additionally calls for key learning sessions

after each phase of the software process and the Independent Testing Organization

provides valuable insights for each of those. In addition, our Test Organization uses a


7

full range of in-process test metrics to manage each testing phase, and to determine

areas for improvement going forward. These metrics are produced at the product level

and rolled up to the corporate level with intermediate levels as necessary. The

corporate level metrics are used to

compare and contrast Bellcore’s performance against software industry trends and to

track overall progress as a company. Individual product metrics are used to determine

the items to be entered in each product’s Quality Improvement Plan and to measure the

product’s progress in achieving these planned improvements.

In addition to metrics, the Bellcore Independent Testing Organization incorporates the

use of a SQA Testing Project Manager. The Project Manager is responsible for guiding

the Independent Testing Management team in identifying key areas for improvement

for the Test Function as opposed to the individual Products, and for identifying and

implementing the steps necessary to move the Testing and overall Quality to best in

class in the industry. Bellcore has dramatically increased the number of test cases

executed per tester in its own testing organization by the effective use of test

automation. Test tools play a critical role in an organization's overall quality program.

Tools make it possible to improve the efficiency and effectiveness of the testing

process. When used prudently, test tools can increase the rate of defect discovery and

decrease test cycle times. However, successfully introducing tools into an organization

requires an overall automation program. The program must be based on (1) goals with

realistic expectations for return on investment, (2) an organizational structure with

strong management support and (3) technology built with sound software engineering

discipline. Experience has shown that automation goals should not be set uniformly

across all application teams. Goals for automating should be established based on the

following factors: application lifecycle, application life-span, relationship to critical

business work-flows, and platform strategy. Developing automated scripts at the wrong

time or for the wrong reasons can be much more expensive than manual testing.

Therefore, the automation program should be initiated with specific and realistic

expectations for the return on investment.


8

Metrics are critical to the successful implementation of a quality testing organization for

several reasons. First of all, metrics give all members of the team visibility into the

project as well as advance warning of problems. In addition, metrics allow the

organization to track its progress on quality improvement and celebrate success. Some

of the metrics that Bellcore uses successfully as its “yardstick” are:

• Field Fault Density

• Severity 1 and 2 Maintenance Requests

• Test Automation

• System Test/Multi-Unit Test Faults Ratio

• Removal Efficiency

• Pre-shipment Fault Density

• Major Releases Shipped

• Test Cost Efficiency

In addition, Bellcore has established a considerable in-process metrics program. This

program allows for the Management and Staff of the product teams to set goals and

evaluate their progress against the goals on a consistent basis. In-process metrics can

be used to determine if additional Management intervention is needed when products

fail to meet these in-process goals. Bellcore has considerable experience in

implementing and managing realistic jeopardy action plans that will guide a product

back onto schedule. Bellcore has found that this set of metrics provides valuable insight

into areas for improvement and the level of product quality.

Bellcore's testing methodology clearly states organizational testing goals, in both

subjective measures and metrics. Goals are formed “top-down” and clearly support the

overall corporate goals. Quarterly corporate test and quality goals are measured and

the corresponding graphs and charts are strategically displayed in all locations so that

each member of the product team (not only the test organization) is aware of the

current status. In particular, the test metrics of effectiveness (number of MRs found by


9

testing, number of MRs found in deployment, etc.) and efficiency (percent of test cases

automated tracked to the business model for automation, etc.) combine to clarify each

testing organization’s goals.

Bellcore’s Proven Results:

Since implementation of our Quality Process and our Independent Software Quality

Assurance (SQA) Testing Organization, we have reduced the number of Severity 1 and

2 troubles our customers experience by 82% from 1995 to 1997, and, 51% through

1998, saving tens of millions of dollars for both our customers and ourselves, and

improving our overall customer satisfaction from 60% to 95%.

To put these results in context, from 1993 to 1998, Bellcore’s Software Systems

organization experienced the following growth in its business:

• 204% increase in products tested

• 267% increase in deployed code

• 145% increase in major releases shipped

In terms of quality improvement, Bellcore’s results are:

• 306% increase in test automation

• 93% improvement in Cumulative Field Faults/KFP

• 81% improvement in Cumulative Field Faults/MLOC

From a cost efficiency standpoint, Bellcore has achieved a 61% decrease in Test Cost

per Line of Code Tested over the same period.


10

Bellcore's proven testing methodology involves culture, process, discipline, and

organizational change. Bellcore acknowledges the difficulty to introduce and

implement this new process. However, this methodology will lead to institutionalizing

test quality assurance, with measurable Quality Improvement.

Bellcore’s Testing Paradigm Change:

Bellcore has created a world class software development quality assurance testing

organization. We know the pitfalls and success factors, especially for an organization

of the size and complexity of Bellcore’s. We know what it takes to make such an

enormous paradigm change to where software testing is a valued part of software

development, software testers are highly critical members of the product/project team,

and where the focus shifts from error detection to error prevention. We have a proven

model for job and career satisfaction for testers and test managers that enables hiring

and retaining excellent staff in Testing.

Benefits of Bellcore’s Independent Testing Organization to the business include:

• Shortening implementation time for launching internal testing programs

• Assuring high quality results by working with a world class process

• Reducing costs of recruiting and retaining top personnel

• Directing investment to the right equipment and tools

• Giving personnel more time to focus on developing their own expertise andcapabilities

• Improved planning and budgeting

• Exceeding customers’ expectations by starting testing earlier so customers’ servicesget to market on time

• Achieving “Best in Class” measurable Quality Improvement and Metrics.


11

In Closing:

Bellcore’s SQA Independent Testing Organization is now industry-recognized as best in

class. We have presented numerous papers describing our SQA Testing approach at

industry conferences such as STAR, Software Quality week, EuroSTAR and ISSRE. In

the last 3 years, we have provided workshops on Independent SQA Testing and have

shared our success story for many large corporations.

References:

Special thanks goes to Anita Sauer, Executive Director, and Chris Spuler, Senior

Director, for their experience and insights in shaping key points in this paper.

John N. RomanakJohn N. Romanak – Senior Executive Director - Software Quality Assurance IndependentTesting for Telcordia's Software Development organization. John leads SQA Testing ProjectManagement, Contract/Proposal Management for Elective Testing Services,Testing/Automation Architecture and Applied Research - Testing Paradigm advancement.

Thirty years’ experience in Software Systems, with the past seven years concentrated inbuilding a world class independent testing organization at Telcordia. Telcordia’s SoftwareDevelopment organization numbers about 3,500 employees, and is ISO9000 and SEI CMMLevel Three certified. Telcordia’s quality journey began in 1993, when testing was viewedas providing low value and there was a very poor relationship between testing anddevelopment. Telcordia’s SQA Independent Testing organization is now industry-recognized as best in class. Quality, timeliness, cost-effectiveness – for years softwareprofessionals have believed you cannot have all three. Telcordia has proven otherwise.John Romanak has given workshops on Independent SQA Testing and has sharedTelcordia’s success story with many large corporations.

ife as a evel 5 test organization - software testing & qa ...€¦ · star ‘99 east -...

Documents