Optimizing Cost of Quality in Legacy An investment to increase ROI
Executive Summary
Quality processes cannot be followed only because someone else is following them, but return on following them has remarkable impact as projects mature. The cost of poor quality can result in raising the total project cost significantly. Most businesses do not know what their quality costs are because they do not keep reliable statistics. Finding and correcting mistakes consume tremendously large portion of resources. Typically, the cost of elimination of a failure in the production stage is five times greater than it is at the development stage. Effective quality management decreases production costs because the sooner an error is found and corrected, the less costly it will be to rectify it. As a legacy system ages, the optimization of cost of quality contributors gets complex. With more and more job abends, less/non SME availability, etc., the cost to maintain good quality goes higher. Eventually it can become very challenging to manage the system. HCL-MMS services provide an end-to-end solution to optimize the factors which are impacting the cost of quality for a legacy system. It not only focuses on the visible parameters impacting the cost of quality, but also the concealed and interlinked parameters, giving a holistic solution.
Purpose
The purpose of this paper is to epitomize how to optimize the CoQ for a legacy system and its benefits in the long run. It also highlights HCL Mainframe and Midrange Services‟ capabilities to optimize the cost of quality in a legacy system.
Background
The theory behind the Cost of Quality (CoQ) Model is that, the more time that is spent on Prevention activities, the less time will need to be spent on Appraisal and dealing with Failures. Prevention activities build quality into the system. Both Appraisal and Failure are “after the event”, and therefore are too late and add cost. The costs associated with quality are divided into two categories: costs due to poor quality, and costs associated with improving quality. Prevention costs and appraisal costs are costs associated with improving quality, while failure costs result from poor quality. An organization‟s primary goal is to maintain high quality services. With a comprehensive understanding of the costs related to quality this goal can be achieved.
2
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Figure 1: Cost of Quality classification
Cost to achieve good quality To ensure that customers will receive a product or service that is worth the money they will spend, firms should spend on prevention and appraisal costs. Prevention costs are associated with preventing defects from occurring. The focus of a prevention cost is to assure quality, and minimize or avoid the likelihood of an event with an adverse impact on the product or service. This also includes the cost of establishing a quality system. A quality system should include the following three elements: training, process engineering, and quality planning. Quality planning is establishing a production process in conformance with design specification procedures, and designing of the proper test procedures and equipment. Training is establishing training programs for employees to keep them efficient on emerging technologies, such as updated languages, platforms, and programs. Appraisal costs are direct costs of measuring quality. This includes: testing, reviews, costs associated with assessment for ISO or other quality award assessments.
Cost due to poor quality The cost due to poor quality is accounted from the failure costs, which is divided into two categories: internal and external. Internal failure costs are expenses incurred from internal failure. This includes cost of rework to fix the pre-shipment defects. External failure costs are associated with product failure after the completion of the production process.
Failure costs take many forms: The effort that developers spend for investigating and
diagnosing defects and then reworking designs and code to correct them.
Schedule slippage as testing uncover defects that require rework and re-testing.
The customer support costs, most of which are for helping customers to deal with all of the defects shipped to them, while developers spend even more time in investigation and rework.
3
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Cost of Quality
Cost due to
poor quality
Cost to achieve
good quality
Internal
failure cost
External
failure cost
Prevention
cost
Appraisal
cost
Statistics According to a study commissioned by the U.S. Department of Commerce's National Institute of Standards and Technology (NIST), software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product,. At the national level, over half of the costs are borne by software users and the remainder by software developers/vendors. The study also found that, although all errors cannot be removed, more than a third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects. These are the savings associated with finding an increased percentage (but not 100 percent) of errors closer to the development stages in which they are introduced. Currently, over half of all errors are not found until "downstream" in the development process or during post-sale software use.
Figure 2: Cost of inadequate testing
If the defects are not fixed early in the life cycle, the rework effort due to external failure can turn out to be very expensive.
CoQ in a Legacy System
In the previous section, we defined the three major costs of quality and discussed their application. We can now examine how they affect mainframe system. The more an IT project invests in preventive measures, the more they are able to reduce failure costs. Furthermore, an investment in quality improvement benefits the organization image, efficiency, performance and growth. This is summed up by the Ludvall-Juran quality cost model, which applies the law of diminishing returns to these costs. The model shows that prevention and appraisal costs have a direct relationship with quality conformance, meaning they increase as quality conformance increases. Thus, quality conformance should have an inverse relationship with failure costs – meaning as quality conformance increases failure costs should decrease. Understanding these relationships and applying the cost of quality process enables an organization to decrease failure costs and assure that their products and services continue to meet customer expectations.
4
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
The important reasons to invest in reducing the cost of quality for a legacy system are:
The cost of the application(s) developed on mainframe system is much higher than the cost of the application(s) developed on distributed systems.
The most critical business processes run on the mainframe. The failure costs of the application(s) on the mainframe system have more impact on the organization than the failure costs of the applications(s) on the distributed environment.
The mainframe environment is much complex than the distributed environment.
The various time indices like development time, time for ROI, application life time are much longer for mainframe system.
Mainframe systems, product suites and architectures are entirely different than distributed environment. Moreover, these applications control the critical business flows.
With emerging business needs, mainframe systems are changing continuously – be it SOA, SOI, or Cloud.
The major attributes to legacy system cost of quality are:
Number of unscheduled outages Complex architecture Inefficient planning, release management, processes, etc. Availability of batch window Time to restore the system in case of failure Time to deploy a new application
HCL-MMS Services to Optimize CoQ for Legacy System
HCL‟s Mainframe & Midrange Services (MMS) is 15+ year old practice with partnerships with leading ISVs and vast experience across portfolio of Services. HCL-MMS services provide an end-to-end solution to optimize the factors which are impacting the cost of quality for a legacy system. It not only focuses on the visible parameters impacting the cost of quality, but also the concealed and interlinked parameters, giving a holistic solution.
5
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Figure 3: CoQ Iceberg
Optimizing Prevention Cost for mainframe projects Managing preventive costs are the most effective way to keep the cost of quality in control. The cost of preventing the defect is much less when compared to rework cost. ASSeT™ is HCL‟s proprietary framework for Knowledge Acquisition Plan / transition. The major phases involve due diligence, Initiation, Absorption, Replication, Observation, Operation. It also provides governance all across the phases during transition. This helps in building a robust knowledge repository of the entire legacy system. It also helps create cross-references between various components.
Figure 4: ASSeTTM Framework
The HCL-MMS has a defect management methodology developed for the mainframe projects, which provides end-to-end solutions for the defects/failures/abends. A detailed root cause analysis is done for the
6
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
defects logged in the defect log, from where the action plan to eliminate the defect is arrived at. Role specific training is conducted for the defect prevention coordinator, project manager, team lead, and team members.
Optimizing Appraisal Cost for mainframe projects
The Mainframe Testing Service Offering targets to reduce the cost of quality in the testing phase of the mainframe project. The factors which influence the CoQ of the mainframe project are studied thoroughly and suggestions are given to optimize them. To achieve the above purpose, HCL-MMS has the following proprietary framework of mainframe testing and also providing the in-depth coverage of various testing types. MMS Mainframe testing framework is built based on the industry best practices, proven methodologies, reusable components, and automation tools.
Figure 5: Mainframe Testing Framework
Optimizing Failure Cost for mainframe projects
HCL-MMS service provides an assortment of services to optimize the failure effort. HCL-MMS has the proprietary Analyzer tool which helps in identifying the redundant code, dead code in the mainframe application, thereby optimizing the existing code. This will help in handling the future enhancements efficiently.
HCL-MMS suggested guidelines for optimizing the number of abends/failures of maintenance projects:
7
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Plan the operations (weekly/monthly/quarterly schedules) Define procedures/policies for the operations and maintenance
department Define SLAs Define roles & responsibilities Developing / procuring the required tools to analyze the abends Building the database of problems and their solutions/work
around Well planned change management Buy-in from relevant stakeholders
CoQ Optimization for Application Development projects The development project can be broadly classified into the following phases: Initiation, Development, Testing, and Closure. The execution phase deals with carrying out the entire project life cycle. For each of these phases, certain practices are identified to optimize certain aspects of the CoQ.
Initiation Phase: Following are the practices adopted in the initiation phase to optimize the cost of quality.
Sl.
No
Practice Cost of Quality
1 Effective requirements strategy Preventive Cost
2 Deploying of SMEs for Design, Architecture
reviews
Preventive Cost
3 Identifying and deploying the Third Party /
In-house tools for code verification, test data
generation, test case generation, generation
of SQL scripts, effective defect management,
SCM
Appraisal cost
4 Identifying the deployment strategy to
minimize the production failures
Preventive Cost
5 Identifying the Training requirements and
implementing
Preventive cost
6 Identifying and implementing the
industry/organization Best Practices &
Lessons Learnt
Preventive cost
Table 1: Initiation phase practices
8
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Development Phase: Following are the practices adopted in the development phase to optimize the cost of quality.
Sl.
No
Practice Cost of Quality
1 Identifying the existing
programs/modules/sub-routines , test cases,
test data, JCLs, templates for re-usability in
the beginning of the development saves a lot
of effort, there by reduces the effort in
reviews, testing
Appraisal cost
2 Defining and communicating the Coding
standards / Guide lines in the beginning of
development
Preventive cost
3 Using the tools for Code analysis, reduces
code review effort
Appraisal cost
4 Implementing efficient DP plan, risk
management
Preventive cost
Table 2: Development phase practices
Testing Phase: Following are the practices adopted in the testing phase to optimize the cost of quality.
Sl.
No
Practice Cost of Quality
1 Prioritize testing according to risk (Risk
Based Testing)
Appraisal cost
2 Implementing automation testing tools Appraisal cost
3 Identify repetitive and time consuming
activities in the testing phase and automate
them
Appraisal cost
Table 3: Testing phase practices
9
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Closure Phase: Following are the practices adopted in the closure phase to optimize the cost of quality.
Sl.
No
Practice Cost of Quality
1 Identifying the Best Practices and Lessons
Learnt from the project
Preventive cost
2 Identifying the newly built reusable
components from the project
Preventive cost
Table 4: Closure phase practices
Project Execution Phase: Following are the practices adopted in the execution phase to optimize the cost of quality.
Sl.
No
Practice Cost of Quality
1 Effective risk monitoring Preventive cost
2 Effective defect monitoring Appraisal cost & Failure
cost
Table 5: Execution phase practices
CoQ Optimization for Application Maintenance & Support projects Usually the efficiency of a legacy application during the maintenance and support period is measured based on the number of job abends, total system failures, problem tickets raised etc. Reduction in these numbers show that the system is working efficiently.
Following are the practices adopted in a maintenance & support project to optimize the cost of quality.
10
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Sl.
No
Practice Cost of Quality
1 Building the database of problems / job
abends with reasons and root cause
analysis
Preventive cost
2 Periodical review of the job abends
database and taking/planning for further
actions
Appraisal cost
3 Plan for enhancements, changes of the
system (if required), well in advance
Preventive cost
Table 6: Maintenance & Support practices
Benefits If a legacy system suffers from a high number of job abends, low quality, and require a great deal of rework effort, then it is unfeasible to accomplish objectives such as faster time-to-market and lower operational and maintenance costs. The MMS CoQ approach allows the customer to identify problem areas which result in poor quality; and also implement a plan of action for optimizing the contributors to the cost of quality. The approach looks at the entire CoQ iceberg giving a holistic solution to address all the aspects.
Appendix In-house Tools used to Optimize CoQ The below table shows some of the HCL-MMS built in-house tools which are used to optimize the cost of quality for a legacy system.
Tool Benefit Impact on CoQ
Interface Analyzer
(AnalyzeIT)
Creates Application level block
diagram depicting cross references.
Allows creating Application blocks
encompassing multiple application
elements of different types.
Reduced appraisal cost
11
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Tool Benefit Impact on CoQ
Documentation
Generator
(DocumentIT)
Reports overall Application level and
element level details, metrics,
complexity and cross referencing.
Generates Application Flow,
Program Flow, Job Flow diagrams.
Produces CICS control flows, CRUD
reports for Data elements and data
stores. Produces Code & Data
Redundancy reports, dead code
analysis reports. Produces Data
Model reports, Dataset information
reports. Produces Program logic and
Paraphrased Business Rules.
Reduced appraisal
cost & failure cost
Test Case Generator
(TestIT)
Generates the Test Cases based on
Application Analysis.
Reduced appraisal
cost
Code Standardization
Tool
It is for COBOL programs. Required
coding standards are given as input.
The tool will scan the program and
reports the coding standard
violations.
Reduced appraisal
cost
Data Migration (PortIT) Automated generation of Extract
Jobs, data migration transformation
programs and Load Jobs. Handles
data migration for multiple
databases. Handles migration of data
into normalized databases.
Generates Audit reports and Data
Reconciliation modules to ensure
data validity. Automated Code Page
conversion of Hitachi or Fujitsu code
page to IBM code page.
Reduced appraisal
cost & failure cost
Database Migration
(MigrateIT)
Will be helpful in database migration
like from IDMS to DB2 or IMS to DB2
or ADABAS to DB2. After migration it
will generate the report with the
details of tables migrated and the
control totals to cross check, thereby
reducing the testing effort.
Reduced appraisal
cost
Table 7: HCL-MMS in-house tools
Code Standardization Tool – An Example
HCL-MMS has developed a code standardization tool (CSTOOL) which can assist in verifying the COBOL Coding standards in a Development or Enhancement Project. CSTOOL works on a generic input file called a “Rule File” which defines the coding standards which need to be enforced by the tool.
12
Optimizing Cost of Quality in Legacy - An investment to increase ROI
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
The “Rule File” is easily customizable to cater to customer specific or application specific COBOL coding guidelines. CSTOOL can verify the applications COBOL programs for rules enforced in “Rule File” and report detailed information on Lines which don‟t conform to the enforced guidelines. CSTOOL is panel driven and easy to use, where the input Rule File can be setup in MS Excel and uploaded as a CSV File. A Rule File can be applied to any number of programs in the Application. Also, Specific rule files can be setup for certain programs. Output Report gives non conformance of enforced rule with detailed information like exact line number, exact issue and exact source code snippet.
Figure 6: Code Standardization Tool
Appraisal Vs. Failure Activities The below table shows the majority of time spent in appraisal/failure activities and the effectiveness of detecting defects and cost to diagnose and remove each defect.
Activity Cost of Quality Effectiveness
Structured Personal
Reviews
Appraisal ***********
Formal Software
Inspections
Appraisal **********
Informal Peer Reviews Appraisal *********
Compiling Failure ********
Unit Testing Failure *******
Integration Failure ******
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
13
Optimizing Cost of Quality in Legacy - An investment to increase ROI
Activity Cost of Quality Effectiveness
Beta Testing Failure *****
System Testing (and
performance & other testing)
Failure ****
Acceptance Testing Failure ***
Walkthroughs Appraisal *
Table 8: Appraisal vs. Failure Activities
The 'Cost of Quality' column indicates whether the majority of the time in that activity is spent appraising or dealing with failures. The 'Effectiveness' column indicates both the percentage of defects that are likely to be detected and the total cost in engineer-hours to detect, diagnose, and remove each defect. The placement of Reviews and Inspections in the list is based on best practices. In some cases, their effectiveness is quite low. The order of Compiling, Unit Testing, Integration, Beta Testing, System Testing and Acceptance Testing in the above list really does indicate their relative efficiency in removing defects (not just their lifecycle order). The placement of Unit Testing in the list is based on best practices. Many developers have never been trained in testing, and are ineffective at it. Walkthroughs are more effective for training purposes than for defect removal.
Cost of Quality Model Categories Herb Krasner described the cost of quality model in his research as:
Major Category Sub category Definition Typical Cost
Sub Items
Dealing with non-
conformances
Internal non-
conformances
Quality problems
detected prior to
product
shipment
Pre-release defect
management,
rework, re-reviews
and retesting.
External non-
conformances
Quality problems
detected after
product
shipment
Post release
technical support,
complaint
investigation,
defect notification,
remedial upgrades
and fixes
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
14
Optimizing Cost of Quality in Legacy - An investment to increase ROI
Major Category Sub category Definition Typical Cost
Sub Items
Appraising the
level of quality
Discovering the
condition of the
product
Discovering the
level of non-
conformances
Testing, SQA,
inspections,
reviews
Assuring the
achievement of
quality
Quality control
gating
Product quality
audits, go/no
decisions to
release or proceed
Preventing poor
quality from
occurring
Quality basis
management
Efforts to define
quality, set
quality goals,
standards and
thresholds.
Quality tradeoff
analysis
Defining release
criteria for
acceptance
testing, and
related quality
standards
Project and
process
Oriented
interventions
Efforts to
prevent poor
product quality,
and/or improve
process quality
Training, process
improvements,
metrics collection
and analysis
Table 9: Quality Model Categorization
References http://asq.org/learn-about-quality/cost-of-quality/overview/overview.html http://www.isixsigma.com/ http://thequalityportal.com/ http://www.performance-measurement.net http://www.projectconnections.com/articles/051005-koch.html http://lifelong.engr.utexas.edu/pdf/sqi/cosq-article-new06.pdf www.nist.gov/director/planning/upload/report02-3.pdf Quality Control Handbook, J. Juran and F. Gryna
AUTHORS Shilpa Kerur, ETS Mainframe CoE HCL Technologies, Hyderabad Satya Srinivas Biyyapu, ETS Mainframe CoE HCL Technologies, Hyderabad
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
15
Optimizing Cost of Quality in Legacy - An investment to increase ROI
ABOUT HCL
HCL Technologies
HCL Technologies is a leading global IT services company, working with clients in the areas that impact and redefine the core of their businesses. Since its inception into the global landscape after its IPO in 1999, HCL focuses on „transformational outsourcing‟, underlined by innovation and value creation, and offers an integrated portfolio of services including software led IT solutions, remote infrastructure management, engineering and R&D services and BPO. HCL leverages its extensive global offshore infrastructure and network of offices in 31 countries to provide holistic, multi-service delivery in key industry verticals including Financial Services, Manufacturing, Consumer Services, Public Services and Healthcare. HCL takes pride in its philosophy of „Employees First‟ which empowers our 72,267 transformers to create real value for customers. HCL Technologies, along with its subsidiaries, had consolidated revenues of US$ 3.1 billion (Rs. 14,101 crores), as on 31st December 2010
About HCL Enterprise
HCL is a $5.7 billion leading global technology and IT enterprise comprising two companies listed in India - HCL Technologies and HCL Infosystems. Founded in 1976, HCL is one of India's original IT garage start-ups. A pioneer of modern computing, HCL is a global transformational enterprise today. Its range of offerings includes product engineering, custom & package applications, BPO, IT infrastructure services, IT hardware, systems integration, and distribution of information and communications technology (ICT) products across a wide range of focused industry verticals. The HCL team consists of over 79,000 professionals of diverse nationalities, who operate from 31 countries including over 500 points of presence in India. HCL has partnerships with several leading Global 1000 firms, including leading IT and Technology firms. For more information, please visit www.hcl.com
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
16
Optimizing Cost of Quality in Legacy - An investment to increase ROI