ieee keynote 2004 09
TRANSCRIPT
-
8/2/2019 IEEE Keynote 2004 09
1/25
"Using Metrics to Drive QualityImprovements and Value --
Company Wide"
Bill WoodworthDirector, IBM Software Test &
Development ExcellenceIBM Corporation
Presented at IEEE ICSM Conference
13 September 2004
-
8/2/2019 IEEE Keynote 2004 09
2/25
-
8/2/2019 IEEE Keynote 2004 09
3/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation3
IBM is a Software Company
In fact, IBM is a HUGE SOFTWARE Company
a) Our products span the entire range: firmware / microcode,operating systems, middleware, systems management,applications, imbedded, etc.
b) Software revenue = $14.3 B in 2003; 1/3 of total company profits
c) World's largest infrastructure (middleware) software provider
d) Second largest software business in the worlde) #1 or 2 in market share in all our software markets (per IT analysts)
f) High double digit thousands of software engineers
g) Thousands of software products
And QUALITY Matters Big Time to Our Customers IBM's systems are generally NOT desktops where you can afford
to crash, burn and re-boot BUT are business-critical systemsthat MUST NEVER crash
Holistic Quality is key to our Success as a Software provider
-
8/2/2019 IEEE Keynote 2004 09
4/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation4
Our Focus on Holistic Quality
Quality is key to IBMs success as a software provider
Total Customer Satisfaction in our products and solutionsContinual improvement = each release better than its predecessor
We apply a quality management system balancing:Quality + Function + Speed to Market + Cost
where each aspect plays an important role
in delivering value to our customers
Function
Quality
Cost
Speed To Market
Software Quality focus must be Holistic:More than reducing defects
Ensuring Correctness
Increasing Availability, Reliability, Integrity, Installability, Testability Improving Problem Determination, Usability, Interoperability,
Portability, Efficiency, ....
Reducing and Preventing Problems
Meeting customer expectations
Disciplined risk management
-
8/2/2019 IEEE Keynote 2004 09
5/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation5
Our Quality System Before 1998
Each unit working to improve Quality of its products
Each release expected to be better than its predecessor
BUT limited company wide sharing on:Best processes and practices Lessons learned
Networking and helping each other
Since then, related complexities began really increasing:Economic need for more effective & efficient operationsQuality expectations of customers in the industryComplexity of technologies and combinations in solutions
New initiatives in OnDemand and Autonomic ComputingDevelopment and Test process pressures from all the above
Need and Opportunity for Software Quality EngineeringImprovements in ALL Units Company Wide SQE Focus
-
8/2/2019 IEEE Keynote 2004 09
6/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation6
Using Metrics to Drive Quality Improvements & Value
Agenda:Background
IBM is a Software Company
Driving & Leading Company Wide SQE Initiatives
Why = ObjectivesWhat = Communities of Testers and of Developers
How = Focuses and Success Factors
Using Metrics to Drive Improvements ValueCritical Role of and Types of Metrics
How Used
More Partnership across the Industry
-
8/2/2019 IEEE Keynote 2004 09
7/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation7
Driving & Leading Company Wide SQE Initiatives
Each Community has Same Basic Vision & Purpose:Vision = Teaming to Become BEST OF BREED
Purpose = Delivering VALUE to Each Business Unit
In 2003: Added Development Process Focus
Created a company wideS/W Developers Community
Modeled on success & operations of ITCL
In 1998: Created Testing Process Focus
Created a company wideIBM Testers Community
Spans all product labs, global services, research
-
8/2/2019 IEEE Keynote 2004 09
8/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation8
Company Wide SQE Initiatives Focus
What we did & are doing = Key Focus Areas People and Community
Processes, Practices and Tools
Metrics and Driving for Improvements
How we were successful = Keys to Success
Recognized need to improve Recognized need to help each other
Unit Leaders = active and empowered
Value demonstrated to all players
Walk before you run mentality
-
8/2/2019 IEEE Keynote 2004 09
9/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation9
Highly Successful SQE CommunitiesDrive and Provide the Means for:
CommunityCommunity
Developing Resourcesneeds basedworkgroups
best technology/source
Driving Improvements
self assessmentusing best resources
remembering lessons learned
Developing Peoplecareer & skill focustraining & recognition
DeliveringValue
Sharingnurturing communications
website, T.R.s, DBsnewsletters; meetings
Networkingin-person meetingsbuilding relationships
contactsLeading & Decidingimpassioned unit leaders
change agents
OperationalInfrastructure
-
8/2/2019 IEEE Keynote 2004 09
10/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation10
Rational UnifiedProcess
IBM RationalProducts
UNIT
Current positionIssues & needs
UNIT
MeasurablyImproved
ITCL & SDCL
Resources
Test & DevTest & Dev
SelfSelf
Assess'tAssess't
Units Becoming Best of Breed
2 Keys within Each Unit Enabling Unit Self-Improvement1. ITCL & SDCL Unit Leaders Aggressive Change Agents
2. Meaningful, disciplined and supported application of the IBMTest and Development Self Assessmentsto affect
the most needed improvements
-
8/2/2019 IEEE Keynote 2004 09
11/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation11
TESTING: From Ideas to Implementation.
Team Building & Networking
DevelopingITCL Resources
DevelopingIBM Test Solutions..
DeployingITCL Resources
DeployingIBM Test Solutions
Driving TestImprovements.
Shifting toRational
_1998_ _1999_ _2000_ _2001_ _2002_ _2003_ _2004_
Yielded 100s of Improvement Actionsand $10,000,000s of Benefits Each Year
-
8/2/2019 IEEE Keynote 2004 09
12/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation12
Using Metrics to Drive Quality Improvements & Value
Agenda:Background
IBM is a Software Company
Driving & Leading Company Wide SQE Initiatives
Why = ObjectivesWhat = Communities of Testers and of Developers
How = Focuses and Success Factors
Using Metrics to Drive Improvements ValueCritical Role of and Types of Metrics
How Used
More Partnership across the Industry
-
8/2/2019 IEEE Keynote 2004 09
13/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation13
PROCESS METRICS
Critical Role in Improving S/W Development
Quality Process Focus & Cost of Quality
Type of Key MetricsDelivered Quality Metrics
In Process Metrics
How Used..Business Decision Making Process
Driving Improvements
Causal Analysis Process Self Assessments
Scorecards / Dashboards
Used to Demonstrate Progress and Value
Gaining Executive Buy In
-
8/2/2019 IEEE Keynote 2004 09
14/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation14
Delivered Quality Metrics
Number of problems whichcustomers report
Basic comparisons whichdemonstrates a given releaseis better than its predecessor
Trend of problems requiringintervention outside thenormal support process
Cost of handling customerreported problems/defects,sometimes normalized byrevenue or dev expense
Number of problems whichcustomers report which
result in code fixes
Number of system failures,usually normalized by usagemonths
Reported survey results
Timing = monthly, quarterly, annuallyOptional grouping by product maturity
Customer Satisfaction
Customer Reported Problems
Customer Reported Defects
Release-to-Release
Crit Sit Trend
Availability and OutagesWarranty / Service Costs
-
8/2/2019 IEEE Keynote 2004 09
15/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation15
In Process Metrics
Process to aid test teams inevaluating their units
strengths / weaknesses
improvement actionsProcess to aid developmentteams in evaluating theirunits strengths / weaknesses
improvement actionsAnalyzing defects / problems =
lessons learned & remembered
Means to understand how
much function or codehas been tested
Means to understand progress vs plan schedule (attempts & completions)
problems / defects (valid & total) outage durations defect arrival rates and density test effectiveness
Design / Code Complete
Functional Test
System Test
Coverage
Causal Analysis
Development Self-Assessment
Test Self-Assessment
Means to understand progress vs plan
schedule, size, functionality design / code peer reviews performed unit test completed
-
8/2/2019 IEEE Keynote 2004 09
16/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation16
PROCESS METRICS
Critical Role in Improving S/W Development
Quality Process Focus & Cost of Quality
Type of Key MetricsDelivered Quality Metrics
In Process Metrics
How Used..Business Decision Making Process
Driving Improvements
Causal Analysis Process Self Assessments
Scorecards / Dashboards
Used to Demonstrate Progress and Value
Gaining Executive Buy In
-
8/2/2019 IEEE Keynote 2004 09
17/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation17
Integrated Product Development (IPD) A management system designed to optimize the development and
the delivery of successful quality products and offerings.
Fact BasedData to ManageYour Business
Project Management Data
Segment Performance Data
Integrated Portfolio Management Team(IPMT)
Market Planning
Information
CustomerFeedback
CompetitorInformation
TechnologyTrends
CurrentProductPortfolio
Understandthe
Marketplace
PerformMarket
Segmentat'n
PerformPortfolioAnalysis
DevelopMarket
Strategy& Plan
Align &OptimizeBusiness
Plans
Manage Market Segment andAssess Performance
StructuredProduct
Development
Product ObjectivesBusiness Strategy
Customer Buying Behavior
R&D Roadmap
Candidate Projects
SatisfiedCustomers
$$$$
Life
CycleLaunchQualifyDevelPlanConcept
Project Development Teams
Resource Allocation
Project Management Data
Performance Data
-
8/2/2019 IEEE Keynote 2004 09
18/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation18
Integrated Product Development (IPD)
Concept
DCP
Plan
DCP
Availability
DCP
EOL
DCP
Strategy OfferingInvestment Execution
ConceptConcept PlanPlan DevelopDevelop QualifyQualify LaunchLaunch LifecycleLifecycle
IPMTand PDTContract
Event Driven Formal 'Decision Check Points' (DCPs)
B S
-
8/2/2019 IEEE Keynote 2004 09
19/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation19
Closed Loop Development Process
Key = Learning & Remembering From MistakesDisciplined & focused causal analysis .Orthogonal Defect Classification (ODC) .
Improving
Emphasis PlanningDesign /DevelopmentExecution
Reducingescapes
TEST
Learning/Changing
Experiences fewerproblems / defectsImproved solutionavailability
Learning/
Changing
CUSTOMERS
Learning a Must
Customer
FocusReducingescapes
Process
Ease of UseAvailabilityReliability
Development
IBM S/W T & D l E ll
-
8/2/2019 IEEE Keynote 2004 09
20/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation20
Used to Demonstrate Value & Progress
Metrics used to understand an aspect of the business
Not metrics for metrics sakeMeans to understand: WHERE are we?
Are we making PROGRESS?
Improvements in Process / PracticesIts all about trends
Progress in what.Customer satisfaction
Customer support
Reducing customer found defects and problems
Benefits of process / practice improvements made.
..
What is managements business / operational focus?Align metrics to develop applicable understanding
IBM S/W T t & D l t E ll
-
8/2/2019 IEEE Keynote 2004 09
21/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation21
Gaining Executive Buy-In to Drive Improvements
3 APPROACHES(in addition to base line management responsibility)1. Tops Down =
Corporate Guidance on Quality of Offerings
2. Peer Pressure ++ =Semi-annual Improving S/W Quality Executive Reviews
3. Partnering =Teaming Line Executives with Executives of
Software Communities
Function
Quality
Cost
Speed
IBM S/W T t & D l t E ll
-
8/2/2019 IEEE Keynote 2004 09
22/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation22
Teaming Line Executives with Community Executives
Line execs responsible for quality of their offerings
Holistic qualityplus operational effectiveness and efficiencies Must inspect what they expect
Active support for Process Self Assessments Set Expectations == Improving is not only OK, but is expected
Communities provide resources (single sourced & anticipatorily) Unit leaders have opportunity to demonstrate leadership
(i.e. look good) and drive improvements from business wise
SQE Community Execs with company wide leadership team Create IBM wide communication fabric
Develop needed resources onceSO ALL UNITS, who need them, can use them
TEAMING TO BECOME BEST OF BREED
IBM S/W Test & De elopment E cellence
-
8/2/2019 IEEE Keynote 2004 09
23/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation23
Using Metrics to Drive Quality Improvements & Value
Agenda:Background
IBM is a Software Company
Driving & Leading Company Wide SQE Initiatives
Why = ObjectivesWhat = Communities of Testers and of Developers
How = Focuses and Success Factors
Using Metrics to Drive Improvements ValueCritical Role of and Types of Metrics
How Used
More Partnership across the Industry
IBM S/W Test & Development Excellence
-
8/2/2019 IEEE Keynote 2004 09
24/25
IBM S/W Test & Development Excellence
Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation24
More Partnership Across the Industry
In spite of industry wide efforts in using metrics to
drive quality, difficult problems still exist (e.g.):
More meaningful and easily collected customer impact metrics
Tools to more easily process multi-language legacy products
Automated test technologies (auto generation, etc.)
Approaches on how to turn academically sound metricsinto successful real world usage
The whole industry should work together to solve
these problems maybe you can help!
-
8/2/2019 IEEE Keynote 2004 09
25/25