1
Predictive Models to Achieve Business Results
Place your image on top of this gray box.
If no graphic i applicable, delete gray box and
notch-out behind gray box, from
the Title Master 19th International Forum on COCOMO and Software Cost Modeling
Cvetan Redzic, Michael Crowley, Nancy Eickelmann, Jongmoon Baik
Motorola, Inc.October 26, 2004
2
Outline
• Overview • Business Goals• Models Used
– COQUALMO
– CoQ-DES
– MotoROI
• Primary Model Inputs– CMM
– Life Cycle Scope
– PCE / PSE
• Results– Cost
– Quality
3
Business Goal – Improved Customer Satisfaction
Must Be
DelightersAttractive
Satisfier Features
Quality
1
2
3
+
-
SW Quality
Type of needs1. Basic Expectations (Must Be)2. Satisfier - Features3. Delighters (Attractive)
Kano Analysis
4
Cause & Effect Diagram
Improved Customer
Satisfaction
5
Integrating Predictive Models
• Models Used– COQUALMO– CoQ-DES– MotoROI
6
COQUALMO
7
Combined COQUALMO Injection Factors
8
CoQ-DES
9
CoQ-DES Simulation
10
MotoROI
Input Data
Validate Inputs
Data Valid?
ExitNo
Analyze Consequences of Software Failure
Yes
Analyze Likelihood of
Software Failure
Risk Analysis & Classification
Potential Maximum Return & ROI
Expected Return & ROI
Output Report
11
MotoROI - DOORS ROI Analysis
Report 1 Report 2 Report 3 Report 4
CMM Level 5 3 3 3Total SLOC 9,780,700 9,780,700 9,780,700 9,780,700Scope of SLOC 9,780,700 9,780,700 9,780,700 9,780,700Software Budget $37,209,669 $37,209,669 $37,209,669 $37,209,669 Investment Budget $400,000 $400,000 $400,000 $400,000 Investment Effectiveness 50% 100% 75% 50%Allocated investment $400,000 $400,000 $400,000 $400,000 Scope of Investment Requirements Requirements Requirements RequirementsPotential Maximum Return $5,953,547 $13,023,384 $13,023,384 $13,023,384 Potential Maximum Return/Scope $2,232,580 $4,837,257 $4,837,257 $4,837,257 Potential Maximum ROI 1:1.6 1:3.5 1:3.5 1:3.5Expected Return $120,000 $520,000 $390,000 $260,000 Expected ROI 1:0.3 1:1.3 1:0.97 1:0.7% COPQ Savings 2% 4% 3% 2%
DOORS – ROI Analysis
12
Model Integration - Primary Model Inputs
• CMM• Life Cycle• PCE / PSE
13
CMM – Process Maturity
0
10
20
30
40
50
60
Co
st a
s a
Pe
rce
nt o
f D
eve
lop
me
nt
1 2 3 4 5 SEI CMM Level
Prevention Appraisal Int Failure Ext Failure TCoSQ
Knox Theoretical Model of TCOQ
(About 50% at CMM Level 3)
• COQUALMO– PMAT (process
maturity has the greatest +/-impact) on injection rates
• CoQ-DES– Not Used directly but
is inherent in organizational calibration
• MotoROI– Process maturity as
represented by the cost of quality/cost of poor quality financial structure is a primary factor.
14
Life Cycle
Requirement
Design
Code
Implementation Unit Test
Component/Integration Test
System Test
Inspections Testing
• COQUALMO– Req., Des., Imp., and Code
• CoQ-DES– Full Life Cycle
• MotoROI– Full Life Cycle or Individual Phases
15
PCE and PSE
• COQUALMO– PCE and PSE as
evidenced by injection and removal rates
• CoQ-DES– PCE and PSE as
evidenced by injection and removal rates
• MotoROI– PCE for DP or PSE
for technology effectiveness
Phase Containment Effectiveness & Phase Screening Effectiveness
16
Measuring and Monitoring Results
• Quality• Cost
17
Defect Removal in REQ, DESIGN & CODE
REQ DESIGN CODE
COQUALMO Average Edge
Defect Injection in REQ, DESIGN & CODE
REQ DESIGN CODE
COQUALMO Average Edge
Quality - Sources of Variation
For Release with about 100 Delta
KLOC, no significant difference estimates & actuals in DI & DR
For large size Release over 100 Delta KLOC, there is significant difference b/w estimates & actuals in DI & DR for Code
REQ DES CODE
Calculated Chi-Square Value 0.19 0.57 11.97
Chi-Square (2;0.05) 5.99
Significance No No Yes
Actual vs. COQUALMO Estimate
18
Quality - Sigma Level
Sigma Level:Defects per Million
OpportunitiesDPMO = 1M * D/(N*O)D = 2464 HS Faults (from PCE)N = 139,595 Delta LOCDPMO = 1M * 2464/139,595
DPMO = 17651 - 3.61 s
Stable processes
Need Leap improvement: SEI CMM Level 5 TCM
From PCE, SRE & CRUD data
Sigma Level by Release - As Is
GSR4 GSR4.1 GSR5.X GSR6 H2 GSR7 EDGE GSR8
Sig
ma
Leve
l
SL HS Faults SL SRE PRs SL CRUD
What is Sigma Level from release perspective ?
Relatively stable across the releases
19
Quality - SRE Goal Setting
20
Quality: As-Is Process
Defect Injection per KLOC - As Is
REQ IDS HLD LLD CODE
Average LB UB
Cum. Defect Injection by Phase
REQ IDS HLD LLD CODE
Cum. Average LB UB
Defect Removal per KOLC by Phase - As Is
REQ IDS HLD LLD CODE PT FT S T P-R CRUD
Average LB UB
Cum. Defect Removal per KOLC by Phase - As Is
REQ IDS HLD LLD CODE PT FT S T P-R CRUD
Cum. Average LB UB
21
Quality - Rayleigh Model Analysis
22
Quality - Impact of Tactical Changes
Frequency Chart
Certainty is 95.00% from 23 to 26
.000
.007
.014
.021
.028
0
7
14
21
28
22 23 25 26 27
1,000 Trials 998 Displayed
Forecast: CRUDoutcome
Monte-Carlo simulation,to include uncertainty &risks In the expert based opinion
23
Quality - New Process Baseline
Cum. Defect Injection per KLOC by Phase - Should Be
REQ IDS HLD LLD CODE
Cum. Average LB UB
Defect Injection per KLOC by Phase - Should Be
REQ IDS HLD LLD CODE
Average LB UB
Defect Removal per KOLC by Phase - Should Be
REQ IDS HLD LLD CODE PT FT S T P-R CRUD
Average UB LB
Cum. Defect Removal per KOLC by Phase - Should Be
REQ IDS HLD LLD CODE PT FT S T P-R CRUD
Cum. Average LB UB
24
Cost - Vital X Monthly Review Charts
GSR8 Program Review - GSR8
FTE Staff
1 3 5 7 9 11 13 15 17 19 21Jul'03
Sep Nov Jan'04
Mar May Jul Sep Nov Jan'05
Mar May
0
20
40
60
80
100
120
140
ppl
23456868
Cum Eff SLOC
1 3 5 7 9 11 13 15 17 19 21Jul'03
Sep Nov Jan'04
Mar May Jul Sep Nov Jan'05
Mar May
0
50
100
150
200
SLO
C (th
ousands)
23456868
Cum All Defects
1 3 5 7 9 11 13 15 17 19 21Jul'03
Sep Nov Jan'04
Mar May Jul Sep Nov Jan'05
Mar May
0
200
400
600
800
1000
1200
1400
CA
D
234568All Defects
1 3 5 7 9 11 13 15 17 19 21Jul'03
Sep Nov Jan'04
Mar May Jul Sep Nov Jan'05
Mar May
0
50
100
150
200
250
AD
234568
Current Plan Actuals Current Forecast Green Control Bound Yellow Control Bound Project: GSR8
SLIM
25
Quality - Vital X Monthly Review ChartsFault Injection & Removal vs. Baselines
Cum. Defect Injection by Phase
REQ IDS HLD LLD CODE
Cum. Average LB UB
Defect Removal per KOLC by Phase
REQ IDS HLD LLD CODE PT FT S T P-R CRUD
Average GSR8 UB LB
Cum. Defect Removal per KOLC by Phase
REQ IDS HLD LLD CODE PT FT S T P-R CRUD
Cum. Average LB Cum. GSR8 UB
Defect Injection per KLOC by Phase - Should Be
REQ IDS HLD LLD CODE
Average LB UB H2
26
Quality - Vital X Monthly Review Charts
SRE
27
CRUD Goal Tracking
CRUD - Projected vs. Actual
Jan-04 Feb-04
Mar-04
Apr-04 May-04
Jun-04 Jul-04 Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Cum. CRUD Goal
28
Summary
• Integrating predictive models provides multiple views of project quality, cost and schedule issues.
• More accurate predictions of defect injection are possible
• More accurate predictions of defect removal are possible
• More accurate predictions of overall staffing and project cost are possible
