©2008 bae systems. organizational systemic causal analysis & resolution group metrics analysis...

1©2008 BAE Systems.

Organizational Systemic Causal Analysis & Resolution Group

Metrics Analysis Group

“Proof” that Process Improvement Works! (Individual Mileage Will Vary!)

Or How I Learned to Stop Worryingand Love Quantitative Management …

Export Approval: #2090010808




Proof that Process Improvement Works!

4 Scenarios In Attaining High Maturity

The following set of scenarios are based on true stories!

These events actually occurred

within real-life organizations on real-life projects

that employed real-life people!

The measurements shown came from

actual reported data!

Project names have been changed to

protect the “innocent”!





(Setting the Stage – Our History with CMM/CMMI)CMM/CMMI

1st S

W-C

MM

Ass

essm

ent

Level 2

Level 3

Level 4

Level 1

Level 5

SW

-CM

M

1989

1992

1994

SW

-CM

M

1996

1997

1998

2000

2002

SW

-CM

M

SW

-CM

M

SE

-CM

M SE

-CM

M

SW

-CM

M

SE

-CM

M

SW

-CM

M

SE

-CM

M

SW

-CM

M

SE

-CM

M

2003

CM

MI V

1.1

Attempted SW-CMM Level 4 but

only achieved SW-CMM Level 3

2006

CM

MI V

1.2

Process Improvement Achievements





COCOMO II Productivity Factor - Unit Cost w/ Org Target Max

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Oct

-98

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

Un

it C

ost

- (

Per

son

Ho

urs

Per

SL

OC

)

Org Target Max

Below Org Target Max Goal

Near Org Target Max Goal

Slightly Above Org Target Max Goal

Well Above Org Target Max Goal

Org Target Maximum Goal

.300

(Setting the Stage – The Measures)Post Delivery Defect Density w/ Org Target Max

0.0

1.0

2.0

3.0

4.0

5.0

6.0

Oct

-98

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

PD

DD

( D

efe

cts

Per

KS

LO

C)

Org Target Max






.75





Scenario 1) Project XProject X’s initial attempt at implementing the company’s standard engineering process set is abandoned after just

2 baseline releases due to a perceived decrease in its Productivity measure.

However, instrumentation for Product Quality was left in place when project attempted to return to its previous

development processes.

Project X’s subsequent release realizes an extremely poor Product Quality measure and Project X decides it should

re-adopt the company’s standard development processes, ultimately becoming a champion and pilot for

more modern development methodologies.





COCOMO II Productivity Factor - Unit Cost – Project X

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Oct

-98

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

Un

it C

ost

- (

Per

son

Ho

urs

Per

SL

OC

)

Org Target Max






.300

Project X undertakes new process set

Project X abandons new process set due to perceived impacts to development team …

Project X has reasonable

productivity but unknown product

quality





Post Delivery Defect Density – Project X

0.0

1.0

2.0

3.0

4.0

5.0

6.0O

ct-9

8

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

PD

DD

( D

efec

ts P

er K

SL

OC

)

Org Target Max






.75

96.866

Project X undertakes new process set

Project X abandons new process set … Project X realizes

unacceptable level of product quality &

re-institutes new process set

implementation





Scenario 2) Project Y Project Y (a sustaining/maintenance effort) has difficulty

implementing the standard development processes which have been designed primary for the needs of full-

scale development efforts.

Subsequently, 4 consecutive releases for Project Y are measured and 7 of 8 data points (4 of 4 data points on Productivity and 3 of 4 data points on Product Quality)

miss the Organizational Target Goals.

Still, improvement trends in both measures, as well as the evident balancing of Productivity vs. Product Quality

performance, prove that Project Y is operating at High Maturity!





COCOMO II Productivity Factor - Unit Cost – Project Y

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75O

ct-9

8

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

Un

it C

ost

- (

Per

son

Ho

urs

Per

SL

OC

)

Org Target Max






.300

High Maturity Projects May Miss Target Goals But

Continuous Improvement Is Their True Focus! And …





Post Delivery Defect Density – Project Y

0.0

1.0

2.0

3.0

4.0

5.0

6.0O

ct-9

8

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

PD

DD

( D

efec

ts P

er K

SL

OC

)

Org Target Max






.75

… High Maturity Projects Learn to Balance Cost &

Quality Indicators Attempting to Optimize Contract Performance





Scenario 3) Project ZProject Z is a Large Scale Development program with

numerous overlapping development efforts and does experience some difficulty implementing the company’s

standard development processes.

Initially, customer / project management pushback is high due to perceived added cost of the more rigorous development, review, measurement and analysis

processes but the engineering team is persistent in implementing higher maturity practices.

Ultimately, Project Z starts to realize increased Process Capability through better analyses of measurements and

appropriate corrective actions.





COCOMO II Productivity Factor - Unit Cost - Project Z

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75O

ct-9

8

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

Un

it C

ost

- (

Per

son

Ho

urs

Per

SL

OC

)

Org Target Max






.300

As High Maturity Projects Stabilize Performance …





Post Delivery Defect Density - Project Z

0.0

1.0

2.0

3.0

4.0

5.0

6.0O

ct-9

8

Jan

-99

Ap

r-99

Jul-

99

Sep

-99

Dec

-99

Mar

-00

Jun

-00

Sep

-00

Dec

-00

Mar

-01

Jun

-01

Sep

-01

Dec

-01

Mar

-02

Jun

-02

Sep

-02

Dec

-02

Mar

-03

Jun

-03

Sep

-03

Dec

-03

Mar

-04

Jun

-04

Sep

-04

Dec

-04

Mar

-05

May

-05

Au

g-0

5

No

v-05

Feb

-06

PD

DD

( D

efec

ts P

er K

SL

OC

)

Org Target Max






.75

… Continuous Improvement Becomes An

Engrained Behavior !





Scenario 4) The Organization

As development and maintenance efforts like Projects X, Y

and Z adopt a more rigorous and uniform set of

engineering development and project management

processes, the Organization as a whole will also start to

realize increased Process Capability and Process

Stability in its Organizational Performance Measures.

Additionally, a standardized company-wide set of

engineering development and project management

processes reduces the impact of the “Learning Curve”

for engineering and support personnel who transfer

between projects, programs and lines of business.





Mean Effort to Resolve DRs (Pri 1,2,3)

18.97

15.75

18.90

12.86

10.28

7.69 7.897.02

7.819.49

11.7810.33 9.77

11.27

29.57

37.79

0

10

20

30

40

50

2002-Q1

2002-Q2

2002-Q3

2002-Q4

2003-Q1

2003-Q2

2003-Q3

2003-Q4

2004-Q1

2004-Q2

2004-Q3

2004-Q4

2005-Q1

2005-Q2

2005-Q3

2005-Q4

Ave

rag

e E

ffo

rt (

En

gin

eeri

ng

Ho

urs

)

Attained SW-CMM Level 4

Maturity Rating


Maturity Rating

Attained CMMI Level 5 Maturity

Rating

Alarming Trend!

As Organizational Process Capabilities Improve …

70 % Reduction

over 4 Years!





Variance in Effort to Resolve DRs (Pri 1,2,3)

0

2000

4000

6000

8000

10000

12000

14000

16000

2002-Q1

2002-Q2

2002-Q3

2002-Q4

2003-Q1

2003-Q2

2003-Q3

2003-Q4

2004-Q1

2004-Q2

2004-Q3

2004-Q4

2005-Q1

2005-Q2

2005-Q3

2005-Q4

Va

ria

nc

e -

Eff

ort

T

o R

es

olv

e D

Rs


Maturity Rating


Maturity Rating

Attained CMMI Level 5 Maturity

Rating

… And Process Variances Stabilize…

Reduced variance in process and product

measurements enables more meaningful

analyses and allows sustainable process

changes!





Organizational Performance Can Be Optimized!

Priority 1- 3 DR Analysis - Average Effort to Resolve

CY 2001 CY 2002 CY 2003 Jan - Apr 2004

Period of Performance

Av

g L

ab

or

Ho

urs

:

Pri 1 DR Avg Effort Pri 2 DR Avg Effort Pri 3 DR Avg Effort Pri 1-5 DR Avg Effort Power (Pri 1-5 DR Avg Effort)

SW CMM Levels 4 & 5

Achieved

CMMI Level 5 Achieved

New Level of Performance

Requires Rebaselining

Operating at SW CMM Level 3





All Inspections - Monthly Equivalent Critical Code Defect Discovery Counts

0

1,000

2,000

3,000

4,000

Jan

-01

Mar

-01

May

-01

Jul-

01

Sep

-01

No

v-01

Jan

-02

Mar

-02

May

-02

Jul-

02

Sep

-02

No

v-02

Jan

-03

Mar

-03

May

-03

Jul-

03

Sep

-03

No

v-03

Jan

-04

Mar

-04

May

-04

Jul-

04

Sep

-04

No

v-04

Jan

-05

Mar

-05

May

-05

Jul-

05

Sep

-05

No

v-05

Def

ects

Dis

cove

red

Per

Mo

nth

Checklist Inspections

Formal Inspections

Total Inspection Defects Discovered

All Inspections - Monthly Average Defect Discovery Per Inspection

0.0

10.0

20.0

30.0

40.0

50.0

Jan

-01

Mar

-01

May

-01

Jul-

01

Sep

-01

No

v-01

Jan

-02

Mar

-02

May

-02

Jul-

02

Sep

-02

No

v-02

Jan

-03

Mar

-03

May

-03

Jul-

03

Sep

-03

No

v-03

Jan

-04

Mar

-04

May

-04

Jul-

04

Sep

-04

No

v-04

Jan

-05

Mar

-05

May

-05

Jul-

05

Sep

-05

No

v-05

Ave

rag

e D

efec

ts D

isco

vere

d P

er In

spec

tio

n

Checklist Inspection Average

Formal Inspection Average

Total Defects Discovered

SW-CMMLevel 3

SW-CMMLevel 4

SW-CMMLevel 5

CMMILevel 5





Post Delivery Defect Density Reduction Goal (2001-2005)

1.27 0.41 0.32 0.66 0.24 0.86 0.76 0.71 0.90 0.69 0.33 0.41 0.72 0.31 0.73 0.34 0.75 0.32 0.16

1.270

1.050 1.050

0.900

0.000

0.250

0.500

0.750

1.000

1.250

1.500

1.750

Basel

ine

Q2 20

01

Q3 20

01

Q4 20

01

Q1 20

02

Q2 20

02

Q3 20

02

Q4 20

02

Q1 20

03

Q2 20

03

Q3 20

03

Q4 20

03

Q1 20

04

Q2 20

04

Q3 20

04

Q4 20

04

Q1 20

05

Q2 20

05

Q3 20

05

Q4 20

05

Po

st-

De

liv

ery

De

fec

t D

en

sit

y (

De

fec

ts/K

SL

OC

)

Recorded Quarter Defect Density

Max. Post-Delivery Defect Density Reduction Goal

4-Quarter Moving Average

75 % Reduction

over 5 Years!





Improved Defect Detection Enables …

Defect Detection Process Capability Comparison

RequirementsAnalysis

Prelim. Design Detailed Design Code/Unit Test SystemIntegration

FAT Post-Delivery (first year)

Software Development Phases

De

fec

t D

ete

cti

on

De

ns

itie

s

Pre-1995 (w/ Ad Hoc Processes)

1995-1999 (w/ Peer Reviews)

1999-2003 (w/ Fagan Process)

2004-present (w/ Multiple ReviewProcesses)

direction of change





Defect Prevention Process Capability Comparison

RequirementsAnalysis

Prelim. Design Detailed Design Code/Unit Test SystemIntegration

FAT Post-Delivery (first year)

Software Development Phases

De

fec

t D

ete

cti

on

De

ns

itie

s

Initial DP Activities Profile

Progressing DP Activities Profile

Advanced DP Activities Profile

dir

ecti

on

of

chan

ge

… Improved Defect Prevention!





Reliability Prediction Model Calibration

Window 1[Attempt to Cover Diamonds with Boxes]

When the Defect Well Starts to Dry Up …

Projects that have cut their teeth on quantitative management of defect injections/detections must ultimately look for new ways to apply their statistical tool set and quantitative management skills.

Integration & Testing Reliability Prediction is one such advanced model.

Weekly Reliability Threshold 0.0500Calculated Lambda - Method 1 0.7061Calculated Lambda - Method 2 1.4123Calculated Lambda - Method 3 1.0147

1.3500 1350.0060 60

0.0051 Lambda/Theta

Spin Controls

Reliability Prediction Model Parameters

Calibration LambdaCalibration Theta

Sum of Squared Differences

Reliability Prediction Model Calibration

Window 2[Attempt to Minimize Blue Area Under Curve]

y ≈ λe-θt





Key Take-Aways!

1. Starting Down the Process Improvement Path Can Be Both Scary & Risky. Danger Lurks Around Every Bend. Choose a Pathfinder That You Can Trust!

2. Every Perceived Failure Can Also Be Viewed as a Potential Opportunity for Improvement. Think Positively!

3. Limiting Process Variance is the Key to Any Successful, Repeatable, Reliable and Sustainable Process Improvement Implementation. Don’t Just Focus on Mean Values!

4. Institutionalization Requires Time & Energy. Build Up & Maintain Momentum. Starting & Stopping & Restarting Again Is A Very Poor Use of Project Funds!





Questions?





Contact Info

• Kevin Domzalski• BAE Systems – C3I Systems

• 10920 Technology Place (MZ: 606-PI)

• San Diego, CA 92127

• Email: [email protected]





• Kevin Domzalski is a seasoned member of the Process Improvement Group at BAE Systems (C3I Systems) headquartered in San Diego, California, where he currently fulfills the role of Metrics Analysis Group Lead

• Kevin joined the company in 1983 (or at least one of the precedent companies that have now been formed into BAE Systems, Inc.) but took a 5-year hiatus between 1993 and 1998 to work as an automotive industry engineering consultant

• Kevin has developed and teaches many company courses as well as the Systems Engineering Software Overview course at the University of California at San Diego (UCSD) Extension Studies Program as a adjunct faculty member

• In addition, Kevin was awarded the PSM Users Group’s “Most Successful PSM Implementation” award in July of 2005 having been nominated by David Card, co-author of PSM and the ISO/IEC 15939 Standard

About the Presenter





Acronyms

CMM® Capability Maturity Model

CMMIsm Capability Maturity Model Integration

COCOMO Constructive Cost Model

ISO/IEC International Organization for Standardization / International Electrotechnical Commission

MAG Metrics Analysis Group

OSCAR Organizational Systemic Causal Analysis & Resolution

PSM Practical Software (& Systems) Measurement

UCSD University of California San Diego

©2008 bae systems. organizational systemic causal analysis & resolution group metrics analysis...

Documents

process improvement

project y project y

project names

bae systems

development team project

institutes new process

project xs subsequent

ksloc org target max