supporting the cmmi metrics framework thru level 5

03-23-05September 2009

Supporting the CMMI Metrics Framework thru Level 5Supporting the CMMI Metrics Framework thru Level 5

EDSEDS--Electronic Data Systems do Electronic Data Systems do BrasilBrasil LtdaLtda. .

MMáárciorcio SilveiraSilveira

September 2009Supporting the CMMI Metrics Framework thru Level 5

AgendaAgendaAgenda

• Objective

• EDS Overall Process Improvement Strategy

• Measurement Elements of the CMMI Model

• M&A and High-Maturity Practices

• Processes and Procedures for M&A and High-Maturity

• Lessons Learned

• Q&A


ObjectiveObjective

This presentation shows the Software Engineering Framework that EDS utilizes at its Applications Development and Maintenance Centers in Brazil that supports the measurement requirements of the CMMI® model thru level 5. During the presentation you will learn the basics and advanced measurement requirements from the CMMI® model and see how EDS implemented the processes and procedures to support these requirements.

® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


EDS Overall Process Improvement StrategyEDS Overall Process Improvement StrategyEDS Overall Process Improvement Strategy

StrategicBusiness

Plan(SBP)

Corporate ObjectivesBusiness Drivers

Technology DriversPI Opportunities

Voice of CustomerVoice of Management

Strategies/InitiativesMeasurement (Monitor & Control)

Inputs/Drivers

OrganizationImprovement

Strategy

Governance Council(Board)

EDGE - Enabling Delivery and Global Excellence


Edge (Work-types, Processes, Activities)EdgeEdge ((WorkWork--typestypes, Processes, , Processes, ActivitiesActivities))


Tailoring the Project ...TailoringTailoring thethe Project ...Project ...

TailoringTool

Requirementsand Approach

Documentation

Project’sDefined Process

File

Starting PointSchedule

OSSP-Org. PlansTailoring Criteria

Project’sDefined Process

File (PDP)

Tailoring Execution

Waivers

Tailoring Decisions

AmendmentHistory

Project WorkbookStructure


M&A e High-Maturity (CMMI-Dev 1.2)M&A e HighM&A e High--Maturity (CMMIMaturity (CMMI--Dev 1.2)Dev 1.2)

PERFORMED (None)Ad Hoc1

• Requirements Development• Technical Solution• Product Integration• Verification• Validation• Integrated Project Management• Risk Mgmt.• Decision Analysis and Resolution• Organizational Process Focus• Organizational Process Definition• Organizational Training

DEFINEDStandard and Consistent

Process3

MANAGED

• Requirements Mgmt.• Project Planning• Project Monitoring and Control• Measurement and Analysis• Process and Product Quality Assurance• Configuration Mgmt.• Supplier Agreement Mgmt.

2 Disciplined Process

LEVEL FOCUS NAME PROCESS AREAS

QUANTIT.MANAGED

• Organizational Process Performance• Quantitative Project ManagementPredictable Process4

• Organizational Innovation & Deployment• Causal Analysis and Resolution

Continuously ImprovingProcess

5 OPTIMIZING


M&A ObjectivesM&A ObjectivesM&A Objectives

• The purpose of Measurement and Analysis is to develop and sustain a measurement capability that is used to support management information needs by:

Specifying the objectives of measurement and analysis such that they are aligned with identified information needs and objectives

Specifying the measures, analysis techniques, and mechanism for data collection, data storage, reporting, and feedback

Implementing the collection, storage, analysis, and reporting ofthe data

Providing objective results that can be used in making informed decisions, and taking appropriate corrective actions


OPP/QPMOPP/QPM

Supporting M&A Goal 1 Supporting M&A Goal 1 Supporting M&A Goal 1

SP 1.2 Specify Measures to Address Measurement Object.SP 1.3 Specify data collection and storage proceduresSP 1.4 Specify analysis procedures

SP 1.2 Specify Measures to Address Measurement Object.SP 1.3 Specify data collection and storage proceduresSP 1.4 Specify analysis procedures

SP 1.1 Establish Measurement ObjectivesSP 1.1 Establish Measurement Objectives

• SPs Goal 1: Align Measurement and Analysis Activities

Performance and Quality

Goals and Objectives



OrganizationalMeasurement

Plans

OrganizationalMeasurement

Plans

Strategic

Business Plan

Strategic

Business Plan

Service Level

Agreement

Service Level

Agreement

ProjectMeasurement

Plans

ProjectMeasurement

Plans

Organization

Project

OrganizationalQuantitative Management

Approach


Approach ProjectQuantitative

Management Plan

ProjectQuantitative

Management Plan


• SPs Goal 2: Provide Measurement Results

SP 2.1 Collect Measurement DataSP 2.3 Store Data and Results

SP 2.1 Collect Measurement DataSP 2.3 Store Data and Results

Project

Source

of Data

Project

Source

of Data

Project

Measurement

Plans

Project

Measurement

Plans

Presage

(Enterprise

Metrics

Repository)

Presage

(Enterprise

Metrics

Repository)

Project Tracking Reports (Local

Repository)


Repository)

SP 2.2 Analyze Measurement data SP 2.3 Store Data and Results SP 2.4 Communicate Results

SP 2.2 Analyze Measurement data SP 2.3 Store Data and Results SP 2.4 Communicate Results





Project Communication

Plan

Project Communication

Plan

PTR Consolidation

Process

OrganizationalMetrics

Analysis Report

OrganizationalMetrics

Analysis Report

Project MetricsAnalysis Report


Project

Organizational Metrics

Repository


Repository

Organization

Supporting M&A Goal 2 Supporting M&A Goal 2 Supporting M&A Goal 2


OPP & QPM ObjectivesOPP & QPM ObjectivesOPP & QPM Objectives• The purpose of Organizational

Process Performance (OPP) is to establish and maintain a quantitative understanding of the performance of the organization’s set of standard processes in support of quality and process-performance objectives by:

Selecting processes to be managed quantitatively

Establishing process performance measurements and quality and process performance objectives

Creating and maintaining Process performance baselines and models

• The purpose of Quantitative Project Management (QPM) is to quantitatively manage the project’s defined process to achieve the project’s established quality and process-performance objectives by:

Establishing and maintaining the project’s quality and process-performance objectives

Identifying suitable sub-processes that compose the project’s defined process based on historical stability and capability data found in process-performance baselines or models

Selecting the sub-processes of the project’s defined process to be statistically managed

Monitoring the project to determine whether the project’s objectives for quality and process performance are being satisfied, and identifying appropriate corrective action

Selecting the measures and analytic techniques to be used in statistically managing the selected sub-processes

Establishing and maintaining an understanding of the variation of the selected sub-processes using the selected measures and analytic techniques

Monitoring the performance of the selected sub-processes to determine whether they are capable of satisfying their quality and process-performance objectives, and identifying corrective action

Recording statistical and quality management data in the organization’s measurement repository


• SPs Goal 1: Establish Performance Baselines and Models

Supporting OPP Goal 1 Supporting OPP Goal 1 Supporting OPP Goal 1

SP 1.4 Establish Process-PerformanceBaselines (PPBs)

SP 1.4 Establish Process-PerformanceBaselines (PPBs)

SP 1.5 Establish Process-Performance Models (PPMs)

SP 1.5 Establish Process-Performance Models (PPMs)

SP 1.1 Select ProcessesSP 1.2 Establish Process-Performance MeasuresSP 1.3 Establish Quality and Process-Performance Object.

SP 1.1 Select ProcessesSP 1.2 Establish Process-Performance MeasuresSP 1.3 Establish Quality and Process-Performance Object.

M&A - SP 1.1 Establish Measurement ObjectivesM&A - SP 1.1 Establish Measurement Objectives




Goals and ObjectivesStrategic

Business Plan

Strategic

Business Plan

Service Level

Agreement

Service Level

Agreement

PPMsPPMsStatistical & Simulation Tools

Statistical & Simulation Tools

Organization

Project


Approach


Approach

ProjectQuantitative

Management Plan

ProjectQuantitative

Management Plan


Repository


RepositoryPPBsPPBs


• SPs Goal 1: Quantitatively Manage the Project

Supporting QPM Goal 1 Supporting QPM Goal 1 Supporting QPM Goal 1

SP 1.4 Manage Project Performance (more in Goal 2)SP 1.4 Manage Project Performance (more in Goal 2)SP 1.1 Establish the Project’s ObjectivesSP 1.2 Compose the Defined ProcessSP 1.3 Select the Sub-processes that w/be Statis. Managed

SP 1.1 Establish the Project’s ObjectivesSP 1.2 Compose the Defined ProcessSP 1.3 Select the Sub-processes that w/be Statis. Managed

Organization

Project

ProjectQuantitative

Management Plan

ProjectQuantitative

Management Plan

Project Defined Process(PDP)

Project Defined Process(PDP)


Repository)


Repository)



ProjectQuantitative

Management Plan

ProjectQuantitative

Management PlanPerformance and

QualityGoals and Objectives




Approach


Approach


• SPs Goal 2: Statistically Manage Sub-process Performance

Supporting QPM Goal 2 Supporting QPM Goal 2 Supporting QPM Goal 2

SP 2.2 Apply Statistical Methods to Understand VariationSP 2.3 Monitor Performance of the Selected Sub-processesSP 2.4 Record Statistical Management Data

SP 2.2 Apply Statistical Methods to Understand VariationSP 2.3 Monitor Performance of the Selected Sub-processesSP 2.4 Record Statistical Management Data

SP 2.1 Select Measures and Analytic TechniquesSP 2.1 Select Measures and Analytic Techniques

Organization

Project


Repository)


Repository)Project MetricsAnalysis Report


Project Storyboard

Project Storyboard

ProjectQuantitative

Management Plan

ProjectQuantitative

Management Plan Statistical & Simulation Tools

Statistical & Simulation Tools PPBs & PPMsPPBs & PPMs


Strategic Business Planning – Key elementStrategic Business Planning Strategic Business Planning –– Key elementKey element

Performance and Quality - Goals

and Objectives

Business Objectives

Market Drivers ,

Improvement Opportunities

Customer’s expectations

+

+

=

Strategic Planning


Performance and Quality Goals and ObjectivesPerformance and Quality Goals and ObjectivesPerformance and Quality Goals and Objectives

• Defines the process-performance and quality goals and objectives from the organization. It is a key output of the SBP activities. It defines:

√ Suggested measurements and objectives at organizational level and project level.

√ High-level measurement plan

√ Reporting mechanism


Organizational Measurement Plans (Pattern)Organizational Measurement Plans (Pattern)

• Defines the process to collect, store, analyze and report the measurements collected by the organization and projects. Basically contains:

√ Linkage between the measurements and the performance and quality goals objectives

√ Detailed information about the metrics that will be collected, stored and analyzed.

√ Analysis and reporting procedures

√ Triggers

√ Roles & responsibilities

√ Organizational Metrics Repository information


Project Measurement PlansProject Measurement Plans

• It is an instantiation of the “Organizational Measurement Plans (Pattern)”.

• Projects reuse the pattern and :

√ Justify what doesn´t apply to the project.

√ Add project specific needs


Organizational Quantitative Management ApproachOrganizational Quantitative Management Approach

• It addresses establishment of process-performance and quality objectives for the organization. These objectives will be used by projects to establish project-level process-performance and quality objectives, and determine the measurements that will be used to support these objectives. Basically contains :

√ Quality goals and process-performance objectives

Common Organizational Objectives

Common Project Objectives (Project Type)

√ Organizational Baselines locations (PPBs)

√ Organizational Performance Models (PPMs) Performance

√ Quantitative Management General Procedures

Organizational Procedures (PPBs, PPMs, Tools, etc.)

Project Procedures


Project Quantitative Management PlanProject Quantitative Management Plan

• It is an instantiation of the “Organizational Quantitative Management Approach”.

• Projects follow the organizational approach by :

√ Selecting the common objectives that are aplicable to the project.

√ Identifying project specifics objectives

√ Identifying processes and sub-processes affected by the objectives.

√ Creating a plan to collect, store, analyze and report on quantitative management data. (DMAIC)

√ Identifying Baselines and Models that will be created and/or reused.

√ Documenting all specific procedures that must be followed.


SAP/MSPS/PresageSAP/MSPS/SAP/MSPS/PresagePresage

SAP

MSPS

Tasks Assignments

Effort

Consolidation&

Upload

Presage


Project Tracking Reports (PTR) – Local RepositoryProject Tracking Reports (PTR) Project Tracking Reports (PTR) –– Local RepositoryLocal Repository


Project Tracking Reports (PTR) – Analysis ProceduresProject Tracking Reports (PTR) Project Tracking Reports (PTR) –– Analysis ProceduresAnalysis Procedures


Effort DataChange

Request data Size data …

Consolidation Process

Project Tracking Reports - Project 1

Project Tracking Reports – Project 2

.

Project Tracking Reports – Project N

Project Tracking Reports (PTR) – ConsolidationProject Tracking Reports (PTR) Project Tracking Reports (PTR) –– ConsolidationConsolidation


Repository


Repository


Organizational Metrics Analysis Report Organizational Metrics Analysis Report Organizational Metrics Analysis Report

Avg Review Effort by WPR Completed (hrs)

2.51

4.17

3.182.87

1.250.84

2.47

0.000.501.001.502.002.503.003.504.004.50

Jan

Feb

Mar

Apr

May

Jun

Avg Review Effort per Work Product Review CompletedOrg Avg Review Effort per Work Product Review CompletedLinear (Avg Review Effort per Work Product Review Completed)

Client Satisfaction Survey 2006 Summary

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Within schedule Meetrequirements

Quality Prompt, clearand effective

communications

Opencommunication

Englishproficiency

Changes effectivelymanaged.

Issues/Riskseffectivelymanaged.

Overall projectmanagement.

Appropriate skills Adequate level ofknowledge

Target

n=34

• Some examples :

Productivity:

Hours per FP per platform

KLOC per resource per month per platform

Quality Assurance:

Average number of non-conformances per audit

Average days for non-conformances closing

Defects:

Defect density (#defects / size * 100)

Defect containment (#defeitos detected in peer reviews/adosem revisões/#total defects) by deliverable/phase.

Estimates:

Effort variation by project/Service Request by platform/language

Duration variation by project/Service Request by platform/language

OPP Baselines and Models

Customer Satisfaction Index


Statistical & Simulation Tools – Minitab/Crystal BallStatistical & Simulation Tools Statistical & Simulation Tools –– MinitabMinitab//CrystalCrystal BallBall

7 1 16 4 056 949 842 73 562 8 52 1 41 4 37 21

6 0

4 0

2 0

0

O b s e rv a tio n

Indi

vidu

al V

alue

_X = 2 .1 4

U C L = 7 .6 7

L B = 0

7 1 16 4 056 949 842 73 562 8 52 1 41 4 37 21

6 0

4 0

2 0

0

O b s e rv a tio n

Mov

ing

Ran

ge

_ _M R = 2 .0 8U C L = 6 .7 9

L C L = 0

111

111111111

11

1

1

11111

111

1

11

11

1

1

1

1

1

11

11

11111

1

11111111111111

11

11

111111

1111

11

11

1111

11

11

1

11

11

1

11

1111

T ic k e ts f r o m 2 0 0 7I -M R C h a r t o f E f f o r t


Process Performance Baselines (PPBs) Process Performance Baselines (Process Performance Baselines (PPBsPPBs) )

Name Description Main Characteristics

Design

Defect Density

It shows the expected range of Defect Density (number of Defects by Pages of Documentation * 100) for the External Reviews.

- Design Phase- Health Care- Design Work Products- Minor Enhancement

Testing Productivity

It shows the range of expected Appl.CRproductivity when a Test Case is created.

- Produce Phase- Health Care, Transportation- Test Case- Minor Enhancement

Effort Variation

It shows the range of expected effort variation that would normally be achieved by a Application CR following the minor modification process with similar characteristics.

- Main Phases (Design, Produce and Testing).

- Health Care, Transportation

- Primary Language (Cobol, Cool:Gen, Java, Shell Script)

- Minor Enhancement

Ticket Effort Resolution

It shows the behaviourof the ticket resolutionprocess.

- On-going support

- Ticket category

- Effort ticket resolution


Process Performance Models (PPMs) Process Performance Models (Process Performance Models (PPMsPPMs) )

Planning(Assignment)

Design

Produce

Testing

Non Dev

Ticket

Dev

Ticket

Predictability

80706050403020100

40

30

20

10

0

-10

Code line Qty _avg

Codi

ng

S 6.09834R-Sq 30.6%R-Sq(adj) 24.6%

Regression95% C I95% PI

Fitted Line Plot - Metavance Defect Team - only SEs, all PCVsCoding = 10.86 - 0.4746 Code line Qty _avg

+ 0.007352 Code line Qty _avg**2

Apr/2007 to Apr/2008

Productivity QualityGAD QMSLife Cycle

0.90.80.70.60.50.40.30.20.10.0

4

3

2

1

0

% Reused Test Cases

Test

Cas

e Cr

eati

on P

rodu

ctiv

ity

S 0.514988R-Sq 66.5%R-Sq(adj) 65.4%

Regression95% CI95% PI

Fitted Line PlotTest Case Creation Productivity = 0.4834 + 2.622 % Reused Test Cases

Pilot and After Pilot

3.02.52.01.51.0

150

100

50

0

-50

Work Product Creator Business Skill

Exte

rnal

Def

ect

Den

sity

S 20.7319R-Sq 70.6%R-Sq(adj) 66.2%

Regression95% PI

Fitted Line PlotDefect Density = 848.4 - 1236 Business Skill

+ 589.2 Business Skill**2 - 91.02 Business Skill**3

3.02.52.01.51.0

150

100

50

0

-50


Exte

rna

l Def

ect

Den

sity

S 20.7319R-Sq 70.6%R-Sq(adj) 66.2%

Regression95% PI



Standard Process

Standard Process

Standard Process

Test Case Creation Reuse

Library Reuse

Knowledge Repository

Training Guideline

Knowledge Base

Business Skill

Business Skill Business Skill

Business Skill

Business Skill

Business Skill

Cheklist Mentoring

Standard Process

Standard Process

Standard Process

Standard Process


Project Storyboard Project Storyboard Project Storyboard

26 / 10 Sep 2008 / EDS INTERNAL APPS Rio – HM Approach Overview

DMAIC Initiatives – Quality

Metavance is the main client of ASFOA Rio. It represents 4 managed projects with 150 resources (about 30% of the Centre).

It was very important to consolidate the perception of this client from ASFOA Rio as a high quality centre.

The Business NeedThe Business Need

•• Increase the quality of the final product;

• Reduce the bottleneck on the application US SME.

Client:

2321191715131197531

100

50

0

-50

Work Product Externally Reviewed

Exte

rnal

Def

ect D

ensi

ty

_X=26.1

UCL=133.9

LCL=-81.8

2321191715131197531

150

100

50

0

Work Product Externally Reviewed

Exte

rnal

Def

ect D

ens

ity

__MR=40.6

UCL=132.5

LCL=0

I-MR Chart of External Defect DensityHealth Care industry, Design phase, Developer role(Period Considered: from April to September 2007)



The analysis of defects raised at ASFOA Rio External Reviews showed that:

• Health Care industry detects 98.5% of defects;

• Design phase is the origin of 68.9% of defects;

• Developer role injects 88.0% of defects.

•>

Root Cause Root Cause AnalysisAnalysis

There was a correlation between External Defect Density and Work Product creator Business Skill.

The Correlation Coefficient (R) is 0.81 and it explains 66.2% of population, what was a good correlation.

The Defect Density increases when the Business Skill is lower than 2.0.

The Main Contributor

3.02.52.01.51.0

150

100

50

0

-50


Exte

rnal

Def

ect

Den

sity

S 20.7319R-Sq 70.6%R-Sq(adj) 66.2%

Regression95% PI





The SimulationThe SimulationIn order to evaluate the alternatives a simulation was performed, evaluating the impact on External Defect Density and Cost. Cristal Ball, a Monte Carlo simulator, was the tool used.

Option 1 Mean Cost: US$ 2.312 ( +7.8%)Option 1 Mean Defect Density: 35.2 (-36%)

Current Mean Cost: US$ 2143Current Mean Defect Density: 55.6

Option 2 Mean Cost: US$ 2.174 ( +1.4%)Option 2 Mean Defect Density: 54.4 (-2,2%)

Option 3 Mean Cost: US$ 2.184 ( +1.9%)Option 3 Mean Defect Density: 55.6 (0%)



The ImprovementThe ImprovementA model was developed and deployed on Health Care projects. The Project Manager will use it to analyze the benefits of perform an internal review to retain defects.

ProjectAppl. CR ID

PhaseSystem Area

Work Product Design documentPOD 50

WP Creator Business Skill 1.30Internal Reviewer Business Skill 3.70

Average

al Defects without Internal Review

Max (UCL)Average

Min (LCL)

The Project Manager decides to perform an internal review considering the possibility of retain 74% of the forecasted defects.

Analysis Result

Regression EquationDefect Density = 848.4 - 1236 (WP Creator Business Skill) + 589.2 (WP Creator Business Skill)**2 - 91.02 (WP Creator Business Skill)**3

50

Forecasted Total of External Defects without Internal Review

19

Forecasted Total of External Defects with Internal Review

19

Organizational Process Performance ModelDefect Density by Business Skill

ABC1234

Design Application Change Billing

Project Data

Confidence Intervals for Forecasted Total of External Defects

0

2

4

6

8

10

12

14

16

18

20

Forecasted Total of External Defectswithout Internal Review

Forecasted Total of External Defects withInternal Review

Fore

caste

d T

ota

l of

Exte

rnal

Defe

cts



The ResultsThe Results

645750433629221581

100

50

0

-50

-100

Work Product Reviewed

Exte

rnal D

efe

ct D

en

sity

_X=25.9

_X=18.6 _

X=3.2

UC L=129.3

UC L=91.2

UC L=17.4

LC L=-77.4LC L=-54.0

LC L=-11.0

Before Improv ement (Apr to Sep, 2007) P ilot (Sep to Nov , 2007) A fter Improvement (F eb to A pr, 2008)

645750433629221581

120

90

60

30

0

Work Product Reviewed

Exte

rnal

De

fect

Densi

ty

__MR=38.9 __

MR=27.3__MR=5.3

UC L=127.0

UC L=89.2

UC L=17.4

LC L=0 LCL=0 LC L=0

Before Improv ement (A pr to Sep, 2007) P ilot (Sep to Nov , 2007) A fter Improv ement (F eb to A pr, 2008)

11

111

I-MR Chart of External Defect Density

Mean from 18.6 to 3,2 defects by 100PODs

88%


Lessons LearnedLessons LearnedLessons Learned1. M&A/OPP/QPM will force you to create a structured process for Strategic

Business Planning. If you don´t have it will be very difficult to implement M&A/OPP/QPM practices

2. Have a long-term strategy to store measurement data, not only M&A but also SPC data.

3. Have an integrated set of tools supporting your Software Engineering projects, otherwise metrics collection will be a nightmare.

4. Reduce the effort, as much as you can, for metrics collection process, if possible, collect at the same time that the task is being performed.

5. Do specialize people on SPC, Six Sigma otherwise levels 4 and 5 will be a very hard journey. Consider to have statisticians around.

6. Excel will not be sufficient in many cases for QPM/OPP.

7. Do start with few performance indicators and then sophisticate and add additional ones, please remember, they must be aligned to companies´sgoals/objectives. 100 performance indicators will not allow you to take any conclusion.

8. Keep these indicators stable, as much as possible, so that you can analyze them periodically.

9. Do not individualize data at personal level, particularly on QPM/OPP analysis where sub-process analysis may lead you to see detailed data on people.


Q&A ? Q&A ? Q&A ?

03-23-05September 2009

eds.comeds.com

supporting the cmmi metrics framework thru level 5

Documents