measurement and the cmmi an impact analysis - … · title: microsoft powerpoint - rabbath -...

20
1 Materiel Policy & Services Materiel Policy & Services Improving the Acquisition and Sustainment of Capital Equipment f Improving the Acquisition and Sustainment of Capital Equipment f or Defence” or Defence” (C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia - - July 2002 July 2002 PSM Users’ Group Conference PSM Users’ Group Conference - - Keystone, Colorado Keystone, Colorado - - July 2002 July 2002 Pascal Rabbath Pascal Rabbath R2 R2 - - 6 6 - - B045, Russell Offices, Canberra ACT 2600, Australia B045, Russell Offices, Canberra ACT 2600, Australia Tel: int + 61 2 6265 6965 Tel: int + 61 2 6265 6965 Mob: int + 61 417 243322 Mob: int + 61 417 243322 e e - - mail: [email protected] mail: [email protected] Directorate of Software Engineering (DSWE) Directorate of Software Engineering (DSWE) Materiel Policy and Services (MPS) Materiel Policy and Services (MPS) Defence Materiel Organisation (DMO) Defence Materiel Organisation (DMO) Australian Department of Defence Australian Department of Defence Measurement and the CMMI Measurement and the CMMI (SM) (SM) : : an Impact Analysis an Impact Analysis

Upload: hoanghanh

Post on 30-Aug-2018

219 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

1Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

PSM Users’ Group Conference PSM Users’ Group Conference -- Keystone, Colorado Keystone, Colorado -- July 2002July 2002

Pascal RabbathPascal RabbathR2R2--66--B045, Russell Offices, Canberra ACT 2600, AustraliaB045, Russell Offices, Canberra ACT 2600, Australia

Tel: int + 61 2 6265 6965Tel: int + 61 2 6265 6965Mob: int + 61 417 243322Mob: int + 61 417 243322

ee--mail: [email protected]: [email protected]

Directorate of Software Engineering (DSWE)Directorate of Software Engineering (DSWE)Materiel Policy and Services (MPS)Materiel Policy and Services (MPS)

Defence Materiel Organisation (DMO)Defence Materiel Organisation (DMO)Australian Department of DefenceAustralian Department of Defence

Measurement and the CMMIMeasurement and the CMMI(SM)(SM)::an Impact Analysisan Impact Analysis

Page 2: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

2Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

OverviewOverview

•• What is the CMMI?What is the CMMI?•• Australian DoD Appraisal BackgroundAustralian DoD Appraisal Background•• Impact of Measurement on the CMMIImpact of Measurement on the CMMI•• Aggregated Appraisal ResultsAggregated Appraisal Results•• Lessons LearntLessons Learnt•• SummarySummary

Page 3: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

3Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

What is the CMMI?What is the CMMI?

•• The Capability Maturity Model Integrated The Capability Maturity Model Integrated (CMMI) is a model of world’s best practice (CMMI) is a model of world’s best practice covering:covering:–– Process ManagementProcess Management–– Project ManagementProject Management–– Systems EngineeringSystems Engineering–– Software Engineering, andSoftware Engineering, and–– Support Functions including QA, CM, Measurement & Support Functions including QA, CM, Measurement &

Analysis, and Decision MakingAnalysis, and Decision Making

Page 4: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

4Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

CMMI CMMI -- Process AreasProcess Areas

ProjectProjectManagementManagement

Process AreasProcess AreasCategoryCategory

Requirements ManagementRequirements Management (SE/SW)(SE/SW)Requirements DevelopmentRequirements Development (SE/SW)(SE/SW)Technical SolutionTechnical Solution (SE/SW)(SE/SW)Product IntegrationProduct Integration (SE/SW)(SE/SW)VerificationVerification (SE/SW)(SE/SW)ValidationValidation (SE/SW)(SE/SW)

EngineeringEngineering

Configuration ManagementConfiguration Management (SE/SW)(SE/SW)Process and Product Quality AssuranceProcess and Product Quality Assurance (SE/SW)(SE/SW)Measurement and AnalysisMeasurement and Analysis (SE/SW)(SE/SW)Decision Analysis and ResolutionDecision Analysis and Resolution (SE/SW)(SE/SW)Organizational Environment for Integration Organizational Environment for Integration (SE/SW/IPPD)(SE/SW/IPPD)Causal Analysis and ResolutionCausal Analysis and Resolution (SE/SW)(SE/SW)

SupportSupport

Project PlanningProject Planning (SE/SW)(SE/SW)Project Monitoring and ControlProject Monitoring and Control (SE/SW)(SE/SW)Supplier Agreement ManagementSupplier Agreement Management (SE/SW)(SE/SW)Integrated Project ManagementIntegrated Project Management (SE/SW/IPPD*)(SE/SW/IPPD*)Risk ManagementRisk Management (SE/SW)(SE/SW)Integrated TeamingIntegrated Teaming (SE/SW/IPPD)(SE/SW/IPPD)Integrated Supplier ManagementIntegrated Supplier Management (SE/SW/IPPD/SS)(SE/SW/IPPD/SS)Quantitative Process ManagementQuantitative Process Management (SE/SW)(SE/SW)

Organizational Process FocusOrganizational Process Focus (SE/SW)(SE/SW)Organizational Process DefinitionOrganizational Process Definition (SE/SW)(SE/SW)Organizational TrainingOrganizational Training (SE/SW)(SE/SW)Organizational Process PerformanceOrganizational Process Performance (SE/SW)(SE/SW)Organizational Innovation and DeploymentOrganizational Innovation and Deployment (SE/SW)(SE/SW)

ProcessProcessManagementManagement

Safety*Safety* Safety ManagementSafety Management (+SAFE)(+SAFE)Safety EngineeringSafety Engineering (+SAFE)(+SAFE)

Page 5: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

5Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

CMMI CMMI -- DMO Model ChoiceDMO Model Choice

•• CMMI was selected for the following reasons:CMMI was selected for the following reasons:–– Defence acquires total system capabilities Defence acquires total system capabilities -- not just not just

softwaresoftware–– Able to perform concurrent or separate appraisalsAble to perform concurrent or separate appraisals–– Can tailor to focus on specific process areasCan tailor to focus on specific process areas–– Can produce a capability profile as well as an Can produce a capability profile as well as an

organisational maturity levelorganisational maturity level•• CMMI sources:CMMI sources:

–– EIA/IS 731, SWEIA/IS 731, SW--CMM, IPDCMM, IPD--CMM, SACMM, SA--CMMCMM–– PDT (Industry, Academia, Government)PDT (Industry, Academia, Government)

Page 6: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

6Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

CMMI Trials CMMI Trials -- ObjectivesObjectives

•• Evaluate the modelEvaluate the model•• Understand the appraisal methodology & Understand the appraisal methodology &

resources required.resources required.•• Encourage (SE/SW) PI in Australian Defence Encourage (SE/SW) PI in Australian Defence

industry.industry.•• Establish appraisal expertise.Establish appraisal expertise.•• Map CMMI to ISO/IEC 15504 Map CMMI to ISO/IEC 15504 -- correspondence correspondence

& translation.& translation.

Page 7: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

7Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

CMMI Trials CMMI Trials -- StatusStatus

•• 15 appraisals completed to date (6 SCAMPI A15 appraisals completed to date (6 SCAMPI A(SM)(SM), , 9 SCAMPI B9 SCAMPI B(SM)(SM)))

•• As a result of Trials Program, DSWE has a team As a result of Trials Program, DSWE has a team of highly experienced appraisersof highly experienced appraisers

•• CMMI to ISO 15504 mapping completedCMMI to ISO 15504 mapping completed•• Development of safety process extension (+SAFE)Development of safety process extension (+SAFE)•• Generated interest from Defence, Defence Generated interest from Defence, Defence

industry, nonindustry, non--Defence industry and other Defence industry and other Australian government departmentsAustralian government departments

Page 8: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

8Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Appraisal OutputsAppraisal Outputs

•• Observations for each practice Observations for each practice -- 300 to 500 + 300 to 500 + (dependent on scope of appraisal)(dependent on scope of appraisal)

•• Findings (global and per Process Area) Findings (global and per Process Area) -- strengths strengths and improvement opportunitiesand improvement opportunities

•• Ratings against goalsRatings against goals•• Capability ProfileCapability Profile•• NonNon--CMMI findingsCMMI findings

Page 9: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

9Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Example Capability ProfileExample Capability Profile

Page 10: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

10Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Impact Analysis of MeasurementImpact Analysis of Measurement

•• Measurement and Analysis mapped onto the rest Measurement and Analysis mapped onto the rest of the CMMI in one of three categories:of the CMMI in one of three categories:–– Explicit dependencyExplicit dependency–– Enabler (including implicit dependency)Enabler (including implicit dependency)–– No explicit dependencyNo explicit dependency

•• Method Used:Method Used:–– Review of CMMI (Normative and Informative sections Review of CMMI (Normative and Informative sections

of the model)of the model)–– Appraisal experienceAppraisal experience

Page 11: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

11Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Measurement MappingMeasurement Mapping

Page 12: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

12Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Impact of Measurement on the CMMIImpact of Measurement on the CMMI

Measurement Mapping Explicit Dependency Enabler No Explicit

Dependency Totals Percentages*

Process Areas 6 - 19 25 24%Generic Goals 0 125 0 125 100%Generic Practices 25 200 200 425 53%Specific Goals 3 43 9 55 84%Specific Practices 26 89 74 189 61%

* Percentages calculated as: (Explicit Dependency + Enabler) / Totals

Page 13: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

13Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Explicit Dependency ExampleExplicit Dependency Example

•• Goal:Goal:–– Verification Specific Goal 2: Perform Peer ReviewVerification Specific Goal 2: Perform Peer Review

•• Description:Description:–– Peer reviews are performed on selected work productsPeer reviews are performed on selected work products

•• Practice implementation indicators:Practice implementation indicators:–– Peer review recordsPeer review records–– Data from peer reviewsData from peer reviews–– Results from analysis conducted on peer review dataResults from analysis conducted on peer review data

Page 14: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

14Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Implicit Dependency ExampleImplicit Dependency Example

•• Goal:Goal:–– Project Planning Specific Goal 1: Establish EstimatesProject Planning Specific Goal 1: Establish Estimates

•• Description:Description:–– Estimates of project planning parameters are Estimates of project planning parameters are

established and maintainedestablished and maintained•• Practice implementation indicators:Practice implementation indicators:

–– Use of estimation parametersUse of estimation parameters–– Estimation rationaleEstimation rationale–– Use of historical dataUse of historical data

Page 15: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

15Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Enabler ExampleEnabler Example

•• Goal:Goal:–– Decision Analysis and Resolution Specific Goal 1: Decision Analysis and Resolution Specific Goal 1:

Evaluate AlternativesEvaluate Alternatives•• Description:Description:

–– Decisions are based on an evaluation of alternatives Decisions are based on an evaluation of alternatives using established criteriausing established criteria

•• Practice implementation indicators:Practice implementation indicators:–– Trigger points for formal decision making process Trigger points for formal decision making process

(based on thresholds)(based on thresholds)–– Trade studies (eg make, buy or reuse)Trade studies (eg make, buy or reuse)

Page 16: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

16Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Aggregated Appraisal Results*Aggregated Appraisal Results*

* Results from 15 Appraisals Conducted over the period May 2001 * Results from 15 Appraisals Conducted over the period May 2001 --February 2002February 2002

Page 17: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

17Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Summary from Aggregated ResultsSummary from Aggregated Results

Specific Goals Explicit Dependency Enabler No Explicit

Dependency Totals Percentages

Majority Satisfied1 0 19 4 23 58%Majority Not Satisfied2 3 12 2 17 42%Totals 3 31 6 40 100%

Generic Goals Goal 1 Goal 2 Goal 3 Totals Percentages

Majority Satisfied1 7 0 0 7 13%Majority Not Satisfied2 11 18 18 47 87%Totals 18 18 18 54 100%

1: Majority of organisations appraised (ie 51% or greater) achieved Goal Satisfaction2: Majority of organisations appraised (ie 51% or greater) did not achieve Goal Satisfaction

Page 18: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

18Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Some Lessons Learnt fromSome Lessons Learnt fromthe CMMI Based Appraisal Programthe CMMI Based Appraisal Program

•• Measurement is currently being done poorly Measurement is currently being done poorly and/or incompletelyand/or incompletely

•• Capability appraisals catalyse adoption of reform Capability appraisals catalyse adoption of reform initiatives such as measurementinitiatives such as measurement

•• PrePre--conceived notion that you should only worry conceived notion that you should only worry about measurement for Maturity Level 2 and about measurement for Maturity Level 2 and aboveabove

•• Resistance to transition from SWResistance to transition from SW--CMM to CMMI CMM to CMMI significantly reduced after organisations realise significantly reduced after organisations realise the impact measurement has on the modelthe impact measurement has on the model

Page 19: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

19Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

SummarySummary

•• Measurement although not explicitly required for Measurement although not explicitly required for a large part of the model is still an effective a large part of the model is still an effective enabler for many practicesenabler for many practices

•• Measurement is key to an effective process Measurement is key to an effective process improvement programimprovement program

•• Measurement is an enabler for all Process Areas at Measurement is an enabler for all Process Areas at Capability Levels 1 Capability Levels 1 -- 55

•• Measurement dependency illustrates the highly Measurement dependency illustrates the highly integrated nature of the CMMIintegrated nature of the CMMI

Page 20: Measurement and the CMMI an Impact Analysis - … · Title: Microsoft PowerPoint - Rabbath - Measurement and the CMMI - An Impact Analysis - July 2002.ppt Author: Dave Created Date:

20Materiel Policy & ServicesMateriel Policy & Services““Improving the Acquisition and Sustainment of Capital Equipment fImproving the Acquisition and Sustainment of Capital Equipment for Defence”or Defence”

(C) Copyright Commonwealth of Australia (C) Copyright Commonwealth of Australia -- July 2002July 2002

Questions?Questions?

(SM): CMMI, SCAMPI A and SCAMPI B are service marks of Carnegie (SM): CMMI, SCAMPI A and SCAMPI B are service marks of Carnegie Mellon UniversityMellon University