applying the cmmi/spi in a cost controlled environment dennis j. frailey

Applying The CMMI/SPI In A Cost Controlled

Environment

Dennis J. Frailey

November 13, 2003 SPI in a Cost Controlled EnvironmentCopyright © 1996-2003, Dennis J. Frailey, All Rights Reserved Slide 2

Agenda

• How to Get There• Lessons Learned• Tangible Results


A Long Journey

BecomeRepeatable

DefineSW

Processes

OptimizeProcesses

Manageby Fact

1 2 3 4 5

ContinuousImprove-

ments

1989 Today

This presentation will discuss how we got where we are today, and give a glimpse into the future.


Status Report - 1989-90

• Software was a universal problem throughout the company

• Common symptoms:–Cost overruns–Schedule overruns–Quality problems

• Customers were unhappy with the industry as a whole and we were no exception

OUR MISSION - FIX ITOUR MISSION - FIX IT


Typical Comments

“We are running by the seat of our pants.”

“We build requirements and plans by throwing engineers at the project and seeing what they come up with.”

“Those of us who are lucky enough to attend training are often not allowed to use it.”

“Software is the last thing we think about.”

“Testing is the last thing we think about -- after software.”


Management ReactedTo: Executive Steering TeamCopy: Jerry Junkins , Chairman, Pres ident & CEOFrom: George HeilmeierSubject: Software Quality at TI

As you know, an interdivis ional group is attempting to addres ss oftware management productivity and quality is s ues . The group haschampioned the us e of the Software Engineering Ins titute’s (SEI)as s es s ment criteria to benchmark the s tatus of each of ourgroups . . .

. . . We need to improve our s oftware capability acros s TI, and thisprogram has rece ived, with a few exceptions , exce llent partic ipationand s upport from the groups . We need to continue thisinterdivis ional e ffort in 1990 to build on the cooperative team e ffortbegun this year. . .

2) Make this a keypriority across thecompany.

Jerry Junkins

1) This is absolutely akiller issue.


Important Early Lessons - We Learned About Ourselves• Not as good as we thought we were• Not as consistent as we thought we were• Much more software than we thought we

had• Many pockets of excellence• But not much transfer of knowledge

Languages

Assembly 23%

Ada 48%

C 21% Others 8%


Most Of Our Software Was Not Where We Thought it

WasDSEG Software

26 Million Lines of Code

Mechanic Eng 2%

Support Eng 3%

Manufacturing 11%Eng Services

Automation 7%

Software Eng 8%

Advanced Tech & Components 2%

Systems/Analysis 9%

Electrical Eng 1%

Test Eng 57%

Where we had concentrated our efforts


Organizational Barriers To Improvement

• No central oversight of software product development

• Several independent business entities within the organization

• “Cottage industry” mentality among software developers– “Not Invented Here”– Perception of uniqueness– Customer-specific or project-specific

standards, policies, and objectives– Dependence on heroes


Organizational Bright Spots• A Cooperative Team of Software Leaders

(Software Improvement Team)– had been meeting to resolve mutual problems

• A Centralized Software Technology Department – had many of the necessary resources in place

• A Corporate Process Group was Forming– developed a corporate wide process framework & goals– nurtured top management support

• A Reorganization was Pending– to put more emphasis on coordination within

engineering functions


SEI was beginning to promote its maturity model and assessment process

External Forces

• Click to add textLevel ResultKey Process Areas

5Optimizing

4Managed

3Defined

2Repeatable

1Initial

ContinuousImprovement

Product and ProcessQuality

Engineering Process

ProjectManagement

Heroes

version 1.1

Productivity

& Quality

Focus

Process change managementTechnology change managementDefect Prevention

Software quality managementQuantitative process management

Organization process focusOrganization process definitionPeer ReviewsTraining programIntergroup coordinationSoftware product engineeringIntegrated software management

Software project planningSoftware project tracking & oversightSoftware subcontract mgmnt.Software quality assuranceSoftware configuration mgmnt.Requirements management

SEI Capability Maturity Model

Risk

»CUSTOMERS were

beginning to demand use

of the SEI model and

assessments

»CUSTOMERS were

beginning to demand use

of the SEI model and

assessments


We Were Not Alone!

81%

12%7%

0% 0%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

1Initial

2Repeatable

3Defined

4Managed

5Optimizing

Software Process Maturity Distribution -- 1991 (from SEI)


A Process For Improving The Process

BIG PICTURE VIEW - FOUR STEPS

Level 1 -> Level 21990-92

Level 4 -> Level 52002-2003

Level 3 -> Level 41994-2001


Assess AssessAssessAssess Assess

CMM

CMMI


Why Does It Take So Long To Move Up One CMM Level?• Each level represents a major

cultural change in the organization– Attitudes– Organizational Structure– Mission of Support Organizations

• And each level requires a lot of work– Five major sub-steps– The business has other priorities as well– Overhead constraints limit process effort


We Don’t Control All Of The Variables

• Especially at higher CMM levels, both the other parts of our organization AND our customers and suppliers must be part of the solution


The Process Within Each Step


Level 4 -> Level 52002-2003

Level 3 -> Level 41994-2001


Establish the Goal

• Vision

• Definition

• Buy-in and Resources

Develop Training

• Process & Methods

• Just in Time

• LanguagesPilot Efforts

• Early Adopters

• Lessons LearnedOrganize for Deployment

• Organization

• Support

• ResourcesFull Deployment• Buy-in• Training• Motivation• Support• Monitor

You must have buy-in at All Levels

before deploying process

improvement

You must have buy-in at All Levels

before deploying process

improvement


Establish the Goal

ResourcesDefinition (What You

Want to Do)GoalGoal

Buy-In / Motivation

Vision

Define the Process

• Definition

• Standard Procedure

• Operating Instructions

Develop Training


• Just in Time


• Various Programs

• Lessons Learned

Organize for Deployment

• Organization / QITs

• SEPG

• Process EngineersFull Deployment

• Site Based Action Plans

• Sponsorship

• Assessments & Training

• Execution of Plans


Training

ProcessTraining

forAll

NewHires

Focused Trainingfor SpecificFunctions

• Software Project Management

• Inspections• Process

Tailoring• and many more

ProgramManagerTrainingSessions

forEach

Project

A Work Force Capable of Applying the Process

“Just in Time”Planning Sessions

forEach Project

• Project Planning• Requirements

Analysis• Preliminary

Design• etc.

Define the Process

• Definition



Develop Training


• Just in Time



• Lessons Learned



• SEPG



• Sponsorship




Pilot Programs

• Identify changes needed for deployment on real programs• Give new ideas to eager, early adopters• Provide extra support to help them

and to learn from them• Develop a small core of vocal,

influential promoters

Define the Process

• Definition



Develop Training


• Just in Time



• Lessons Learned



• SEPG



• Sponsorship




• May require significant organizational changes–establish oversight bodies that really do

pay attention to process improvement progress and results

–shift resources from organization-wide SEPG to site-specific and project-specific counterparts

– reassign influential and experienced personnel to key process deployment roles

Plan For Deployment

WHYCHANGE

?

Define the Process

• Definition



Develop Training


• Just in Time



• Lessons Learned



• SEPG



• Sponsorship




DeploymentDefine the Process

• Definition



Develop Training


• Just in Time



• Lessons Learned



• SEPG



• Sponsorship



%

with

the

pro-gram

time

Early Adopters

Bulk of the Population

Laggards

The Deployment Learning Curve


Deployment Elements

• Buy-in• Training• Motivation• Support• Monitor

Effective deployment is perhaps the most

significant difference between successful and

unsuccessful process improvement initiatives

Effective deployment is perhaps the most

significant difference between successful and

unsuccessful process improvement initiatives

Lessons Learned


Leaders Must Lead• Organizational leaders must be highly

visible at the head of the parade– Participating actively in all phases from goal

setting to the nitty-gritty of deployment– Communicating that they know and care

• Foster an attitude of success and unity• It must be clear up and down the

organization that the improvements are not going away


Set Clear Expectations• Who is expected to do What,

When, Where, How and Why• Distinguish between requirements,

desires, and options• Document what is expected• Communicate and enforce the

consequences• Clarify to whom the expectations

apply


Tie to Organizational Initiatives

• Synergy with organizational programs for overall corporate improvement

• Support for what you want to do instead of competition for resources

• Acknowledgement and appreciation of what you are doing from top management


Don’t Take On Too Much

• Higher Maturity Levels• 100% Deployment• Different Types of

Software•Mission Critical•Test•Maintenance•Simulation•etc.

Higher CMM Level

100

%

A llTy pe s

»The magnitude of the effort is sobering»The magnitude of the effort is sobering


Educate

• Every person playing a role should be competent and knowledgeable

• They should understand the principles behind the methods– So they know when NOT to do it as

prescribed

• When they know what to do and why, people will dare to achieve and improve


Deploy Behaviors, Don’t Teach Models (CMM, CMMI,

etc.)– One behavior every few weeks

• Frequent, short briefings• With plenty of buy-in up the line

– Over a period of time– With strong reinforcement

• Set expectations• Reporting and monitoring• Behaviors become integrated parts of the

process

– And training available NOW


Frequent Feedback

• Surveys to understand how things are going

• Measurements and monitoring• Be willing to change if it isn’t

working• Walk the talk


The SEPG Must Not be anIvory Tower

• Populated with people from real projects– Job rotation into and out of the SEPG– Project people participate in defining the

processes and procedures

• Actively out there, supporting projects– Playing a role in the day to day activity– Providing subject matter experts who get

involved

• Their world view must include the real world, not be limited to writing procedures and processes


The Process Must be Efficient and Appropriate

• Large, thick documents are never read - and they cost a lot of money

• It takes time to mature a process • Tailoring is an essential part of an

effective process - and it saves money– Tailoring can be reviewed, approved,

justified, etc.


The WEB Can BeWonderful or Horrible

• You must have a systematic approach to using web resources– Plan it, don’t just tell people to use it– Standard architectures, formats, etc.– Consistent look and feel– Systematic methods of developing

and maintaining web pages and sites


Just-in-Time Training

• Short training modules on individual topics

• Taught right before people need to know

• Taught by people who know how it works in practice

• May be integrated with project planning or discussion activities


Keep Your Eye on the Ball

• Regular reporting and feedback mechanisms

• Track progress on deployment

ProjectImprovements

A B C …

Alpha

Beta

…

Nobody wants to be “red” on a

report card


Outside Consultants See What You Overlook

• Bring them in early!– Lack of adequate evidence of good

practices– Inadequate access to data– Ad-hoc, inconsistent behaviors– Misunderstandings– …

• Dry run your assessments


Evolving Role Of The SEPG• Initially, it was to define the process

for software development and the process for improving the organization

• As time went on, it became a support organization to help with the rest of the steps of the improvement process

• And now it provides many process maintenance functions:– Software Metrics Data Base - collection &

analysis– Software Process Assets Library– Lessons Learned and Problem Reports– Integrated with Tool Support Organization

Tangible Results


Some Of The Tangible Signs•Consistency in measurements and

processes•It begins to be possible to collect

meaningful data at the organizational level

•Quantifiable results:–Measurable improvements in quality,

productivity and cycle time–Measurable improvements on customer

satisfaction surveys


Software Product Cycle Time All Software

23-Nov-90 11-Jun-91 28-Dec-91 15-Jul-92 31-Jan-93 19-Aug-93 7-Mar-94 23-Sep-94 11-Apr-95 28-Oct-95 15-May-96 1-Dec-96 19-Jun-97

Product Start (Pre. Design) Date

Ca

len

da

r D

ay

s

Req. Analysis Design Integration System Integration Post Release Duration Expon. (Duration)


DLOC Productivity (Delivered Lines Of Code)

BIC 1 - SWED Products

Dec-91 Jul-92 Jan-93 Aug-93 Mar-94 Sep-94 Apr-95 Oct-95 May-96 Dec-96 Jun-97

Product (Pre. Design) Start Date

Pro

du

ctiv

ity

(EL

OC

/ E

ffo

rt H

ou

rs)

Req. Analysis Design CUT I&T Sys I&t Released Expon.


Mission Software Aggregate Fault Density

Jul-91 Oct-91 Jan-92 Apr-92 Jul-92 Oct-92 Jan-93 Apr-93 Jul-93 Oct-93 Jan-94 Apr-94 Jul-94 Oct-94 Jan-95 Apr-95

Fau

lts

/KP

SL

OC

Goal: Six Sigma

SEI LEVEL 2 5% 30% 50% 70% 80% 90% 100% 100% 100% 100% 100% 100% 100% 100% 100%

Release Date:

SEI LEVEL 3 25% 30% 55% 63% 75% 100% 100% 100% 100%


Alignment Of Methods

•Common measurements•Consistent terminology•Compatible reward systems

VS.


Reductions in Rework

•Rework = Cost and Schedule Slips•Process improvement helped us

–See the rework–Quantify the rework– Identify the causes of rework–Motivate spending to reduce rework–Reap the benefits of less rework


What Process Does For Us•Codifies the expertise of our

predecessors and ourselves•Deals with the details so we can

concentrate on what is important•Enables us to stand on the

shoulders of giants•Gives us the freedom and the support we need to do our job in a professional manner


Misuse of Process

• Process should be a Guideline, not a Mandate -- like a map for an explorer

• It should be like a high level language for the process of developing software

• Processes should depend on the knowledge and expertise of practitioners -- not substitute for it


Other Fruits of our Labor• Variation between best and worst

projects has been reduced substantially

• Overall performance on cost, schedule, and quality have improved– It helps to have a way to measure

these!• Our processes are followed more

uniformly and are respected by our colleagues in other disciplines


Final Remarks•The journey has been tough but the

results are showing in very quantitative ways

•A long term outlook is required for success

•Never underestimate the power of a supportive and insistent customer

•We are well past the “point of no return” - turning back is not an option


References – Slide 1 of 3Bate, Roger R. (1991), STEP - Software Engineering

Process, Guest Editorial, Texas Instruments Technical Journal, Vol. 8, no. 3, ISSN 0893-7877, 2-4.

Benno, Steve and Dennis Frailey (1994). “Software Process Improvement in DSEG, 1989-95,” Texas Instruments Technical Journal, Vol. 12, no. 2, 20-28.

Department of Defense (1988), Defense System Software

Development, Dod-STD 2167A, 29 Feb. 1988, Department of Defense, Washington D.C., 20201.

Fortin, Pauline with David Carter and Dennis Frailey (1994), Who's On First - Covering All the Bases for Software Process Improvement, Proceedings, Fourth IEEE Workshop on Technology Transfer, April, 1994.


References – Slide 2 of 3Frailey, Dennis J. (1993), "Concurrent Engineering and the

Software Process," Proceedings, 2nd International Conference on the Software Process, IEEE Computer Society Press (ISBN 0-8186-3600-9, IEEE catalog no. 93TH0509-0), 103-114.

Frailey, Dennis J. (1991a), "Defining a Corporate-wide Software Process," Proceedings, 1st International Conference on the Software Process, IEEE Computer Society Press (ISBN 0-8186-2490-6; order no. 2490), 60-67.

Frailey, Dennis J. (1991b), "Activities of the STEP Basic Software Development Cycle," Texas Instruments Technical Journal, Vol. 8, no. 3, ISSN 0893-7877, 28-37.

Humphrey, Watts S. (1989), Managing the Software Process, Addison-Wesley, Reading, MA, 1989.


References – Slide 3 of 3Humphrey, Watts S. (1987a), Characterizing the Software Process: A

Maturity Framework, Software Engineering Institute, CMU/SEI-87-TR-11, ADA182895, June 1987.

Humphrey, Watts S. and W.L. Sweet (1987b), A Method for Assessing the Software Engineering Capability of Contractors, Software Engineering Institute, CMU/SEI-87-TR-23, ADA187320, September 1987.

IEEE (1992), IEEE Standard for Developing Software Life-Cycle Processes, IEEE-STD-1074-1991. IEEE Computer Society, New York.

Paulk, Mark, et. al. (1993), Capability Maturity Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-2, February, 1993.

SEI (1991), State of the Practice: A Software Process Maturity Perspective. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA 15213 (presentation at the 1991 SEI Affiliates Symposium, August 27-29, 1991.

applying the cmmi/spi in a cost controlled environment dennis j. frailey

Documents

placea corporate process

projectspecific standards

sei model

companycommon symptoms

necessary resources

engineering functionsspi

transfer of knowledge

universal problem