early adoption experience with cmmi-acq · early adoption experiences with cmmi for acquisition 4...
TRANSCRIPT
Early Adoption Experiences with CMMI for Acquisition 081508
Sponsored by the U.S. Department of Defense© 2008 by Carnegie Mellon University
This material is approved for public release.
Early Adoption Experiences with CMMI® for Acquisition (CMMI-ACQ)
by Jon L Gross and Matt Fisher
SM CMM Integration, SCAMPI, SCAMPI Lead Appraiser, TSP, and IDEAL are service marks of Carnegie Mellon University.
® CMMI, Capability Maturity Model, CMM, and Carnegie Mellon are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. For more information on CMU/SEI Trademark use, please visit http://www-internal.sei.cmu.edu/teams/comms/trademarks-usage.html
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
2081508
Outline of Presentation
• Process improvement needs in an acquisition organization
• High level summary of CMMI for Acquisition (CMMI-ACQ) and differences with CMMI for Development (CMMI-DEV)
• Description of early adoption experiences and risks identified• Teaching• Appraisals• Process Improvement Consulting
• Mitigation of early adoption risks
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
3081508
Outline of Presentation
• Process improvement needs in an acquisition organization
• High level summary of CMMI-ACQ and differences with CMMI- DEV
• Description of early adoption experiences and risks identified• Teaching• Appraisals• Process Improvement Consulting
• Mitigation of early adoption risks
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
4081508
Some definitions:
Acquisition - “The process of obtaining products or services through a supplier agreement.” This would also include, outsourcing and/or supply chain; where supplier agreements are established.
Supplier agreement - “A documented agreement between the acquirer and supplier (e.g., contract, license, or memorandum of agreement).”
Source: CMMI-ACQ V1.2 model
Note: The laws and regulations of the country and organization are expected to be followed when applying CMMI-ACQ in improved processes.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
5081508
What We Mean By Acquisition
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
These are all potentials areas for acquisition work process improvement.
Acquisition Phases
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
6081508
Lifecycles Overlay – A complex environment
Lifecycle Milestones
• Product
• Acquisition
• Project
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMsAwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
AwardPhase
Pre-AwardPhase Post-Award Phase
RFP
ContractAward
ContractCompletion
Initiation
Deliverables...
...
Proposals
• Acquisition planning• Bidders conference• RFP preparation• Solicitation
• Contract administration• Monitoring technical performance• Technical Interchange Meetings• Test and evaluation of deliverables
• Proposal evaluation• Best and final offers• Source selection
RFI TIMs
0 1 2 N N+1
These all potentials areas for acquisition work process improvement.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
7081508
A Survey of Software Projects
1994
1996
1998
2000
2002
2004
2006
0% 40% 60% 80% 100%20%
Software projects completed on time (Success) Projects canceled before completion (Failed)Late and over budget (Challenged)
16%
27%
26%
28%
31%31%
40%40%
28%28%
23%23%
53%
33%
46%
49%
29% 18%18% 53%
35% 19%19% 46%
34%
Ref: CHAOS Study, Standish Group, 1994 - 2006
Average cost growth exceeds 89%Average final product contains 61% of the originally specified features
15%15% 51%
What role did acquisition organization play in results? We do not know, not measured.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
8081508
The State of Acquisition Practice
Is There an Acquisition Crisis?Investigation of one acquisition program showed:—System complexity and the program’s lack of experience in
procuring major systems caused serious cost growth.—Program lacked systems engineering and program management
expertise.—Requirements stabilization process was absent.—Program management does not enforce timely milestones,
timelines, and deliverables.—Program’s lack of process control made assessment of technical
risk impossible.—Program’s lack of short- and long-term budget tracking made cost
assessment nearly impossible.—Program did not manage risk.
Source: GAO Report
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
9081508
Supplier
Mismatch
Mismatch
mature acquirer mentors low maturity supplier
less mature acquirer derails mature supplier; encourages short cuts
Matchedacquirer and supplier are both high maturity
Disasterno disciplineno process
Technical & Management Skill
Low
Low
Hig
h
High
outcome not predictable
highest probability of success
no product supplier compromises processes
Acquirer/Supplier Mismatch
Acq
uire
r
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
10081508
Addressing Mismatch - Why Look in the Mirror?
“He who has the gold makes the rules” and pays the price.”
• All acquisitions begins inside the organization needing a development or an outsourcing.
• Acquisition work processes lead to supplier agreements.
• Insanity -- “… doing the same thing over and over and expecting different results. "-Benjamin Franklin
• Agencies have spent decades using magnifying glasses to look at the performance of the suppliers/contractors.
• We still have problems with program success.
It is time to start looking in a mirror
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
11081508
Acquisition Management
Acquisition activities deal with:
Acquisition project team
Supplier activities deal with:
Supplier project team
The overall project performance, schedule, cost
The Supplier’s performance, schedule, cost
WHO IS IN CHARGE?And who is eventuality accountable
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
12081508
• Based on the premise that “The quality of a system or product is highly influenced by the quality of the process used to develop or maintain it.” (CMMI-ACQ V1.2)
We could improve the process and practices of the Supplier•But the developers have a head start (CMMI-DEV based improvements are widespread) •Has not solved all the problems
• We could improve the processes and practices of the Acquirer by: •increasing the visibility of the Acquirer’s contribution to program success •defining, implementing, measuring and evolving effective acquisition processes and practices
What Can Be Done?
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
13081508
As the Acquirer, how would you like to work?
Processes can make the difference.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
14081508
Ineffective Approach to Quality Acquisition
Historically, many acquisition organizations have approached quality by performing product (contractor or supplier) measurement only. But, this approach does not improve quality nor guarantee quality. It’s a strategy of defect detection, not defect prevention.
Measurement of final product attributes is too late ...the quality, good or bad, is already in the product.
100% inspection is only 80% effective — Dr. Joseph Juran
Acquisition Processes
Contractor/ Supplier Products and Processes
Examining with a ‘magnifying glass’
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
15081508
Expected Acquisition Organization Result?
As you move up in maturity level, project cost will decrease and distribution will become tighter and more predictable
Most acquisition organization do not measure their work processes.Most acquisition organizations tend to be optimistic in the program projections.
Benefits of measuring:• Better estimation• Less deviation from
targetNotional for acquisition processes.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
16081508
Outline of Presentation
• Process improvement needs in an acquisition organization
• High level summary of CMMI-ACQ and differences with CMMI-DEV
• Description of early adoption experiences and risks identified
— Teaching
— Appraisals
— Process Improvement Consulting
• Mitigation of early adoption risks
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
17081508
CMMI V1.1March 2002
CMMI-AMV1.1
May 2005
CMMI-DEV V1.2August 2006
CMMI-ACQV1.2
Nov 2007
Evolution of CMMI Models*
CMMI-SVCTBD
Adapting CMMI for Acq
Jun 2006
SW-CMM
V1.11993
*Notional Time Scale
EIA 7311998
Guidance For Acquisition Improvement Has Evolved For Over a Decade.
SA-CMMV1.011996
SA-CMMV1.021998
SA-CMMV1.032002
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
18081508
Traditional Acquisition Activities
Acquisition Organization
The Supplier Agreement or Contract
SupplyingOrganization
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
19081508
Comparisons of CMMI-ACQ, V1.2 and CMMI-DEV, V1.2
CMMI-ACQ CMMI-DEV
For the acquirer (i.e., those who acquire, procure, or otherwise select and purchase products and services for business purposes, or those who outsource development and support)
For the product and service developer (i.e., those who develop or maintain products and services for business purposes)
Focus on the acquisition of products and services Focus on the development and maintenance of products and services
Generic practices are covered only in the Generic Goals and Generic Practices section
Generic practices are covered both in the Generic Goals and Generic Practices section and at the end of each process area
Explicit coverage of services Implicit coverage of services through the definition of the term “product,” which covers both products and services
Contains an Acquisition process area category, but no Engineering category
Contains an Engineering process area category, but no Acquisition category
Stages Acquisition Requirements Development at maturity level 2
Stages Requirements Development at maturity level 3
Categorizes Requirements Management as a Project Management process area
Categorizes Requirements Management as an Engineering process area
Contains typical work products and typical supplier deliverables
Contains typical work products
http://www.sei.cmu.edu/cmmi/models/ACQ-v12-comparetoDEV.html
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
20081508
ProjectManagement
Process AreasCategory
<Acquisition Requirements Development><Acquisition Technical Management><Acquisition Verification><Acquisition Validation><Agreement Management><Solicitation and Supplier Agreement Development>
Acquisition
Configuration ManagementProcess and Product Quality AssuranceMeasurement and AnalysisDecision Analysis and Resolution Causal Analysis and Resolution
Support
<Project Planning><Project Monitoring and Control><Integrated Project Management>Requirements ManagementRisk ManagementQuantitative Project Management
Organizational Process Focus<Organizational Process Definition>Organizational TrainingOrganizational Process PerformanceOrganizational Innovation and Deployment
ProcessManagement This chart shows all PAs
in CMMI-ACQ. All PAs in the Acquisition Category are unique to CMMI-ACQ.
The other PAs with blue and surrounded by <> contain additional specific practices that are only in the CMMI-ACQ model.
Requirements Management is in the Project Management category in CMMI-ACQ.
http://www.sei.cmu.edu/cmmi/models/ACQ-v12-comparetoDEV.html
CMMI-ACQ V1.2 Continuous Representation: PAs by Category
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
21081508
Supplier
Mismatch
Mismatch
mature acquirer mentors low maturity supplier
less mature acquirer derails mature supplier; encourages short cuts
Matchedacquirer and supplier are both high maturity
Disasterno disciplineno process
Technical & Management Skill
Low
Low
Hig
h
High
outcome not predictable
highest probability of success
no product supplier compromises processes
Acquirer/Supplier Mismatch
Acq
uire
r
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
22081508
Acquirer is in Charge
Acquisition Planning
RFP Prep.
Solicita- tion
Source Selection
System Acceptance
Program Leadership Insight / Oversight Transition
Plan Design Integrate & TestDevelop Deliver
Supplier Processes
CMMI for Acquisition
Operational and Business Need
Developer or Supplier Processes(CMMI for Development based process improvement.)
Acquirer Processes
Operational and Business Need Satisfied
Task Orders Do This After Basic Contract Awarded
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
23081508
Today’s Approach to Quality Acquisition
• A strategy of problem prevention: focus on the acquisition work processes.
• Fixing the “acquisition processes” lowers the probability bad products or services will be produced in the future acquisitions and supplier agreements.
Contractor/ Supplier Products and Processes
Acquisition Work Processes
Examining self with a
‘mirror’
The Supplier Agreement or Contract
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
24081508
Outline of Presentation
• Process improvement needs in an acquisition organization
• High level summary of CMMI-Acquisition and differences with CMMI-DEV
• Description of early adoption experiences and risks identified• Teaching• Appraisals• Process Improvement Consulting
• Mitigation of early adoption risks
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
25081508
Early Experiences from Teaching - 1
Learning to Look in a Mirror is Tough
• “It is the suppliers problem, that is why we hired them.”
A few acquisition organizations are willing to focus on their internal improvement.• Some US Government Agencies• Some Foreign Government Agencies• Some commercial companies
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
26081508
Early Experiences from Teaching - 2
• They are using experiences from CMMI-DEV process improvement and applying them to acquisition work processes.
• Still holding the supplier accountable for contract performance.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
27081508
Early CMMI-ACQ Appraisal Experiences - 1
General Motors performed SCAMPI A, B and C Appraisals, using process areas from CMMI-DEV and the Adapting CMMI for Acquisition Organizations.• SCAMPI Method and Classes work, without change• Depends on acquisition organization defining their work
processes
Gaining continuing Senior Level Sponsorship is Key, but can be politically sensitive.• External pressure can be essential (e.g., GAO, Congress,
CEO)• Strategic View is important “I need to improve”
A few US Government acquisition organizations are doing model based process improvement.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
28081508
Early CMMI-ACQ Appraisal Experiences - 2
The SCAMPI Appraisal Family of methods works for CMMI-ACQ, in the same way it works for CMMI-DEV.
SCAMPI C: provides a wide range of options, including characterization of planned approaches to process implementation according to a scale defined by the user
SCAMPI B: provides options in model scope and organizational scope, but characterization of practices is fixed to one scale and is performed on implemented practices
SCAMPI A: Is the most rigorous method, and is the only method that can result in ratings.
breadth of tailoring
C
A
B depth of investigation
SCAMPI Lead Appraiser must be authorized for CMMI-ACQ.
SCAMPI Family
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
29081508
Early CMMI-ACQ Appraisal Experiences - 3
SCAMPI Appraisal Approach for CMMI-ACQ:
• Assure the appraisal team members have experience and understanding of acquisition work.
• Assure model understanding by the organizational unit in preparing for appraisal.
— SCAMPI Class B and C appraisals are the best starting point— Uncovers model issues in a less intense appraisal environment— Allows time to align work processes with appraisal objective
evidence expectations.
• Appraisal Team Members need to complete SEI Authorized Introduction to CMMI-ACQ V1.2 Training.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
30081508
Early CMMI-ACQ Process Consulting Experiences• Transition from current process improvement model to CMMI-ACQ
• One Government organization is developing a transition process, with entry and exit criteria
• A commercial company is transitioning from acquisition processes based on CMMI-DEV to new processes based on CMMI-ACQ
• Some organizations are starting with blank sheets of paper
• Lack of knowledge of model detail by senior management
• Sponsors often do not have enough detailed knowledge of the models to understand the significance, benefits, or effort required
• Budget and/or senior management changes often affect process improvement programs first
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
31081508
Outline of Presentation
• Process improvement needs in an acquisition organization
• High level summary of CMMI-ACQ and differences with CMMI-DEV
• Description of early adoption experiences and risks identified
— Teaching
— Appraisals
— Process Improvement Consulting
• Mitigation of early adoption risks
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
32081508
The Bottom Line or Mission Must Drive ImprovementImprovement means different things to different organizations:
• What are your mission or business goals?
• How do you measure progress?
Improvement is a long-term, strategic effort:
• What is the expected impact on the mission or bottom line?
• How will impact be measured?
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
33081508
Getting Started: Key Success Factors for an Effective PI Effort
Before major change can even be contemplated, top management must recognize the need and be sold on the idea.
The Must Haves:
1. A compelling reason for change
2. Leadership of the change effort by the top executive in the organization — responsibility cannot be delegated
3. Informed commitment of the top management team
4. Designation of a primary PI Leader and an adequate mandate for change
5. Sound performance measures that drive change
Source: Mastering Process Improvement Course
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
34081508
Getting There Means CHANGING!Acquisition organizations have many barriers to address to reach the culture of measured process performance.
Improvement of acquisition work processes is still in the “Innovator” or “Earlier Adopter” stage of transition.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
35081508
Why So Long for Process Improvement?
Magnitude of Technological Change Sought
Years Months Weeks Small Large
Time to Adjust
Skills
Procedures
Structure
Strategy
Culture
Levels of Learning Required
Diagram adapted from Adler
Most Acquisition Organizations are Large.
Do you have long term sponsorship?
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
36081508
Organizing Process Improvement
InitiateInitiate DiagnoseDiagnose EstablishApproachEstablishApproach ActAct LearnLearn
Get Started Where Am I? What ShouldI Do?
Take Action Learn fromExperience
Re-assessed as needed for further improvement
Shows benefit of process improvement
Helps get management buy-in
CMMI-ACQ appraisal baselines acquisition work processes
Provides an agenda for improvement
Capability levels in Process Areas help set priorities and goals
Define Process Areas and characteristics of desirable practices
Provides a framework to allow fine- tuning to improvement process
Reference: IDEAL
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
37081508
Ongoing Communication is Key to Adoption
How individuals and organizations commit to change
Time
ContactAwareness
Understanding
Adoption
Institutionalization
Com
mitm
ent
Installation
Daryl R. Conner and Robert W. Patterson. “Building Commitment to Organizational Change,”Training and Development Journal (April 1983):18-30.
Acquisition organizations
are just starting the adoption
process.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
38081508
Applying the CMMI-ACQ
Use the same basic steps:
• CMMI-ACQ terminology must be mapped onto local acquisition organization terminology and the local way of doing business.
• The generic organization terms (e.g., project manager, project, senior manager) of the CMMI-ACQ must be mapped onto the local organization’s management structure.
• The CMMI-ACQ should be applied using intelligence, common sense, and professional judgment. • The CMMI-ACQ is not carved in stone.
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
39081508
Process Improvement Consulting
It is all about:
• getting and continuing long term sponsorship
• a willingness and courage to look internally
• a willingness to establish documented processes
• establishing an internal process improvement support effort
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
40081508
Conclusion
CMMI-ACQ provides the opportunity to do the following:
• Improve acquisition program results, by improving operating processes and practices in acquisition organizations
• Improve the management and technical work skills— acquirer-supplier— acquirer-acquirer— acquirer-user
• Encourage strategic acquirer-supplier teamwork• Extend enterprise-wide process improvement
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
41081508
Contact Information
Mr. Jon L. GrossSoftware Engineering InstituteCarnegie Mellon University4301 Wilson Blvd, Suite 200Arlington, VA 22203-1867USAOffice: (703) 908-8238Email: [email protected]
Dr. Matthew FisherSoftware Engineering InstituteCarnegie Mellon University4500 Fifth AvePittsburgh, PA 15213-3890USAOffice: (412) 268-5877Email: [email protected]
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
42081508
BACK-UP SLIDES
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
43081508
CMMI-ACQ v1.2 Acquisition Category Process Areas
Acquisition Requirements Development
Solicitation & Supplier
Agreement Development
Agreement Management
Acquisition Technical
Management
Acquisition Validation
Acquisition Verification
CMMI Framework
16 Project, Organizational,
and Support Process Areas
© 2008 Carnegie Mellon University
Early Adoption Experiences with CMMI for Acquisition
44081508
Why Is a Process Model Important?
A process model (CMMI-ACQ) provides
• a place to start
• the benefit of a community’s prior experiences
• a common language and a shared vision
• a framework for prioritizing actions
• a way to define what “improvement” means for your organization
Remember: “All models are wrong; some are useful.” - George Box