measures for success: lessons learned from … - sept 2004.pdfmeasures for success: lessons learned...

PSMCRS 5.0g

Practical Software and Systems Measurement

PSM - IFPUG, 1PSM ™ All rights reserved.

Measures for Success: Lessons Learned from Applying PSM

Practical Software and Systems MeasurementObjective Information for Decision Makers

Department of Defense US Army



Measurement Principles

• Measurement Is a Consistent but Flexible Process that Must Be Tailored to the Unique Information Needs and Characteristics of the Project or Organization

• Decision Makers Must Understand What is Being Measured and Trust the Information

• Measurement Must Be Used to be Meaningful

PSMCRS 5.0g



PSM - ISO Standards - CMMIPractical Software

and Systems Measurement

ISO/IEC SC7 Standards12207 (revision - supporting process)15288 (measurement concepts)9126 (terminology coordinated)14598 (terminology coordinated)

-----

ISO/IEC 15939, Software Measurement

Process

CMMIMeasurement and Analysis



Fact-Based Management

• Requires Measurement Information to Support Critical Project and Organizational Business and Technical Decisions

• Measurement Results Must Be Communicated and Used Across the Entire Corporate Entity

• Measurement Results Must Be Incorporated with Risk Management and Context Information

PSMCRS 5.0g



Consistent but Flexible Process



Measurement Activities

Core Measurement Process

Information Needs

EvaluateMeasurement

ImprovementActions

AnalysisResults and Performance Measures

Scope of PSM

User FeedbackAnalysis Results

Establish andSustain

CommitmentPlan

MeasurementPerform

Measurement

Technical and Management

Processes

Measurement Plan

NewIssues

PSMCRS 5.0g



Plan Measurement

• Measures Are Defined to Provide Insight into a Project or Organization’s Information Needs by- Identifying what the decision makers need to

know- Relating these to entities that can be

measured- Selecting and specifying measures based on

project and organizational processes



Information Categories

• Schedule and Progress• Resources and Cost• Product Size and Stability • Product Quality• Process Performance• Technology Effectiveness• Customer Satisfaction

PSMCRS 5.0g



Plan Measurement - Lessons Learned

• Start by Implementing a Small Set of Measures- Reduces the level of change, resources

required, and impact on the workload- Add additional measures later, incrementally

• The Measurement Plan Should Specify Both “What” Will be Measured and “How” the Process Will Work



Multi-Level Measurement Requirements

Enterprise Management

OrganizationalManagement

ProjectManagement

Performance MeasurementNormative Performance BaselinesTechnical and Business PolicyInvestment Decisions & Analysis

Process ImprovementProject Planning GuidelinesPerformance Based GuidelinesOrganizational Norms & Benchmarks

Project Estimation & PlanningProject Performance TrackingProject Tradeoff AnalysisResource Management

RiskManagement

Process

Information - DrivenMeasurement

Process

PSMCRS 5.0g



Plan Measurement - Lessons Learned (cont).• Workshops Are Good For Defining

Information Needs• Allow Tailoring of Organizational

Measures• Successful Measurement Programs

Integrate the Needs of all Decision-Makers- This simplifies data collection and reduces

duplication



Perform Measurement

• Collecting Measurement Data, Performing Analysis, and Presenting Results

• Analysis Can Include Estimation, Feasibility Analysis of Plans, and Performance Analysis of Actual Data Against Plans

PSMCRS 5.0g



PSM Analysis ModelTechnology

EffectivenessProcess

Performance

Product Size and Stability

Resourcesand Cost

Schedule and Progress

ProductQuality

Customer Satisfaction



Size - EffortEstimating Relationship

0.1

1.0

10.0

100.0

1,000.0

10,000.0

0.285 K .00285K .0285K .285K 2.85K 28.5 K

Function Points

Staf

f Mon

th

-1 Standard Deviation

Project: PSM Data as of 31 Mar 98

+1 Standard Deviation Average

Project under consideration

Measurement Feasibility Analysis

PSMCRS 5.0g



Lessons Learned - Perform Measurement

• Automate Data Collection Whenever Possible• It Take About 6 to 9 Months to Establish

Measures- Initial focus is on ensuring data is provided- Next focus in on data problems- After these are resolved, focus can move to

performance issues• Clearly Identify the Reporting Mechanisms at

Both the Project and Organizational Levels• Aggregation Approaches Need to be

Specified, Especially When Measures Are Tailored



Establish and Sustain Commitment

• Includes Establishing the Resources, Training, and Tools to Implement a Measurement Program Effectively and Ensuring Commitment to Use the Information that is Produced

• Need to have Commitment Reinforced and Demonstrated Regularly

• Need to Adopt an Action Orientation

PSMCRS 5.0g



Gain Support for Measurement

• The Resulting “Culture Shock” from Implementing any new Process causes a Natural Reaction of Personal Resistance- Overcome resistance to change

• Provide the Participants with an Understanding of the Measurement Process and the Benefits to Their Projects- Training programs should help project

representatives identify information needs and measures

- Planning workshops should include representatives at all levels of the organization



Lessons Learned - Evaluate Measurement

• Both the Measurement Process and the Specific Measures Should be Periodically Evaluated and Improved

• Measurement is an Iterative Process: the Measures are Refined as Information Needs Change and the Organization Implements Improvement Actions

PSMCRS 5.0g



Understand and Trust Measurement Information



Information Model: High-Level View

Adapted from ISO/IEC 15939 - Software Measurement Process

InformationProduct

Information Need

MeasurementConstruct

MeasurableConcept

AttributeEntity

PSMCRS 5.0g



Measurement Construct - LevelsInformation

Product

DerivedMeasure

Indicator

Base Measure

Attribute

Measurement Construct



MeasurementMethod

MeasurementMethod

Estimate or Evaluation thatProvides a Basis for DecisionMaking

-

Operations Quantifying anAttribute Against a Scale

Indicator

Derived Measure

Derived Measure

BaseMeasure

Algorithm Combining Two or More Base Measures

A Measure of a Single AttributeBy a Specific Method

Quantity Defined as a Function ofTwo or More Measures

Algorithm Combining Measures and Decision Criteria

MeasurementFunction

Entities

Information Needs

Adapted from ISO/IEC 15939 - Software Measurement Process

BaseMeasure

Interpretation

AnalysisModel

InformationProduct

Attribute Attribute Property Relevant to Information Needs

Measurement Construct

PSMCRS 5.0g


PSM Version 5.0f, 23

Measurement Construct - Productivity

AnalysisModel

Compute Mean and StandardDeviation for Historical ProjectCompared to Estimate for newProjects

Adjusted Mean and TwoStandard Deviation Confidence Limits

ATTRIBUTES

BASE MEASURES

INDICATORS

DERIVED MEASURES

Count TotalHours

Count Semicolons

Divide SizeBy Effort

Repeat for Each Historical Project

Value for Project X

Value for Project X

Value for Project XProject X Actual

Productivity

MeasurementFunction

SizeProject X

MeasurementMethod

MeasurementMethod

Count TotalHours

Count inputs, outputs, inquires, files, and interfaces


Repeat for Each Historical Project

Value for Project X

Value for Project X

Value for Project XProject X

Actual Productivity

MeasurementFunction

SizeProject X

SourceCodeSemicolonHoursTimecards Function

PointsHoursTimecards

MeasurementMethod

MeasurementMethod

Value for Project X

Value for Project X

SizeProject X

Value for Project X

Value for Project X

SizeProject X

Count TotalHours


Value for Project Y

Value for Project Y

Value for Project YProject X Actual

Productivity

MeasurementFunction

SizeProject X

MeasurementMethod

MeasurementMethod

Count TotalHours


Repeat for Each New Project Project Y

EstimatedProductivity

MeasurementFunction

SizeProject X

Design SpecificationSemicolonHoursEffort

PlannedFunction

PointsHours

MeasurementMethod

MeasurementMethod

SizeProject X

SizeProject Y

Count inputs, outputs, inquires, files, and interfaces



Lessons Learned

• Provide Well Defined Base Measures to Ensure Consistency

• Present Information in a Format that is Understandable to the Decision Maker

• People Doing Measurement Aren’t Generally Making the Decisions - It Is Important to Present Information Clearly and Concisely

• Just Because the Measurement Information is Accurate and Objective, Does Not Mean It Will be Acted On

PSMCRS 5.0g



Measurement Must be Used



Lessons Learned

• The Measurement Process Should be an Integral Part of the Way Business Is Conducted

• Data Must be Provided Early Enough to Allow Managers to Take Action

• Results Must Be Communicated Throughout the Organization in a Timely Manner

• Decisions Should Not Wait for Perfect Data, but Should be Based on Accurate Data, Augmented by Risk Management and Context Information

PSMCRS 5.0g



Lessons Learned (cont.)

• The Measurement Results Should Help Decision-Makers Optimize Overall Performance

• Most Organizations are Composed of a Portfolio of Distinct Projects, So Project-Level Information Must Be Aggregated to Appropriate Levels of the Organization to be Used Effectively



Project-Level Focus

Discovered vs. Closed Defects

0

50

100

150

200

250

300

350

400

450

Nov-02 Dec-02 Jan-03 Feb-03 Mar-03 Apr-03 May-03 Jun-03 Jul-03 Aug-03 Sep-03

Date

# of

Def

ects

Discovered Closed-Implemented

PSMCRS 5.0g



Aggregated Data Highlights Issues

Time to Fix vs. Time to CloseClosed DefectsAs of 25 Sep 03

16

14

19 8

6

0 5 10 15 20 25 30 35 40 45

Calendar Days

Fixed Rework Time to Close

Time to Fix

Time to Fix Time to Rework Time to Finish

Time to Finish

93% of the PCRs follow the Normal Flow

7% of the PCRs involve Rework



Summary

PSMCRS 5.0g



Opportunities / Threats

• More Use/Mis-Use of Measurement• Continuous Requests for the “List” of Best

Measures• Constrained Environments that Don’t Let

You Act on the Measurement Results• Changing DoD Requirements for

Measurement- Greatly reduced requirements/standards (e.g.

DoDI-5000) - Continued emphasis on performance

requirements



Recommendations for Improvements

• Align Information Needs Across All Perspectives

• Put Your Energy into Making Sure Information Is the Best It Can Be (Versus The Most It Can Be) - Fewer but better measures

• Design the Measurement Process to React to Continuous Change

• Use the measurement results- Use the measurement results- Use the measurement results - ….

PSMCRS 5.0g



PSM Products



PSM Products Available

• PSM Book - Version 5 - Published by Addison Wesley

• PSM Insight• PSM Guidebook - Version 4• 1 and 2 day Training Sessions• Measurement Workshops for Programs or

Organizations• Measurement Experience Reports• Sample Measurement Specifications

PSMCRS 5.0g



PSM Products Available (cont)

• DoD Implementation Guidance• Measurement White Papers

- Evolutionary Acquisition- Interoperability- Object-Oriented Design- Product Lines

• Applying PSM to Enterprise Measurement (PSM and the Balanced Scorecard)

• Rational Unified Process (RUP) Measurement Plug-In



PSM Products Under Development

• Measurement White Papers- Acquisition Measurement- Safety and Security Measurement

(UK/Australian collaboration)- Systems Engineering Technical Measures

(INCOSE collaboration)- System of System Measurement (DAU

collaboration)- Process Improvement Measurement- Organizational / Enterprise Measurement

PSMCRS 5.0g



For More Information

Cheryl JonesQuality Engineering & System Assurance (QA&SA)US Army ARDECAMSRD-AAR-QEW-DBuilding 62Picatinny Arsenal, NJ 07806-5000(973) 724-2644 (Voice)(973) 724-2382 (FAX)

[email protected] Home Page: www.psmsc.com

PSM Support Center47196 Middle Bluff PlacePotomac Falls, VA 20165



Backup Material

PSMCRS 5.0g



PSM Participating Organizations

• US Navy NAVSEA• US Navy NAWC• US Navy NRL• US Navy NSWC • US Navy NUWC• US Navy OPTEVFOR• OSD NII• OSD PA&E • ODUSD (S&T)• Aerospace Corp. • Dept of Homeland

Security, Customs & Border Protection

DoD and Government• US Air Force AFMC• US Air Force SSSG• US Air Force STRATCOM • US Air Force STSC• US Army AMC• US Army CECOM• US Army RDECOM-ARDEC • US Army SAALT• US Army SMDC• USMC MCTSSA• US Navy AEDC• US Navy ASN RDA• US Navy FMSO• US Navy NAVAIR

• DAU-DSMC• DCMA• DFAS • DISA• DLA• FAA• IDA• MITRE Corp.• NASA• Sandia National Lab• Soc. Sec. Admin. • Software Engineering

Institute




• Hawaiian Electric• IEEE• IFPUG• IITRI• INCOSE • ITABHI• Independent Engineering, Inc. • IBM• Jacobs Sverdrup• James Gregory Assoc. • Kodak Health Imaging• L3 Communications • Lockheed Martin• Nat. Renewable Energy Lab.• Northrop Grumman • OAO Corporation• Pragma Systems Corporation• PRICE Systems, LLC• QSM• Quality Plus Tech.• Raytheon - Hughes• Reifer Consultants• Robbins, GIOIA• Rockwell Collins• SAIC

• Sallie Mae• SETA• Sentel• Software Engineering Assoc., Inc.• Software Management Solutions • Software Metrics, Inc.• Software Productivity Consortium• Tecolote Research. Inc.• TeraQuest Metrics• Texas Guaranteed Student Loan

Corporation• Titan Corporation• Tivoli• United Defense • Univ. Southern CA• Upsart Systems, LLC• User Trust Network• US West• VPI - State University• West Virginia High Tech. Consortium• West Virginia University• Whittaker Group• Wind River Systems• Xcel Energy

Industry• ACS GSG• Accenture, Quality and PI• Alion Science and Technology • American Sys. Corp.• Ameritrade Corporation• Argon Engineering Associates• Assurance Engineering• BAE Systems• Bloodworth Int. Tech.• Booz Allen Hamilton• CMIS• CTA• Carnegie Mellon Univ.• Computer Sciences Corp.• David Consulting Group• Distributive Software• Federal Reserve Bank• First Line Partners• Frunhofer Ctr. For Experimental

Software Engineering• GTE• Galorath, lnc. • General Dynamics• General Scientific Corporation• Graeme & Garland

PSMCRS 5.0g




International• Liveware I.SS.A.

(Argentina) • ADI Limited (Australia)• Australian Defence Force

Academy (Australia)• Defence Materiel

Org.-Australian DoD(Australia)

• Jacobs SverdrupTechnology (Australia)

• Motorola SOC Design Center (Australia)

• Amdocs (Israel)• Tangram Hi-Tech

Solutions (Israel)• Centro de Investigacion en

Matematicas (Mexico)• Ericsson Espana SA (Spain)• Defense Science and

Technology Labs (UK)• UK Ministry of Defense (UK)• University of York/ York

Metrics Ltd. (UK)

• S-3 Consulting Pty. Ltd (Australia)

• Saab Systems Pty. Ltd. (Australia)

• Software Improvements Pty. Ltd. (Australia)

• Software Quality Institute (Australia)

• Tenix ESD (Australia)• ti Metricas (Brazil)• General Dynamics

(Canada)• NRC (Canada)



Transition Organizations

DoD and Government• US Army RDECOM – ARDEC (PSM Support Center)• US Air Force Software Technology Support Center• US Navy NAVAIR• Naval Undersea Warfare Center• Aerospace Corporation • Defense Contract Management Agency• Defense Acquisition University -

Defense Systems Management College• Federal Aviation Administration • MITRE Corporation• Software Engineering Institute

PSMCRS 5.0g



Industry• Argon Engineering Associates • Computer Sciences Corporation • David Consulting Group• Distributive Software • INCOSE• Lockheed Martin • Northrop Grumman • Quality Plus Technologies, Inc.• Software Management Solutions • Software Productivity Consortium• TeraQuest Metrics• Texas Guaranteed Student Loan Center• Titan Corporation




International• Liveware I.S.S.A (Argentina)• ADI Limited (Australia)• Australian Defence Force Academy (Australia)• Defence Materiel Organisation - Australian DoD (Australia)• Jacobs Sverdrup Technology (Australia)• Motorola SOC Design Center (Australia)• S-3 Consulting Pty. Ltd. (Australia)• Saab Systems Pty. Ltd (Australia)• Software Improvements Pty Ltd. (Australia)• Software Quality Institute (Australia)• Tenix ESD (Australia)• ti Metricas (Brazil)• Tangram Hi-Tech Solutions (Israel)• Centro de Investigacion en Matematicas (Mexico)• University of York/York Metrics Ltd (UK)


measures for success: lessons learned from … - sept 2004.pdfmeasures for success: lessons learned...

Documents