measuring roi in low maturity - wibas...basics of improvement: process focus the engineering process...
TRANSCRIPT
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 1
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 1
Copyright StatementSM Architecture Tradeoff Analysis Method, ATAM, CMM Integration, COTS Usage
Risk Evaluation, CURE, EPIC, Evolutionary Process for Integrating COTS-based systems, Framework for Software Product Line Practice, IDEAL, Interim Profile, OAR, OCTAVE, Operationally Critical Threat, Asset, and Vulnerability Evaluation, Options Analysis for Reengineering, Personal Software Process, PLTP, Product Line Technical Probe, PSP, SCAMPI, SCAMPI Lead Appraiser, SCE, SEPG, SoS Navigator, T-Checks, Team Software Process, TSP, are service marks of Carnegie Mellon University
TM Carnegie Mellon Software Engineering Institute (stylized), Carnegie Mellon
ROI in Low Maturity
g g g ( y ), gSoftware Engineering Institute (and design), Simplex, Stylized hexagon are trademarks of Carnegie Mellon UniversityQ:PIT and the Q:PIT logo are trademarks of Q:PIT Ltd, Milton Keynes, UK
® Capability Maturity Model, Capability Maturity Modeling, Carnegie Mellon, CERT, CERT Coordination Center, CMM, CMMI, FloCon, OCTAVE are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University
©Q:PIT Ltd 2005-2008 2
Contents5 Reasons for FailureBasics for ImprovementPrinciples of MeasurementRequirements for Measurement
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 3
qProcess for MeasurementPutting into Practice
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 2
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 4
Five Reasons to FailObjectives
Focus on the satisfying the model, not on the business
BudgetDemonstrate return on investment within the year
Technical change
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 5
Technical changeLet the techies do this: it is their problem
We know what is neededDon’t get the customer involved today, they will be happy that we improve
Do it quicklyWe can’t afford to wait
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 6
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 3
Basics of Improvement:Business Focus
Focus on what you need to deliver to your market:Less defects, increased qualityReduced costs, better investmentQuicker time to marketBeing Maturity Level X is rarely a business issue
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 7
Basics of Improvement: Culture
Process Improvement is not a technical activity:Focus on the cultural change that is neededEnsure that the cultural aspects are understood by managementGeneric goals are more important in the long-term than specific goals
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 8
p gWe need people to focus on quality and on improvement, not on complianceWe need management to demonstrate what they want
Basics of Improvement: Leadership
Keep your eye on the horizonLead by exampleLearn from mistakesEncourage the right attitude rather than the short-term result
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 9
Lead so that people can follow
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 4
Basics of Improvement: Quality Assurance
Assurance that people know what they are doingAssurance that you know how they are doingAssurance that they know why they are doing itIt is only by knowing what is really happening that we
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 10
y y g y pp gcan learn and progressQuality assurance needs to be seen as a proactive support, not just compliance audits and punishment
Basics of Improvement: Configuration Management
Configuration management gives us the foundations on which the rest can be built
Every area, every process needs to ensure the integrity of the systemThis is more than just version controlThis does require investment in a tool
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 11
This does require investment in a tool
Basics of Improvement: Process Focus
The Engineering Process Group needs to be implemented at Level 1
Even though OPF is a maturity level 3 process area, the ownership and co-ordination of processes and improvement paths need to be implemented from the beginningH i th OPF i l d t th t d t
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 12
Having the OPF in place does not mean that you need to jump over ML2 and have everyone operating at ML3
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 5
Basics of Improvement: Process Framework
Set up the framework for the futureConsider what you are planning on doingConsider how you want to structure the processesSet up a process architectureSet up a process repositoryS t f ti d t ti l
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 13
Set up a means for reporting processes and potential improvementsIdentify what and how you want to measure
Basics of Improvement: Measurement
If you don't know what you want to achieve, you will not achieve itIf you cannot measure your achievement, you will not achieve itIf you don't know how far you have come you will not
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 14
If you don t know how far you have come, you will not know if you have arrivedIf you cannot demonstrate that you have improved, you will lose your support
Setting Up the BasicsSetting up all these things requires a significant investment
How much should you invest?What will you get out of it?How to persuade management that this is an investment and not a cost?
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 15
and not a cost?How to persuade the board / the shareholders that they are getting benefits out of this?How to ensure that this is improving quality of products and the life of the participants, not just creating more bureaucracy?
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 6
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 16
Purpose of MeasurementEnsure that you are improving what needs to be improvedDemonstrate that the improvements are useful to the businessDemonstrate that progress is supporting the business
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 17
Demonstrate that progress is supporting the businessDemonstrate that this is an investment, not a cost
Principles Of Measurement 1/2
Specifying the objectives of measurement and analysis such that they are aligned with identified information needs and objectivesSpecifying the measures, data collection and storage mechanisms, analysis techniques, and reporting and feedback mechanisms
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 18
feedback mechanismsImplementing the collection, storage, analysis, and reporting of the dataProviding objective results that can be used in making informed decisions, and taking appropriate corrective actions
Adapted from CMMI® - SEI 2003
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 7
Principles Of Measurement 2/2
Objective planning and estimatingTracking actual performance against established plans and objectivesIdentifying and resolving process-related issues
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 19
Providing a basis for incorporating measurement into additional processes in the future
Adapted from CMMI® - SEI 2003
What to MeasureThere is not that much we can measure:
Timeo Elapsedo Effort
CostQuality (defects)
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 20
Quality (defects)Size
Progress and BaselinesAbsence of measures is the key reason that organizations are ML1: if you knew the cost of quality, you would already have improved itAbsence of baselines is the key reason that ML1 organizations do not measure
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 21
We don't measure, because we have no measurements!The time to establish the baselines is start measuring is now
But where to start?
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 8
Establishing MeasuresThink outside the box:
What do you need?What do you have?How long do you need to establish a baseline?
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 22
Quick BaselinesThink outside the box
Establish a 1-week baseline
Could the productivity of the coming week be representative of your general productivity?
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 23
p y
Measure the TrendStart measuring the trend without a baseline:
What is the change between two projects?What is the different with yesterday or last week?
Y d t d t h
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 24
You do not need to have a baseline in order to determine the difference between two or three points
02468
101214
M.01 M.02 M.03 M.04 M.05 M.06 M.07 M.08 M.09 M.10 M.11
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 9
Reverse BaselineMeasure your previous project to establish a baseline
What happened previously?What were the results?What went wrong?What needs to be changed?
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 25
Estimate the OppositeEstimate the cost of note having done it:
What if you had not identified this risk?How long would it take if you had only identified this requirements defect in acceptance testing, not in the requirements review?What would have been the cost if QA had not identified
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 26
Qthis?
What Do You Have?Consider what you already have
Milestone slippagesDefects reportedTime in testingBudgetO ti
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 27
Overtime
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 10
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 28
CMMI®Align Measurement and Analysis Activities
Establish Measurement ObjectivesSpecify MeasuresSpecify Data Collection and Storage ProceduresSpecify Analysis Procedures
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 29
Provide Measurement ResultsCollect Measurement DataAnalyse Measurement DataStore Data and ResultsCommunicate Results
Adapted from CMMI® - SEI 2003
FAA-ICMMGoals
Measures related to goals, objectives and major issues are establishedMeasurement data are collected, analysed, and results are reportedMeasurement data and results are stored for use
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 30
Some general categories of measures:Schedule and progressResources and costProduct size and stabilityProcess performanceProduct/service qualityContractor performanceTechnology effectiveness
Adapted from FAA-ICMM - Published by the Federal Aviation Administration, 2001.
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 11
Trillium Measurement MapLevel 2
CostsProduct MeasurementProject MeasurementProcess MeasurementProcess and Product FailuresCustomer Satisfaction
Level 3Measurement PlanningProcess MeasurementProduct MeasurementData AnalysisProcess and Product FailuresCustomer Satisfaction
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 31
Level 5Statistical Process Control
Level 4Process MeasurementQuality Management
Adapted from Trillium (Roadmap 3.4 – Measurements) - © Copyright Bell Canada 1994
Trillium Level 2 Measurements 1/2
CostsProject cost factors
Product MeasurementSize, system resources, performanceAdditions, changes and deletions to requirements,
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 32
, g q ,design, code and test specifications Activities for managing requirements (e.g. status, change activity, cumulative number of changes)
Project MeasurementMilestone completion, work completed, effort extendedEffort and resources, change activity for the development plan
Adapted from Trillium (Roadmap 3.4 – Measurements) - © Copyright Bell Canada 1994
Trillium Level 2 Measurements 2/2
Process MeasurementSupplier costs and delivery date compared to planQuality system costs and number of audits compared to planConfiguration management effort and funds, completion of milestones compared to plan
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 33
p p
Process and Product FailuresFailures and statistics throughout the life-cycle of the product
Adapted from Trillium (Roadmap 3.4 – Measurements) - © Copyright Bell Canada 1994
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 12
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 34
The 6-Sigma Approach: DMAICDefine
Understand what are the key processes
MeasureIdentify and collect measurements of success
Analyse
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 35
AnalyseSee what are the issues
ImproveMake useful improvements
ControlManage the process
DMAIC : DefineCharterCustomer needs and requirementsHigh-level process mapAgree on the high-level steps
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 36
g g p
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 13
DMAIC : DefineCharter
Business case Why do we want to change anything? What is the business reason for spending time on this?
Problem statementA detailed description of what is the problemPossibly include a quantitative description of the expected benefits and targetsWhat is the scope of this project hat is incl ded
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 37
Project scope What is the scope of this project, what is included, where are the limits?
Goals and objectives Description of what we hope to achieve through this approach
Milestones Dates and deliverables for each of the main phases
Roles and responsibilities
Who is taking responsibility for this problem? Who is the sponsor? Who are the process improvement team members?
DMAIC : DefineCustomer needs and requirements
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 38
DMAIC : DefineHigh-level process map 1/2
Name the processEstablish the start and stop points of the processDetermine the output(s) of the processDetermine the customer(s) of the processDetermine the supplier(s) of the processD t i th i t( ) f th
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 39
Determine the input(s) of the process
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 14
DMAIC : DefineHigh-level process map 2/2
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 40
DMAIC : MeasureDefine how you are going to measure:
Create a data collection mapCreate data collection forms
Implement your data collection planStart making some statistics:
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 41
Start making some statistics:Calculate baseline sigmaEstablish a basic control chart
DMAIC : MeasureData collection map creation
What to measureType of measureType of dataOperational definitions
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 42
pTargets / specificationsData collection formsSampling
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 15
DMAIC : Measure Data collection plan
Data collection formsDetermine what a defect isDetermine reason codes for defectsDetermine the time frame for data collectionDetermine a grid for data collection
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 43
DMAIC : Measure Calculating Baseline
Standard Sigma Baseline:1000000 * (number of defects / (number of units * number of opportunities))
The Sigma baseline is not a requirement but a possibility
O h d f t t bli h
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 44
Once you have made a few measurements, establish a baselineThis does not have to be based on a statistically representative sample, but it does need to be understood for what it is
DMAIC : AnalyseData Analysis
DiscreteContinuous
Process analysisSub-process mapping
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 45
p pp gNature of work
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 16
DMAIC : AnalyseData Analysis 1/2
DiscreteParetoPie chart
ContinuousFrequency distribution check sheet
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 46
q yo Machines in the processo Materials used in the processo Methods in the processo Nature or environmento The measurement itselfo People
Run chart
DMAIC : AnalyseData Analysis 2/2
8%
16%
13%
11%23%
11%
8%
6% 4%
0
5
10
15
20
25
0 1 2 3 4 5 6 7 8 9 10
Proj
ect S
taff
Initial PlanEstimatedSpent
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 47
0 1 2 3 4 5 6 7 8 9 10
Months
-50
0
50
100
150
200
0 3 6 9 12 15 18 21 24
Months
Cum
ilativ
e K
LOC
Unit Tested (New and Modified) Integrated
DMAIC : AnalyseProcess Analysis
Sub-process mappingNature of work
Sub-process selectiono Customer considers the sub-process importanto There is a physical changeo It is done right the first time
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 48
gNon-value-added activities
o Internal failureo External failureo Delayso Controls and inspectionso Preparation and set upo Moveso Value enabling
Micro-problem statement
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 17
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 49
Define ObjectiveBasic objectives for process improvement:
Cycle time (efficiency)Cycle time slippage (repeatability)Effort to find and fix defectsEffort slip rate (predictability)C t t d d f t ( lit )
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 50
Customer reported defects (quality)Feature slip rate
CharterBusiness case Improving the satisfaction of existing customers will directly
impact repeat business
Problem statement Customers are currently not satisfied because we regularly deliver late
Project scope All projects within the local development centre
Goals and objectives Identify how we can reduce the time to deliver
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 51
Goals and objectives Identify how we can reduce the time to deliver
Milestones
Define:Measure:Analyse:Improve:Control:
1 week from now3 weeks from now4 weeks from now2 months from now6 months from now
Roles and responsibilities
Manager to provide dataMeasurement specialist to provide analysis
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 18
Define Customer NeedsCustomer
Satisfaction
Meet Promises Faster thanCompetition
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 52
Deliver on Time Few Defects
Able to promisewhat we can deliver
Plan reasonabledeadlines Meet deadlines
All Functionality
Define High-Level Process Map
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 53
Define Data Collection MapWhat to measure
Cycle time
Type of measureTime
Type of dataMinutes
Operational definitions
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 54
Operational definitionsTime between moment customer enters the restaurant and first bite into productTime taken to complain and respond is added to value
Targets / specificationsBring total time below 4 minutes
Data collection formsSee attached
SamplingMeasure the time for every 20th customer
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 19
Measure ResultsNr Entered Ordered Served Complaints Total
42 12:15 12:17 12:18 - 3
43 12:25 12:27 12:32 - 7
44 12:35 12:40 12:43 - 8
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 55
45 12:50 12:58 13:05 00:02 17
Average:
Variation
15
20
25
Cus
tom
ers Time to Serve
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 56
0
5
10
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Minutes
Num
ber o
f C
TrendMean = Σx / NStd Deviation = (Σx2 / N)-M2Control Limit = ±(3 SD)
14
16
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 57
0
2
4
6
8
10
12
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 20
Identify Sub-ProcessesSub-Processes:
QueuingChoosingWaiting for selection
o Sandwicho Fries
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 58
o Frieso Drinko Deserts
Paying
Identify the detailed measurements
Now Do ItDefine your improvementsKeep on measuringDemonstrate that you are doing the right thingBecause, if you cannot measure it, you cannot improve
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 59
, y , y pit
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 60
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 21
Measurement ChartDefine what you need to measureDefine what are the added-value elements
Step Value Added Non VA Time
Request bid X 24
Bids are reviewed
Xwait 48
Finalists contacted X 1
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 61
Re-negotiate bid Xerror 24
Try alternative vendor
Xprep. 48
Wait for final submission
Xwait 48
Award contract X 1
Contact vendor X 1
Total 196
Cause and Effect Diagram
Why are there somany waits
Machine Methods Measurement
Drive throughmicrophone
problems
Fryer Failure
Fryer warm-up
Basket holes
Order delays
Pushproduction
ProductionLayout
CommunicatingOrders
Cooking timeaccuracy
Altitude adjustmentin cooking time
Turnover
Lack ofmotivation
SubstandardOil
Food toofrozen
Need tore cook
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 62
Environment People Materials
re-cook
Identify the root causes of problems, classify them into general categoriesSuggested general categories are Machine, Methods, Measurement, Materials, People and Environment, but others are possible
HistogramClassify sequential events, see the “bell-curve”, in which the most frequent case is bound by examples of lesser frequency tapering off in both directions
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 63
0
5
10
15
20
25
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Minutes
Num
ber o
f Cus
tom
ers Time to Serve
SPIN - Darmstadt - 18 Jan 2008
©Q:PIT Ltd 2008 22
Pareto Chart
3.6
3
2.2
no money
forgotten item
misc
Group elements based on frequency of a problem occurring, allows to rapidly identify the biggest issue.
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 64
36.3
22
13.3
11.2
8.4
0 5 10 15 20 25 30 35 40
price check
override
wrong item
register tape
bag
Some Additional ReadingSix-Sigma for Everyone
George Eckes, 2003John Wiley and Sons, EditorISBN 0-471-28156-5
Application of the Indicator Template for Measurement and Analysis
ROI in Low Maturity ©Q:PIT Ltd 2005-2008 65
yWoflhart Goethert and Jeannine Siviy, 2004Software Engineering Measurement and Analysis InitiativeSEI Technical Note CMU/SEI-2004-TN-024
Metrics and Models in Software Quality EngineeringStephen H. Kan 2003Addison Wesley / Pearson EducationISBN 0-201-72915-6
Contact InformationPeter Leeson
Q:PIT Ltd1 Harby CloseEmerson ValleyMilton KeynesMK4 2BBUnited Kingdom
ROI in Low Maturity
United KingdomDirect Line: +44 (0)20 8433 4120Tel / Fax: +44 (0)1 908 506 908Mobile/Cell: +44 (0)773 998 98 67E Mail: [email protected]: http://www.qpit.ltd.uk
©Q:PIT Ltd 2005-2008 66