steve lawrence - agile metrics
DESCRIPTION
Agile metrics can be used to the advantage or the detriment of teams and an organisation’s Agile success. This session looks at several of the core Agile metrics used to measure success to help you understand what success looks like, why the metric is desirable and what the metrics can tell us. Understanding why we want these metrics is critical to capturing something of value, rather than just doing 'because'. What will leaders and decision makers do with these metrics? What value do they add? Steve will also dive into the negative impacts of some of the Agile metrics we are sometimes forced to capture, how chasing velocity leads to gaming the system etc. He’ll look at bad metrics such as the seven deadly sins of Agile measurement and how to avoid them in your enterprise.TRANSCRIPT
Agile MetricsSteve LawrenceSlice Consulting Pty Ltd
Your pic
flickr.com/angelabethell/
STEVELAWRENCE
@goriansteve
A Metrics talk by Steve Lawrence
“ “When done correctly; an economic
framework will shine a bright light into all the dark corners of product
development
Don Reinertsen, The Principles of Product Development FlowVapor trail
Establish an Economic Framework
Our overall goal is to influence economic outcomes
Our most important decisions involve tradeoffs between multiple measures of performance
Why do we want to capture metrics
What is the purpose
How will they be used
Who are they for
Why Metrics?To support decision making processes
Measure Value (Product) or Process
To affirm and reinforce Lean and Agile principles
To measure outcomes
To follow trends, not numbers
Reveal, rather than conceal, context and significant variables
Provide fuel for meaningful conversation
At the current churn rate, 75% of the S&P will be replaced by 2027
If a measurement happens at all, it is because it must have some conceivable effect on decision and behavior. If we can't identify what decisions could be affected by a proposed measurement and how that measurement could change them, then the measurement simply has no value.
“How to Measure Anything” By Douglas W. Hubbard
““
What For?
Support organisational objectives
Optimise learning
Should be simple, big & visible
Well-understood and easily adopted
Guide actions and decisions
“Escape Velocity” By Geoffrey Moore
Bad Metrics
Collected because they always have been
Encourage gaming of the system
Result in bad behavior
Ignore the system
Don’t align with the Why!!
@goriansteve
1. Using metrics as levers2. Using a convenient metric (rather than one
that provides critical insight)
3. Bad analysis4. Motivating people to hide
information5. Too costly measures6. Too many measures (information overload)
7. Too few measures (unbalanced)
The Seven Deadly Sins of Agile Measurement
Larry Maccherone, Rally Software
Types of Metrics
Internal vs. External
Qualitative vs. Quantitative
System Efficiency vs. Local Efficiency
Hypotheses, Experiments, and Little Bets
Some Common Measures
estimatingwe are not so good at
Remember
comparisonwe are good at
ww
w.cp
cstrate
gy.co
m
Productivity Metrics
Value Points Delivered per Time Period
Cost per Value Point
Concept To Cash
Revenue per Employee
Lead Time per Story
Mean Time to Ticket Resolution
SLA Achievement Metrics
Velocity
i-x
J-A
10-1
Stable Teams Creates an almost 2:1 difference in throughput in Teams that are 95% or more dedicated compared to teams that are less than 50% dedicated.
AND60% better productivity40% better predictability60% better responsiveness.
Predictability
Say / Do Ratio
Velocity Variance
Cycle Time per Story Point
Feature Comparison
Epic Comparison
flickr.com/uggboy
Customer Satisfaction
Net Promoter Score Trending
Customer Satisfaction Trending
Repeats / Renewals
flickr.com/amne//
Quality
Maintenance Complexity
Trending
Defect Density
Issue Re-introduction Rate
Defect Arrival / Kill Rate
Unit Test Coverage
Auto Functional Test Coverage
flickr.com/ivyfield/
Employee Satisfaction
Employee Satisfaction Trending (surveys)
Attrition Rates
Pain Scale
Retrospective Outcomes
Responsiveness
Cycle Time per Story
Lead Time per Story
Queue / Batch Size
Average Impediment Lifetime
Mean Time to Release
Mean Time to Fixflickr.com/y500
By themselves Metrics don’t tell the story. Trends Do!!
Velocity - a guide not a target
How Velocity Works
Whoa - What does your velocity look like over the last 3 iterations?
Well over the last 3 iterations we completed 45, 80 and 70 story points.
This iteration we completed 90 story points and next iteration we will do 160
Ok, so what has changed in your team or your work that makes you think you can achieve 160 story points?
Nothing, but to satisfy our customer we have to.
Velocity
poor cadence, unpredictable
Velocity
better cadence, more predictable
Program Velocity?
23 points per sprint
235 points per sprint
Team A
Team B
Examples of BadnessI want the ‘Blue’ team to work on my projects because their velocity is higher
I want to compare the output of people in the Team
We committed to 120 points and completed them all and carried forward 30?
HUH
As a manager I have to constantly drive my teams to ensure they meet the goals ‘we’ set
Burndown Charts
@goriansteve #agileaus
Manage Flow - Watch your WIP
Lean Start Up: Experiments
“Lean Start Up” By Eric Ries
Proposed Solution
The Lean Canvas 30-Jun-2014Iteration #1
Impact Mapping Communications Plan
Problem Conceptual Solution
NotesUnique Value PropositionSingle, clear, compelling message that states why the solution will be different and worth experimentation
Metrics Outcomes
This is a sample text, insert your
own text,
This is a sample text, insert your
own text,
This is a sample text, insert your
own text,
Tasks
This is a sample text, insert your own
text,
Lean Canvas is adapted from The Business Model Canvas (http://www.businessmodelgeneration.com) and is licensed under the Creative Commons Attribution-Share Alike 3.0 Un-ported License.
PRODUCT
Sample text
Sample Text
FeatureSample
Text
Sample Text
FeatureDelta
Anticipated Improvement
Sample Text
Sample text
Sample Text
Deploy / Pivot / RIP
Top features
This is a sample text, insert your
own text,
TasksTasks
TasksTasks
Tasks
Feature
TasksTasks
Tasks
Expected Outcome
Relative Size S/M/L
Learnings
V0.81
WSJF Prioritising
user value + time value + RR | OE
job size
“Images used with permission of Scaled Agile, Inc. See ScaledAgileFramework.com for more information
Feature User Business
Value
Time Criticality
Risk Reduction/Opportuni
ty Enablement
Cost of Delay
Job Size
WSJF
Presentation5 8 6 19 10 1.90
Sales Proposal 10 10 9 29 7 4.14
Remote Training
feedback 3 3 3 9 4 2.25
My Example
Value
Time
In our experience no single sensitivity is more eye-opening than cost of delay
Don Reinertsen, The Principles of Product Development Flow
“
Cost of Delay
“
AGILE EVM
Measures Schedule and Performance – not Value
Forecasts in financial units
Expects everything to be defined up front
No Assertion of Quality
PV, EV, CPI, SPI, ETC, EAC
Capacity Planning as a Metric
Waiting Times more than double as utilisation moves from 80% to 90% and double again as it moves from 90% to 95%
Don Reinertsen, The Principles of Product Development Flow
Control Queues Not Capacity Utilisation
Don Reinertsen, The Principles of Product Development Flow
Waiting on SME’s Longer Cycle Times
Waiting on Sign Offs Increased Risk
Management Reviews More Variability
Big Upfront Analysis More Overhead
Waiting on Releases Lower Quality
Waiting More Administration
Waiting Less Motivation
Waiting Flow On effects
Why do we want to capture metrics What decisions or behaviours do you wish to impact
How will they be usedLever or to facilitate feedback
Who will use the metricsAre the metrics fit for purpose
@goriansteve
Why do we want to capture metrics
What is the purpose
How will they be used
Who are they for
Defect Density
My Hypothesis = 95-100%
Draw 2 parallel lines
At one end of the parallel lines, draw a line at right
angles to them
At the other end of the parallel lines, draw an
inverted V
At one end of the inverted V, draw 2 parallel lines
On the end of the parallel lines you just drew, draw a line at right angles to them
Customer collaboration over contract negotiation
Individuals and interactions over processes and tools
Better Outcomes
Steve LawrenceSlice Consulting Pty Ltd
Your pic
Thanks for listening…