cycle time analytics making decisions using lead time and cycle time to avoid needing estimates for...
TRANSCRIPT
Cycle Time AnalyticsMaking decisions using Lead Time and Cycle Time to avoid needing estimates for every
story
Troy Magennis @t_magennis
Slides at bit.ly/agilesim
LKCE 2013 – Modern Management Methods
2 @t_Magennis slides at bit.ly/agilesim
Actual Maximum
Actual Minimum
1
3
2
4
Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
@t_Magennis slides at bit.ly/agilesim
Actual Maximum
Actual Minimum
1
3
2
4
Highest sample
Lowest sample
Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
?
?
?
?
@t_Magennis slides at bit.ly/agilesim
Actual Maximum
Actual Minimum
1
3
2
4
25% chance higher than highest seen
25% lower than highest and higher than second highest
25% higher than lowest and lower than second lowest
25% lower than lowest seen
Highest sample
Lowest sample
Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
A. 50%
% = (1 - (1 / n – 1)) * 100@t_Magennis slides at bit.ly/agilesim
Actual Maximum
Actual Minimum
1
3
2
12
5% chance higher than highest seen
5% lower than lowest seen
Highest sample
Lowest sample
Q. What is the chance of the 12th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
?
?
A. 90%
% = (1 - (1 / n – 1)) * 100
4
5
6
7
8
9
10
11
@t_Magennis slides at bit.ly/agilesim
# Prior Samples Prediction Next Sample Within Prior Sample Range
3 50%4 67%5 75%6 80%7 83%8 86%9 88%
10 89%11 90%12 91%13 92%15 93%17 94%20 95%
@t_Magennis slides at bit.ly/agilesim
8
Four people arrange a restaurant booking after work
Q. What is the chance they arrive on-time to be seated?
@t_Magennis slides at bit.ly/agilesim
Commercial in confidence9
Person 1 Person 2 Person 3 Person 41
in 1
6 EV
ERYO
NE
is O
N-T
IME
15 T
IMES
mor
e lik
ely
at le
ast o
n pe
rson
is la
te
10
Prioritized by Opinion / Mandatory
Team Exists & Fixed in size?
No Estimate!
Date Fixed or New Team?Forecast Team Size/Skillset
Options
Economic Prioritization Used?
Calculated Value / Cost of Delay?
Date Fixed or New Team?
Forecast Team Size/Skillset Options
Team Fixed in size?
Forecast Date, Manage Risks
Foggy Value / Cost of Delay?
Foggy Estimate GUESS!
70% Chance you need to
Forecast(You just performed a probabilistic forecast!
3.5 cases out of 5)
Estimate or #NoEstimate?
16
Forecasts are attempts to answer questions about
future events. They are an estimate within a stated
uncertainty
85% Change of
15th August 2013
Definitely Greater
than $1,000,000
At Least 2 Teams
@t_Magennis slides at bit.ly/agilesim
There is NO single forecast result
Uncertainty In = Uncertainty Out
There will always be manypossible results, some more likely and this is the
key to proper forecasting@t_Magennis slides at bit.ly/agilesim
18
Probabilistic Forecasting combines many uncertain inputs to find many possible outcomes, and what
outcomes are more likely than others
Time to Complete Backlog
50% Possible Outcomes
50% Possible Outcomes
Like
lihoo
d
@t_Magennis slides at bit.ly/agilesim
19
Did the Obama 2012 Campaign Fund Advertising to Achieve 50% Chance of Re-election?
Time to Complete Backlog
85% Possible Outcomes
15%
Like
lihoo
d
@t_Magennis slides at bit.ly/agilesim
20
Task Uncertainty – Summing Variance
Source attribution: Aidan Lyon, Department of Philosophy. University of Maryland, College Park. “Why Normal Distributions Occur” http://aidanlyon.com/sites/default/files/Lyon-normal_distributions.pdf
1 2 3 4
@t_Magennis slides at bit.ly/agilesim
22
Staff
Dev
Cos
t
Cost
of D
elay
Decision Induced Uncertainty
Forecast Completion DateJuly August September October November December
Planned / Due Date
Actual Date
Every choice we make changes the outcome
@t_Magennis slides at bit.ly/agilesim
24
MODELING AND CYCLE TIMEWhat is modelling and how to use cycle time
A model is a tool used to mimic a real world process
Models are tools for low-cost experimentation
@t_Magennis slides at bit.ly/agilesim
Commercial in confidence27
Depth of Forecasting modelsPartitioned Cycle Time
Simulated process
Sim
ple
Dia
gnos
tic
28
Simple Cycle Time Model
Amount of Work(# stories)
Lead Time or Cycle Time
ParallelWork in Proc.(WIP)
Random Chance / Risk / Stupidity @t_Magennis slides at bit.ly/agilesim
Use with attribution29
Capturing Cycle Time and WIPStory Start Date Completed
DateCycle Time
(days)1 1 Jan 2013 15 Jan 20132 5 Jan 2013 12 Jan 20133 5 Jan 20134 6 Jan 20135 3 Jan 2013 7 Jan 20136 7 Jan 2013 18 Feb 20137 10 Jan 2013 22 Jan 20138 10 Jan 2013 18 Jan 20139 13 Jan 2013 26 Jan 2013
10 15 Jan 2013
Date “Complete” – Date “Started”
14
Use with attribution30
Capturing Cycle Time and WIPStory Start Date Completed
DateCycle Time
(days)1 1 Jan 2013 15 Jan 20132 5 Jan 2013 12 Jan 20133 5 Jan 20134 6 Jan 20135 3 Jan 2013 7 Jan 20136 7 Jan 2013 18 Feb 20137 10 Jan 2013 22 Jan 20138 10 Jan 2013 18 Jan 20139 13 Jan 2013 26 Jan 2013
10 15 Jan 2013
Date WIP1 Jan3 Jan4 Jan5 Jan6 Jan7 Jan8 Jan9 Jan
10 Jan…
15 JanCount of Started, but
Not completed
5
Use with attribution31
Capturing Cycle Time and WIPStory Start Date Completed
DateCycle Time
(days)1 1 Jan 2013 15 Jan 2013 142 5 Jan 2013 12 Jan 2013 7 3 5 Jan 20134 6 Jan 20135 3 Jan 2013 7 Jan 2013 46 7 Jan 2013 18 Feb 2013 427 10 Jan 2013 22 Jan 2013 128 10 Jan 2013 18 Jan 2013 89 13 Jan 2013 26 Jan 2013 13
10 15 Jan 2013
Date WIP1 Jan 13 Jan 24 Jan 25 Jan 36 Jan 47 Jan 58 Jan 59 Jan 5
10 Jan 7… …
15 Jan 7
Sum Random Numbers
25112943342631452227
31436545
87
34735448
1912242721
39
202329
187410295Sum:
…..
Historical Story Cycle Time Trend
Days To Complete
Basic Cycle Time Forecast Monte Carlo Process
1. Gather historical story lead-times2. Build a set of random numbers based on pattern3. Sum a random number for each remaining story to build a potential outcome4. Repeat many times to find the likelihood (odds) to build a pattern of likelihood outcomes
𝑇𝑜𝑡𝑎𝑙𝐷𝑎𝑦𝑠❑=𝑆𝑢𝑚(𝑆𝑡𝑜𝑟𝑦𝑛×𝑅𝑎𝑛𝑑𝑜𝑚𝑛)
𝐸𝑓𝑓𝑜𝑟𝑡
More often
Less often
34
Backlog
Feature 1
Feature 2
Feature 3
1. Historical Cycle Time
3. The Work (Backlog)
2. Planned Resources/ WIP
Monte Carlo Analysis = Process to Combine Multiple Uncertain Measurements / Estimates
4. Historical Scope Creep Rate
(optional) 5. Historical Defect Rate and Cycle Times(optional)
6. Phases
@t_Magennis slides at bit.ly/agilesim
35 @t_Magennis slides at bit.ly/agilesim
Commercial in confidence36
How certain based on model
forecast
Further calculations to
make economic tradeoffs
38
Y-Axis: Number of Completed
Stories
Project Complete Likelihood
Range of complete stories probability
0 to 50%
50 to 75%
>75%X-Axis: Date
@t_Magennis slides at bit.ly/agilesim
39
FORECASTING STRATEGIES
Janu
ary
Febr
uary
Mar
ch
April
May
June July
Augus
tSe
ptem
ber
The Future
The Past
1. Model Baseline using historically
known truths
2. Test Model against historically
known truths3. Forecast
When you have historical data
42
Compare Model vs Actual Often
@t_Magennis slides at bit.ly/agilesim
Range of complete probability
Actual results to compare if model is predictable
43
Forecast Trend Report CardWatch GREAT Watch Ok Bad Watch WORRY Watch
Cycle Time Trend versus Model ↘ ↘ ↗ ↗ ↘ ↘ ↗ ↗ACTIVE WIP Actual vs Model ↘ ↗ ↘ ↗ ↘ ↗ ↘ ↗Story Count Trend versus Model ↘ ↘ ↘ ↘ ↗ ↗ ↗ ↗
Arrow Legend: Green: Heading in beneficial direction Red: Heading in damaging direction
For “Watch” – the WIP and story count trends need to be analyzed. They are 1:1 ratio of compensating importance, and may offset each other.
@t_Magennis slides at bit.ly/agilesim
When you have no historical dataJa
nuar
y
Febr
uary
Mar
ch
April
May
June July
Augus
tSe
ptem
ber
The Future
@t_Magennis slides at bit.ly/agilesim
46
Simple Cycle Time Model
Amount of Work(# stories, scope creep, defects)Cycle
Time(range guess or samples)
ParallelWork in Proc.(WIP on board)
@t_Magennis slides at bit.ly/agilesim
48
If we understand how cycle time is statistically distributed, then an
initial guess of maximum allows an inference to be made
Alternatives -
• Borrow a similar project’s data• Borrow industry data• Fake it until you make it… (AKA guess range)
@t_Magennis slides at bit.ly/agilesim
49
Probability Density Function
Histogram Gamma (3P) Lognormal Rayleigh Weibull
x1301201101009080706050403020100-10
f(x)
0.32
0.28
0.24
0.2
0.16
0.12
0.08
0.04
0
Note: Histogram from actual data
@t_Magennis slides at bit.ly/agilesim
Commercial in confidence50
Exponential Distribution (Weibull shape = 1)The person who gets the work can complete the workTeams with no external dependenciesTeams doing repetitive work E.g. DevOps, Database teams,
Commercial in confidence51
Weibull Distribution (shape = 1.5)Typical dev team ranges between 1.2 and 1.8
Commercial in confidence52
Rayleigh Distribution (Weibull shape = 2)Teams with MANY external dependenciesTeams that have many delays and re-work. E.g. Test teams
53
Probability Density Function
Histogram Weibull
x1201101009080706050403020100
f(x)
0.28
0.24
0.2
0.16
0.12
0.08
0.04
0
Scale – How Wide in Range. Related to the
Upper Bound. *Rough* Guess: (High – Low) / 4
Shape – How Fat the distribution. 1.5 is a good starting point.
Location – The Lower Bound
@t_Magennis slides at bit.ly/agilesim
54
What Distribution To Use...
• No Data at All, or Less than < 11 Samples (why 11?)– Uniform Range with Boundaries Guessed (safest)– Weibull Range with Boundaries Guessed (likely)
• 11 to 30 Samples– Uniform Range with Boundaries at 5th and 95th CI– Weibull Range with Boundaries at 5th and 95th CI
• More than 30 Samples– Use historical data as bootstrap reference– Curve Fitting software
@t_Magennis slides at bit.ly/agilesim
55
Questions…• Download the slides (soon) and software at
http://bit.ly/agilesim• Contact me
– Email: [email protected]– Twitter: @t_Magennis
• Read:
Commercial in confidence56
CURRENT FORECASTING PRACTICES
Why traditional forecasting practices fail to deliver and why this is the only outcome possible…
Total Story Lead Time
30 days
Development Time5 Days (~ 15%)
Testing Time 2 Days
Defect Rework 2 Days
Release / DevOps Time 1 Day
Blocked and Waiting Time 9 Days
Waiting Time 3 Days
Waiting Time8 Days
Total Story Lead Time
30 days
Development Time5 Days (~ 15%)
Testing Time 2 Days
Defect Rework 2 Days
Release / DevOps Time 1 Day
Blocked and Waiting Time 9 Days
Waiting Time 3 Days
Waiting Time8 Days
Total Story Lead Time
30 days
Story / Feature Inception5 Days
Waiting in Backlog25 days
System Regression Testing & Staging 5 Days
Waiting for Release Window5 Days
“Active Development”30 days
Pre Work
30 days
Post Work
10 days
Total Story Lead Time
30 days
Story / Feature Inception5 Days
Waiting in Backlog25 days
System Regression Testing & Staging 5 Days
Waiting for Release Window5 Days
“Active Development”30 days
Pre Work
30 days
Post Work
10 days
9 days (70 total)approx 13%
Commercial in confidence63
ESTIMATING DISTRIBUTIONS USING HISTORIC DATA
How to use historic data to estimate the different distributions
Commercial in confidence64
What Distribution To Use...
• No Data at All, or Less than < 11 Samples (why 11?)– Uniform Range with Boundaries Guessed (safest)– Weibull Range with Boundaries Guessed (likely)
• 11 to 50 Samples– Uniform Range with Boundaries at 5th and 95th CI– Weibull Range with Boundaries at 5th and 95th CI– Bootstapping (Random Sampling with Replacement)
• More than 100 Samples– Use historical data at random without replacement– Curve Fitting
Commercial in confidence65
Tools
• EasyFit from http://mathwave.com $499• R http://cran.r-project.org Free• Statistics feature of KanbanSim
http://focusedobjective.com Free• Excel of course• Monte Carlo features of KanbanSim $495-995• Google for Monte Carlo Simulation Tools
– Oracle, Palisade, HubbardResearch (he may be here, so a big recommendation!)
Commercial in confidence66
Sampling at Random Strategies
• If you pick what samples to use, you bias the prediction…
• Strategies for proper random sampling –– Use something you know is random (dice, darts)– Pick two groups using your chosen technique and
compute your prediction separately and compare– Don’t pre-filter to remove “outliers”– Don’t sort the data, in fact randomize more if
possible
Commercial in confidence67
Concurrent WIP Sample : Find the smallest and the biggest or take at least 11 samples to be 90% sure of range
Estimating Concurrent Effort from Cumulative Flow Chart
20% 40%
60% 80%
Commercial in confidence70
Scope Creep Over TimeLook at the rate new scope is added over time
PREDICTION INTERVALS
What is the likelihood the next sample I take will be within the range of previous samples
Actual Maximum
Actual Minimum
1
3
2
Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
4
Actual Maximum
Actual Minimum
1
3
2
4
Highest sample
Lowest sample
Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
?
?
?
?
Actual Maximum
Actual Minimum
1
3
2
4
25% chance higher than highest seen
25% lower than highest and higher than second highest
25% higher than lowest and lower than second lowest
25% lower than lowest seen
Highest sample
Lowest sample
Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
A. 50%
% = (1 - (1 / n – 1)) * 100
Actual Maximum
Actual Minimum
1
3
2
12
5% chance higher than highest seen
5% lower than lowest seen
Highest sample
Lowest sample
Q. What is the chance of the 12th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)
?
?
A. 90%
% = (1 - (1 / n – 1)) * 100
4
5
6
7
8
9
10
11
# Prior Samples Prediction Next Sample Within Prior Sample Range
3 50%4 67%5 75%6 80%7 83%8 86%9 88%
10 89%11 90%12 91%13 92%15 93%17 94%20 95%
Commercial in confidence77
THE SHAPE OF CYCLE TIMEWhat distribution fits cycle time data and why…
Commercial in confidence78
Probability Density Function
Histogram Gamma (3P) Lognormal Rayleigh Weibull
x1301201101009080706050403020100-10
f(x)
0.32
0.28
0.24
0.2
0.16
0.12
0.08
0.04
0
Note: Histogram from actual data
Commercial in confidence79
Probability Density Function
Histogram Weibull
x1301201101009080706050403020100-10
f(x)
0.32
0.28
0.24
0.2
0.16
0.12
0.08
0.04
0
Normal, or it will be after a few
thousand more simulations
Base + 1 Delay
Base + 2 Delays
Base + 3 Delays
Base + 4 Delays
Base + 5 Delays
Commercial in confidence87
ASSESSING AND COMMUNICATING RISK
Speaking Risk To Executives
• Buy them a copy of “Flaw of Averages”• Show them you are tracking & managing risk• Do
– “We are 95% certain of hitting date x”– “With 1 week of analysis, that may drop to date y”– “We identified risk x, y & z that we will track weekly”
• Don’t – Give them a date without likelihood
• “February 29th 2013”
– Give them a date without risk factors considered• “To do the backlog of features, February 29th, 2013”
We spend all our time estimating here
1 2 3
**Major risk events have the predominate role in deciding where deliver actually occurs **
Plan Performance Issues
External Vendor Delay
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Risk likelihood changes constantly
1 2 3
95th Confidence
Interval
Commercial in confidence95
Models Start with Assumptions
• Cycle Time– Guessed– Last releases performance– Another teams performance
• Amount of Work– # Stories guessed– Scope creep guessed– Similar sizing distribution to last release assumed
• Concurrent WIP– Guessed– Measured from last release– Similar process to last release assumed
Commercial in confidence96
Forecast Completion DateJuly August September October November December
Planned / Due Date
Actual Date CCost to Develop
Staff Option A
Staff Option C
Staff Option B
Staff C : $
Staff B : $$
Staff A : $$$$$$$$Actual Date BActual Date A
Commercial in confidence97
Can’t I just use Average Cycle Time?
Forecast (Days)
Median 525Mean 670MC >50% 677MC >85% 793MC >95% 851
Projection of cycle time to complete 25 stories in a sequential fashion based upon historical cycle time data
Story / Feature Inception5 Days
Waiting in Backlog25 days
System Regression Testing & Staging 5 Days
Waiting for Release Window5 Days
“Active Development”30 days
Tota
l Cyc
le T
ime
Story / Feature Inception5 Days
Waiting in Backlog25 days
System Regression Testing & Staging 5 Days
Waiting for Release Window5 Days
“Active Development”30 days
Test
Dev
Le
adTi
me
Commercial in confidence102
Backlog
Feature 1
Feature 2
Feature 3
1. Historical Cycle Time
2. Planned Resources/ Effort
3. The Work (Backlog)
5. Historical Defect Rate & Cycle Times
4. Historical Scope Creep Rate
A Process to Combine Multiple Uncertain Measurements / Estimates is Needed
(optional)(optional)
Design Develop Test
Design Develop Test