evidence based decision-making - lean product development

Evidence-based Decision-making for Lean Product Development

presented byKevin Burns@ MN AEG

Jan. 17, 2017

[email protected], @kevinbburns

Kevin BurnsCoach

Org Change Agent

[email protected], @kevinbburns 2

My work history and experience


Peace Corp Recruitment and Public AffairsStory of how we used technology to improve Peace Corp recruitment.

Switch from USPS to email

Switch from manual data entry to wild-card search in gopher email system and screen scraping results, an early for of ETL.

Conduct direct email campaigns when spam still meant ‘meat in a can’


Open Discussion

Who’s measuring value, outcomes, and/or impacts today?

How are you measuring them?

If you’re not measuring them, why not?

Who’s measuring cost?

How are you measuring cost?


The 1st Agile Principle

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

(should we change valuable to beneficial impact?)

How do we define value (impact) and how do we measure it?

Not all Projects (or Features) are created equal.


Is value determined by delivery on time, on budget, and on scope?

Is the scope delighting the customer?Are they using everything we delivered?


In a survey of 4 products, 65% of the features were rarely or never used.

How much money could have been saved if we never built them?

In the Waterfall project world, we have to ask for everything we can think of because capital will end at the end of the project. Instead we should be asking what has the most value in terms of the business outcome and/or impact and how are we going to measure it.


Marty Cagan Quotes

• Customers don’t know what they want. It’s very hard to envision the solution you want without actually seeing it.

• At least 2/3 of our ideas are never going to work. The other 1/3 will take 3-4 iterations to get right.

• The role of the product manager is to discover a product that is valuable, usable, and feasible. Product, design, and engineering work together to arrive at optimal solution.


What we measure is changing

Business CustomerPO, SM, BL

Software EngineeringAD, DD, DA

UserUX, BA, QA, SME

BusinessValuable

Design Usable

TechnicallyFeasible

INNOVATIVESOLUTION


Lean Startup

• Are we asking what are Minimum Viable (Valuable) Product and how do we know when we’ve delivered it?

• Use a scientific method to measure, learn and pivot or preserver.

• Use meaningful quantitative objective measure to evaluate impact.

• Can you use A/B testing?


MVPInnovation

UserUX, BA, QA, SME

Business Valuable

Design Usable

Software EngineeringAD, DD, DA

Business CustomerPO, SM, BL

Use scientific method (measurable) to learn

and discovery your Minimum Viable

(Valuable) Product (MVP)

Technically Feasible

MVP innovations emerge from Conversations


Value and/or Impact driven culture

• Are we measuring the Cost vs Benefit at all levels of our work items?• Portfolio• Program• Project• Feature/Capability• Story/Requirement• Tasks/Test

• Are we measuring the Impact our features have on our customers?

• The act of sizing helps us define done and value

• Use story telling and test statements create understanding of value and DoD


Don Reinertsen’s Seven Big Ideas2nd Generation Lean Product Development1. Understand your economics2. Manage your queues3. Exploit variability4. Enable smaller batches5. Control WIP and start rates6. Prioritize based on economics7. Accelerate feedback


1. Understand Your Economics

• In product development all difficult decisions involve multiple variables.

• Making decisions that affect multiple variables requires quantification.

• Doing quantification, isn’t as hard as you might think.


A Typical Question

Should we operate our testing process at 80% utilization with a 2-week queue, or 90% utilization with a 4 week queue?


Making Economic Decisions

• Waste• Cycle time• Variability• Efficiency• Revenue• Unit Cost• Value-Added

Proxy Variable Space

Tran

sfor

mat

ions

Life Cycle Profits

Economic Space


Modeling Process

Model Expense Overrun

Model Cost

Overrun

Model Value

Shortfall

Model Schedule

Delay

Model Risk

Change

Create Baseline Model

Determine Total Profit Impact of Missing a MOP

Calculate Sensitivity Factors


Guidelines for decision-making

• Set strategic goals/guidelines for decision-making so low-level tactical decision can be decentralized while still being aligned within strategic goals/objectives.

• While good intentioned, centralized decision-making is often slow and suboptimal because it lacks the context of all the variable at play at tackle level


2. Manage Your Queues

• Many product developers assume higher utilization leads to faster development.

• They neither measure nor manage the invisible queues in their process.

• Consequently, they underestimate the true cost of overloading their processes.

• Such overloads severely hurt all aspects of development performance.


Effect of Capacity Utilization

10 20 30 40 50 60 70 80 90 100

05

1015

20

Que

ue S

ize

% Capacity [email protected], @kevinbburns 26

Managing Queues

Cost of Excess Delay

Total Cost

Dollars

Excess Product Development Resources

Minimize Total Cost to Maximize Profits


Why Queues Matter

• Queues Create…• Longer cycle time• Lower Quality• More variability• Increased risk• More overhead• Less motivation

Managing queues is the key to improving product development economics


3. Exploit Variability

• In manufacturing it is always desirable to reduce variability

• In product development eliminating variability eliminates innovation

• We must understand the specific conditions that make variability valuable and manage our process to create these conditions

• We need development process that function in the presence of variability


Asymmetric Payoff and Option Pricing

Expected Price Payoff vs Price

Expected Payoff

Expe

cted

Pay

off

Price

PricePrice

Prob

abili

ty

Payo

ff Strike Price

Strike Price


Higher Variability Raises the PayoffEx

pect

ed P

ayof

f

Price

Strike Price

Payoff SD = 15

Payoff SD = 5

Option Price = 2, Strike Price = 50, Mean Price – 50, Standard Deviation = 5 and 15


4. Enable Smaller Batches

• When work products are invisible, batch sizes are invisible• When batch sizes are invisible, product developers pay little attention

to them• Many companies institutionalize large batch sizes• Batch size reduction is attractive because it is fast, easy, cheap,

granular, leveraged, and reversible• It is a great starting point for LPD

Batch Size Queues Cycle Time

X 0.5 X 0.5 X [email protected], @kevinbburns

32

Drawing Review Process

200

10 Weeks

20

1 Week

Unreviewed Drawings

Large Batch Small Batch

[email protected], @kevinbburns33

Benefits of Small Batch Testing

Higher ValidityFewer Open Bugs

Faster Cycle Time

Early Feedback

Less Debug Complexity

More Efficient Debugging

More Uptime

Smaller Change

Fewer Status Reports

Less Requirement Changes

Faster Learning

Lower Cost Changes

Cheaper Debugging

Cheaper Testing

Less Non-Value-Added

Better Code

Cheaper Correction

Better Economics


Setting Batch Size

Transaction Cost

Cost

Items per Batch

Economic Batch Size

1 2 3 4 5 6 7 8 9 10

05

1015

20

Total Cost


5. Control WIP and Start Rates

• Many developers incorrectly assume that the sooner they start work, the sooner they will finish it

• They are constantly tempted to start too much work

• This dilutes resources and causes long transit time through their processes

• A long transit time hurts efficiency, quality and responsiveness


Little’s Formula

• By constraining WIP in development processes we can control cycle time

• This approach, which is known as Lean Kanban, is currently growing rapidly in software development

https://en.wikipedia.org/wiki/Little%27s_law

MeanResponseTime = MeanNumberInSystem / MeanThroughput


Control Number of Active Projects

1

2

3

4

1

2

3

4

COD Savings of Project 1 and 2 Late Start Advantages for Project 3 and 4

Time to Deliver

Time to Deliver Time to Deliver


Avoid Long Planning Horizons

• The further out you plan, the less likely your forecast will be accurate

• Don’t do detailed analysis on things beyond a quarter

• Market conditions change everyday, this can change requirements

• Changing requirements cause churn (waste)


Visual WIP Control Boards

Ready Queue Coding Ready to Test Testing Done

WIP constraints = 10

13

14

15

16

11

10

9

8

7

6

5

4

3

2

112

?


6. Sequence Work Correctly

• The sequence in which work is processed is called the queuing discipline

• By changing the queuing discipline we can reduce the cost of a queue without decreasing the size of the queue

• Since manufacturing has homogeneous flows it always uses FIFO (First-In-First-Out)

• For the non-homogeneous flows of product development other approaches have better economics


Use FIFO for Homogeneous Flow

First-In First-Out

Cost of

Delay

12

3A

B

Time

Cost

Delay CostLast-In First-Out

Cost of

Delay 12

3

AB

Time

Cost

Project Duration Cost of Delay

1 3 3

2 3 3

3 3 3


Weighted Shortest Job First (WSJF) for Non-homogenous flow

High Weight First

Cost of

Delay

1

23

AB

Time

Cost

Delay Cost

Low Weight First

Cost of

Delay

A

B

Time

Cost

Project Duration Cost of Delay

Weight = COD/Duration

1 1 10 10

2 3 3 1

3 10 1 0.1

1

23

160 796 % Reduction in COD


7. Create Faster Feedback

• When queues and batch sizes are large feedback is slow

• Slow feedback hurts quality, efficiency, and cycle time

• Feedback speed has enormous economic leverage in product development, but it is rarely explicitly managed


The Front-Loaded Lottery

• A lottery ticket pays $3000 to winning three digit number• You can pick the number in two ways:

• Pay $3 to select all three digits at once• Pay $1 for the first digit, find out if it is correct, then choose if you wish to pay

$1 for the second digit, and then choose if you wish to pay $1 for the third digit.


Value of Feedback

100%

Spend $1

Savings = $0.90Savings = $0.99

10%1%

0 $1 $2 $3

Probabilityof

Occurrence

Cumulative [email protected], @kevinbburns 46

?

[email protected]@kevinbburns612-396-7724


References

• Don Reinertsen• https://www.youtube.com/watch?v=L6v6W7jkwok&spfreload=1#t=16.242347• https://www.amazon.com/gp/product/1935401009/ref=as_li_tl?ie=UTF8&camp

=1789&creative=9325&creativeASIN=1935401009&linkCode=as2&tag=reinertassoci-20&linkId=U56AHKXXQVN4VZFY

• https://en.wikipedia.org/wiki/Little%27s_law
