Agile Metrics - Lessons Learned from my FitBit

Goodhart’s Law

When a measure becomes a target, it ceases to be a good measure.

Goodhart’s Law

The Danger of Targets and Incentives

• They kill intrinsic motivation

• Achievement of the target starts to supersede the overarching goal

• Can lead to gaming of the system

https://www.youtube.com/watch?v=u6XAPnuFjJc

The Hawthorn Effect

The alteration of behavior by the subjects of a study due to their awareness of being observed.

That which is measured, will improve… but at what cost?

Metrics - Good or Bad?

What is velocity?

Velocity – Measure of a Complex System

The Human Body – A Complex System

Measuring your Weight … a Lagging Indicator

Lean – Optimize the Whole

Focus on the Baton

Percent Utilization

Why should we measure things?

Reduce uncertainty Manage our work Provide transparency Answer questions from stakeholders

Metrics are not for management…

They are for the people doing the work!

Management sets the context and strategic objectives

Teams should decide how they will measure themselves, in alignment with strategic objectives.

Our Strategic Objective

Increase Predictability of Delivery

1. Self-organize into Teams 2. Pick a Team Name 3. Identify three metrics your team will use to align with our Strategic Objective

10 Minutes

Activity

Metric Ideas

• Velocity

• Throughput

• Team Member Happiness

• Team Backlog Health

• Planned / Accepted ratio

• Cycle Time

• Queue Size

• Team Turnover

• Scope Change Rate

• …

Activity

Activity

Scaling Metrics

Scoreboard Index

1. Define a set of Qualitative Metrics aligned to Objectives

2. Define a set of Quantitative Metrics aligned to Objectives

3. For each metric, define a lower and upper bound • Lower Bound = Unhappy

• Upper Bound = Very Happy

4. Collect measurements on a consistent, frequent basis

5. Express every measurement as a point between 0 and 100

6. Calculate an index of the metrics

A Basic Scoreboard Index Example

Team Apple Team Orange Team Pear

Kanban Scrum Scrum

Our Strategic Objective

Increase Predictability of Delivery

Metric Team Apple Team Orange Team Pear

Throughput

Cycle Time

Velocity

Scope Change Rate

Planned vs. Accepted

Team member Turnover

Lower Upper Iteration 1 Iteration 2 Iteration 3

Throughput

Cycle Time

Turnover

0

10

2

12

2

0

10

0

4.5

83.3%

68.8%

100%

84.0%

5

7.2

1

71.3% 35.0%

50.0%

42.2%

11

4.3

0

91.7% 41.7%

100%

87.6%

= (Value – Lower) / (Upper – Lower)

= (10 – 0) / (12 – 0)

Lower Upper Iteration 1 Iteration 2 Iteration 3

Cycle Time

Velocity

Scope Change

10

0

100%

2

16

0.0%

8.8

33%

8

15.0%

50.0%

67.0%

44.0%

9

7

40%

87.5% 43.8%

60.0%

38.8%

4.8

14

10%

65.0% 12.5%

90.0%

80.8%

Lower Upper Iteration 1 Iteration 2 Iteration 3

Throughput

Velocity

Planned/Accepted

0

0

0%

9

22

100%

8

89%

20

88.9%

90.9%

89.0%

89.6%

7

14

75%

100% 63.6%

75.0%

72.1%

10

22

100%

111% 77.8%

100%

103.7%

+ + Iteration 1 Iteration 2 Iteration 3

Team Apple 80.0% 42.9% 86.3%

Team Orange 44.0% 38.8% 80.8%

Team Pear 89.6% 72.1% 103.7%

71.2% 51.3% 90.3%

It’s not about the numbers…

There is a story behind every number Focus on the conversations Rely on a Data Informed

approach not a Data Driven approach. Focus on the trends over time

References

• Doc Norton – Agile Metrics – Velocity is NOT the Goal • https://vimeo.com/97505655

• http://www.intelliberg.com/wp-content/uploads/2013/07/7DeadlySins_ofAgileMeasurement.pdf

Use both Leading and Lagging Indicators

Leading

• Team Member Happiness

• Queue Size

• Team Backlog Health

• Team Turnover

Lagging

• Lead Time

• Cycle Time

• Velocity

• Throughput

Use Both Qualitative & Quantitative Metrics

Qualitative

• Interviews

• Focus Groups

Quantitative

• Velocity

• Cycle Time

• Throughput

• Sprint Burn Up/Down

• Cumulative Flow Diagram

• Measuring percent utilization

• Measuring individual team members

• Measuring Developers based on lines of code

• Measuring based on defects logged

Avoid…

• Going after hard to collect data

• Relying on low quality data

• Relying on only one metric

• Tracking too many metrics

• Applying targets and/or incentives to metrics

Avoid…

http://www.intelliberg.com/wp-content/uploads/2013/07/7DeadlySins_ofAgileMeasurement.pdf