metrics driven prioritization

Metrics Driven PrioritizationEngineering For Product Success — #QConSF 2014Sam McAfee, Principal - @sammcafee

Innovate or die.

Who is this guy?• Dot-com boom survivor. • Ran a dev shop for 10 years after that. • Agile has always been the “norm”. • Read ‘4 Steps’ before #LeanStartup. • Participated in 3 different startups. All dead. • Ran the Change.org engineering team. • Learned LeanUX from @clevergirl herself. • Now a Principal at Neo Innovation (neo.com)

Tweet every word! @sammcafee #qconsf

Slides available at bit.ly/metricsdriven

“If you build it…”Is anyone even going care about this product?

Nope.

Competing priorities

Everyone loves quarterly planning meetings

A Metrics Driven Approach

Pre-requisitesWhat is product success, anyway?

A successful product solves a painful problem for a specific customer in a

sizable reachable market.

UsableDoes it delight the user?

FeasibleCan we build it?

1

2

ValuableDoes anyone want it?

3

Test Your Assumptions• Use lightweight personas to test the customer segment.

• Use problem interviews to test the problem.

• Use sketches and paper prototypes to test the proposed solution.

• Use landing pages to test the market.

• And do all this before you build anything!

Cross-functionalThe team includes design, engineering, and product.

CollocatedEvery works in the same location.

DedicatedThe team only works on one product at a time.

Assemble the Team

A Metrics Driven Approach

Measure the BusinessUse an economic framework to measure the

impacts of your decisions on core KPIs.

Measure the TeamsMeasure lead time from concept to cash, and use

outliers to set realistic expectations for delivery.

Run ExperimentsYour backlog is a list of experiments you need to run

in order to validate your business assumptions.

Measure the Business

Business Model Canvas

Model Your Funnel

Build an Economic Framework

An Example Product Funnel

• $20 / month subscription.

• 10% conversion from landing page.

• 5% referral rate

• 10% churn rate (or 90% retention rate)

Information Value = Economic impact *

uncertainty.

An Example Product Funnel

• Each visit is worth $2.00

• Each share is worth $0.10

• Customer lifetime value (LTV) is depleted $1 per month, on average, per customer.

Quantify Cost of Delay

Example: Cost of Delay

Measure the Teams

Most delay in product development is wait time.

Measuring Cycle Time

Weighted Shortest Job First

Running Experiments

The Value of Experiments• Each visit is worth $2 (10% of visitors sign up for $20/month).

• Increasing conversion by 50% is worth $1 per user to us.

• We have only a 10% chance of being able to increase it by 50%.

• Historically, only 10% of our experiments on that page succeed.

• $2 * 1.5 * 0.1 * 0.1 = $0.01 = experiment is only worth a penny!!

Prioritizing an experiment requires data from the

funnel, the team, and any past experiments.

Thank you!

@sammcafee

Slides available at bit.ly/metricsdriven