product management by numbers: using metrics to optimize your product by dan olsen

Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC11

Dan OlsenDan OlsenCEO & Founder, YourVersionCEO & Founder, YourVersionDec 17, 2008Dec 17, 2008

Product Management by Numbers:Product Management by Numbers:Using Metrics to Optimize your ProductUsing Metrics to Optimize your Product

CopyrightCopyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC

Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC

Using Metrics to Optimize your ProductUsing Metrics to Optimize your Product

Product ManagementProduct Management’’s Roles Role

Integrating metrics & optimizationIntegrating metrics & optimization

Using metrics to optimize value creationUsing metrics to optimize value creationBusiness ValueBusiness Value

Customer ValueCustomer Value

UI DesignUI Design

Illustrate with real world examplesIllustrate with real world examples

My BackgroundEducation

BS, Electrical Engineering, NorthwesternMS, Industrial Engineering, Virginia TechMBA, StanfordPHP, MySQL, JavaScript, XHTML, CSS, UI design

17 years of Product Management ExperienceManaged submarine design for 5 years5 years at Intuit, led Quicken Product ManagementLed Product Management at FriendsterOlsen Solutions LLC, PM consultant for start‐upsCEO & Founder of YourVersion, Social Discovery start‐up


Product ManagementProduct Management’’s s RoleRole


A Product Manager by any Other Name Would Smell as Sweet

Product managers are sometimes calledProduct marketing manager

Program manager

Project manager

Label and definition of role can varyBased on industry or company

Based on B2C (consumer) vs. B2B (enterprise)

Based on stage of company

Can be area of responsibility vs. actual position


Product Management isCritical Link in Value Creation

Market• Current customers

• Prospectivecustomers

• Competitors

Product Management

Development Team


A Process View of Product Management

BusinessStrategy

ProductStrategy

BusinessObjectives

ProductObjectives

ProductDevelopment

Service/Support

Market/Sell

LongTerm

ShortTerm

“Inbound”Product

Management“Outbound”

Product Management


PM’s Job: A Successful ProductKnow market and customer better than anyone elseTranslate business objectives and customer needs into product requirementsBe the clearinghouse for all product ideas Identify, plan & prioritize product ideas to maximize ROI on engineering resourcesWork with team to design and build a great productEvangelize product within and outside the companyKeep the product team motivated


Integrating Metrics and Integrating Metrics and Optimization into your Optimization into your

Product ProcessProduct Process


Adding Metrics and Optimization to your Product Process

PlanPlan

DesignDesign

DevelopDevelop

BusinessObjectives

ProductObjectives

Prioritized Feature List

Scoping

Requirements & Design

Code Test Launch

Site Level

Feature Level

OptimizeOptimize Metrics & User Feedback


Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement

Measurethe metric

Analyzethe metric

Identify top opportunitiesto improve

Design & develop the enhancement

Launch theenhancement

Learning

Gaining knowledge:

• Market

• Customer

• Domain

• Usability


OptimizingOptimizingBusiness Value Creation Business Value Creation

with Metricswith Metrics


Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise

Given reality as it exists today,Given reality as it exists today,

optimize our business resultsoptimize our business results

subject to our resource constraints.subject to our resource constraints.

Profit = Profit = RevenueRevenue ‐‐ CostCost

Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor

Impressions/VisitorImpressions/Visitor x Effective CPM / 1000x Effective CPM / 1000

Visits/Visitor x Visits/Visitor x PageviewsPageviews/Visit x Impressions/PV/Visit x Impressions/PV

New VisitorsNew Visitors + Returning Visitors+ Returning Visitors

Invited VisitorsInvited Visitors + Uninvited Visitors+ Uninvited Visitors

# of Users Sending Invites x Invites Sent/User x Invite Clic# of Users Sending Invites x Invites Sent/User x Invite Clickk‐‐through Ratethrough Rate


Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””


How to Track Your MetricsHow to Track Your Metrics

Track each metric as daily time seriesTrack each metric as daily time series

Create ratios from primary metrics: X / YCreate ratios from primary metrics: X / YExample: How good is your registration page?Example: How good is your registration page?Okay:Okay: # of registered users per day# of registered users per dayBetter:Better: registration conversion rate =registration conversion rate =

# registered users / # # registered users / # uniquesuniques to to regreg pagepage

DateDateUnique Unique VisitorsVisitors

Page Page viewsviews

Ad Ad RevenueRevenue

New User New User SignSign‐‐upsups ……

4/24/084/24/08 10,10010,100 29,60029,600 2525 490490

4/25/084/25/08 10,50010,500 27,10027,100 2424 480480

……


Sample Signup Page Yield DataSample Signup Page Yield Data

Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10

Dai

ly S

ignu

p Pa

ge Y

ield

Changedmessaging

Added questionsto signup page

Started requiringregistration


Using Metrics to Optimize the Using Metrics to Optimize the Equation of your BusinessEquation of your Business

What are the metrics for your business?What are the metrics for your business?Where is current value for each metric? Where is current value for each metric? How many resources to How many resources to ““movemove”” each metric?each metric?

DeveloperDeveloper‐‐hours, time, moneyhours, time, moneyWhich metrics have highest ROI opportunities?Which metrics have highest ROI opportunities?

Return

Return

InvestmentInvestment

Return

Return

InvestmentInvestmentRe

turn

Return

InvestmentInvestment

Metric AMetric AGood ROIGood ROI

Metric BMetric BBad ROIBad ROI

Metric CMetric CGreat ROIGreat ROI

• Which metric has highest ROI opportunity?


Case Study:Case Study:Optimizing Viral Loop MetricsOptimizing Viral Loop Metrics

ActiveUsers

Prospective Users

Invite Click

Succeed

Inviteclick-through rate

Conversion rate

Don’t Click

Fail

Invites per sender

% of users sending invites

• Multiplied together, these metrics determine your viral ratio

Users

% of users who are active

= 15% = 2.3

= 85%

Registration Process


Maximum Upside Potential of a MetricMaximum Upside Potential of a Metric

0

100%

Registration conversion rate

0

100%

% of users sending invitations

0

?

Avg # of invites sent per sender

2.3

85%

15%

Max possible improvement

0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%


Doubling Number of Invitations Sent per Sender by Doubling Number of Invitations Sent per Sender by Adding Address Book ImporterAdding Address Book Importer


OptimizingOptimizingCustomer Value Creation Customer Value Creation

with Metricswith Metrics


The Fuzzy Math of Customer ValueThe Fuzzy Math of Customer Value

You create customer value with a product thatYou create customer value with a product thatSatisfies customersSatisfies customers’’ needsneedsIs easy to useIs easy to useHas a good priceHas a good priceIs better than other alternativesIs better than other alternatives

Not fuzzy: # of users and frequency of useNot fuzzy: # of users and frequency of useApplying metrics approach to customer valueApplying metrics approach to customer value

Need framework of user needs & satisfactionNeed framework of user needs & satisfactionQuantitative: site analytics, usage metrics, surveysQuantitative: site analytics, usage metrics, surveysQualitative: 1Qualitative: 1‐‐onon‐‐1 user feedback sessions are best1 user feedback sessions are best


Is the site up when I want to use it?

Is the site fast enough?

Does the functionality work?

Does the functionality meet my needs?

Olsen’s Hierarchy of Web User Needs(adapted from Maslow)

Customer’s Perspective What does it mean to us?

Uptime

Page Load Time

Absence of Bugs

Feature Set

Usability & Design

Decreasing

Dissatisfaction

Increasing S

atisfaction

How easy to use is it?


Kano Model: User Needs & Satisfaction

User SatisfactionUser Satisfaction

User DissatisfactionUser Dissatisfaction

Performance Performance (more is better)(more is better)

Delighter (wow)Delighter (wow)

NeedNeednot metnot met

NeedNeedfully metfully met

Must HaveMust Have

Needs & features Needs & features migrate over timemigrate over time


Importance vs. SatisfactionAsk Users to Rate for Each Feature

98

8784

8679 847055 80

7280

75

4150

55

60

65

70

75

80

85

90

95

100

40 50 60 70 80 90 100

Satisfaction

Impo

rtan

ce

Recommended reading: “What Customers Want” by Anthony Ulwick

BadBad

GreatGreat


What is Your Value Proposition?What is Your Value Proposition?

Which user benefits are most important?Which user benefits are most important?How well does your product deliver?How well does your product deliver?

Benefit 1

ImportanceImportanceto Userto User

Current UserCurrent UserSatisfactionSatisfaction

Benefit 2

User BenefitUser BenefitUpsideUpside

PotentialPotential

Benefit 4

Benefit 5

Benefit 6

High

High

Med

Low

Low

Low High – +

Doesn’t matter

Benefit 3 Lowopportunity

opportunity

?

?


Analyzing Product Ideas by ROIAnalyzing Product Ideas by ROI

Investment (developer-weeks)

Ret

urn

(Val

ue C

reat

ed)

Idea C

Idea B

Idea D

Idea A

Idea F

1

1

2 3 4

2

3

4?


Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI

Investment (developer-weeks)

Ret

urn

(Val

ue C

reat

ed)

Idea B

Idea A

Idea C

1

1

2 3 4

2345678

5


The UI Design IcebergThe UI Design Iceberg

VisualDesign

InteractionDesign

InformationArchitecture

ConceptualDesign

Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net

What most people seeand react to

What good PMs and Designers think about


UI questions are never yes/no! (not binary)UI questions are never yes/no! (not binary)Should ask: Should ask: ““What percentage of users What percentage of users ……??””UI changes impact your metricsUI changes impact your metrics

Impact can be positive, negative, small, largeImpact can be positive, negative, small, largeSeek highSeek high‐‐ROI UI changesROI UI changes

Approaching UI Design AnalyticallyApproaching UI Design Analytically

Typical UI design question:Typical UI design question:““When using web pages, do users scroll down?When using web pages, do users scroll down?””

‐‐ YesYes‐‐ NoNo


Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold

Landing Page ALanding Page A Landing Page BLanding Page B

Key conversion action is above the fold

Key conversion action is below the fold

The Fold

The Fold: “Chrome” Takes Away Pixels

768pixels

1024 pixels

Only 600

pixels


The Fold IsnThe Fold Isn’’t Binary Eithert Binary Either

600 px

768 px

1024 px

Data courtesy of Data courtesy of ClickTaleClickTaleFree trial at Free trial at www.clicktale.comwww.clicktale.com

% of U

sers The chrome steals

about 170 pixels

Analyze User Screen Height Distributionto Select Design Height for Key Pages

1030, 5%

866, 25%

600, 100%

1024, 20%

1000, 20%

864, 29%

807, 29%

768, 92%

772, 38%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

600 650 700 750 800 850 900 950 1000 1050 1100Monitor Screen Height (pixels)

% o

f U

sers

with

Giv

en (o

r Hig

her)

Scr

een

Hei

ght

Going above768 px = big drop

Page Design Height (pixels)


Case Study: Account Signup Process RedesignCase Study: Account Signup Process Redesign

100%

62.3%58.8%

50.9%

34.4% 32.7%

0%

20%

40%

60%

80%

100%

% of U

sers

% of U

sers

Sign in /Sign in /RegistrationRegistration

Account Account TypeType

Cash vs.Cash vs.MarginMargin

5 Partner5 PartnerPagesPages

3 Partner3 PartnerPagesPages

Biggest drop


OpenAccount

Sign in

Account Selection

Register

56%

44%

Forget Password

Registration Process

45% drop off(20% of total)

36% overall drop off for this step

70%(32% of Total)

17% drop off (10% of total)

20% drop off(6% of total)

30%(14% of Total)

80%(26% of Total)

55%(24% of Total)

64%of Total

Analysis of Sign In/Registration FlowAnalysis of Sign In/Registration Flow

Change Password

83%(46% of Total)

Abandonment Rate (7 Day Moving Average)

0%

10%

20%

30%

40%

50%

60%

70%

80%

10/7

/02

10/1

4/02

10/2

1/02

10/2

8/02

11/4

/02

11/1

1/02

11/1

8/02

11/2

5/02

12/2

/02

12/9

/02

12/1

6/02

12/2

3/02

12/3

0/02

1/6/

03

1/13

/03

1/20

/03

Aba

ndon

men

t Rat

e (7

Day

Mov

ing

Ave

rage

)

Steps 1-2


Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion RateRegistration Conversion Rate

37% improvement in conversion rate

ReleasedNew Design


Product Management by NumbersProduct Management by NumbersCheat SheetCheat Sheet

Define what success meansDefine what success meansCustomer value propositionCustomer value proposition

Equation of your businessEquation of your business

Instrument your site and track key metricsInstrument your site and track key metrics

Identify opportunities and prioritize by ROIIdentify opportunities and prioritize by ROI

Launch, learn, and iterateLaunch, learn, and iterate

Discover ShareBookmark

YourVersion proactively discovers relevant web content for you based on your interests.

http://www.yourversion.com/

Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC4040 CopyrightCopyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC

[email protected]@yourversion.comwww.yourversion.comwww.yourversion.com




http://www.yourversion.com

product management by numbers: using metrics to optimize your product by dan olsen

Business

users

time

created

weeks

pixels

developer