early stage web product management by dan olsen

Copyright © 2009 YourVersion

Dan OlsenDan OlsenCEO, YourVersionCEO, YourVersionJuly 24, 2009July 24, 2009

Early Stage Web Product Management

http://www.yourversion.com/


What You Wanted to Hear About

1 2 3 4 5 6 7 8 9 10

Usability

Understanding userneeds

Prioritizing features

Translating learningsfrom metrics into

action

Maximizing ROI ondevelopment

resources

Setting up andtracking the right

metrics

UI design


AgendaAgenda

Understanding customer needs

Prioritizing, scoping, and maximizing ROI on engineering resources

UI design

Usability testing

Using metrics for optimization

Will post slides to slideshare.net/dan_o


My BackgroundEducation

BS, Electrical Engineering, NorthwesternMS, Industrial Engineering, Virginia TechMBA, StanfordPHP, MySQL, JavaScript, XHTML, CSS, UI design

18 years of Product Management ExperienceManaged submarine design for 5 years5 years at Intuit, led Quicken Product ManagementLed Product Management at FriendsterOlsen Solutions LLC, PM consultant for startupsCEO & Founder of YourVersion, real‐time discovery startup


For an Early Stage Startup,Which is More Important?

Qualitative Qualitative Information?Information?

Quantitative Quantitative Information?Information?

oror

“When you’re Small, start with Qual!”


How is developing Early Stage Products Different from Later Stage Products?

With an early stage product, you haveWAY MORE uncertainty about:Who your target customers really are

Which customer needs you should address

How to best meet those needs

What design works best

These are qualitative learnings/decisions

Quantitative is also valuable (later)


What is Product Management?


Product Management isCritical Link in Value Creation

Market• Current customers

• Prospectivecustomers

• Competitors

Product Management

Development Team


The Product Manager’s Job:A Successful Product

Be the expert on the market and the customer

Translate business objectives and customer needs into product requirements

Be the clearinghouse for all product ideas

Work with team to design & build great product

Define and track key metrics

Identify, plan & prioritize product ideas to maximize ROI on engineering resources


What’s the Formulafor a Winning Product?

A product that:Meets customers’ needs

Is better than other alternatives

Is easy to use

Has a good value/price


Russians: pencil

NASA: space pen($1 M R&D cost)

Example:Ability to write in space (zero gravity)

Problem Space vs. Solution Space

Problem SpaceA customer problem, need, or benefit that the product should address

A product requirement

Solution SpaceA specific implementation to address the need or product requirement


Problem Space vs. Solution SpaceProduct Level

Problem Space(user benefit)

Solution Space(product)

TurboTax

TaxCut

Pen and paperPrepare

my taxes

File my taxes


Problem Space vs. Solution SpaceFeature Level

Problem Space(user benefit)

Solution Space(feature)

Gmail importerMake it easy

to share a link with my

friends

Allow me to reuse my

email contacts

Design#1

Design#2

Design#3

DesignPreview with checkboxes

User can edit before import

#1 No No

#2 Yes No

#3 Yes Yes


Types of Customer BenefitsFunctional Benefits

Allow the user to do something they couldn’t do beforeDeliver a benefit more quickly, conveniently, or cheaply

Booking travel more easily (travel sites)Talking for less money (VOIP software)

Emotional BenefitsControl (Quicken)Feeling informed (news sites)Confidence / security (anti‐virus software)Enjoyment / entertainment (YouTube)Self‐expression (MySpace, Facebook)


The Customer Benefits “Ladder”

HigherHigher‐‐levellevelbenefitbenefit

(more abstract)(more abstract)

Quicken makes it easy for me Quicken makes it easy for me to balance my checkbookto balance my checkbook

……which gives me a clear picture which gives me a clear picture of how much money I haveof how much money I have

……which makes me feel more in which makes me feel more in control of my financescontrol of my finances

……which means one less thing to which means one less thing to worry about in my hectic lifeworry about in my hectic life

LowerLower‐‐levellevelbenefitbenefit

(more specific)(more specific)


How Do You Prioritize User Benefits and Product Features?

Need a framework for prioritizationWhich user benefits should you address?

Which product features to build (or improve)?

Importance vs. SatisfactionImportance of user need (problem space)

Satisfaction with how well a product meets the user’s need (solution space)

Opportunity =High Importance need with low Satisfaction


What is Your Value Proposition?What is Your Value Proposition?

Which user benefits are you going to provide?Which user benefits are you going to provide?How will your product be better than rest?How will your product be better than rest?

Benefit 1

ImportanceImportanceto Userto User

Current UserCurrent UserSatisfactionSatisfaction

Benefit 2

User BenefitUser BenefitUpsideUpside

PotentialPotential

Benefit 4

Benefit 5

Benefit 6

High

High

Med

Low

Low

Low High – +

Doesn’t matter

Benefit 3 Lowopportunity

opportunity

?

?


Importance vs. SatisfactionAsk Users to Rate for Each Feature

98

8784

8679 847055 80

7280

75

4150

55

60

65

70

75

80

85

90

95

100

40 50 60 70 80 90 100

Satisfaction

Impo

rtan

ce

Recommended reading: “What Customers Want” by Anthony Ulwick

BadBad

GreatGreat


Kano Model: User Needs & Satisfaction

User SatisfactionUser Satisfaction

User DissatisfactionUser Dissatisfaction

Performance (more is better)

Delighter (wow)

NeedNeednot metnot met

NeedNeedfully metfully met

Must Have

Needs & features migrate over time


Is the site up when I want to use it?

Is the site fast enough?

Does the functionality work?

Does the functionality meet my needs?

Olsen’s Hierarchy of Web User Needs(adapted from Maslow)

Customer’s Perspective What does it mean to us?

Uptime

Page Load Time

Absence of Bugs

Feature Set

Usability & Design

Decreasing

Dissatisfaction

Increasing S

atisfaction

How easy to use is it?


Prioritization and Scope

Customer value is only half the equation

How much engineering effort will it take?

Need to consider value and effort (ROI)

Ruthlessly prioritize: rank order

Be deliberate about scope & keep it smallIt’s easy to try to do too much

Strategy = deciding what you’re NOT doing

Break features down into smaller chunks

Smaller scope → faster iterations → better


Have to Prioritize Across Multiple Dimensions At The Same TimeCu

stom

er Value

Custom

er Value

TimeTime

Customer Customer UnderstandingUnderstanding

Functionality

Quality

Ease of Use


Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI

Investment (developer‐weeks)

Return (V

alue

Created

)

Idea C

Idea B

Idea D

Idea A

Idea F

1

1

2

3

4

2 3 4

?


User Benefits vs. Ease of Use

Q: If two products equally deliver the exact same user benefits, which product is better?

A: The product that’s easier to use

‘Ease of use’ provides benefitsSaves time

Reduces cognitive load

Reduces frustration

Not many companies excel at UI design

‘Ease of use’ can be differentiator


The Design Gap at Most Start‐ups

Define Design Code

Product Mgmt Engineering



PM Eng

EngPM

UI

Level

1

2

3

4


The UI Design IcebergThe UI Design Iceberg

VisualDesign

InteractionDesign

InformationArchitecture

ConceptualDesign

Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net

What most people seeand react to

What good PMs and Designers think about


Elements of User Interface Design

Consists of Three Distinct Elements:Information Architecture

Structure and layout at both site and page levelHow site is structured (sitemap)How site information is organized (site layout)How each page is organized (page layout)

Interaction DesignHow user and product interact with one anotherUser flows (e.g., navigation across multiple pages)User input (e.g., controls and form design)

Visual Design“How it looks” vs. “What it is”, often called “chrome”Fonts, colors, graphical elements


Information Architecture

Documents usedSitemap

Show how sections of website are organized

Show major navigation patterns

WireframesShow the layout of components on a page

Does NOT focus on visual designBlack & White

No graphics

Templates for overall website and individual pages

Tools: Visio, OmniGraffle, Axure, Powerpoint, Word, Excel, Photoshop, Balsamiq, WriteMaps, whiteboard


Information ArchitectureSitemap


Wireframe


Napkin Wireframe


Interaction Design

Documents usedFlowchartCombination of Wireframes & Flowcharts

Tools: Visio, OmniGraffle, Powerpoint, Photoshop, whiteboardMay build prototype using HTML, jQuery, Ruby on Rails, Flash, or paperUsability testing can help find problems


Flowchart showing conditional logic


Napkin Flowchart


Combo Wireframe & Flowchart


Visual Design

Documents usedMockups (aka comps)

Tools used: Photoshop, Illustrator, Fireworks

Designer may also deliver HTML & CSS version (with no back‐end code)


Visual DesignMockup


Visual DesignUI Spec


Visual DesignSpecifying Color Palette


The Value of Usability Testing

Critical to talk with customers 1‐on‐1Gain better understanding ofUsability issues with your productCustomer needs and problemsWhat alternatives customers are using,pros & cons of each, customer preferencesQA: use cases & bugs you haven’t seen

Really a “user learning” sessionMake test as real for user as possible


“Ramen” Usability for Startups

Anyone can do it!

Ingredients:1 user with their laptop

1 desk

1 person to conduct usability

Pen and paper

N optional observers

N+2 chairs


Typical Format for Customer Session

5 ‐ 10 min: Ask questions to understand user needs and solutions they currently use

30 ‐ 50 min: Usability testNon‐directed as much as possible

When necessary, direct user to attempt to perform a specific task

5 ‐ 10 min: Wrap‐upAnswer any user questions that came up

Point out/explain features you want to highlight


Dos & Don’ts of Conducting Usability

DoExplain to the user:

Their usability test will help improve the productNot to worry about hurting your feelings“Think Aloud Protocol”

Ask user to attempt the task, then be a fly on the wallAsk non‐leading, open‐ended questionsTake notes and review them afterwards for take‐aways

Don’tAsk leading questions“Help” the user or explain the UI (e.g., “click over here”)Respond to user frustration or questions (until test is over)Get defensiveBlame the user


Usability Case Study: Travel SitesConsider 3 major travel sites

Expedia, Travelocity, OrbitzAll 3 try to provide same user benefits

Easily find flights that match your scheduleEasily find the lowest price flights

UI Design differences make some betterCase study: round trip from San Fran to NYC

Want to find best combo of price and airports3 possible airports for each city9 possible one‐way airport combinations81 possible round‐trip airport combinations


Expedia:Only One Airport Combo at a Time

•• Have to Have to manually manually check all 9 check all 9 comboscombos

••3 clicks to 3 clicks to change airportchange airport

••Then wait for Then wait for new resultsnew results

••24 clicks +24 clicks +8 page reloads 8 page reloads to see all 9 to see all 9 comboscombos


Travelocity:Can Select which Airports to Include


Travelocity:Can only view results 1 combo at a time

•• Clicking button brings up Clicking button brings up results for this comboresults for this combo

••Problem: browser Back Problem: browser Back button loses other button loses other airports!airports!

••Have to go through Have to go through ‘‘Change SearchChange Search’’ process =process =9 clicks + 4 page reloads 9 clicks + 4 page reloads for each combofor each combo

••72 clicks + 32 page loads 72 clicks + 32 page loads to see other 8 combosto see other 8 combos


Orbitz:Binary choices to include nearby airports

•• Good news: all combos Good news: all combos appear in resultsappear in results

••8080‐‐mile radius might mile radius might include airports I doninclude airports I don’’t t want (3 for SFO, 5 for NYC) want (3 for SFO, 5 for NYC) but turned out not to be but turned out not to be an issuean issue


All 3 Travel Sites have Identical“Airline vs. Number of Stops” Price Grids

TravelocityTravelocity

OrbitzOrbitz

ExpediaExpedia


Options for Sorting Flight Results

Website

Can Sort By Expedia Travelocity Orbitz

Airline Y

Departure Time Y Y Y

Arrival Time Y Y

Travel Time Y Y Y

Price Y Y Y


Summary Comparison of Travel Sites

User Benefit Expedia Travelocity Orbitz

Ability to include other nearby airports Yes Yes Yes

Flight Results Sorting Options Med High Low

Ability to pick specific nearby airports

High(by changing)

High(can pre‐select) Low

Ease of trading off airport combos vs. price Low High Med

Ease of seeing results for multiple airports Med Low High

Airline vs. Number of Connections Price Grid Yes Yes Yes

Overall ability to easily find best airport combo Med Low High


And now,And now,for something completely for something completely

differentdifferent……

Quantitative!Quantitative!(finally)(finally)


Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise

Given reality as it exists today,Given reality as it exists today,

optimize our business resultsoptimize our business results

subject to our resource constraints.subject to our resource constraints.


Profit = Profit = RevenueRevenue ‐‐ CostCost

Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor

Impressions/VisitorImpressions/Visitor x Effective CPM / 1000x Effective CPM / 1000

Visits/Visitor x Visits/Visitor x PageviewsPageviews/Visit x Impressions/PV/Visit x Impressions/PV

New VisitorsNew Visitors + Returning Visitors+ Returning Visitors

Invited VisitorsInvited Visitors + Uninvited Visitors+ Uninvited Visitors

# of Users Sending Invites x Invites Sent/User x Invite Conv# of Users Sending Invites x Invites Sent/User x Invite Conversion Rateersion Rate

Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””

Advertising Business Model:


( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conve( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Ratersion Rate

Paying UsersPaying Users x x Revenue per Paying UserRevenue per Paying User

New Paying UsersNew Paying Users + + Repeat Paying UsersRepeat Paying Users

Previous Paying Users x ( 1 Previous Paying Users x ( 1 –– Cancellation Rate )Cancellation Rate )Trial UsersTrial Users x x ConvConv RateRate

Profit = Profit = RevenueRevenue ‐‐ CostCost

Equation of your BusinessEquation of your BusinessSubscription Business ModelSubscription Business Model


How to Track Your MetricsTrack each metric as daily time series

Create ratios from primary metrics: X / YExample: How good is your registration page?Okay: # of registered users per dayBetter: registration conversion rate =

# registered users / # uniques to reg page

DateUnique Visitors

Page views

Ad Revenue

New User Sign‐ups …

4/24/08 10,100 29,600 25 490

4/25/08 10,500 27,100 24 480

…


Sample Signup Page Yield DataSample Signup Page Yield Data

Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10

Dai

ly S

ignu

p Pa

ge Y

ield

Changedmessaging

Added questionsto signup page

Started requiringregistration


Identifying the Identifying the ““Critical FewCritical Few”” MetricsMetrics

What are the metrics for your business?Where is current value for each metric? How many resources to “move” each metric?

Developer‐hours, time, moneyWhich metrics have highest ROI opportunities?

Return

Return

InvestmentInvestment

Return

Return

InvestmentInvestmentRe

turn

Return

InvestmentInvestment

Metric AMetric AGood ROIGood ROI

Metric BMetric BBad ROIBad ROI

Metric CMetric CGreat ROIGreat ROI


Using Web Analytics Tools to Understand Your Users

Tracking visitors and traffic

Seeing where users are clicking

Measuring key conversions

Monitoring user feedback


Basic Tracking of Traffic:Google Analytics

•Unique visitors

•New vs. returning

•Pageviews

•Time on site

•Top referrers

•Top geos


Seeing Where Users are Clicking: CrazyEgg Heatmap

•Shows Click Density

•Color indicates % of clicks

•See which links perform best

•See impact of UI changes


Measuring Key Conversions:Conversion Funnel

•Tie user actions to business goals

•Instrument key steps in user flow

•See where users are dropping off

•Quantify improvement from changes


Monitoring User Feedback Quantitatively with Kampyle

Simple popup

UnobtrusiveSolicitation Average Grade over time

Average Grade for each page on your site


UI questions are never yes/no! (not binary)Should ask: “What percentage of users …?”UI changes impact your metrics

Impact can be positive, negative, small, largeSeek high‐ROI UI changes

Typical UI design question:“When using web pages, do users scroll down?”

‐ Yes‐ No

Approaching UI Design Analytically


The Fold: “Chrome” Takes Away Pixels

768pixels

1024 pixels

Only 600

pixels

Copyright © 2009 YourVersionCopyright Copyright ©© 2009 Olsen Solutions LLC2009 Olsen Solutions LLC

Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold

Landing Page ALanding Page A Landing Page BLanding Page B

Key conversion action is above the fold

Key conversion action is below the fold

The Fold


The Fold IsnThe Fold Isn’’t Binary Eithert Binary Either

600 px

768 px

1024 px

Data courtesy of Data courtesy of ClickTaleClickTaleFree trial at Free trial at www.clicktale.comwww.clicktale.com

% of U

sers The chrome steals

about 170 pixels


Analyze User Screen Height Distributionto Select Design Height for Key Pages

1030, 5%

866, 25%

600, 100%

1024, 20%

1000, 20%

864, 29%

807, 29%

768, 92%

772, 38%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

600 650 700 750 800 850 900 950 1000 1050 1100Monitor Screen Height (pixels)

% o

f U

sers

with

Giv

en (o

r Hig

her)

Scr

een

Hei

ght

Going above768 px = big drop

Page Design Height (pixels)


Marrying UI Design with MetricsThink of “Ease of use”as distinct from the functionalityUser Interface elements matter

PositionLayoutSizeColorFontText copyInteraction designNavigation


Automated Multivariate Testing:Google Website Optimizer


Case Studies:Case Studies:Translating Metrics Into ActionTranslating Metrics Into Action


Case Study 1: Quicken BrokerageCase Study 1: Quicken BrokerageOptimizing Sign In/Registration FlowOptimizing Sign In/Registration Flow

100%

62.3%58.8%

50.9%

34.4% 32.7%

0%

20%

40%

60%

80%

100%

% of U

sers

% of U

sers

Sign in /Sign in /RegistrationRegistration

Account Account TypeType

Cash vs.Cash vs.MarginMargin

5 Partner5 PartnerPagesPages

3 Partner3 PartnerPagesPages

Biggest drop


OpenAccount

Sign in

Account Selection

Register

56%

44%

Forget Password

Registration Process

45% drop off(20% of total)

36% overall drop off for this step

70%(32% of Total)

17% drop off (10% of total)

20% drop off(6% of total)

30%(14% of Total)

80%(26% of Total)

55%(24% of Total)

64%of Total

Mapping the Flow to See WhereMapping the Flow to See WhereUsers Were Dropping OffUsers Were Dropping Off

Change Password

83%(46% of Total)


Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion Rate 37%Registration Conversion Rate 37%

37% improvement in conversion rate

ReleasedNew Design


• Which metric has highest ROI opportunity?

Case Study 2: FriendsterCase Study 2: FriendsterOptimizing Viral GrowthOptimizing Viral Growth

ActiveUsers

Prospective Users

Invite Click

Succeed

Inviteclick-through rate

Conversion rate

Don’t Click

Fail

Invites per sender

% of users sending invites

• Multiplied together, these metrics determine your viral ratio

Users

% of users who are active

= 15% = 2.3

= 85%

Registration Process


The Upside Potential of a Metric

0

100%

Registration Process Yield

0

100%

% of users sending invitations

0

?

Avg # of invites sent per sender

2.3

85%

15%

Max possible improvement

0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%


Okay, so how can we improve the metric?

How do we increase the average number of invites being sent out per sender?

For each idea:What’s the expected benefit? (how much will it improve the metric?)

What’s the expected cost? (how many engineer‐hours will it take?)

You want to identify highest benefit/cost idea


Case Study 2: FriendsterCase Study 2: FriendsterDoubled Number of Invitations Sent per SenderDoubled Number of Invitations Sent per Sender


ContinuousContinuousImprovementImprovement


Adding Metrics and Optimization to your Product Process

PlanPlan

DesignDesign

DevelopDevelop

BusinessObjectives

ProductObjectives

Prioritized Feature List

Scoping

Requirements & Design

Code Test Launch

Site Level

Feature Level

OptimizeOptimize Metrics & User Feedback


Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement

Measurethe metric

Analyzethe metric

Identify top opportunitiesto improve

Design & develop the enhancement

Launch theenhancement

Learning

Gaining knowledge:

• Market

• Customer

• Domain

• Usability


Early Stage Product Management Early Stage Product Management Cheat SheetCheat Sheet

Truly understand your customer needs

Ruthlessly prioritize and launch v1

Talk with users 1‐on‐1 and get feedback

Define equation of your business

Identify and track key metrics

Identify opportunities & prioritize by ROI

Launch, learn, and iterate


Questions?@[email protected] www.yourversion.com

http://www.yourversion.com/

http://www.yourversion.com