Copyright © 2009 YourVersion
Dan OlsenDan OlsenCEO, YourVersionCEO, YourVersionJuly 24, 2009July 24, 2009
Early Stage Web Product Management
Copyright © 2009 YourVersion
What You Wanted to Hear About
1 2 3 4 5 6 7 8 9 10
Usability
Understanding userneeds
Prioritizing features
Translating learningsfrom metrics into
action
Maximizing ROI ondevelopment
resources
Setting up andtracking the right
metrics
UI design
Copyright © 2009 YourVersion
AgendaAgenda
Understanding customer needs
Prioritizing, scoping, and maximizing ROI on engineering resources
UI design
Usability testing
Using metrics for optimization
Will post slides to slideshare.net/dan_o
Copyright © 2009 YourVersion
My BackgroundEducation
BS, Electrical Engineering, NorthwesternMS, Industrial Engineering, Virginia TechMBA, StanfordPHP, MySQL, JavaScript, XHTML, CSS, UI design
18 years of Product Management ExperienceManaged submarine design for 5 years5 years at Intuit, led Quicken Product ManagementLed Product Management at FriendsterOlsen Solutions LLC, PM consultant for startupsCEO & Founder of YourVersion, real‐time discovery startup
Copyright © 2009 YourVersion
For an Early Stage Startup,Which is More Important?
Qualitative Qualitative Information?Information?
Quantitative Quantitative Information?Information?
oror
“When you’re Small, start with Qual!”
Copyright © 2009 YourVersion
How is developing Early Stage Products Different from Later Stage Products?
With an early stage product, you haveWAY MORE uncertainty about:Who your target customers really are
Which customer needs you should address
How to best meet those needs
What design works best
These are qualitative learnings/decisions
Quantitative is also valuable (later)
Copyright © 2009 YourVersion
What is Product Management?
Copyright © 2009 YourVersion
Product Management isCritical Link in Value Creation
Market• Current customers
• Prospectivecustomers
• Competitors
Product Management
Development Team
Copyright © 2009 YourVersion
The Product Manager’s Job:A Successful Product
Be the expert on the market and the customer
Translate business objectives and customer needs into product requirements
Be the clearinghouse for all product ideas
Work with team to design & build great product
Define and track key metrics
Identify, plan & prioritize product ideas to maximize ROI on engineering resources
Copyright © 2009 YourVersion
What’s the Formulafor a Winning Product?
A product that:Meets customers’ needs
Is better than other alternatives
Is easy to use
Has a good value/price
Copyright © 2009 YourVersion
Russians: pencil
NASA: space pen($1 M R&D cost)
Example:Ability to write in space (zero gravity)
Problem Space vs. Solution Space
Problem SpaceA customer problem, need, or benefit that the product should address
A product requirement
Solution SpaceA specific implementation to address the need or product requirement
Copyright © 2009 YourVersion
Problem Space vs. Solution SpaceProduct Level
Problem Space(user benefit)
Solution Space(product)
TurboTax
TaxCut
Pen and paperPrepare
my taxes
File my taxes
Copyright © 2009 YourVersion
Problem Space vs. Solution SpaceFeature Level
Problem Space(user benefit)
Solution Space(feature)
Gmail importerMake it easy
to share a link with my
friends
Allow me to reuse my
email contacts
Design#1
Design#2
Design#3
DesignPreview with checkboxes
User can edit before import
#1 No No
#2 Yes No
#3 Yes Yes
Copyright © 2009 YourVersion
Types of Customer BenefitsFunctional Benefits
Allow the user to do something they couldn’t do beforeDeliver a benefit more quickly, conveniently, or cheaply
Booking travel more easily (travel sites)Talking for less money (VOIP software)
Emotional BenefitsControl (Quicken)Feeling informed (news sites)Confidence / security (anti‐virus software)Enjoyment / entertainment (YouTube)Self‐expression (MySpace, Facebook)
Copyright © 2009 YourVersion
The Customer Benefits “Ladder”
HigherHigher‐‐levellevelbenefitbenefit
(more abstract)(more abstract)
Quicken makes it easy for me Quicken makes it easy for me to balance my checkbookto balance my checkbook
……which gives me a clear picture which gives me a clear picture of how much money I haveof how much money I have
……which makes me feel more in which makes me feel more in control of my financescontrol of my finances
……which means one less thing to which means one less thing to worry about in my hectic lifeworry about in my hectic life
LowerLower‐‐levellevelbenefitbenefit
(more specific)(more specific)
Copyright © 2009 YourVersion
How Do You Prioritize User Benefits and Product Features?
Need a framework for prioritizationWhich user benefits should you address?
Which product features to build (or improve)?
Importance vs. SatisfactionImportance of user need (problem space)
Satisfaction with how well a product meets the user’s need (solution space)
Opportunity =High Importance need with low Satisfaction
Copyright © 2009 YourVersion
What is Your Value Proposition?What is Your Value Proposition?
Which user benefits are you going to provide?Which user benefits are you going to provide?How will your product be better than rest?How will your product be better than rest?
Benefit 1
ImportanceImportanceto Userto User
Current UserCurrent UserSatisfactionSatisfaction
Benefit 2
User BenefitUser BenefitUpsideUpside
PotentialPotential
Benefit 4
Benefit 5
Benefit 6
High
High
Med
Low
Low
Low High – +
Doesn’t matter
Benefit 3 Lowopportunity
opportunity
?
?
Copyright © 2009 YourVersion
Importance vs. SatisfactionAsk Users to Rate for Each Feature
98
8784
8679 847055 80
7280
75
4150
55
60
65
70
75
80
85
90
95
100
40 50 60 70 80 90 100
Satisfaction
Impo
rtan
ce
Recommended reading: “What Customers Want” by Anthony Ulwick
BadBad
GreatGreat
Copyright © 2009 YourVersion
Kano Model: User Needs & Satisfaction
User SatisfactionUser Satisfaction
User DissatisfactionUser Dissatisfaction
Performance (more is better)
Delighter (wow)
NeedNeednot metnot met
NeedNeedfully metfully met
Must Have
Needs & features migrate over time
Copyright © 2009 YourVersion
Is the site up when I want to use it?
Is the site fast enough?
Does the functionality work?
Does the functionality meet my needs?
Olsen’s Hierarchy of Web User Needs(adapted from Maslow)
Customer’s Perspective What does it mean to us?
Uptime
Page Load Time
Absence of Bugs
Feature Set
Usability & Design
Decreasing
Dissatisfaction
Increasing S
atisfaction
How easy to use is it?
Copyright © 2009 YourVersion
Prioritization and Scope
Customer value is only half the equation
How much engineering effort will it take?
Need to consider value and effort (ROI)
Ruthlessly prioritize: rank order
Be deliberate about scope & keep it smallIt’s easy to try to do too much
Strategy = deciding what you’re NOT doing
Break features down into smaller chunks
Smaller scope → faster iterations → better
Copyright © 2009 YourVersion
Have to Prioritize Across Multiple Dimensions At The Same TimeCu
stom
er Value
Custom
er Value
TimeTime
Customer Customer UnderstandingUnderstanding
Functionality
Quality
Ease of Use
Copyright © 2009 YourVersion
Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI
Investment (developer‐weeks)
Return (V
alue
Created
)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2
3
4
2 3 4
?
Copyright © 2009 YourVersion
User Benefits vs. Ease of Use
Q: If two products equally deliver the exact same user benefits, which product is better?
A: The product that’s easier to use
‘Ease of use’ provides benefitsSaves time
Reduces cognitive load
Reduces frustration
Not many companies excel at UI design
‘Ease of use’ can be differentiator
Copyright © 2009 YourVersion
The Design Gap at Most Start‐ups
Define Design Code
Product Mgmt Engineering
Product Mgmt Engineering
Product Mgmt Engineering
PM Eng
EngPM
UI
Level
1
2
3
4
Copyright © 2009 YourVersion
The UI Design IcebergThe UI Design Iceberg
VisualDesign
InteractionDesign
InformationArchitecture
ConceptualDesign
Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net
What most people seeand react to
What good PMs and Designers think about
Copyright © 2009 YourVersion
Elements of User Interface Design
Consists of Three Distinct Elements:Information Architecture
Structure and layout at both site and page levelHow site is structured (sitemap)How site information is organized (site layout)How each page is organized (page layout)
Interaction DesignHow user and product interact with one anotherUser flows (e.g., navigation across multiple pages)User input (e.g., controls and form design)
Visual Design“How it looks” vs. “What it is”, often called “chrome”Fonts, colors, graphical elements
Copyright © 2009 YourVersion
Information Architecture
Documents usedSitemap
Show how sections of website are organized
Show major navigation patterns
WireframesShow the layout of components on a page
Does NOT focus on visual designBlack & White
No graphics
Templates for overall website and individual pages
Tools: Visio, OmniGraffle, Axure, Powerpoint, Word, Excel, Photoshop, Balsamiq, WriteMaps, whiteboard
Copyright © 2009 YourVersion
Information ArchitectureSitemap
Copyright © 2009 YourVersion
Information ArchitectureSitemap
Copyright © 2009 YourVersion
Wireframe
Copyright © 2009 YourVersion
Napkin Wireframe
Copyright © 2009 YourVersion
Interaction Design
Documents usedFlowchartCombination of Wireframes & Flowcharts
Tools: Visio, OmniGraffle, Powerpoint, Photoshop, whiteboardMay build prototype using HTML, jQuery, Ruby on Rails, Flash, or paperUsability testing can help find problems
Copyright © 2009 YourVersion
Flowchart showing conditional logic
Copyright © 2009 YourVersion
Napkin Flowchart
Copyright © 2009 YourVersion
Combo Wireframe & Flowchart
Copyright © 2009 YourVersion
Visual Design
Documents usedMockups (aka comps)
Tools used: Photoshop, Illustrator, Fireworks
Designer may also deliver HTML & CSS version (with no back‐end code)
Copyright © 2009 YourVersion
Visual DesignMockup
Copyright © 2009 YourVersion
Visual DesignUI Spec
Copyright © 2009 YourVersion
Visual DesignSpecifying Color Palette
Copyright © 2009 YourVersion
The Value of Usability Testing
Critical to talk with customers 1‐on‐1Gain better understanding ofUsability issues with your productCustomer needs and problemsWhat alternatives customers are using,pros & cons of each, customer preferencesQA: use cases & bugs you haven’t seen
Really a “user learning” sessionMake test as real for user as possible
Copyright © 2009 YourVersion
“Ramen” Usability for Startups
Anyone can do it!
Ingredients:1 user with their laptop
1 desk
1 person to conduct usability
Pen and paper
N optional observers
N+2 chairs
Copyright © 2009 YourVersion
Typical Format for Customer Session
5 ‐ 10 min: Ask questions to understand user needs and solutions they currently use
30 ‐ 50 min: Usability testNon‐directed as much as possible
When necessary, direct user to attempt to perform a specific task
5 ‐ 10 min: Wrap‐upAnswer any user questions that came up
Point out/explain features you want to highlight
Copyright © 2009 YourVersion
Dos & Don’ts of Conducting Usability
DoExplain to the user:
Their usability test will help improve the productNot to worry about hurting your feelings“Think Aloud Protocol”
Ask user to attempt the task, then be a fly on the wallAsk non‐leading, open‐ended questionsTake notes and review them afterwards for take‐aways
Don’tAsk leading questions“Help” the user or explain the UI (e.g., “click over here”)Respond to user frustration or questions (until test is over)Get defensiveBlame the user
Copyright © 2009 YourVersion
Usability Case Study: Travel SitesConsider 3 major travel sites
Expedia, Travelocity, OrbitzAll 3 try to provide same user benefits
Easily find flights that match your scheduleEasily find the lowest price flights
UI Design differences make some betterCase study: round trip from San Fran to NYC
Want to find best combo of price and airports3 possible airports for each city9 possible one‐way airport combinations81 possible round‐trip airport combinations
Copyright © 2009 YourVersion
Expedia:Only One Airport Combo at a Time
•• Have to Have to manually manually check all 9 check all 9 comboscombos
••3 clicks to 3 clicks to change airportchange airport
••Then wait for Then wait for new resultsnew results
••24 clicks +24 clicks +8 page reloads 8 page reloads to see all 9 to see all 9 comboscombos
Copyright © 2009 YourVersion
Travelocity:Can Select which Airports to Include
Copyright © 2009 YourVersion
Travelocity:Can only view results 1 combo at a time
•• Clicking button brings up Clicking button brings up results for this comboresults for this combo
••Problem: browser Back Problem: browser Back button loses other button loses other airports!airports!
••Have to go through Have to go through ‘‘Change SearchChange Search’’ process =process =9 clicks + 4 page reloads 9 clicks + 4 page reloads for each combofor each combo
••72 clicks + 32 page loads 72 clicks + 32 page loads to see other 8 combosto see other 8 combos
Copyright © 2009 YourVersion
Orbitz:Binary choices to include nearby airports
•• Good news: all combos Good news: all combos appear in resultsappear in results
••8080‐‐mile radius might mile radius might include airports I doninclude airports I don’’t t want (3 for SFO, 5 for NYC) want (3 for SFO, 5 for NYC) but turned out not to be but turned out not to be an issuean issue
Copyright © 2009 YourVersion
All 3 Travel Sites have Identical“Airline vs. Number of Stops” Price Grids
TravelocityTravelocity
OrbitzOrbitz
ExpediaExpedia
Copyright © 2009 YourVersion
Options for Sorting Flight Results
Website
Can Sort By Expedia Travelocity Orbitz
Airline Y
Departure Time Y Y Y
Arrival Time Y Y
Travel Time Y Y Y
Price Y Y Y
Copyright © 2009 YourVersion
Summary Comparison of Travel Sites
User Benefit Expedia Travelocity Orbitz
Ability to include other nearby airports Yes Yes Yes
Flight Results Sorting Options Med High Low
Ability to pick specific nearby airports
High(by changing)
High(can pre‐select) Low
Ease of trading off airport combos vs. price Low High Med
Ease of seeing results for multiple airports Med Low High
Airline vs. Number of Connections Price Grid Yes Yes Yes
Overall ability to easily find best airport combo Med Low High
Copyright © 2009 YourVersion
And now,And now,for something completely for something completely
differentdifferent……
Quantitative!Quantitative!(finally)(finally)
Copyright © 2009 YourVersion
Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise
Given reality as it exists today,Given reality as it exists today,
optimize our business resultsoptimize our business results
subject to our resource constraints.subject to our resource constraints.
Copyright © 2009 YourVersion
Profit = Profit = RevenueRevenue ‐‐ CostCost
Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor
Impressions/VisitorImpressions/Visitor x Effective CPM / 1000x Effective CPM / 1000
Visits/Visitor x Visits/Visitor x PageviewsPageviews/Visit x Impressions/PV/Visit x Impressions/PV
New VisitorsNew Visitors + Returning Visitors+ Returning Visitors
Invited VisitorsInvited Visitors + Uninvited Visitors+ Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Conv# of Users Sending Invites x Invites Sent/User x Invite Conversion Rateersion Rate
Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””
Advertising Business Model:
Copyright © 2009 YourVersion
( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conve( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Ratersion Rate
Paying UsersPaying Users x x Revenue per Paying UserRevenue per Paying User
New Paying UsersNew Paying Users + + Repeat Paying UsersRepeat Paying Users
Previous Paying Users x ( 1 Previous Paying Users x ( 1 –– Cancellation Rate )Cancellation Rate )Trial UsersTrial Users x x ConvConv RateRate
Profit = Profit = RevenueRevenue ‐‐ CostCost
Equation of your BusinessEquation of your BusinessSubscription Business ModelSubscription Business Model
Copyright © 2009 YourVersion
How to Track Your MetricsTrack each metric as daily time series
Create ratios from primary metrics: X / YExample: How good is your registration page?Okay: # of registered users per dayBetter: registration conversion rate =
# registered users / # uniques to reg page
DateUnique Visitors
Page views
Ad Revenue
New User Sign‐ups …
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…
Copyright © 2009 YourVersion
Sample Signup Page Yield DataSample Signup Page Yield Data
Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10
Dai
ly S
ignu
p Pa
ge Y
ield
Changedmessaging
Added questionsto signup page
Started requiringregistration
Copyright © 2009 YourVersion
Identifying the Identifying the ““Critical FewCritical Few”” MetricsMetrics
What are the metrics for your business?Where is current value for each metric? How many resources to “move” each metric?
Developer‐hours, time, moneyWhich metrics have highest ROI opportunities?
Return
Return
InvestmentInvestment
Return
Return
InvestmentInvestmentRe
turn
Return
InvestmentInvestment
Metric AMetric AGood ROIGood ROI
Metric BMetric BBad ROIBad ROI
Metric CMetric CGreat ROIGreat ROI
Copyright © 2009 YourVersion
Using Web Analytics Tools to Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Measuring key conversions
Monitoring user feedback
Copyright © 2009 YourVersion
Basic Tracking of Traffic:Google Analytics
•Unique visitors
•New vs. returning
•Pageviews
•Time on site
•Top referrers
•Top geos
Copyright © 2009 YourVersion
Seeing Where Users are Clicking: CrazyEgg Heatmap
•Shows Click Density
•Color indicates % of clicks
•See which links perform best
•See impact of UI changes
Copyright © 2009 YourVersion
Measuring Key Conversions:Conversion Funnel
•Tie user actions to business goals
•Instrument key steps in user flow
•See where users are dropping off
•Quantify improvement from changes
Copyright © 2009 YourVersion
Monitoring User Feedback Quantitatively with Kampyle
Simple popup
UnobtrusiveSolicitation Average Grade over time
Average Grade for each page on your site
Copyright © 2009 YourVersion
UI questions are never yes/no! (not binary)Should ask: “What percentage of users …?”UI changes impact your metrics
Impact can be positive, negative, small, largeSeek high‐ROI UI changes
Typical UI design question:“When using web pages, do users scroll down?”
‐ Yes‐ No
Approaching UI Design Analytically
Copyright © 2009 YourVersion
The Fold: “Chrome” Takes Away Pixels
768pixels
1024 pixels
Only 600
pixels
Copyright © 2009 YourVersionCopyright Copyright ©© 2009 Olsen Solutions LLC2009 Olsen Solutions LLC
Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold
Landing Page ALanding Page A Landing Page BLanding Page B
Key conversion action is above the fold
Key conversion action is below the fold
The Fold
Copyright © 2009 YourVersion
The Fold IsnThe Fold Isn’’t Binary Eithert Binary Either
600 px
768 px
1024 px
Data courtesy of Data courtesy of ClickTaleClickTaleFree trial at Free trial at www.clicktale.comwww.clicktale.com
% of U
sers The chrome steals
about 170 pixels
Copyright © 2009 YourVersion
Analyze User Screen Height Distributionto Select Design Height for Key Pages
1030, 5%
866, 25%
600, 100%
1024, 20%
1000, 20%
864, 29%
807, 29%
768, 92%
772, 38%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
600 650 700 750 800 850 900 950 1000 1050 1100Monitor Screen Height (pixels)
% o
f U
sers
with
Giv
en (o
r Hig
her)
Scr
een
Hei
ght
Going above768 px = big drop
Page Design Height (pixels)
Copyright © 2009 YourVersion
Marrying UI Design with MetricsThink of “Ease of use”as distinct from the functionalityUser Interface elements matter
PositionLayoutSizeColorFontText copyInteraction designNavigation
Copyright © 2009 YourVersion
Automated Multivariate Testing:Google Website Optimizer
Copyright © 2009 YourVersion
Case Studies:Case Studies:Translating Metrics Into ActionTranslating Metrics Into Action
Copyright © 2009 YourVersion
Case Study 1: Quicken BrokerageCase Study 1: Quicken BrokerageOptimizing Sign In/Registration FlowOptimizing Sign In/Registration Flow
100%
62.3%58.8%
50.9%
34.4% 32.7%
0%
20%
40%
60%
80%
100%
% of U
sers
% of U
sers
Sign in /Sign in /RegistrationRegistration
Account Account TypeType
Cash vs.Cash vs.MarginMargin
5 Partner5 PartnerPagesPages
3 Partner3 PartnerPagesPages
Biggest drop
Copyright © 2009 YourVersion
OpenAccount
Sign in
Account Selection
Register
56%
44%
Forget Password
Registration Process
45% drop off(20% of total)
36% overall drop off for this step
70%(32% of Total)
17% drop off (10% of total)
20% drop off(6% of total)
30%(14% of Total)
80%(26% of Total)
55%(24% of Total)
64%of Total
Mapping the Flow to See WhereMapping the Flow to See WhereUsers Were Dropping OffUsers Were Dropping Off
Change Password
83%(46% of Total)
Copyright © 2009 YourVersion
Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion Rate 37%Registration Conversion Rate 37%
37% improvement in conversion rate
ReleasedNew Design
Copyright © 2009 YourVersion
• Which metric has highest ROI opportunity?
Case Study 2: FriendsterCase Study 2: FriendsterOptimizing Viral GrowthOptimizing Viral Growth
ActiveUsers
Prospective Users
Invite Click
Succeed
Inviteclick-through rate
Conversion rate
Don’t Click
Fail
Invites per sender
% of users sending invites
• Multiplied together, these metrics determine your viral ratio
Users
% of users who are active
= 15% = 2.3
= 85%
Registration Process
Copyright © 2009 YourVersion
The Upside Potential of a Metric
0
100%
Registration Process Yield
0
100%
% of users sending invitations
0
?
Avg # of invites sent per sender
2.3
85%
15%
Max possible improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2009 YourVersion
Okay, so how can we improve the metric?
How do we increase the average number of invites being sent out per sender?
For each idea:What’s the expected benefit? (how much will it improve the metric?)
What’s the expected cost? (how many engineer‐hours will it take?)
You want to identify highest benefit/cost idea
Copyright © 2009 YourVersion
Case Study 2: FriendsterCase Study 2: FriendsterDoubled Number of Invitations Sent per SenderDoubled Number of Invitations Sent per Sender
Copyright © 2009 YourVersion
ContinuousContinuousImprovementImprovement
Copyright © 2009 YourVersion
Adding Metrics and Optimization to your Product Process
PlanPlan
DesignDesign
DevelopDevelop
BusinessObjectives
ProductObjectives
Prioritized Feature List
Scoping
Requirements & Design
Code Test Launch
Site Level
Feature Level
OptimizeOptimize Metrics & User Feedback
Copyright © 2009 YourVersion
Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement
Measurethe metric
Analyzethe metric
Identify top opportunitiesto improve
Design & develop the enhancement
Launch theenhancement
Learning
Gaining knowledge:
• Market
• Customer
• Domain
• Usability
Copyright © 2009 YourVersion
Early Stage Product Management Early Stage Product Management Cheat SheetCheat Sheet
Truly understand your customer needs
Ruthlessly prioritize and launch v1
Talk with users 1‐on‐1 and get feedback
Define equation of your business
Identify and track key metrics
Identify opportunities & prioritize by ROI
Launch, learn, and iterate
Copyright © 2009 YourVersion
Questions?@[email protected] www.yourversion.com