[@indeedeng] data-driven product design
DESCRIPTION
@IndeedEnd April: Wednesday, April 24, 2013 Video available: http://youtu.be/Y6NI8y21xhg Advertising pioneer John Wanamaker famously said, "Half the money I spend on advertising is wasted; the trouble is I don't know which half." At Indeed, our experience shows that more than half of product features are not only wasted but actually have a negative impact on user experience. Thanks to rigorous A/B testing we know precisely which half, so we can ensure that every feature we add delivers measurable value. With 100 million monthly unique visitors and 3 billion monthly searches, our data provides empirical answers to (nearly) all of our product questions. In this talk, we will cover: * a motivation for why we A/B test everything * the tools and systems we have developed to measure and analyze test results and the key metrics we focus on, and * multiple case studies illustrating how years of testing has shaped Indeed's design and product thinking. We will include specific examples of how small, overlooked changes have impacted our metrics in significant and unexpected ways, and how implementing simple "no-brainer" updates on our site decreased key business metrics and caused us to re-evaluate our solutions. This talk is for designers, product managers, and engineers from companies of any size who want to understand how data-driven design makes better products. Speakers J Christopher Garcia is the User Experience Lead at Indeed. Chris ensures Indeed's products are easy to use for millions of people all over the world. He has focused on user experience at Indeed for over 6 years. As the company has grown from start-up underdog to the world's leading job search engine, he has contributed to every major product Indeed offers. Chris Hyams is SVP of Product & Engineering at Indeed. Prior to Indeed, Chris was founder of B-Side, an Austin-based company that provided digital distribution, marketing, and analytics technology for the film industry. Prior to B-Side, he was VP of Engineering at Trilogy Software.TRANSCRIPT
Data-DrivenProduct Design
J Christopher GarciaChris Hyams
DesignJ Christopher Garcia
User Experience Lead, Indeed
2008
2013
114 tests244 variations
84,000 combinations
Direct observation can be inefficient
Before and afterdoesn't work
Traffic doesn't look like this
Hourly traffic looks like this...
...and daily traffic looks like this
...and monthly traffic looks like this
Local maximum
What we've learnedUI Principles
Clickable items should look clickable
Users read
a word is worth a thousand icons
+14%more search results pages
BEWARE MULTIPLE CHANGES
Searches: -2.0%Job Clicks: -1.7%Saved jobs: -7.6%
Searches: -1.8%Job Clicks: -0.8%Saved jobs: 0%
Really?
Really.
ProductChris Hyams
SVP, Product & Engineering
I help people get jobs.
What's best for the job seeker?
How do we know what's best for job seekers?
Test and measure EVERYTHING
Measurement
1. Log everything
jobsearch indexabredistimeacmetimeaddltimeadscadsdelayadsibadscbadsiboostojcboostojibsjcbsjcwiabsjibsjindappliesbsjindappviewsbsjrevbsjwiackcntckszcountsctkagectkagedaysdayofweekdcpingtimedomTotalTimeds-mpo
dsmissdstimefeatempfjfreekwacfreekwarevfreesjcfreesjrevfrmtimegalatdelayiplatiplongjslatdelayjsvdelaykwackwacdelaykwaikwarevkwcntlacinsizelacsgsizelmstimempotimemprtimenavTotTimendxtime
ojcojclongojcshortojcwiaojiojindappliesojindappviewsojwiaoocscpageprcvdlatencyprimfollowcntprvwojiprvwojlatprvwojopentimeprvwojreqradscradsirecidlookupbudgetrectimeredirCountredirTimerelfollowcntrespTimereturnvisitrojc
rojirqcntrqlcntrqqcntrrsjcrrsjirrsjrevrsavailrsjcrsjirsusedrsviableserpsizesjcsjcdelaysjclongsjcntsjcshortsjcwiasjisjindappliessjindappviewssjrevsjwiasllatsllong
sqcsqisugtimesvjsvjnostarsvjstartadsctadsitimetimeofdaytotcnttotfollowcnttotrevtottimetsjctsjcwiatsjitsjindappliestsjindappviewstsjrevtsjwiaunqcntvpwacinsizewacsgsize
acmepageacmereviewmodacmeserviceacmesessionadclickadcrequestadcrevadschanneladsclickadsenseclickadveadvtagghttpaggjiraaggjobaggjob_waldorfaggsherlockaggsourcehealthagstimingapiapijsvapisearcharchiveindexarchiveindex_shingled_testbincarclicksclickclickanalyticscobranddctmismatchdrawdupepairsdupepairs_minidupepairs_olddupepairsalldupepairsall_miniejcheckeremilyops
feedbridgeglobalnavgooglebot_organichomepageimpressionindeedapplyjhstjobalertjobalertorganicjobalertsearchjobalertsponsoredjobexpirationjobexpiration2jobexpiration3jobprocessedjobqueueblockjobsearchjssquerykeywordAdlocsvclucyindexermainmechanicalturkmindyopsmobhomepagemobilmobilemobileorganicmobilesponsoredmobrecjobsmobsearchmobviewjobmyindeedmyindfunnelmyindpagemyindrezcreatemyindsessionoldopsesjasx
organicorgmodelorgmodelsubsetorgmodelsubset90passportaccountpassportpagepassportsigninramsaccessrecjobsrecommendserviceresumedataresumesearchrexcontactsrexfunnelreximpressionrexsearchrezSrchSearchrezalertrezalertfunnelrezfunnelrezjserrrezsrchrequestrezviewsearchablejobsseosessionsjmodelsponsoredsysadappinfosysadapptimingtestndxtestndx1testndx2tmpusrsvccacheusrsvcrequestviewjobwebusersignin
16 million jobs100 million UV
3 billion searches
1TB per day
Everything you ever wanted to know about logging but were afraid to ask
Logrepo: Enabling Data-Driven Decisions
2. Index data
3. Build tools toquery indexes
[another tech talk]
Tools answer questions
Mobile searches per hour inJP vs. UK?
Resume creation by country?
Impressions in job alerts byemail domain?
Percent of app downloads fromiOS, Android, Windows?
How quickly does a datacenter take on traffic after a failover?
Test EVERYTHING
TextVisual designLayoutControl flowNew featuresAlgorithms [also another tech talk]
Everything
5% 5%grp:bluebtntst0 grp:bluebtntst1
split:
log:
Measure
statistical relevance
p-value
Big changes require less data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
source: http://go.indeed.com/jyusg
Small changes require more data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
Product Principles
There's no such thing as a no-brainer
-30%
Corollary: if a test fails,it means the test failed
-30%
+25%
Little things can makea big difference
organic
sponsored
+34%
Corollary: be skepticalof big changes
Does color really matter that much?
a A
a B
a C +34%
+10%
Are people just boredof blue?
+29% +22%
Will the effect wear off?
Get it right,then do it right
Location backfill: job clicks
+8%
Location backfill: search time
+1,500%
Location backfill: performance fix
+1,500%
Location backfill: performance fix
+1,500% +0%
Location backfill: job clicks
+8%
Location backfill: clicks after fix
+8% +9%
Measure everything(aka, Beware of unintended consequences)
April Fools' 2010
Organic clicks
+1.8%
Job alert signups
-17%
Revenue
+15.5%
Revenue
+15.5%
Job alert signups: top v. bottom
-17% +6%-21%
Corollary: attention is a zero-sum game
(usually)
SearchClickRevenueEmail signupResume create...
Relative value of actions
Different for everyone
Location auto-complete
Zero-results page
Zero-results pages
-2.7%
Organic clicks
+0%
Clicks w/ auto-complete
+8%
Keyword ad revenue
+1,410%
Keyword ad revenue
+1,410%
keyword ads
Keyword ad revenue after fix
Measure everything,every time
Getting Started
Fewer tests
Focus on big wins
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
Straw man
Determine # of trials in a weekDetermine minimum improvementOne new test each weekIf success, roll out; if not, kill & move on
What We Get
Quality
Risk tolerance
Culture of measurement
(Empirically)better products
fin