Anatomy Of Relevance: From Data To Action

Saïd Radhouani, Ph.D. radhouani

Yellow Pages, Canada

Agenda

•  Yellow Pages & Myself •  The importance of relevance for our business •  Relevance Building Blocks

•  Measuring and Improving Relevance

•  Conclusion

Who we are

•  Leading media and marketing solutions company in Canada

•  Champion the neighbourhood economy by connecting businesses and consumers

7.3 million Unduplicated unique visitors to our network of websites

6.5 million Downloads of our mobile apps

26% Internet users reached in Canada each month

300 000 Local and national advertisers

Who I am

•  Entrepreneur

•  Ph.D. in Search and Knowledge Management •  IT Director – Content, Search and Relevance

•  Noble cause: leverage data to help build smarter organizations

Agenda

•  Yellow Pages & Myself •  The importance of relevance for our business •  Relevance Building Blocks

•  Measuring and Improving Relevance

•  Conclusion

Local Search

The Importance Of Relevance

Businesses Consumers

Source:  unknown  

Agenda

•  Yellow Pages & Myself •  The importance of relevance for our business •  Relevance Building Blocks

•  Measuring and Improving Relevance

•  Conclusion

User Objective Understand the user intent to maximize their satisfaction Data •  Who the user is (individual or group) •  Where they come from (direct, SEs, …) •  Their context (Location, TOD/DOW/Season, Device, …) •  Their interactions history

KPIs •  Coverage: %tracked user, %tracked user interactions •  Freshness: Daily updates of User profiles & interactions

User  

Content Objective Have accurate and “fresh” content to satisfy your audience needs Data •  Enriched Directory Content (Merchants, Products, Deals, etc.) •  Rich Editorial Content (Neighborhoods, Smart Tips, etc.) •  Third-party content (social networks, etc. ) •  User-Generated Content

KPIs •  Coverage: %Canadian businesses, %Neighborhoods •  Completeness: e.g., for each merchant, have all required info •  Freshness: Frequency of validation & updates Content   User  

Knowledge Objective Help to bridge the gap between user queries and content data Data •  Linguistic (synonyms, acronyms, etc.) •  Rich and “fresh” geo data (location names, polygons, etc.) •  Multilingual

KPIs •  Coverage: %searched entities (keywords, location names, etc.) •  Completeness: e.g., for each location: centroid, polygon,

synonyms, relationships, etc. •  Freshness: Frequency of validation & updates

Content  Knowledge   User  

Search Objective Action-driven, personalized and contextualized relevant search experience KPIs Offline •  Recall & Precision Online •  #returned results •  Mean Reciprocal Rank (MRR) – the multiplicative inverse of

the rank of the first correct answer. Perfect SE should have a MRR of 1

Content  Knowledge  

Search  

User  

Presentation Objective Allow users to access quickly to actionable content through optimal & dynamic search results presentation e.g., “Movie Theater” vs. “Restaurant” KPIs [that really matter] •  Page Views •  Unique Visitors •  Visit Duration •  Bounce Rate •  Return Visits Frequency •  (Useful) Interactions per Visits Content  

Knowledge  

Presenta4on  Search  

User  

Optimization

UserSatisfaction improveRelevance() { While (alive) { Act; Measure; Learn; Grow; } }

Content  Knowledge  

Op4miza4on  

Presenta4on  Search  

User  

Agenda

•  Yellow Pages & Myself •  The importance of relevance for our business •  Relevance Building Blocks

•  Measuring and Improving Relevance

•  Conclusion

A  problem  well-­‐stated  is  half-­‐solved.  

Where To Start?

Charles  Ke2ering  

Data-Driven Actions

•  Implicit User Feedback (analytics & logs)

•  Stable Test Collection

•  Side-By-Side Comparison

•  A/B Testing

•  Explicit User Feedback (Survey)

•  Find gaps in data and improve them

•  Improve processes

•  Improve presentation

•  Improve algorithms

•  Confirm hypothesis

Measure Act

KPIs  

Know Your Data

•  Web & Mobile Analytics

•  Search logs Analysis

•  Build KPIs

From Data To Action

Examples: •  Missing geo-location •  Missing businesses •  Seasonal trends (adapt search algo)

Manual  or  automa8c  ac8ons  

Test Collection KPIs: Recall & Precision •  Predefined test collection built manually

•  Variety of use cases that cover most users needs

•  Evaluate new updates before going to production

Side-By-Side Comparison

•  Comparison of two (versions of the) Search Engines

•  Pre-defined set of real searches selected from logs

•  Pre-classified (keywords, business names, language, geo location, etc.)

•  “Blind” comparison

•  Significant number of testers per search

A/B Testing

Principle •  Experiment one or many ingredients at a

time with real users •  Some non-trivial amount of traffic is

always being tested

KPIs •  Page Views •  Unique Visitors •  Visit Duration •  Bounce Rate •  Return Visits Frequency •  Useful Interactions per Visits

Simulation •  Run new versions on a significant

search logs sample

•  Estimate the impact prior deployment to prod and avoid bad surprises

Relevance In Action

e.g., Bounce Rate – Why the user left quickly? •  Who is the user? •  Where they come from (direct vs. SE) •  Their devices •  Which content was provided? •  Which page? •  etc.

Valuable signals, NOT only for marketing!

Agenda

•  Yellow Pages & Myself •  The importance of relevance for our business •  Relevance Building Blocks

•  Measuring and Improving Relevance

•  Conclusion

Conclusion

Stop coding and make your data talk!

Anatomy Of Relevance: From Data To Action

Saïd Radhouani, Ph.D. radhouani

Yellow Pages, Canada