Recommendation Architecture

Jeremy Schiff

RecSys 2014Large Scale Recommender Systems10/10/2014

BEFORE DURING AFTER

DIN

ERS

REST

AURA

NTS

Sharing & Remembering

Understanding& Evolving

Attracting &Planning

Discovery &Convenience

DelightfulDining

OpenTable:Becoming an Experience Company

Proprietary 2

BEFORE DURING AFTER

DIN

ERS

REST

AURA

NTS

Understanding& Evolving

Attracting &Planning

Deliver great experiences at every step, based on who you are

Proprietary 3

OpenTable in Numbers

•Our network connects diners with approximately 32,000 restaurants worldwide.•Our diners have spent more than $25 billion at our partner restaurants.•OpenTable seats more than 15 million diners each month.• Every month, OpenTable diners write more than 400,000 restaurant reviews

4

Recommendations ≠

Collaborative Filtering

5

So what are recommendations?

6

Building Recommendation Systems

• Importance of A/B Testing

• Generating Recommendations

• Recommendation Explanations

• Recommendation Infrastructure

7

What’s the Goal

Minimizing Engineering Time to Improve The Metric that Matters

•Make it Easy to Measure•Make it Easy to Iterate• Reduce Iteration Cycle Times

8

Importance of A/B Testing

• If you don’t measure it, you can’t improve it

•Metrics Drive Behavior

• Continued Forward Progress

9

Pick Your Business Metric

Revenue, Conversions or Satisfaction• OpenTable • Amazon

Engagement• Netflix• Pandora• Spotify

10

Satisfaction Metric

Recommend RestaurantUser Query

User Review

Verify / Refine

Models

11

Measuring & The Iteration Loop

A/B Testing

Weeks

Measure

12

Measuring & The Iteration Loop

Optimize Models

A/B Testing

Days Weeks

Predict Measure

13

Measuring & The Iteration Loop

Analyze & Introspect

Optimize Models

A/B Testing

Hours Days Weeks

Insights Predict Measure

14

Fundamental Differences in Usage

Right now vs. Planning

Search vs. Recommendations

Cost of Being Wrong

15

Recommendation Stack

Query Interpretation

Retrieval

Ranking – Item & Explanation

Index Building

Context for Query & User

Model Building

Explanation Content

Visualization

CollaborativeFilters

Item / User Metadata

16

Query Interpretation & Retrieval

•Get User Intent• Two Solutions

- Spelling Correction- Auto Complete

•One Box, Many Types

- Name- Cuisine

17

Ranking Objectives

Objectives: • Training Error

- RMSE

•Generalization Error- Precision at K

• A/B Metric- Conversion

18

Ranking

Phase 1: Bootstrap through heuristics

Phase 2: Learn to Rank• E [ Revenue | Query, Position, Item, User ]• E [ Engagement | Query, Position, Item,

User ]• Modeling Diversity is Important

19

Modeling Support: Example

20

Modeling Support: Example

21

Modeling Support: Example

22

Modeling Confidence

•Understand Intersection of

- Support of User- Support of Item

•How does support affect variability of prediction?

- 23

Frequency, Sentiment, and Context

• High End Restaurant for Dinner

- High Sentiment, Low Frequency

• Fast, Mediocre Sushi for Lunch

- High Frequency, Moderate Sentiment

24

How to use this data

• Frequency Data: - General: Popularity- Personalized: Implicit CF

• Sentiment Data: - General: Good Experience- Personalized: Explicit CF

• Good Recommendation- Use both to drive your Business Metric

25

Collaborative Filtering Architecture

Hyper-Parameter Tuning

(Many Days)

Predicted Rating

Full Trainer(Many hours)

Incremental Trainer

(A few seconds)Model

(User, Item)

26

Analyzing Review Content

27

This really is a hidden gem and I'm not sure I want to share but I will. :) The owner, Claude, has been here for 47 years and is all about quality, taste, and not overcharging for what he loves. My husband and I don't often get into the city at night, but when we do this is THE place. The Grand Marnier Souffle' is the best I've had in my life - and I have a few years on the life meter. The custard is not over the top and the texture of the entire dessert is superb. This is the only family style French restaurant I'm aware of in SF. It also doesn't charge you an arm and a leg for their excellent quality and that also goes for the wine list. Soup, salad, choice of main (try the lamb shank) and choice of dessert - for around $42 w/o drinks.

“SUPERB!”

Bay Area ReviewsPost Jan 2013

Reviews come in all shapes and sizes!

28

29

The ingredients of a spectaculardining experience…

30

… and a spectacularly bad one

Content Features

Pandora•Music Genome Project

Natural Language Processing• Topics & Tags

31

Generating Topic Features

• Stop Words & Stemming• Bag of Words Model• TF/IDF • Topic Modeling•Describe Restaurants as Topics

32

Stop Words & Stemming

The food was great! I loved the view of the sailboats.

33

Stop Words & Stemming

The food was great! I loved the view of the sailboats.

34

The food was great! I loved the view of the sailboats.

Bag of Words Model

food great chicken sailboat view service

1 1 0 1 1 0

35

TF-IDF

• Term Frequency - Inverse Document Frequency

• Final Value = TF(t) IDF(t)36

TF-IDF Example

food great chicken sailboat view service

.02 0.05 0 0.5 0.25 0

The food was great! I loved the view of the sailboats.

37

Topic Modeling Methods

We applied two main topic modeling methods:• Latent Dirichlet Allocation (LDA)

- (Blei et al. 2003)

•Non-negative Matrix Factorization (NMF)

- (Aurora et al. 2012)38

Topics with NMF using TF-IDFWord 1 Word … Word N

Review 1 0.8 0.9 0

Review … 0.6 0 0.8

Review N 0.9 0 0.8

ReviewsX

Words

ReviewsX

Topics

TopicsX

Words

39

Describing Restaurants as Topics

Each review for a given restaurant has certain topic distribution

Combining them, we identify the top topics for that restaurant.

Topic 01 Topic 02 Topic 03 Topic 04 Topic 05

Topic 01 Topic 02 Topic 03 Topic 04 Topic 05

Topic 01 Topic 02 Topic 03 Topic 04 Topic 05

review 1

review 2

review N

...

Topic 01 Topic 02 Topic 03 Topic 04 Topic 05

Rest

aura

nt

40

Examples of Topics

41

Varying Topic By Region• San Francisco

• `

• London

• Chicago

• New York

42

Compelling Recommendations

43

Recommendation Explanations• Amazon

• Ness

• Netfl

• ix

• Ness - Social

44

Summarizing Content

• Essential for Mobile• Balance Utility With Trust?

- Summarize, but surface raw data

• Example: - Initially, read every review- Later, use average star

rating

45

Summarizing Restaurant Attributes

46

Active Learning for Summarization

• Incremental Supervised Learning• Know Precision & Recall• Always Forward Progress• Generate Dataset: False Positive/Negative or Difficult to

Discriminate

Provide Labels

Train Model

Evaluate AccuracyOn Full Dataset

Generate New

Dataset

47

Devil is in the Details

Attribute Tag – Dim Lighting

“I love the relaxed feel of this place – dark, small, and cozy – like a comfortable living room.”

48

Dish Recommendation•What to try once I have arrived?

49

Edit via the Header & Footer menu in PowerPoint

5050

Infrastructure

User Interactions

Service Logs

51

Infrastructure

User Interactions

Service Logs

Queue

52

Infrastructure

User Interactions

Service LogsBatched Data For Analysis

Real-Time Processing

Queue

53

Infrastructure

User Interactions

Service Logs

Queue

Batched Data For Analysis

Real-Time Processing

Model Training

A/B Testing

Analytics

54

Multi-Datacenter Infrastructure

55

Building Recommendation Systems

• Importance of A/B Testing

• Generating Recommendations

• Recommendation Explanations

• Recommendation Infrastructure

56

Team Composition

Team• Data Scientist

- Math & Applied Machine Learning- Relevancy and Accuracy

• Data Science Engineer- Software Development- Infrastructure, Speed and Maintainability

Everyone works on production systems57

Questions?

Jeremy Schiffjschiff@opentable.com