personalizing search at linkedin

Recruiting SolutionsRecruiting SolutionsRecruiting Solutions

Ganesh Venkataraman Viet Ha-Thuc

Personalizing Search @ LinkedIn

Bigger Picture

▪ LinkedIn’s vision– Create economic opportunity for every member of the

global workforce▪ Connect members to other members, knowledge and

opportunity and help them be great at what they do

Economic Graph

▪ Organize people, companies, jobs, knowledge and map out the economic graph

3

Role of Search

▪ At the heart of the economic graph, search makes the economic graph accessible, useful and actionable

▪ Powers searching people, jobs, companies, schools etc. ▪ On linkedin.com consumer, recruiter, sales solutions

4

Powered by Search

5

Basic Nomenclature

6

TypeAhead/TYAH Full Search/SERP

Search is ...

7

8

Search is about understanding the user intent

9

LinkedIn Search - An Overview

10

Query Processing

Retrieval

Ranking

Federated Page Construction

Search Assist● Instant Results● Guided suggestions● Autocomplete

suggestions

Entity View/Action

Let’s talk intent - Navigational

▪ Navigational - exactly one result in mind

11

Two types of Intent - Exploratory

▪ Exploratory - Typically more than one entity in mind

12

How to handle navigational queries?

Be Fast

Type Less

Be Lenient

13

Handling Navigational Queries

▪ Type Less– Index prefixes (‘ga’, ‘gan’, ‘gane’ => ‘ganesh’)

▪ Be Fast– Do not retrieve all documents– Order documents in posting list by static rank – Modify query for targeted retrieval

▪ Be Lenient– Smart spell correction

14

Exploratory Queries

▪ If possible guide users to more structured queries▪ Above query could go into different verticals if these are selected▪ User intent becomes much clearer

15

Exploratory Queries

16

Unclear intent - Federating TYAH results

17

LinkedIn Search - Bird’s eye view

18

Query Processing

Retrieval

Ranking

Federated Page Construction

Search Assist● Instant Results● Guided suggestions● Autocomplete

suggestions

Entity View/Action

Query Processing - things not strings

1919

TITLE CO GEO

TITLE-237software engineersoftware developer

programmer…

CO-1441Google Inc.

Industry: Internet

GEO-7583Country: US

Lat: 42.3482 NLong: 75.1890 W

(RECOGNIZED TAGS: NAME, TITLE, COMPANY, SCHOOL, GEO, SKILL )

Retrieval

▪ Custom search engine to handle 100’s of millions of documents (Galene)

▪ Key Features:– Offline indexing pipeline– Supports live updates with fine granularity– Static Ranking

▪ Posting list organized by static rank for each document

▪ Enables early termination20

LinkedIn Search - Bird’s eye view

21

Query Processing

Retrieval

Ranking

Federated Page Construction

Search Assist● Instant Results● Guided suggestions● Autocomplete

suggestions

Entity View/Action

Ranking

▪ Manually tuning vs. Learning to Rank (LTR)

▪ Why Learning to Rank?– Hard to manually tune with very large number of features– Challenging to personalize– LTR allows leveraging large volume of click data in an

automated way

22

Training Data: Human Label

What if the searcher is a job seeker?Or a recruiter?

Training Data: Human Label

▪ Relevance depends on who’s searching

▪ Difficult to scale

Training Data: Human Label

Training Data: Click StreamApproach: Clicked = Relevant, Skipped = Not Relevant

User eye scan direction

Unfair penalized

Training Data: Click StreamApproach: Graded relevance

Uncertain (middle level)

Non-relevant

Relevant

Feature Overview

▪ Textual features▪ Social features▪ Homophily features

– Geo– Industry

▪ Inferred Searcher Interests▪ etc.

Inferred Searcher Interests

Interests * Locations * Industry ...

Learning Algorithm

▪ Coordinate Ascent Algorithm– Listwise approach

▪ Objective function: Normalized Discounted Cumulative Gain (NDCG)– Defined on graded relevance

– Intuition: more useful to show more-relevant documents at higher positions

LinkedIn Search - Bird’s eye view

31

Query Processing

Retrieval

Ranking

Federated Page Construction

Search Assist● Instant Results● Guided suggestions● Autocomplete

suggestions

Entity View/Action

32

Federated Search Page

▪ Why do we need this?– Not to overwhelm the user with too much information –Make results personally relevant

33

Motivation

▪ Challenges–Query can be ambiguous

–Incomparability across vertical objects▪Compare objects of different nature: individual job vs. people cluster▪Objects associate with different signals

– Comparability across verticals

34

Motivation

35

Overall Approach

Learning Federation Model

▪ Predicts: p(click| individual result OR vertical cluster, query, searcher)▪ Training data: click logs▪ Features

–Relevance scores from base rankers–Searcher intent–Query intent–etc.

Features▪ Searcher Intents

– Mine searcher profiles and past behavior to infer intent▪ Title recruiter -> recruiting intent▪ Search for jobs -> job seeking intent

– Machine-learned models predict member intents:▪Job seeking▪Recruiting ▪Content consuming

37

Features▪Query Intents: e.g. p(job vertical| “software engineer”)

–Mine from historical searches and actions

38

Features▪Query Intents: e.g. p(job vertical| “software engineer”)

–Mine from historical searches and actions

▪Personalized Query Intents–p(job vertical| “software engineer”, searcher)

39

Features

▪ Query Intents: e.g. p(job vertical| “software engineer”)–Mine from historical searches and actions

▪Personalized Query Intents–p(job vertical| “software engineer”, searcher)

–Individual searcher → searcher group▪p(job vertical| “software engineer”, job seeking searcher)

40

Calibrate Signals across Verticals▪ Relevance scores from vertical rankers are incomparable

41

Calibrate Signals across Verticals▪ Relevance scores from vertical rankers are incomparable▪ Construct composite features

People relevance score of searcher if result is Peoplef 1= ⎨0, otherwise

42

Calibrate Signals across Verticals

▪ Verticals associate with different signals

43

People Result

Job Result

Group Result

Recruiting Intent

Job Seeking Intent

Content Consuming

Intent

Calibrate Signals across Verticals

▪ Verticals associate with different signals

44

People Result

Job Result

Group Result

Recruiting Intent

Job Seeking Intent

Content Consuming

Intent

Calibrate Signals across Verticals

▪ Verticals associate with different signals

45

People Result

Job Result

Group Result

Recruiting Intent

Job Seeking Intent

Content Consuming

Intent

Conclusions▪ Search personalization is at the core of our economic graph

vision–Connect talent with opportunity at massive scale

▪ Click data is useful sources for personalized training data–Need to correct position bias

▪ Personalized features are keys

▪Create composite features to calibrate across verticals

47

We are hiring!