fast, lenient, and accurate – building personalized instant search experience at linkedin

Fast, Lenient, and AccurateBuilding Personalized Instant Search Experience at LinkedIn

Ganesh Venkataraman, Abhi Lad, Lin Guo, Shakti SinhaLinkedIn

Agenda

● LinkedIn● LinkedIn Search

○ Navigational vs Exploratory searches○ Typeahead vs SERP

● Big picture and problem statement● Instant search – Search-as-you-type

○ Query autocomplete○ Entity-aware suggestions○ Instant results

● Conclusions & Future work

LinkedIn – Professional Identity

LinkedIn – Professional Graph

LinkedIn – Jobs

LinkedIn – And much more...

Companies

Skills

Professional Content

LinkedIn – Massive Scale

LinkedIn Search

Navigational Search

Looking for someone specific by name.

Query has a single correct result.

Exploratory SearchFinding people that match a given set of criteria.

Multiple results match the user’s query.

Instant Search – Search-as-you-typeSatisfy navigational searches: Show instant search results.

Help frame exploratory searches: Complete the user’s query and show search suggestions.

Big PicturePartial query

Instant results Autocomplete

Search suggestions

Query tagger

Full-text search

Search results

Manuallyenteredquery

Big PicturePartial query

Instant results Autocomplete

Search suggestions

Query tagger

Full-text search

Search results

Manuallyenteredquery

Focus today:● Autocomplete● Search suggestions● Instant results

Problem StatementPartial query

Instant results Autocomplete

Search suggestions

Query tagger

Full-text search

Search results

Manuallyenteredquery

Focus today:● Autocomplete● Search suggestions● Instant results

How can we build an instant search experience that scales to 450+ million members, and is fast, lenient, and accurate?

● Instant search = Query autocomplete + search suggestions + instant results● Fast = Search-as-you-type latencies● Lenient = Handle spelling errors and common variations● Accurate = Highly relevant and personalized results

Query Tagging

PERSON

TITLE(ID=126)

COMPANY(ID=1337)

Entity types identified: Person name, job title, company, school, skills, locations.

Key part of query processing!Impacts: autocomplete, spelling correction, search suggestions,query rewriting, ranking.

Sequential prediction model(CRF – Conditional Random Fields)

Training data:● Standardized dictionaries (people names,

companies, schools, titles, skills, locations)● Query logs● Clickthrough (CTR) data● Crowdsourced labels

Query Autocomplete

● Fast● Relevant and contextual● Resilient to spelling errors

Query Autocomplete – Offline processing

linkedin software engineersoftware engineerbig datadata scientistdata engineerexpert systems..

[linkedin] [software engineer]

Query logs Entities Index

FST – Finite State Transducers

Compact + fast retrieval + fuzzy match (via Levenstein Automata)

Query Autocomplete – Online processingTwo step process:

1. Retrieval (Candidate generation)

User’s query: [big data e]

Candidates = C(big data e) U C(data e) U C(e)= big data engineer, big data expert systems, big data entry, ...

linkedin software engineersoftware engineerbig datadata scientistdata engineerexpert systems..

Query logs

Query Autocomplete – Online processingTwo step process:

2. Scoring (Ranking)

User’s query: [big data e]Candidate completions: “big data engineer”, “big data expert”, “big data entry”

Score(“big data engineer”):

P(s1, s2, s3…) ≈ P(s1)·P(s2|s1)·P(s3|s2).. // Bigram language model

Use entities : P([engineer] | [big data])Fall back to words : P(engineer | data)·P(data | big)

Query Suggestions – Autocomplete + query tagger

“linke” ⇒ “Linkedin” ⇒ COMPANY

“had” ⇒ “Hadoop” ⇒ SKILL

Instant Results

● Fast retrieval over 450+ million members● Highly personalized● Balance personalization & popularity● Resilient to spelling variations

Instant Results – Indexing

NAME: richardPREFIX: r, ri, ric, rich, richa, ...NAME: bransonPREFIX: b, br, bra, bran, brans, ...

● Inverted Index (Maps token to list of docs that contain that token):NAME:richard => [1, 4, 10, 15, …] // Everyone named “richard”PREFIX:ri => [1, 2, 4, 7, 10, 15, …] // Everyone whose name starts with “ri”…

● Retrieval approachUser’s query – richard bRewritten query – +NAME:richard +PREFIX:b

● Prefix-based tokenization:

DOCID 4

(posting lists)

Instant Results – Indexing

CONN: 1, 10, 15

● Inverted IndexCONN:4 => [1, 10, 15] // Everyone connected to Richard BransonCONN:1 => [4, ...]CONN:10 => [4, ...]...

● Retrieval approachUser’s query – richard bRewritten query – +NAME:richard +PREFIX:b +CONN:1

(Everyone named richard b… and connected to User:1)

● Connections Index:

DOCID 4

Instant Results – Indexing

Early Termination

Problem: A query like [PREFIX:ri] might retrieve too many candidate documents.

How can we retrieve the most promising documents first so that we don’t need to score all of them?

Static Rank: Order documents based on their prior (query independent) likelihood of relevance:

A combination of:● Profile views● Spam and security related scores● Editorial rules (Celebrities, influencers, …)

numToScore: The number of documents to retrieve and score for any query

Balancing Popularity and Personalization

Query: richard b…

Are you looking for Richard Branson, or a colleague name Richard Burton?

(Assume searcher’s ID = 1)

Rewritten Query:

● +NAME:richard +PREFIX:b +CONN:1 // Too restrictive. Only find searcher’s connections.

● +NAME:richard +PREFIX:b ?CONN:1[50%] // Try to retrieve 50% results from searcher’s connections

Instant Results – Retrieval

Custom search operator: “Weighted OR”

Instant Results – Spelling Variations

weiner ⇔ wiener

catherine ⇔ kathryn

dipak ⇔ deepak

Name Clusters

Offline process to cluster together similar sounding or similarly spelt names.

Two step process:

1. Coarse clustering (optimized for broad coverage)Normalization: repeated chars, accented chars, common phonetic variations (c ⇔ k, ph ⇔ f)Combination of edit distance & double metaphone (sound)E.g. (dipak = deepak), (wiener = weiner), (catherine = kathryn), (jeff = joff)

2. Fine-grained clustering (optimized for precision)Split up clusters based on more sophisticated rulesPosition and character-aware edit distanceQuery reformulation data (q1 → q2 → click)E.g. (jeff ≠ joff)

Instant Results – Spelling Variations

Instant Results – Spelling Variations

NAME: kathrynCLUSTER: katharine

Potential queries:katherinekathrynkatharinecatharine

Rewritten queries:?NAME:katherine ?CLUSTER:katharine?NAME:kathryn ?CLUSTER:katharine?NAME:katharine ?CLUSTER:katharine?NAME:catharine ?CLUSTER:katharine

Either match original query term or match the name cluster

Query time

Indexing time

Clicked result treated as positive.

All other shown results treated as negative.

Since this is navigational search, we assume there’s only 1 correct result => low presentation bias.

Learning to Rank (Machine-learned ranking)

Training data● Click data from previous typeahead sessions● <searcher, query, doc> ⇒ positive/negative

Features / signals● Textual match against various fields● Network distance, number of shared connections● Global popularity● Compound features

Instant Results – Scoring

+

–

–

–

Conclusions● Instant search experience

○ Directly satisfy navigational search uses in typeahead via Instant Results

○ Help the user frame exploratory search queries via Query Autocomplete & Search

Suggestions

● Combination of techniques○ Query tagger for entity extraction – “Things not Strings”○ FST-based query completion○ Inverted index-based instant results + Early termination + Weighted OR○ Name clusters for fuzzy name matching

Future Work● Personalized query completions

○ m ⇒ machine learning

○ m ⇒ machinist

● Multi-entity query suggestions○ Now : [linkedin] ⇒ “Find people who work at LinkedIn”

○ Future : [linkedin data scientist] ⇒ “Find data scientists at LinkedIn”

● Better blending○ Autocomplete + query suggestions + instant results○ Query features – what does the query mean?○ Results features – what results come back from each system?

Thank You!

LinkedIn – The Economic Graph

LinkedIn Search – SERP (Jobs)

LinkedIn Search – Typeahead

LinkedIn Search – SERP