1

Web Crawling for Search:What’s hard after ten years?

Raymie Stata

Chief Architect, Yahoo! Search and Marketplace

2

Agenda

• Introduction• What makes crawling hard for “beginners”• What remains hard for “experts”

3

Introduction

• Web “crawling” is the primary means of obtaining data for Search Engines– Tens of billions of pages downloaded– Hundreds of billions of pages “known”– Average page <10 days old

• Web crawling is as old as the Web– “Large scale” crawling is about ten-years old

• Lots published, but still exists “secret sauce”• Must support RCF

– Relevance, comprehensiveness, freshness

4

Components of a crawler

Downloaders

Web DB

Page processing

Page storage

Prioritization

Feeds

I’net*

*Internet = DNS as well as HTTP

Enrichment

Click streams

5

Baseline challenges: overall scale

• 100s machines dedicated to each component• Must be good at logistics (purchasing and

deployment), operations, distributed programming (fault tolerance included), …

6

Baseline challenges: downloaders

• DNS scaling (multi-threading)• Bandwidth

– Async I/O vs. threads– Clustering/distribution

• Non-conformance• Politeness

7

Baseline challenges: page processing

• File-cracking– HTML, Word, PDF, JPG, MPEG, …

• Non-conformance• Higher-level processing

– JavaScript, sessions, information extraction, …

8

Baseline challenges:Web DB and enrichment

• Scale– Update rate– Extraction rate

• Duplication detection• Alias detection• Checkpoints

9

Baseline challenges: prioritization

• Quality ranking• Spam and crawler traps

10

Evergreen problems

• Relevance– Page quality, spam

• Page processing, prioritization techniques

• Comprehensiveness– Sheer scale

• Sheer machine count (expensive)• Scaling of the Web DB

– Deep Web, information extraction• Page processing

• Freshness– Discovery, frequency, “long tail”

11

Web DB: more details

• For each URL, the Web DB contains:– In- and outlinks– Anchor text– Various dates: last downloaded, last changed, …– “Decorations” from various processors

• Language, topic, spam scores, term-vectors, fingerprints, “shingleprints,” many more…

• Subset of the above stored for several instances– That is, we keep track of the history of a page

12

Web DB: update volume

• When a page is downloaded, we need to update inlink and anchor-text info for each page it points to

• A page has ~20 outlinks on it• We download 1,000’s pages per second• At peak, need well over 100K updates/sec

13

Web DB: scaling techniques

• Perform updates in large batches• Solves bandwidth problems…• …but introduces latency problems

– In particular: time to discover new links

• Solve latency with “short-circuit” for discovery– But this by-passes the full prioritization logic,

which introduces quality problems that need to be solved with more special solutions and before long, Oi, it’s all getting very complicated…

14

DHTML: the enemy of crawling

• Increasing use of client-side scripting (aka, DHTML) is making more of the Web opaque to crawlers– AJAX: Asynchronous JavaScript and XML

• (The end of crawling?)

• Not (yet) a major barrier to Web search, but is a barrier to shopping and other specialized search, where we also have to deal with:– Form-filling and sessions– Information extraction

15

Conclusions

• Large-scale Web crawling not trivial• Smart, well-funded people could figure it out

from the literature• But secret sauce remains in:

– Prioritization– Scaling the Web DB– JavaScript, form-filling, information extraction

16

The future

• Will life get easier?– Ping plus feeds

• Will life get harder?– DHTML -> Ajax -> Avalon

• A little bit of both?– Publishers regain control– But, net, comprehensiveness improves