creating models of real-world communities with referralweb henry kautz university of washington bart...

Creating Models of Real-World Communities with ReferralWebCreating Models of Real-World Communities with ReferralWeb

Henry KautzUniversity of Washington

Bart SelmanCornell University

2

Recommender SystemsRecommender Systems

New category of software: programs that make personalized recommendations of goods, services, and people

• Amazon.com - books

• Jango.com - stores

• Whowhere.com - friends

Current methodsContent-based: find things similar to ones you like

Collaborative-filtering: find things liked by people who are similar to you

Explosive growth• Viewed as crucial for e-commerce sites

• Excite: 100,000,000+ recommendations per day!

3

Anonymous OpinionsAnonymous Opinions

Most recommender systems hide the identity of the sources of the recommendations

• E-communities: fictitious identities

• Matchmaker systems: deliberately hide true identities

• Collaborative filtering: aggregation - no one to trust (or blame!)

Result: anonymous opinions• Okay choosing a movie or CD

• But would you bet your job on that “recommendation”?!

– Gee, boss, the project failed, but somebody on the net, I don't know who, said it was a good approach...

4

Trusted RecommendationsTrusted Recommendations

For serious life / business decisions, you want the opinion of a trusted expert

• If an expert not personally known, then want to find a reference to one via a chain of friends and colleagues

Referral-chain provides:• Way to judge quality of expert's advice

• Reason for the expert to respond in a trustworthy manner

Finding good referral-chains is slow, time-consuming, but vital

• business gurus on “networking”

5

Example TasksExample Tasks

• You are an associate editor for JAIR. Find a reviewer for a paper that claims new results on “expander graphs”.

• You are considering transferring to a different division of your company. Is that division head a good guy to work for?

• You are putting together a project team to launch a new internet service. Who in your company should you tap for expertise on image compression?

6

ReferralWebReferralWeb

Set of all possible referral-chains = a social network

System for modeling, visualizing, and searching social networks

• in a company

• in an e-community

• in the WWW as a whole

Integrates IR search with a model of personal connections

7

Social NetworksSocial Networks

Social network model specifies:• Who knows who

• Who knows what

How to create?• Ask users to register with system and provide lists

of contacts and interests– sixdegrees, 6DOS, Firefly, Whowhere?

• High startup cost

• Incomplete, out of date, untrustworthy information

• Best experts will actively avoid– a network of the lonely and disenfranchised?

8

Mining Social NetworksMining Social Networks

Alternative: automatically generate network models from pre-existing data

• Email logs (not)

• Bibliographic databases

• Corporate records of organizational structure, project teams, in-house documents

• Arbitrary web pages– personal web pages more accurate / up to date than

official corporate records!

Can extract evidence for both relationships and expertise

9

Discovering NamesDiscovering Names

Proper name extraction• Can accurately identify names in arbitrary

documents

• Frequency of co-occurrence of names can be quickly determined using IR search engines

Canonizing names• John Zack, J. C. Zack, Jim Zack

• Match names / initials / nicknames as long as unambiguous

– closed world assumption

• Improvement: use context– “Henry A. Kautz” matches “Harry A. Kautz”

if both strongly linked to “Bart Selman”

10

Disambiguating NamesDisambiguating Names

Problem: different individuals with the same name

Observation: Within even large organizations the vast majority (90%+) of full names are unique

• 3,000 employees in R&D at AT&T

• 10,000 research scientists in AI, NL, and theory

For medium size networks - considered as noise• Key interface issue: ability to explain each link in

path to users

Further scaling: name + additional context

11

User ProfilesUser Profiles

Manually-entered profiles incomplete, impossible to maintain

• impossible in principle to create complete a-priori list of kinds of expertise

Many services today create highly specialized profiles

• your book buying habits

Simple, robust profile: “bag of words” of all documents in which your name appears

• standard IR vector space model to match queries, people

12

Test NetworksTest Networks

1. Proof of concept: 1,000 node network • Created by combination of web crawling and Altavista

queries, centered on a professor at M.I.T.

• Test group of users could usually find experts on given topics

– but small size of network led to distant referrals

2. 10,000 Researchers in AI, Theory, and NL• Based on 30,000 bibliography entries from high-

quality conferences– AAAI, STOC, FOCS, ACL...

• links between co-authors (not citations)

• http://www.research.att.com/kautz/referralweb

• “paper-reviewer finder”

13

Exploring the NetworkExploring the Network

14

Who can I ask to review a paper on “expander graphs”?Who can I ask to review a paper on “expander graphs”?

15

Experts on Expander GraphsExperts on Expander Graphs

16

Paths to ExpertsPaths to Experts

17

Request Details on FriezeRequest Details on Frieze

18

Frieze Home PageFrieze Home Page

19

ObservationsObservations

Quickly found short chains to experts• Could not be found using IR search alone

User can select chain that is most likely to succeed

• Do not want to bother busiest, most famous experts with every request

Chains cross disciplines• Kautz - AI

• Kearns - AI, Machine Learning

• Blum - Machine Learning, Theory

• Frieze - Theory, Mathematics

Useful tool for strengthening ties both within and between communities

20

Why Does it Work?Why Does it Work?

The Small World PhenomenaMilgram (1967) - any two individuals in the U.S.A. are

linked by a chain of 6 or fewer first-name acquaintances

– “6 degrees of separation”

– Erdös numbers

– “6 degrees of Kevin Bacon”

But• No formal model to explain short paths!

• Due to high average degree?

• True for acquaintances or co-stars, but false for our computer science co-author database!

– 100’s versus 61 versus 4.28!

21

Small-world NetworksSmall-world Networks

Due to randomness?• Random graphs have short average path lengths

• But social networks are not random– nodes are highly clustered (many cliques)

– random graph model predicts that high clustering corresponds to long average paths!

Better model: Small-world networks• Idea: a highly structured (clustered) network with

just a few random links (Watts & Stogatz, 1998)

• Result: high clustering + short paths!

• Random edges correspond to shortcuts– direct relationships between people who primarily

participate in different sub-communities

22

Small-world vs. Random NetworksSmall-world vs. Random Networks

Clustering Coefficient = Average value of C(n) over all nodes, where

2) of neighbors ofnumber (

) of neighborsbetween edges ofnumber ()(

n

nnC

Size Avg Avg Path ClusteringDegree Length Coeff.

CS Co-authors 8,070 4.28 7.9 0.72Random 1 8,070 4.28 6.4 0.072

Film Co-stars 225,000 61 3.65 0.79Random 2 225,000 61 2.99 0.00027

Neurons 282 14 2.66 0.28Random 3 282 14 2.25 0.05

23

Corporate CommunitiesCorporate Communities

Finding good internal experts a strategic business problem

• “intellectual assets” worthless if not consulted!

AT&T: 170,000 employees, 3,000 in the R&D community

• How to build a project team?

• What R&D people to consult for a new business venture?

• What business people to contact about a new technological breakthrough?

In practice: successful projects based on grassroots cross-organizational networking

24

Modeling the AT&T Corporate Network

Modeling the AT&T Corporate Network

Model integrates information from• Official organizational charts (online)

• Personal web pages (+ crawling)

• External publication databases

• Internal technical document databases

Informal structure will prove vital for• finding shorter paths to experts

• finding people who can reliably evaluate experts

• synergy between official and unofficial channels

25

Who can tell me about the Director of Speech Processing research at AT&T?

Who can tell me about the Director of Speech Processing research at AT&T?

26

Paths With All Link TypesPaths With All Link Types

27

Filtering link typesFiltering link types

28

Paths With Only Organizational LinksPaths With Only Organizational Links

29

Paths With Only Web/Article LinksPaths With Only Web/Article Links

30

ObservationsObservations

Official company hierarchy only a sparse subset of the corporate social network

Shortest (and often best) paths involve a combination of official and unofficial links

• Conditions for trust and evaluation may greatly differ

• Global social network is the union of many different kinds of sub-networks

Search greatly aided when user can choose different views of the network

+ types of edge

+ strength of edge

31

Who can help out my project with some great image compression software?

Who can help out my project with some great image compression software?

32

A Note on BelievabilityA Note on Believability

Observation: the recommendations made by (any) recommender system tend to be either astonishingly accurate, or absolutely ridiculous

• true for any AI-complete problem

How can a recommender system be trusted enough for “serious” use?

• Make system transparent: able to explain its reasoning

• indicate to user where the data is ambiguous

• Any link or node can be explained by viewing the data on which it is based

33

Checking the Expert’s ExpertiseChecking the Expert’s Expertise

34

Checking the Reason for an EdgeChecking the Reason for an Edge

35

Verifying the Edge ContextVerifying the Edge Context

36

SummarySummary

Many uses of recommender system require connecting people to people, not just providing “oracular” advice

• Find people, not just documents - access to information that may not even be online!

• Help users evaluate quality of information

Need to automatically model existing, real-world communities

• Cannot require everyone to sign up in advance!

• Can improve and strengthen the “weak ties” that are crucial for effective organizations

ReferralWeb: a tool for generating and searching social networks

37

Status and Future WorkStatus and Future Work

ReferralWeb• Version 2.0 for the Computer Science research

communityhttp://www.research.att.com/~kautz/referralweb

• Corporate version undergoing trials in AT&T Labs

Current research topics• Automatic clustering - discovery of sub-

communities

• Combining uncertain information

• Scale-up to WWW-size communities

• Analysis of more accurate formal models of small-world networks

– accurately predict search performance

38

BibliographyBibliography

• Kautz, H., Selman, B. & Shah, M. 1997. The Hidden Web. AI Magazine 18(2): 27-36.

• Milgram, S. 1967. The Small-World Problem. Psychology Today 1(1): 60-76.

• Resnick, P., ed. 1996. Special Section on Recommender Systems. Communications of the ACM 30(3).

• Watts, D. & Stogatz, S. 1998. Collective dynamics of ‘small-world’ networks. Nature 393: 440-442.