reputation systems

Reputation Systems

Guest LecturePaul Resnick

Associate ProfessorUniv. of Michigan

School of Informationpresnick@umich.edu

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Learning Objectives Understand

– What a reputation system is– Theory about when and why it should work– Open research questions

Participate in design– Recognize situations when it might be helpful– Raise some of the difficult design challenges

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Outline What is a reputation system? Theory: when/why they should work Empirical results Design space Case study: NPAssist recommender

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Definition A Reputation System…

– Collects– Distributes– Aggregates

…information about behavior

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Examples BBB Bizrate eBay Expertise sites

– Epinions “top reviewers”– Slashdot karma system

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Why Reputation Systems Interacting with strangers Sellers (Exchange Partners) Vary

– Skill– Effort– Ethics

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Other Trust-Inducing Mechanisms in E-commerce Insurance Escrow Fraud Prosecution

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

How Reputation Systems Should Work Information Incentive Self-selection

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Some Issues Anonymity Name changes Name trades Lending reputations Eliciting evaluation Honesty of evaluations

Interaction Type

ID changes

Anonymous every xaction

Pseudonyms

at will

Identified never

Anonymity Analysis

Interaction type

ID changes

Reputation Sharing

Trust/ cooperation

Anonymous every xaction

Pseudonyms

at will

1L Pseudonyms each arena

Identified never

Interaction type

ID changes

Reputation Sharing

Trust/ cooperation

Anonymous every xaction

none none

Pseudonyms

at will + only + only

1L Pseudonyms each arena

Identified never + and – + and 0

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

1L Pseudonyms Third-party issues pseudonyms

– No cost– Not replaceable– Reveal name to third party– Don’t reveal mapping of name to

pseudonym

Interaction type

ID changes

Reputation Sharing

Trust/ cooperation

Anonymous every xaction

none none

Pseudonyms

at will + only + only

1L Pseudonyms

each arena

+ and – within arena

+ and 0 within arena

Identified never + and – + and 0

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Empirical Results: eBay Feedback is provided It’s almost all positive Reputations are informative Reputation benefits

– Effect on probability of sale– Effect on price

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Provision of Feedback

Negatives: paid but did not receive; seller cancelled; not as advertised; communication

Neutrals: slow shipping, not as advertised, communication

Buyer of Seller Seller of Buyer Frequency Percent Frequency Percent

negative 111 0.3 353 1.0 neutral 62 0.2 60 0.2 positive 18,569 51.2 21,560 59.5 none 17,491 48.3 14,260 39.4 Total 36,233 36,233

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Feedback Profiles of Buyers and Sellers

Group N (Sellers)

Percent neutral and negative

(Sellers)

N (Buyers)

Percent neutral and negative

(Buyers) 0-9 positive 4,018 2.83% 13,306 1.99%

10-49 positive 3,932 1.25% 7,366 1.09% 50-199 positive 3,728 0.95% 3,678 0.76%

200-999 positive 1,895 0.79% 738 0.60% 1000+ 122 1.18% 15 0.92%

All 13,695 0.93% 25,103 0.83%

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Predicting Problematic Transactions

Logistic Regression

f(0,0) = 1.91% f(100,0) = .18% f(100,3) = .53%

N = 36233Beginning Block Number 0. Initial Log Likelihood Function-2 Log Likelihood 2194.3468-2 Log Likelihood 2075.420

Dependent Variable.. NEGNEUT---------------------- Variables in the Equation -----------------------

Variable B S.E. Wald df Sig R Exp(B)

LNNPOS .7712 .1179 42.7907 1 .0000 .1363 2.1624LNPOS -.5137 .0475 116.8293 1 .0000 -.2288 .5983Constant -3.9399 .1291 931.3828 1 .0000

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Predictive Value

1-specificity (% of unproblematic transactions rejected)

Sensitivity (% of problematic transactions rejected)

Cutoff predicted probability

% of accepted transactions that are problematic

75% 94.2% .20% .11% 50% 81.5% .31% .18% 25% 57.2% .54% .27% 10% 32.4% 1.09% .36% 0% 0% Accept all .48%

Predicting Problem Transactions

1 - Specificity

1.00.75.50.250.00

Sen

sitiv

ity

1.00

.75

.50

.25

0.00

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Some Recently Completed Work Experiment: does reputation affect

profit?– Many positives: Yes, but only a little (8.1%)– One or two negatives: No

Incentives for quality feedback provision– Can pay based on agreement among

raters

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Studies Currently Underway Feedback provision (empirical)

– Reciprocation, altruism, and free riding Dynamics: learning and selection

(empirical) Geography: trust and trustworthiness by

state

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Design Space Rating scales Aggregation of ratings Who rates? Incentives for raters Identification/Anonymity

– Exchange partners– Evaluation providers

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Case Study Goal: help Michigan non-profits select

consultants and other service providers Is this a good candidate for a reputation

system?

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Case Study Goal: help Michigan non-profits select

consultants and other service providers Is this a good candidate for a reputation

system?Interacting with strangersSellers (Exchange Partners) Vary

SkillEffortEthics

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Case Study Design Choices Rating scales Aggregation of ratings Who rates? Incentives for raters Identification/Anonymity

– Exchange partners– Evaluation providers

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Summary RS inform, incent, select Opportunity for RS: interactions with

strangers Design space

– Scales, aggregation, raters, incentives, anonymity