reputation systems

Reputation Systems

Guest LecturePaul Resnick

Associate ProfessorUniv. of Michigan

School of [email protected]

SCHOOL OF INFORMATION

UNIVERSITY OF MICHIGANsi.umich.edu

Learning Objectives Understand

– What a reputation system is– Theory about when and why it should work– Open research questions

Participate in design– Recognize situations when it might be helpful– Raise some of the difficult design challenges



Outline What is a reputation system? Theory: when/why they should work Empirical results Design space Case study: NPAssist recommender



Definition A Reputation System…

– Collects– Distributes– Aggregates

…information about behavior



Examples BBB Bizrate eBay Expertise sites

– Epinions “top reviewers”– Slashdot karma system



Why Reputation Systems Interacting with strangers Sellers (Exchange Partners) Vary

– Skill– Effort– Ethics



Other Trust-Inducing Mechanisms in E-commerce Insurance Escrow Fraud Prosecution



How Reputation Systems Should Work Information Incentive Self-selection



Some Issues Anonymity Name changes Name trades Lending reputations Eliciting evaluation Honesty of evaluations

Interaction Type

ID changes

Anonymous every xaction

Pseudonyms

at will

Identified never

Anonymity Analysis

Interaction type

ID changes

Reputation Sharing

Trust/ cooperation


Pseudonyms

at will

1L Pseudonyms each arena

Identified never

Interaction type

ID changes

Reputation Sharing

Trust/ cooperation


none none

Pseudonyms

at will + only + only

1L Pseudonyms each arena

Identified never + and – + and 0



1L Pseudonyms Third-party issues pseudonyms

– No cost– Not replaceable– Reveal name to third party– Don’t reveal mapping of name to

pseudonym

Interaction type

ID changes

Reputation Sharing

Trust/ cooperation


none none

Pseudonyms

at will + only + only

1L Pseudonyms

each arena

+ and – within arena

+ and 0 within arena

Identified never + and – + and 0



Empirical Results: eBay Feedback is provided It’s almost all positive Reputations are informative Reputation benefits

– Effect on probability of sale– Effect on price



Provision of Feedback

Negatives: paid but did not receive; seller cancelled; not as advertised; communication

Neutrals: slow shipping, not as advertised, communication

Buyer of Seller Seller of Buyer Frequency Percent Frequency Percent

negative 111 0.3 353 1.0 neutral 62 0.2 60 0.2 positive 18,569 51.2 21,560 59.5 none 17,491 48.3 14,260 39.4 Total 36,233 36,233



Feedback Profiles of Buyers and Sellers

Group N (Sellers)

Percent neutral and negative

(Sellers)

N (Buyers)

Percent neutral and negative

(Buyers) 0-9 positive 4,018 2.83% 13,306 1.99%

10-49 positive 3,932 1.25% 7,366 1.09% 50-199 positive 3,728 0.95% 3,678 0.76%

200-999 positive 1,895 0.79% 738 0.60% 1000+ 122 1.18% 15 0.92%

All 13,695 0.93% 25,103 0.83%



Predicting Problematic Transactions

Logistic Regression

f(0,0) = 1.91% f(100,0) = .18% f(100,3) = .53%

N = 36233Beginning Block Number 0. Initial Log Likelihood Function-2 Log Likelihood 2194.3468-2 Log Likelihood 2075.420

Dependent Variable.. NEGNEUT---------------------- Variables in the Equation -----------------------

Variable B S.E. Wald df Sig R Exp(B)

LNNPOS .7712 .1179 42.7907 1 .0000 .1363 2.1624LNPOS -.5137 .0475 116.8293 1 .0000 -.2288 .5983Constant -3.9399 .1291 931.3828 1 .0000



Predictive Value

1-specificity (% of unproblematic transactions rejected)

Sensitivity (% of problematic transactions rejected)

Cutoff predicted probability

% of accepted transactions that are problematic

75% 94.2% .20% .11% 50% 81.5% .31% .18% 25% 57.2% .54% .27% 10% 32.4% 1.09% .36% 0% 0% Accept all .48%

Predicting Problem Transactions

1 - Specificity

1.00.75.50.250.00

Sen

sitiv

ity

1.00

.75

.50

.25

0.00



Some Recently Completed Work Experiment: does reputation affect

profit?– Many positives: Yes, but only a little (8.1%)– One or two negatives: No

Incentives for quality feedback provision– Can pay based on agreement among

raters



Studies Currently Underway Feedback provision (empirical)

– Reciprocation, altruism, and free riding Dynamics: learning and selection

(empirical) Geography: trust and trustworthiness by

state



Design Space Rating scales Aggregation of ratings Who rates? Incentives for raters Identification/Anonymity

– Exchange partners– Evaluation providers



Case Study Goal: help Michigan non-profits select

consultants and other service providers Is this a good candidate for a reputation

system?



Case Study Goal: help Michigan non-profits select

consultants and other service providers Is this a good candidate for a reputation

system?Interacting with strangersSellers (Exchange Partners) Vary

SkillEffortEthics



Case Study Design Choices Rating scales Aggregation of ratings Who rates? Incentives for raters Identification/Anonymity

– Exchange partners– Evaluation providers



Summary RS inform, incent, select Opportunity for RS: interactions with

strangers Design space

– Scales, aggregation, raters, incentives, anonymity

reputation systems

Documents