approval-rating systems that never reward insincerity

Approval-rating systems that never

reward insincerity

Rob LeGrandWashington University in St. Louis

(now at Bridgewater College)

legrand@cse.wustl.edu

Ron K. CytronWashington University in St. Louis

cytron@cse.wustl.edu

COMSOC ’083 September 2008

2

Approval ratings

3

Approval ratings

• Aggregating film reviewers’ ratings– Rotten Tomatoes: approve (100%) or disapprove (0%) – Metacritic.com: ratings between 0 and 100– Both report average for each film– Reviewers rate independently

4

Approval ratings

• Online communities– Amazon: users rate products and product reviews– eBay: buyers and sellers rate each other– Hotornot.com: users rate other users’ photos– Users can see other ratings when rating

• Can these “voters” benefit from rating insincerely?

5

Approval ratings

6

Average of ratings

9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r

0 172.0

outcome: 72.0)( vfavg

data from Metacritic.com: Videodrome (1983)

7

Average of ratings

9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r

0 164.0

outcome: 64.0)( vfavg

Videodrome (1983)

8

Another approach: Median

9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r

0 18.0

outcome: 8.0)( vfmed

Videodrome (1983)

9

Another approach: Median

9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r

0 18.0

outcome: 8.0)( vfmed

Videodrome (1983)

10

Another approach: Median

• Immune to insincerity– voter i cannot obtain a better result by voting– if , increasing will not change– if , decreasing will not change

• Allows tyranny by a majority– – – no concession to the 0-voters

ii rv imed vvf )(

imed vvf )( iv

iv

1,1,1,1,0,0,0v1)( vfmed

)(vfmed

)(vfmed

11

Declared-Strategy Voting[Cranor & Cytron ’96]

electionstate

cardinal

preferences

rational

strategizer

ballot

outcome

12

Declared-Strategy Voting[Cranor & Cytron ’96]

electionstate

cardinal

preferences

rational

strategizer

ballot

outcome

• Separates how voters feel from how they vote• Levels playing field for voters of all sophistications• Aim: a voter needs only to give sincere preferences

sincerity strategy

13

Average with Declared-Strategy Voting?

• Try using Average protocol in DSV context

• But what’s the rational Average strategy?• And will an equilibrium always be found?

electionstate

cardinal

preferences

rational

strategizer

ballot

outcome

14

Rational [m,M]-Average strategy

• Allow votes between and• For , voter i should choose to move

outcome as close to as possible• Choosing would give• Optimal vote is

• After voter i uses this strategy, one of these is true:– and– – and

0m

)),,min(max( Mmvnrvij jii

iavg rvf )(

ij jii vnrv

iavg rvf )(

Mvi

mvi iavg rvf )(

iavg rvf )(

1Mni 1 iv

ir

15

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

72.0

9.0,8.0,8.0,7.0,4.0v

Videodrome (1983)

16

Equilibrium-finding algorithm

01

9.0,8.0,8.0,7.0,4.0r

0,0,0,0,0v

17

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

2.0

1,0,0,0,0v

18

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

4.0

1,1,0,0,0v

19

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

6.0

1,1,1,0,0v

20

• Is this algorithm guaranteed to find an equilibrium?

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

7.0

1,1,1,5.0,0vequilibrium!

21

• Is this algorithm guaranteed to find an equilibrium?• Yes!

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

7.0

1,1,1,5.0,0vequilibrium!

22

• These results generalize to any range

Expanding range of allowed votes

1 2

9.0,8.0,8.0,7.0,4.0r

8.0

2,2,2,1,1 v

23

• Will multiple equilibria always have the same average?

Multiple equilibria can exist

outcome in each case:

7.0)( vfavg

1,1,1,5.0,0v 9.0,8.0,7.0,7.0,4.0r

1,1,9.0,6.0,0v

1,1,75.0,75.0,0v

24

• Will multiple equilibria always have the same average?• Yes!

Multiple equilibria can exist

outcome in each case:

7.0)( vfavg

1,1,1,5.0,0v 9.0,8.0,7.0,7.0,4.0r

1,1,9.0,6.0,0v

1,1,75.0,75.0,0v

25

Average-Approval-Rating DSV

9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r

0 17.0

outcome: 7.0)1,0,( vfaveq

Videodrome (1983)

26

• AAR DSV is immune to insincerity in general

Average-Approval-Rating DSV

9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r

0 1

outcome: 7.0)1,0,( vfaveq

7.0

27

• Expanded vote range gives wide range of AAR DSV systems:

• If we could assume sincerity, we’d use Average• Find AAR DSV system that comes closest• Real film-rating data from Metacritic.com

– mined Thursday 3 April 2008– 4581 films with 3 to 44 reviewers per film– measure root mean squared error

• Perhaps we can come much closer to Average than Median or [0,1]-AAR DSV does

Evaluating AAR DSV systems

10 a 10 b)(, vba

28

Evaluating AAR DSV systems

5.0,aRMSE

a

3240.0aminimum at

5.0b

29

Evaluating AAR DSV systems: hill-climbing

a

3647.0aminimum at

4820.0b

4820.0,aRMSE

30

Evaluating AAR DSV systems: hill-climbing

4820.0bminimum at

3647.0a

bRMSE ,3647.0

b

31

Evaluating AAR DSV systems

)(4820.0,3647.0 v

)(vfavg

32

AAR DSV: Future work

• New website: trueratings.com– Users can rate movies, books, each other, etc.– They can see current ratings without being tempted to

rate insincerely– They can see their current strategic proxy vote

• Richer outcome spaces– Hypercube: like rating several films at once– Simplex: dividing a limited resource among several uses– How assumptions about preferences are generalized is

important

Thanks! Questions?

33

What happens at equilibrium?

• The optimal strategy recommends that no voter change

• So• And

– equivalently,

• Therefore any average at equilibrium must satisfy two equations:– (A)– (B)

1)( ii vrvi

ii rvvi 0)(0)( ii vrvi

irvinv : nvrvi i :

34

Proof: Only one equilibrium average

irinA :)( nriB i :)(

212211 )()()()( BABA

• Theorem:

• Proof considers two symmetric cases:– assume– assume

• Each leads to a contradiction

21 12

35

Proof: Only one equilibrium average

21 case 1:

ii rri 12)( ii riri 12 :: ii riri 12 ::

irin 22 : nri i 11:

nririn ii 1122 :: nn 12

12 21 , contradicting

)( 2A)( 1B

36

Proof: Only one equilibrium average

21 Case 1 shows that

Case 2 is symmetrical and shows that 12

21 Therefore

Therefore, given , the average at equilibrium is uniquer

37

An equilibrium always exists?

• At equilibrium, must satisfy

Given a vector , at least one equilibrium indeed always exists.

A particular algorithm will always find an equilibrium for any . . .

)),,min(max()( Mmvnrviij jii

v

r

r

38

An equilibrium always exists!

Equilibrium-finding algorithm:• sort so that• for i = 1 up to n do

• Since an equilibrium always exists, average at equilibrium is a function, .

• Applying to instead of gives a new system, Average-Approval-Rating DSV.

r

)),,)(min(max( Mmminvnrvik kii

ji rrji )(

),,( Mmrfaveq

v

r

aveqf

(full proof and more efficient algorithm in dissertation)

39

• What if, under AAR DSV, voter i could gain an outcome closer to ideal by voting insincerely ( )?

• It turns out that Average-Approval-Rating DSV is immune to strategy by insincere voters.

• Intuitively, if , increasing will not change .

ii rv

Average-Approval-Rating DSV

iaveq vMmvf ),,(

iv),,( Mmvfaveq

40

• If ,– increasing will not change .– decreasing will not increase .

• If ,– increasing will not decrease .– decreasing will not change .

• So voting sincerely ( ) is guaranteed to optimize the outcome from voter i’s point of view

AAR DSV is immune to strategy

iiaveq rvMmvf ),,(

),,( Mmvfaveq

),,( Mmvfaveq

iviv

iiaveq rvMmvf ),,(

iv

iv

),,( Mmvfaveq

),,( Mmvfaveq

ii rv

(complete proof in dissertation)

41

• [m,M]-AAR DSV can be parameterized nicely using a and b, where and :

mMa

1

mM

mb

1

a

bbM

1

a

bbm

Parameterizing AAR DSV

x

bb

x

bbvfv aveq

axba

1,,lim)(,

10 a 10 b

42

• For example:

Parameterizing AAR DSV

)1,0,()(,1 vfv aveqb

vfv med

)(

2

1,0

11,10,)(2

1,

21

1 vfv aveq

vv

min)(1,0

vv

max)(0,0

2,1,)(2

1,3

1 vfv aveq

43

• Real film-rating data from Metacritic.com– mined Thursday 3 April 2008– 4581 films with 3 to 44 reviewers per film

Evaluating AAR DSV systems

10 a 10 b

2,, vfvvSE avgbaba

V

VV

v

vba

ba v

vSEvRMSE

,

,

44

Higher-dimensional outcome space

• What if votes and outcomes exist in dimensions?

• Example:• If dimensions are independent, Average, Median

and Average-approval-rating DSV can operate independently on each dimension– Results from one dimension transfer

1d

1010:, 2 yxyx

45

Higher-dimensional outcome space

• But what if the dimensions are not independent?– say, outcome space is a disk in the plane:

• A generalization of Median: the Fermat-Weber point [Weber ’29]

– minimizes sum of Euclidean distances between outcome point and voted points

– F-W point is computationally infeasible to calculate exactly [Bajaj ’88] (but approximation is easy [Vardi ’01])

– cannot be manipulated by moving a voted point directly away from the F-W point [Small ’90]

1:, 222 yxyx