vote elicitation with probabilistic preference models: empirical estimation and cost tradeoffs
DESCRIPTION
Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs. Tyler Lu and Craig Boutilier University of Toronto. Introduction. New communication platforms can transform the way people make group decisions. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/1.jpg)
Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost
Tradeoffs
Tyler Lu and Craig BoutilierUniversity of Toronto
![Page 2: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/2.jpg)
IntroductionNew communication platforms can transform
the way people make group decisions.
How can computational social choice realize this shift?
ChoicesPeople
Computational Social Choice
Consensus
2
![Page 3: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/3.jpg)
Introduction• Computational social choice
– Aggregate full preferences (rankings)– Mostly study rank-based schemes (Borda, maximin, etc…)
• Rank-based voting schemes rarely used in practiceProblem: Cognitive and communication burdenOur approach (recent work): Elicit just the right preferences to make good enough group decisionsThis work: Multi-round elicitation and probabilistic preference models to further reduce burdensAlice Bob Cindy
>12
1
3
![Page 4: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/4.jpg)
Outline
• Preliminaries
• Multi-round Probabilistic Vote Elicitation
• Methodology and Analysis for One-round
• Experimental Results
4
![Page 5: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/5.jpg)
Preliminaries• Voters N = {1..n}; alternatives/items A = {a1…am}
• Vote vi is a ranking of A
• Complete profile v = (v1, …, vn)
Alice Bob1
23
voting rule r
5
Cindy
![Page 6: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/6.jpg)
Score-based Rules
• Many rules have score-based interpretation– Surrogate for “total group satisfaction”– E.g. Borda, Bucklin, maximin, Copeland, etc…
• Associates a score for each item given full rankings s(a, v)
• Winner has highest score6
s( , v) = 7
s( , v) = 6
s( , v) = 5
Alice Bob1
23
Cindy Borda scores
![Page 7: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/7.jpg)
Partial Preferences
• Partial vote pi is a partial order of A– Represented as a (consistent) set of pairwise comparisons– Higher order: top-k, bottom-k, …– Easy for humans to specify
• Partial profile p
7
Alice
>
> >
How to make decision with partial preferences?
![Page 8: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/8.jpg)
Decision with Partial Preferences
• Possible and necessary co-winners [Konczak, Lang’05]
• Recently: minimax regret (MMR) [Lu, Boutilier’11]
– Provides worst-case guarantee on score loss w.r.t. true winner
– Small MMR means good enough decision– Zero MMR means decision is optimal
8
![Page 9: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/9.jpg)
Minimax Regret
9
Adversarial
Bestresponse
![Page 10: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/10.jpg)
Vote Elicitation
• MMR: good choices with “right” partial votes– How to minimize amount of partial preference
queries to make good decision?
• MMR-based incremental elicitation [Lu, Boutilier’11]
– Problem: must wait for response before next query
10
![Page 11: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/11.jpg)
Incremental Elicitation Woes
• Each query is a (voter, pairwise comparison) pair– Exploits MMR, depends on all previous responses
11
ElicitorYES NO
…> ? > ?…
Bob annoyed at having to come back to answer query“interruption cost”
![Page 12: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/12.jpg)
Our Solution:Multi-Round Batching
• Send queries to many voters in each round
12
Elicitor Round: 1Give your top 2
1. 1. 1.
Round: 2Give your next top 1
3. 3. 3.
MMR ≤ εRecommendation:
Interruption cost reduced
2. 2. 2.
![Page 13: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/13.jpg)
Multi-Round Probabilistic Vote Elicitation
• Query class: “rank top-5”, “is A > B?”, etc…– Single request of preferences from voter– Have different cognitive costs
• In each round π selects a subset of voters, and corresponding queries– Can be conditioned on previous round responses
• Function ω, selects winner and stops elicitation• How to design elicitation protocol with provably good
performance?– Worst-case not useful (for common rules)– Use probabilistic preference models to guide design
13
![Page 14: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/14.jpg)
Multi-Round Probabilistic Vote Elicitation
• Distribution P over vote profiles– Induced distribution over runs of protocol (π, ω)
• Can define distribution over performance metrics
14
Quality of winner: Max regret, expected regret
Amount of information elicited: equivalent #pairwise comparisons, or bits.
Number of rounds of elicitation
Tradeoffs!
Depends on what costs are
important.
![Page 15: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/15.jpg)
One-Round Protocol
• Query type: top-k– “Rank your top-k most preferred”
• Simple top-k heuristics [Kalech et al’11]
– Necessary and possible co-winners– No theoretical guarantees on winner quality– Don’t provide guidance on good k– No tradeoff between winner quality and k
15
![Page 16: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/16.jpg)
Probably Approximately Correct (PAC) One-Round Protocol
• Any rank-based voting rule• Any distribution P over profiles
• What is a good k?– p[k] are partial votes after eliciting top-k
k*: smallest k, with prob. ≥ 1 - δ, MMR(p[k]) ≤ ε
• As long as we can sample from P, we can find “approximately” good k…– Samples can come from historical datasets, surveying, or generated
from learned distribution
16
![Page 17: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/17.jpg)
Probably Approximately Correct One-Round Protocol
General Methodology• Input: sample of vote profiles: v1, …, vt
• MMR accuracy ε > 0• MMR confidence δ > 0• Sampling accuracy ξ > 0• Sampling confidence η > 0
Find best the smallest k with
17
![Page 18: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/18.jpg)
Probably Approximately Correct One-Round Protocol
Theorem: if sample size
then for any P, with probability 1 - η, we have
(a) ≤ k* (b) P[ MMR(p[ ]) ≤ ε ] ≥ 1 - δ - 2ξ
18
![Page 19: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/19.jpg)
Practical Considerations
• Sample size from theorem typically unnecessarily large
• Empirical methodology can be used heuristically
• Can generate histograms of MMR for profile samples from runs of elicitation– Can “eyeball” a good k– Can “eyeball” tradeoffs with MMR
19
![Page 20: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/20.jpg)
Experimental Results
• First experiments with Mallows distribution– Rankings generated i.i.d.– Unimodal, with dispersion parameter– t = 100 profiles (for guarantees, use bounds for t)
• Borda voting• Simulate runs of elicitation– Measure max regret and true regret– Normalize regret by number of voters
20
![Page 21: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/21.jpg)
Experimental Results
21
x-axis is MMR per voter
![Page 22: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/22.jpg)
Experimental Results
22
![Page 23: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/23.jpg)
Experimental Results
23
![Page 24: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/24.jpg)
Experimental Results
24
![Page 25: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/25.jpg)
Experimental Results
25
![Page 26: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/26.jpg)
Experimental Results
26
Sushi 10 alternatives50 profiles, each with 100 rankings
![Page 27: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/27.jpg)
Experimental Results
27
Dublin North12 alternatives73 profiles, each with 50 rankings
![Page 28: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/28.jpg)
Concluding Summary
• Model of multi-round elicitation protocol– Highlights tradeoffs between quality of winner,
amount of information elicited, and #rounds– Probabilistic preference profiles to guide design and
performance instead of worst-case• One-round, top-k elicitation– Simple, efficient empirical methodology for choosing k– PAC guarantees and sample complexity– With MMR solution concept, enables probabilistic and
anytime guarantees previous works cannot achieve
28
![Page 29: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/29.jpg)
Future Work
• Multi-round elicitation, top-k or pairwise comparisons
• Fully explore above tradeoffs (associative different costs)
• Assess expected regret and max regret
29
![Page 30: Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs](https://reader035.vdocuments.net/reader035/viewer/2022062501/5681668d550346895dda580a/html5/thumbnails/30.jpg)
The End
30