information aggregation: experiments and industrial applications kay-yut chen hp labs
Post on 21-Dec-2015
214 Views
Preview:
TRANSCRIPT
Information Aggregation:Experiments and Industrial Applications
Kay-Yut ChenHP Labs
Experimental Economics Program
Agenda
• Lessons from HP Information Markets
(Chen and Plott 2002)
• Scoring Rules and Identification of Experts
(Chen, Fine and Huberman 2004)
(Chen and Hogg 2004)
• Public Information (Chen, Fine and Huberman 2004)
Experimental Economics Program
HP Information Markets (Chen and Plott)• Summary of Events
– 12 events, from 1996 to 1999
– 11 events sales related
– 8 events had official forecasts
• Methodology & Procedures
– Contingent state asset (i.e. winning ticket pays $1, others $0)
– Sales amount (unit/revenue) divided into (8-10) finite intervals
– Web-based real time double-auction
– 15-20 min phone training for EVERY subject
– Market open for one week at restricted time of the day (typically lunch and after hours)
– Market size: 10-25 people
Experimental Economics Program
Event 2
0
0.05
0.1
0.15
0.2
0.25
0.3
0 100 200 300
$
P
IAM Distribution
Actual Outcome
HP OfficialForecast
IAM Prediction
Experimental Economics Program
Results
Abs % Errors of IAM Predictions
Last Interval Ignored Last Interval Mass at Lower Bound
Event Absolute % errors of HP
forecasts
Average last 60%
trade
Average last 50%
trade
Average last 40%
trade
Average last 60%
trade
Average last 50%
trade
Average last 40%
trade2 13.18% 4.61% 4.57% 4.68% 5.63% 5.68% 5.80%3 59.55% 57.48% 55.72% 54.60% 59.25% 57.46% 56.32%4 8.64% 7.84% 8.15% 8.52% 6.45% 6.77% 7.13%5 32.08% 30.93% 31.57% 31.83% 29.74% 30.33% 30.48%6 29.69% 24.23% 24.54% 25.30% 22.94% 23.22% 23.93%7 4.10% 7.33% 7.02% 6.71% 5.35% 4.91% 4.55%8 0.11% 2.00% 2.35% 1.83% 1.53% 1.39% 1.00%9 28.31% 23.85% 24.85% 24.39% 17.55% 17.32% 16.54%
T-test P-value 0.079 0.084 0.071 0.034 0.026 0.022
Random variable x=official error – market error
H0: mean of x=0 Alternate: mean of x>0
Experimental Economics Program
Business Constraints and Research Issues
• Not allowed to “bet” players’ own money -> stakes limited to an average of $50
per person
• Time horizon constraints -> 3 months to be useful
• Recruit the “right” people
• Asset design affects the results (How to set the intervals?)
• Thin markets (sum of price ~ $1.11 to $1.31 over the dollar)
– Few players
– Not enough participation
Experimental Economics Program
Reporting with Scoring Rule
Reports of Probability Distribution
A B C
Outcome
p1 p2 p3
Pays C1+C2*Log(p3)
Experimental Economics Program
Information Aggregation Function
ssss
sss
N
N
ppp
pppIsP
...
...|
21
21
If reports are independent, Bayes Law applies …
Experimental Economics Program
Two Complications
• Non-Risk Neutral Behavior
• Public Information
Experimental Economics Program
Dealing with Risks Attitudes:Two-Stage Mechanism
Event 1
Event 2
Event 3
Event 4
Event 5
Event 6
Event 7
Event 8
Stage 1: Information Market
Call Market to Solicit Risk Attitudes
Stage 2: Probability Reporting & Aggregation
Individual Report of Probability DistributionNonlinear Aggregated Function
Tim
e
Experimental Economics Program
Second Stage: Aggregation Function
Bayes Law with Behavioral Correction
i=r(V i / i)c
Holding value/Risk- measure relative risk of individuals
Normalizing constant for individual risks
“market” risk~sum of prices/winning payoff
sNss2s1
Nss2s1
N21
N21
p...pp
p...ppI|sP
Experimental Economics Program
Experiments:Inducing Diverse Information
A B C
Outcome
Box of
Balls
A
B
C
C
C
* In actual experiments, there are TEN states
Random DrawsProvide Info
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 1.453
Comparison To All Information Probability
Experiment 4, Period 17No Information
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I J
States
Pro
bab
ilit
yOmniscient
No Info
Experimental Economics Program
Kullback-Leibler Measure
• Relative entropy
• Always >=0
• =0 if two distributions are identical
• Addictive for independent events
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Kullback-Leibler = 1.337
Comparison To All Information Probability
Experiment 4, Period 171 Player
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 1.448
Comparison To All Information Probability
Experiment 4, Period 172 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 1.606
Comparison To All Information Probability
Experiment 4, Period 173 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 1.362
Comparison To All Information Probability
Experiment 4, Period 174 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 0.905
Comparison To All Information Probability
Experiment 4, Period 175 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 1.042
Comparison To All Information Probability
Experiment 4, Period 176 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 0.550
Comparison To All Information Probability
Experiment 4, Period 177 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 0.120
Comparison To All Information Probability
Experiment 4, Period 178 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1 2 3 4 5 6 7 8 9 10
Series1
Series2
Kullback-Leibler = 0.133
Comparison To All Information Probability
Experiment 4, Period 179 Players Aggregated
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I JStates
Pro
bab
ilit
yOmniscient
IA Mechanism
Comparison To All Information Probability
Experiment 4, Period 17
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
A B C D E F G H I J
Pro
bab
ilit
yOmniscient
IA Mechanism
market
Best Individual
Experimental Economics Program
KL Measures for Private Info Experiments
1.977 (0.312) 1.222 (0.650) 0.844 (0.599) 0.553 (1.057)
1.501 (0.618) 1.112 (0.594) 1.128 (0.389) 0.214 (0.195)
1.689 (0.576) 1.053 (1.083) 0.876 (0.646) 0.414 (0.404)
1.635 (0.570) 1.136 (0.193) 1.074 (0.462) 0.413 (0.260)
1.640 (0.598) 1.371 (0.661) 1.164 (0.944) 0.395 (0.407)
No Information
Market Prediction
Best PlayerNonlinear Aggregation
Function
Experimental Economics Program
Group Size Performance
Experimental Economics Program
Did the Markets Pick out Experts?
Group Exp 1 Exp 2 Exp 3 Exp 4 Exp 5
Random 1.36 0.93 1.18 1.12 1.15
Payoff 1.45 1.09 1.24 1.13 1.39
Value 0.72 0.91 0.94 1.13 1.22
Optimal 0.53 0.72 0.75 0.83 0.77
•KL measure of all query data
•Pick groups of 3
Experimental Economics Program
Did Previous Queries Pick out Experts?
Group Exp 1 Exp 2 Exp 3 Exp 4 Exp 5
Random 1.15 0.92 1.18 1.07 1.21
Query 0.78 0.89 0.71 0.92 0.81
Optimal 0.60 0.59 0.69 0.72 0.72
•KL measure of second half of query data
•Pick groups of 3
Experimental Economics Program
Public Information
• Information observed by more than one
• Double counting problem
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
A B C D E F G H I J
States
Pro
bab
ilit
yOmnicient
Public
No Info
Information Aggregation with Public InformationKullback-Leibler = 2.591
Public Info Experiment 3, Period 911 Players Aggregated
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J
States
Pro
bab
ilit
yOmnicient
Public
IAM
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J
States
Pro
bab
ilit
yOmnicient
Public
IAM
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J
States
Pro
ba
bil
ity
Omnicient
Public
IAM
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J
States
Pro
ba
bil
ity
Omnicient
Public
IAM
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J
States
Pro
ba
bil
ity
Omnicient
Public
IAM
Experimental Economics Program
Dealing with Public Information:Add a Game to the Second Stage
Event 1
Event 2
Event 3
Event 4
Event 5
Event 6
Event 7
Event 8
Stage 1: Information Market
Call Market to Solicit Risk Attitudes
Stage 2: Probability Reporting & Aggregation
Individual Report of Probability DistributionMatching Game to Recover Public Information
Modified Nonlinear Aggregated Function
Tim
e
Experimental Economics Program
Assumptions
• Individuals know their public information
• Private & Public Info Independent
• Structure of Public Info Arbitrary
Experimental Economics Program
Matching Game
Reports of Probability Distribution
A B C
Outcome
q11 q12 q13Player 1: q1
q21 q22 q23Player 2: q2
q31 q32 q33Player 3: q3...
.
.
.
.
.
.
.
.
.
Player 1’s Payoff: (match function)*(C1+C2*Log(q33))Match function: f(q1,q2)=(1-0.5*sum(abs(q1i-q2i)))^2
Choose player (3) byMax (match function)
Experimental Economics Program
Matching Game
• Any match function f(q1,q2) with property
– Max when q1=q2
• Multiple Equilibria
• Payoff increases as entropy decreases
• Hopefully, individuals report public information
Experimental Economics Program
Aggregation Function withPublic Information Correction
Bayes Law with a) Behavioral Correction b) Public Info Correction
s Ns
Ns
s2
s2
s1
s1
Ns
Ns
s2
s2
s1
s1
N21
N21
qp
...qp
qp
qp
...qp
qp
I|sP
i=r(V i / i)c
Holding value/Risk- measure relative risk of individuals
Normalizing constant for individual risks
“market” risk~sum of prices/winning payoff
Experimental Economics Program
Public Information Experiments
• 5 Experiments
• Various Information Structures
– All subject received 2 private draws & 2 public draws
– All subject received 3 private draws & 1 public draws
– All subject received 3 private draws & half of the
subjects receive 1 public draws
– All subject received 3 private draws & 1 public draws.
2 groups of independent public information.
• 9 to 11 participants in each experiments
Correcting for Public Information
Public Info Experiment 3, Period 911 Players Aggregated
Kullback-Leibler = 0.291
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J
Omnicient
sim aggr
IAM
IAM (true public)
Experimental Economics Program
ExptPrivate
InfoPublicInfo No Info
Market Prediction
BestPlayer
Nonlinear Aggregation
Function
Public Info
Correction
Perfect Public Info Correction
1 2 draws for all
2 draws for all
1.332 (0.595)
0.847 (0.312)
0.932 (0.566)
2.095(1.196)
0.825 (0.549)
0.279 (0.254)
2 2 draws for all
2 draws for all
1.420 (0.424)
0.979 (0.573)
0.919 (0.481)
2.911 (2.776)
0.798 (0.532)
0.258 (0.212)
3 3 draws for all
1 draws for all
1.668 (0.554)
1.349 (0.348)
1.033 (0.612)
2.531 (1.920)
0.718 (0.817)
0.366 (0.455)
4 3 draws for all
1 draws for half
1.596 (0.603)
0.851 (0.324)
1.072 (0.604)
0.951 (1.049)
0.798 (0.580)
0.704 (0.691)
53 draws for all
Two groups
of public info
1.528 (0.600)
0.798 (0.451)
1.174 (0.652)
0.886 (0.763)
1.015 (0.751)
0.472 (0.397)
KL Measures for Public Info Experiments
Experimental Economics Program
Summary
• IAM with public info correction did better than
best person.
• IAM with public info correction did better than
markets in 4 out of 5 cases.
• IAM corrected with true public info did significant
better than all other methods.
Experimental Economics Program
Implied Probabilities of Revenue Bins, September 2003
0%
5%
10%
15%
20%
25%
30%
35%Actual Value
$1053mOfficial Projection
Experimental Economics Program
Implied Probabilities of Operating Profit Bins, September 2003
0%
10%
20%
30%
40%
50%
60%
70%
$0 -$37.1m
$37.1 -$46.1m
$46.1m -$54.4m
$54.4 -$62.0m
$62.0 -$69.3m
$69.3 -$73.6m
$73.6 -$77.8m
$77.8 -$82.0m
$82.0 -$86.2m
$86.2 -$90.4m
$90.4 -$94.7m
$94.7 -$102.0m
$102.0 -$109.6m
$109.6 -$117.9m
$117.9 -$126.9m
$126.9mor above
Actual Value$113m
Official Projection
Experimental Economics Program
Supplementary
Experimental Economics Program
Previous Research
• Academic Studies– Information Aggregation in Markets
• Plott, Sunder, Camerer, Forsythe, Lundholm, Weber,…
– Pari-mutuel Betting Markets
• Plott, Wit & Yang
• Real World Applications– Iowa Electronic Markets
– Hollywood Stock Exchange
– HP Information Markets
– Newsfuture
– Tradesport.com
– …
Experimental Economics Program
Risk Attitudes
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
A B C D E F G H I J
Risk Loving
Risk Neutral
Risk Averse
Experimental Economics Program
Dealing with Risks Attitudes:Two-Stage Mechanism
Event 1
Event 2
Event 3
Event 4
Event 5
Event 6
Event 7
Event 8
Stage 1: Information Market
Call Market to Solicit Risk Attitudes
Stage 2: Probability Reporting & Aggregation
Individual Report of Probability DistributionNonlinear Aggregated Function
Tim
e
Experimental Economics Program
Probability Reporting
Reports of Probability Distribution
A B C
Outcome
p1 p2 p3
Pays C1+C2*Log(p3)
Experimental Economics Program
Second Stage: Aggregation Function
Bayes Law with Behavioral Correction
i=r(V i / i)c
Holding value/Risk- measure relative risk of individuals
Normalizing constant for individual risks
“market” risk~sum of prices/winning payoff
sNss2s1
Nss2s1
N21
N21
p...pp
p...ppI|sP
Experimental Economics Program
Private Information Experiments
• 5 Experiments
• Various Information Conditions
– All subject received 3 draws
– Half received 5 draws, half received 1 draw
– Half received 3 draws, half received random number of draws
• 8 to 13 participants in each experiments
Experimental Economics Program
Next Step
• Field Test (Fine and Huberman) …
top related