Using Informative Priors to Enhance Wisdom in Small Crowds

Eyewitness testimony

Common problems Reliance on error-prone eyewitnesses Reliance on very small number of eyewitnesses (often one)

This research Integrating recalled memories across multiple individuals Focus on situations involving small number of individuals with

potentially poor memory

Approach Incorporate informative priors into a wisdom of crowd

aggregation model Prior knowledge is extracted from a different group of individuals

2

Illustrative example: Galton’s Ox

Galton’s original experiment estimate weight of ox (true answer is 1198 pounds) large number of individuals (800) median answer came within 9 pounds of true weight

Thought experiment Suppose we only have small numbers of individuals Suppose we know something about ox weight a priori

3

Galton’s Ox with Prior Information

4

),(~ 00 Ntrue

),(~ memtruej Ny true

0

jyj=1,..,N

Goal: infer the true mean from the observed estimates

Assumptions: 8.54,75,1150 00 mem

0

mem

Bayesian Inference

Standard solution:

Measure mean absolute difference

5

ywwtrue )1(ˆ 0

220

20

1/

1

mem

Nw

|ˆ| truetrue

Simulation

Vary the number of individuals (N) and strength of the assumed prior by the researcher (σ0)

6

N 75 150 300 ∞1 35.4 39.2 42.2 43.72 27.4 29.3 30.5 30.83 23.3 24.4 24.9 25.25 18.7 19.1 19.3 19.5

10 13.4 13.6 13.8 13.920 9.6 9.8 9.7 9.8

0

uninformative prior

(based on 50,000 repetitions of Galton’s experiment in each cell)

Application to Episodic Memory

We apply the same ideas to an episodic memory task: serial recall

Serial recall experiment (N=28) Study event sequence in video Order the test images from memory

Norming experiment (N=16) No study phase Order test images in as natural order as possible Allows us to build a model for prior probability of each sequence

7

Materials

Type I 3 videos with stereotyped event sequences (e.g. wedding) associated with “strong prior”

Type II 3 videos with less predictable event sequence associated with “weak prior”

Extracted 10 images for testing

8

Example Type I Sequence: Bus video

9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 284 3 13 1 9 5 3 12 11 10 10 2 5 1 1 10 2 1 2 4 2 5 1 1 1 1 2 1 6 1 1 0 1 2 2 1 2 1 1 1 2 0 1 1A B A A A B B A B B A A D B A B B A A A B B A A A A A A A A A A A B B A A A B A B A A AB C C B C D A D A A C B B A B A A B B B A A B B B B B B G B B B B A A B B B A C A B B BE A E C D A D G C E E D A C D E C C C G C C C C C C C C C C C C C C C C C C C B C C C CD D F D E E C C E C D C C D C F D D D C D D D D D D D D B D D D D E D D D D D D D D E DC E G E G C E E D G G E E E E D E E G D E H E E F F F E D F E E E D E E F E E E E E D EG G H G F G G F J F F G G F F G G G E E G F G G E E G G E E G F G F G G G G F F G F F GF F I F H F F H I H H F F G G H F F F F F E F F G G E F F G F G F G F F E F G G F G G FH H D H I H H I H J I H H H H I H H H H H G H H H H H H H H H H H H H H H H H H H H H HI I J I J I I J F I J I I I I J I I I I I I I I I I I I I I I I I I I I I I I I I I I IJ J B J B J J B G D B J J J J C J J J J J J J J J J J J J J J J J J J J J J J J J J J J

prior serial recall

Note: first row is subject id, second row is Kendall tau to true ordering

A B C D E

F G H I J

Example Type II Sequence: Clay Video

10

prior serial recall1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 2823 29 28 30 24 14 27 18 23 30 16 28 23 33 31 31 9 16 4 10 4 5 14 6 8 12 7 10 13 7 0 12 12 16 13 3 10 24 6 24 13 1 2 23H H H I G D H F H H D H D F J H A A A A A A B B A F A A H A A A J A F A A I A F J A A EB D I E H E B D J J A D E I I E D C B F B D C C B A C D A C B B A I A B B C C A B B B DJ E D G I B A I A G F E G J E D B G D C C B A D F C D E F B C F D C D C D J B I C C D JA J E D B G J B F C I J H G D J F E F D F F I F C D F F C D D E C J E F J B E C A E C GI I F F F A I A B B C I B E G I E I C B E C F A I E E G B F E G B B G D C F F J D D F FF C A J A H G C G F B C F H B C H D E I D E H G D G B B D H F I E F B E H A D G F F E BC F J A C C E J C I H F J A C B C H G G G G G E E I I C E G G C F D H G E G I E G G G CD B C H D I D E E A J A I B F G J F H E H I D H H H G J I I H H G E I H F E H H E H H AE A G C J J F G D D E B C D A F G J J H J H E I G B H H G J I J H G C J G D G B H I I HG G B B E F C H I E G G A C H A I B I J I J J J J J J I J E J D I H J I I H J D I J J I

A B C D E

F G H I J

Empirical Results: distribution of tau distances

11Note: bin width is two so first bar is the relative proportion to tau = 0 and tau = 1

Type I

Type II

0 4 8 12 16 20 24 28 320

0.1

0.2

0.3

0.4

0.5

0.6wedding/morning/bus

0 4 8 12 16 20 24 28 320

0.05

0.1

0.15 clay/pizza/yogurt

PriorEpisodic MemoryChance

Mallows Model

12

)|(

)ψ(

1),|( ωyωy dep

),Mallows(~,| ωωy

Kendall tau distance

Scaling parameter Normalizing constant

Latent truthRecalled order

ω yj θjj=1,..,M

Model 1: Mallows Model with Uninformative Prior

)/1,Gamma(~ 0 j

)Uniform(~ ω

),Mallows(~,| jjj ωωy

ω yj θjj=1,..,M

θ*

ωoyojθojj=1,..,N

Model 2: Mallows Model with Informative Prior

(prior knowledge data) (memory data)

),Mallows(~ *0 ωω

),Mallows(~,| jjj ωωy

),Mallows(~,| 00000jjj ωωy

Example calibration result

15

-3 -2 -1 0 1

0

5

10

15

20

25

log

R=-0.969

(clay video; all subs)

-3 -2 -1 0 1

0

5

10

15

20

25

log

R=-0.986

Model 1 Model 2

Wisdom of Crowds Effect (Model 1)

16

0

2

4

6

8

10

12

14

16

Individuals

Mea

n

ModelIndividuals

Pick worst K individuals (from episodic task)

17(means of taus across mcmc samples; averaged across 3 videos for each type)

1 2 5 10 280

5

10

15

20

25

K

Mea

n

Type I

ChanceModel 1Model 2

1 2 5 10 280

5

10

15

20

25

K

Mea

n

Type II

Modeling Results(pick worst K individuals; means of taus across mcmc samples)

18

Type Sequence K=1 K=2 K=5 K=10 K=28 K=1 K=2 K=5 K=10 K=28

I Bus 16.53 9.08 2.77 1.99 1.00 6.68 4.44 2.26 1.90 1.00Morning 22.11 18.64 4.06 3.00 0.00 10.36 7.47 3.57 3.00 0.00Wedding 19.13 16.42 9.52 5.26 0.00 16.06 15.23 9.13 4.91 0.00

II Yogurt 22.11 20.51 18.80 16.82 4.53 19.16 19.04 17.68 16.32 4.88Pizza 23.00 25.46 23.78 18.16 3.69 23.17 23.12 23.32 16.91 3.38Clay 23.57 23.71 20.56 15.99 1.09 24.76 25.84 22.53 15.59 1.10

Model 1 Model 2

Pick random K individuals

19(means of taus across mcmc samples; averaged across 3 videos for each type)

1 2 5 10 280

5

10

15

20

25

K

Mea

n

Type I

ChanceModel 1Model 2

1 2 5 10 280

5

10

15

20

25

K

Mea

n

Type II

Pick random K individuals

20means of taus across mcmc samples

Type Sequence K=1 K=2 K=5 K=10 K=28 K=1 K=2 K=5 K=10 K=28

I Bus 14.01 3.194 1.025 0.664 1.00 4.09 1.916 1.048 0.882 1.00Morning 14.01 3.919 0.577 0.295 0.00 5.71 2.27 0.418 0.264 0.00Wedding 14.73 5.693 1.951 0.37 0.00 15.85 6.794 1.557 0.627 0.00

II Yogurt 17.08 12.97 8.487 5.029 4.53 15.41 13.03 7.495 4.897 4.88Pizza 17.92 14.31 8.383 5.931 3.69 16.73 15.81 9.37 5.819 3.38Clay 17.2 13.69 6.568 3.303 1.09 22.34 17.45 7.261 3.614 1.10

Model 1 Model 2

Left over

21

Pick worst K individuals (single tau)

22

1 2 5 10 280

5

10

15

20

25

K

Mea

n

Type I

ChanceModel 1Model 2

1 2 5 10 280

5

10

15

20

25

K

Mea

n

Type II

Modeling Results(pick worst K individuals; single tau – use Borda count to aggregate samples)

23

Type Sequence K=1 K=2 K=5 K=10 K=28 K=1 K=2 K=5 K=10 K=28

I Bus 7 5 2 2 1 3 2 2 2 1Morning 23 12 4 3 0 5 4 3 3 0Wedding 14 9 10 5 0 13 13 8 5 0

II Yogurt 23 19 18 16 5 17 19 18 16 5Pizza 25 27 24 19 3 24 23 19 13 3Clay 24 25 20 14 1 30 29 22 14 1

Model 1 Model 2

This research

Wisdom of crowds Recover original sequence of events based on recollected order

from a group of individuals

Prior knowledge We measure prior knowledge about event sequences Incorporate this prior knowledge into our wisdom of crowd

aggregation model informative priors

24