prediction and change detection
DESCRIPTION
Prediction and Change Detection. Mark Steyvers Scott Brown Mike Yi University of California, Irvine. This work is supported by a grant from the US Air Force Office of Scientific Research (AFOSR grant number FA9550-04-1-0317). Overview. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/1.jpg)
Prediction and Change Detection
Mark Steyvers Scott Brown Mike Yi
University of California, Irvine
This work is supported by a grant from the US Air Force Office of Scientific Research (AFOSR grant number FA9550-04-1-0317)
![Page 2: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/2.jpg)
Overview
• Prediction in non-stationary time-series data
– statistical properties changing over time
– Example: stock market, traffic, weather
• Accurate prediction requires detection of change
• How well can people predict future outcomes?
• What are the individual differences?
![Page 3: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/3.jpg)
Previous Work
• Much work on perception of stationary random sequences:
– Gambler’s fallacy
– Hot hand belief (e.g. Gilovich et al.)
• Shows how people often perceive changes in arguably stationary sequences: overfitting
![Page 4: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/4.jpg)
Our Approach
• Non-stationary random sequences:
– Distribution changes over time at random points
• Allows for perception of
– too little structure: underfitting
– too much structure: overfitting
![Page 5: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/5.jpg)
Basic Task
• Given a sequence of random numbers, predict the next one
![Page 6: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/6.jpg)
Experiment 1
1
2
3
4
5
6
7
8
9
10
11
12
12 PossibleLocations
• Where next blue square will arrive on right side?
![Page 7: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/7.jpg)
Experiment 1
• 15 blocks of 100 trials
• 21 subjects: all get same sequence
• Window shows history of 30 trials
• Each trial is subject initiated
• Points are given for correct or near-to-correct predictions.
![Page 8: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/8.jpg)
0
5
10
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
Sequence Generation
• Locations are drawn from a binomial distribution of size 11, with probability of success θ drawn from [0,1].
• Each time step carries a 10% chance that θ will be changed to a new random value in [0,1]
• Example sequence:
Time
θ=.12 θ=.95 θ=.46 θ=.42 θ=.92 θ=.36
![Page 9: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/9.jpg)
Optimal Strategy
• Optimal strategy: detect change points for θ then identify the mode within each section
• Bayesian model formalizes this strategy(Steyvers & Brown, NIPS, in press)
![Page 10: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/10.jpg)
0
5
10
Optimal Bayesian Solution
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
= observed sequence
Optimal Bayesian Solution= prediction
0
5
10
Optimal Bayesian Solution
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
Subject 4 – change detection too slow0
5
10
Optimal Bayesian Solution
0
5
10
Subject 4
40 50 60 70 80 90 100 110 120 1300
5
10
Time
Subject 12
Subject 12 – change detection too fast
(sequence from block 5)
![Page 11: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/11.jpg)
Tradeoffs
• Detecting the change too slowly will result in lower accuracy and less variability in predictions than an optimal observer.
• Detecting the change very quickly will result in false detections, leading to lower accuracy and higher variability in predictions.
![Page 12: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/12.jpg)
0 0.5 1 1.5 2
1.2
1.4
1.6
1.8
2
2.2
2.4
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
3
4
56
7
8
910
11
12
13
14
15
16
17
1819
2021
OPTIMAL SOLUTION
Average Error vs. Movement
= subject
Relatively many changes
Relatively few changes
![Page 13: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/13.jpg)
A simple model
1. Make new prediction some fraction α of the way between old prediction and recent outcome.
2. Fraction α is a linear function of the error made on last trial
3. Two free parameters: A, B
A<B bigger jumps with higher error
A=B constant smoothing
1 (1 )t t tp p y
t t tError y p
α
0
1
A
B
BA
![Page 14: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/14.jpg)
Sweeping the parameter space
0 0.5 1 1.5 2
1.2
1.4
1.6
1.8
2
2.2
2.4
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
3
4
56
7
8
910
11
12
13
14
15
16
17
1819
2021
OPTIMAL SOLUTION
= subject
= model
![Page 15: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/15.jpg)
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
B parameter
A p
ara
me
ter
1
2
34
56
7
89
10
11
12
13
14
15
16
17
18
19
2021
Best fitting parameters for individual subjects
t t tError y p
α
0
1A=B
A<B
A p
aram
eter
B parameter
α ≈ constant(bad strategy– no
jumps)
Jumps with large errors: good strategy
![Page 16: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/16.jpg)
Effect of A and B parameters
0 0.5 1 1.5 2
1.2
1.4
1.6
1.8
2
2.2
2.4
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
3
4
56
7
8
910
11
12
13
14
15
16
17
1819
2021
OPTIMAL SOLUTION
= subject
= model A ≈ B
= model A << B
![Page 17: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/17.jpg)
Model misses some trends in data…
0
2
4
6
8
10
12 = observed sequence
= prediction
False perception of motion: if successive blocks go up, then extrapolate the trend
(subject 12, block 3)
![Page 18: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/18.jpg)
= observed data
= prediction
Experiment 2: two-dimensional prediction
• Touch screen monitor
• 1500 trials • Self-paced • Same sequence
for all subjects
![Page 19: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/19.jpg)
0.5 1 1.5 2 2.5 3 3.52.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
34
5
6
7
8 9
OPTIMAL SOLUTION
Average Error vs. Movement
= subject
![Page 20: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/20.jpg)
Average Error vs. Movement
0.5 1 1.5 2 2.5 3 3.52.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
Mean Absolute Movement
Me
an
Abs
olu
te T
ask
Err
or
12
34
5
6
7
8 9
OPTIMAL SOLUTION
= subject
= model
![Page 21: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/21.jpg)
Conclusion
• Individual differences
– Overfitters: hypotheses too complex
– Underfitters: hypotheses too simple
• Relation to perception of real-world phenomena?
• Relation to personality characteristics?
![Page 22: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/22.jpg)
Best fitting parameters for individual subjects
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
B parameter
A p
ara
me
ter
12
3
4
5
6
7
8
9
t t tError y p
α
0
1A=B
A<B
![Page 23: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/23.jpg)
Responses across subjects
= observed sequence
= #subjects with that prediction
(sequence from block 5)
60 70 80 90 100 110 120 130
0
2
4
6
8
10
12
Time
![Page 24: Prediction and Change Detection](https://reader030.vdocuments.net/reader030/viewer/2022032414/56813381550346895d9a8c46/html5/thumbnails/24.jpg)
Responses across subjects
40 50 60 70 80 90 100 110 120 130
0
2
4
6
8
10
12
= observed sequence
= #subjects with that prediction
(sequence from block 5)