computer vision i - tracking - heidelberg university · 11/12/2015 computer vision i: tracking 3....
TRANSCRIPT
![Page 1: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/1.jpg)
Computer Vision I -Tracking
Carsten Rother
Computer Vision I: Tracking11/12/2015
11/12/2015
[slide credits: Alex Krull]
![Page 2: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/2.jpg)
What is Tracking?
• „Tracking an object in an image sequence means continuously identifying its location when either the object or the camera are moving“ [Lepetit and Fua 2005]
• This can mean estimating in each frame:
• 2D location• Autostereoscopic Display [1] : eye detection (0.1s) / tracking (40ms)
11/12/2015 Computer Vision I: Tracking 2
[1] H. Heidrich, et al. “Eye position detection system,” Stereoscop. Displ. Virt. Real. Syst., 2000.
![Page 3: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/3.jpg)
What is Tracking?
• „Tracking an object in an image sequence means continuoslyidentifying its location when either the object or the camera aremoving“ [Lepetit and Fua 2005]
• This can mean estimating in each frame:
• 2D location or window
• 6D rigid body transformation
• More complex parametric models• Active Appearance Models
• Skeleton for human pose etc.
11/12/2015 Computer Vision I: Tracking 3
![Page 4: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/4.jpg)
Tracking vs Localization
• Tracking of objects is closely related to:
• camera pose estimation in a known environment
• localization of agents (eg. Robots) in a known environment
• Reminder: SLAM has unknown location (agent, camera) and unknown environment
11/12/2015 Computer Vision I: Tracking 4
![Page 5: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/5.jpg)
Outline
• This lecture
• The Bayes Filter• Explained for localization
• Next lecture
• The Particle Filter
• The Kalman Filter
• Pros and Cons
• Pose Estimation & Pose Tracking [Eric]: • 6-DOF Model Based Tracking via Object Coordinate Regression
11/12/2015 Computer Vision I: Tracking 5
![Page 6: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/6.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 6
• We have:
• Probabilistic model for movement
• Probabilistic model formeasurement• Based on map of the environment
• Where is the ship?
• Using all previous andcurrent observations
![Page 7: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/7.jpg)
1tztz
1txtx
The Hidden Markov Model
11/12/2015 Computer Vision I: Tracking 7
Observations:Depth measurement
States:Positions
time
![Page 8: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/8.jpg)
The Hidden Markov Model
11/12/2015 Computer Vision I: Tracking 8
Observation Model:What is the likelihoodof an observation, given a state?
𝑝 𝑧𝑡 𝑥𝑡 =1
𝑐𝑒− 𝑧𝑡−𝑑 𝑥𝑡
2/(2𝜎2)
1tztz
1txtx
time
𝑑 𝑥𝑡 is the known true depth
![Page 9: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/9.jpg)
The Hidden Markov Model
11/12/2015 Computer Vision I: Tracking 9
Motion Model:Probability forstate transition
𝑝 𝑥𝑡+1 𝑥𝑡
1tztz
1txtx
time
![Page 10: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/10.jpg)
Probabilities - Reminder
• A random variable is denoted with 𝑥 ∈ {0,… , 𝐾}
• Discrete probability distribution: 𝑝(𝑥) satisfies 𝑥 𝑝(𝑥) = 1
• Joint distribution of two random variables: 𝑝(𝑥, 𝑧)
• Conditional distribution: 𝑝 𝑥 𝑧
• Sum rule (marginal distribution): 𝑝 𝑧 = 𝑥 𝑝(𝑥, 𝑧)
• Independent probability distribution: 𝑝 𝑥, 𝑧 = 𝑝 𝑧 𝑝 𝑥
• Product rule: 𝑝 𝑥, 𝑧 = 𝑝 𝑧 𝑥 𝑝(𝑥)
• Bayes’ rule: 𝑝 𝑥|𝑧 =𝑝(𝑧|𝑥)𝑝 𝑥
𝑝(𝑧)
11/12/2015 10Computer Vision I: Tracking
![Page 11: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/11.jpg)
Probabilities - Reminder
• Sum rule (marginal distribution): 𝑝 𝑧 = 𝑥 𝑝(𝑥, 𝑧)
• Product rule: 𝑝 𝑥, 𝑧 = 𝑝 𝑧 𝑥 𝑝(𝑥)
• Bayes’ rule: 𝑝 𝑥|𝑧 =𝑝(𝑧|𝑥)𝑝 𝑥
𝑝(𝑧)
• Sum rule (marginal distribution): 𝑝 𝑧|𝐴 = 𝑥 𝑝(𝑥, 𝑧|𝐴)
• Product rule: 𝑝 𝑥, 𝑧|𝐴 = 𝑝 𝑧 𝑥, 𝐴 𝑝(𝑥|𝐴)
• Bayes’ rule: 𝑝 𝑥|𝑧, 𝐴 =𝑝(𝑧|𝑥,𝐴)𝑝 𝑥|𝐴
𝑝(𝑧|𝐴)
11/12/2015 11Computer Vision I: Tracking
![Page 12: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/12.jpg)
Probabilities - Reminder
11/12/2015 12Computer Vision I: Tracking
𝐴 𝐵
𝐴
𝐶
𝐵
Independence
𝐴 and 𝐵 are not connected in graph
• 𝐴 does not contain informationabout 𝐵
• 𝑝 𝐴, 𝐵 = 𝑝 𝐴 𝑝 𝐵
• 𝑝 𝐴|𝐵 = 𝑝 𝐴
• 𝑝 𝐵|𝐴 = 𝑝(𝐵)
Conditional Independence
𝐴 and 𝐵 are connected only via 𝐶
• 𝐴 does not contain informationabout 𝐵 when 𝐶 is known
• 𝑝 𝐴, 𝐵|𝐶 = 𝑝 𝐴|𝐶 𝑝 𝐵|𝐶
• 𝑝 𝐴|𝐵, 𝐶 = 𝑝 𝐴|𝐶
• 𝑝 𝐵|𝐴, 𝐶 = 𝑝(𝐵|𝐶)
Two dice rolls
It rains
People have umbrellas
Tram is crowded
![Page 13: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/13.jpg)
Probability distribution for the state given all previous and currentobservations:
• This is what we are interested in
• Eg. use maximum as current estimate
The Posterior distribution
11/12/2015 Computer Vision I: Tracking 13
tz
tx
1tz
1tx
0z
0x
...
...
𝑝 𝑥𝑡 𝑧0:𝑡
𝑥𝑡∗ = 𝑎𝑟𝑚𝑎𝑥𝑥𝑡
𝑝 𝑥𝑡 𝑧0:𝑡
![Page 14: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/14.jpg)
The Prior distribution
11/12/2015 Computer Vision I: Tracking 14
1txtx
1tz
1tx
0z
0x
...
...
𝑝 𝑥𝑡+1 𝑧0:𝑡
Probability distribution for the next state given only previousobservations:
• Intermediate step for calculating the next posterior
tz
![Page 15: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/15.jpg)
Important Distributions
11/12/2015 Computer Vision I: Tracking 15
𝑝 𝑥𝑡+1 𝑧0:𝑡
1txtx
1tz
1tx
0z
0x ...
tztz
tx
1tz
1tx
0z
0x ...
𝑝 𝑥𝑡 𝑧0:𝑡
1txtx
tz
tx
𝑝 𝑧𝑡 𝑥𝑡𝑝 𝑥𝑡+1 𝑥𝑡
Observation Model: Motion Model:
Posterior: Prior:
• Likelihood ofobservation givenstate
• Continuous Gaussianaround real depth
• Probability of newstate given old one
• Discrete Gaussian
• Probability of state given previousand current observations
• Probability of state given onlyprevious observations
![Page 16: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/16.jpg)
Step by Step
11/12/2015 Computer Vision I: Tracking 16
𝑡 = 0
• Assume prior for first frame:
• Make first measurement: 𝑧0
𝑝(𝑥0)
![Page 17: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/17.jpg)
Step by Step
11/12/2015 Computer Vision I: Tracking 17
𝑡 = 0
• Calculate likelihood 𝑝(𝑧0|𝑥0) for every possible state 𝑥0:
![Page 18: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/18.jpg)
Step by Step
11/12/2015 Computer Vision I: Tracking 18
𝑡 = 0
• Calculate the posterior by• multiplying with prior• Normalizing
• Reducing uncertainty𝑝 𝑥0 𝑧0 =
𝑝(𝑥0)𝑝(𝑧0|𝑥0)
𝑥 𝑝(𝑥0 = 𝑥)𝑝(𝑧0|𝑥0 = 𝑥)
![Page 19: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/19.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 19
𝑡 = 0
𝑝 𝑥1 𝑧0 =
𝑥
𝑝 𝑥0 = 𝑥 𝑧0 𝑝 𝑥1 𝑥0 = 𝑥
• Calculate the prior by Convolutionwith motion model
• Adding uncertainty
![Page 20: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/20.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 20
𝑡 = 1
• Make new measurement: 𝑧1
![Page 21: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/21.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 21
𝑡 = 1
• Calculate likelihood 𝑝(𝑧1|𝑥1) for every possible state 𝑥1:
![Page 22: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/22.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 22
𝑡 = 1
• Calculate the posterior by• multiplying with prior• Normalizing
• Reducing uncertainty𝑝 𝑥1 𝑧0:1 =
𝑝(𝑥1|𝑧0)𝑝(𝑧1|𝑥1)
𝑥 𝑝(𝑥1 = 𝑥|𝑧0)𝑝(𝑧1|𝑥1 = 𝑥)
![Page 23: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/23.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 23
𝑡 = 1
• Calculate the prior by Convolutionwith motion model
• Adding uncertainty
𝑝 𝑥2 𝑧0:1 =
𝑥
𝑝 𝑥1 = 𝑥 𝑧0:1 𝑝(𝑥2|𝑥1 = 𝑥)
![Page 24: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/24.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 24
𝑡 = 2
• Make new measurement: 𝑧2
![Page 25: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/25.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 25
𝑡 = 2
• Calculate likelihood 𝑝(𝑧2|𝑥2) for every possible state 𝑥2:
![Page 26: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/26.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 26
𝑡 = 2
• Calculate the posterior by• multiplying with prior• Normalizing
• Reducing uncertainty𝑝 𝑥2 𝑧0:2 =
𝑝(𝑥2|𝑧0:1)𝑝(𝑧2|𝑥2)
𝑥 𝑝(𝑥2 = 𝑥|𝑧0:1)𝑝(𝑧2|𝑥2 = 𝑥)
![Page 27: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/27.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 27
𝑡 = 2
![Page 28: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/28.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 28
Algorithm:
1. Make observation
2. Calculate likelihood for every position
3. Multiply with last prior and normalize
• Calculate posterior
4. Convolution with motion model
• Calculate new prior
5. Go to 1.
![Page 29: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/29.jpg)
𝑝 𝑥0 𝑧0
Calculating the Posterior
11/12/2015 Computer Vision I: Tracking 29
𝑝(𝑥0)
0z
1x0x
1z
...
=𝑝(𝑥0)𝑝(𝑧0|𝑥0)
𝑥 𝑝(𝑥0 = 𝑥)𝑝(𝑧0|𝑥0 = 𝑥)
(Multiply and normalize)Observation model
(see proof 1)
![Page 30: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/30.jpg)
Proof 1
11/12/2015 Computer Vision I: Tracking 30
![Page 31: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/31.jpg)
Calculating the next Prior
11/12/2015 Computer Vision I: Tracking 31
𝑝(𝑥0)
𝑝 𝑥0 𝑧0
𝑝 𝑥1 𝑧0
...
=
𝑥
𝑝 𝑥0 = 𝑥 𝑧0 𝑝 𝑥1 𝑥0 = 𝑥
(Convolution)
Observation model
0z
1x0x
1z
(see proof 2)
![Page 32: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/32.jpg)
Proof 2
11/12/2015 Computer Vision I: Tracking 32
![Page 33: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/33.jpg)
Calculating the Posterior
11/12/2015 Computer Vision I: Tracking 33
𝑝(𝑥0)
𝑝 𝑥0 𝑧0
𝑝 𝑥1 𝑧0
...
Observation model
1x0x
Observation model
𝑝 𝑥1 𝑧0:1 =𝑝(𝑥1|𝑧0)𝑝(𝑧1|𝑥1)
𝑥 𝑝(𝑥1 = 𝑥)𝑝(𝑧1|𝑥1 = 𝑥)
1z0z
![Page 34: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/34.jpg)
𝑝 𝑥𝑡 𝑧0:𝑡
Calculating the Posterior (General Case)
11/12/2015 Computer Vision I: Tracking 34
𝑝(𝑥𝑡|𝑧0:𝑡−1)
tz
1txtx
1tz
...
=𝑝(𝑥𝑡|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡)
𝑥 𝑝(𝑥𝑡 = 𝑥|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡 = 𝑥)
(Multiply and normalize)Observation model
...
(see proof 3)
![Page 35: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/35.jpg)
Proof 3
11/12/2015 Computer Vision I: Tracking 35
![Page 36: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/36.jpg)
Calculating the next Prior (General Case)
11/12/2015 Computer Vision I: Tracking 36
𝑝(𝑥𝑡|𝑧0:𝑡−1)
𝑝 𝑥𝑡 𝑧0:𝑡
𝑝 𝑥𝑡+1 𝑧0:𝑡
...
=
𝑥
𝑝 𝑥𝑡 = 𝑥 𝑧0:𝑡 𝑝(𝑥𝑡+1|𝑥𝑡 = 𝑥)
(Convolution)
Observation model
...
tz
1txtx
1tz
(see proof 4)
![Page 37: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/37.jpg)
Proof 4
11/12/2015 Computer Vision I: Tracking 37
![Page 38: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/38.jpg)
Overview Continuous Approaches
11/12/2015 Computer Vision I: Tracking 38
• How to apply it in continuous space?
• Two popular alternatives:
• Particle filter• Represent prior and posterior with samples
• Kalman Filter • Represent prior and posterior distributions as Gaussians
![Page 39: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/39.jpg)
Important Distributions (Particle Filter)
11/12/2015 Computer Vision I: Tracking 39
𝑝 𝑥𝑡+1 𝑥𝑡
Observation Model: Motion Model:
Posterior: Prior:
• Likelihood ofobservation givenstate
• Continuous Gaussianaround real depth
• Probability of newstate given old one
• Continuous Gaussian
• Probability of state given previousand current observations
• Continuous, represented as set ofSamples (Particles)
• Probability of state given onlyprevious observations
• Continuous, represented as set ofSamples (Particles)
tz
tx
1tz
1tx
0z
0x ...
𝑝 𝑥𝑡 𝑧0:𝑡 𝑝 𝑥𝑡+1 𝑧0:𝑡
1txtx
1tz
1tx
0z
0x ...
tz
1txtx
tz
tx
𝑝 𝑧𝑡 𝑥𝑡
![Page 40: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/40.jpg)
Particle Filter
11/12/2015 Computer Vision I: Tracking 40
![Page 41: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/41.jpg)
Discrete Bayes Filter vs. Particle Filter
11/12/2015 Computer Vision I: Tracking 41
• Discrete Bayes Filter:
1. Make observation
• Particle Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery sample -> weights
3. Resampling according toweights
4. Randomly move samplesaccording to motion model(Sampling)
5. Go to 1.
![Page 42: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/42.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 42
𝑡 = 0
• Represent continuous prior with particles 𝑥𝑡1 , … , 𝑥𝑡
𝑛
• Make measurement: 𝑧𝑡
![Page 43: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/43.jpg)
Particle Filter / Resampling
11/12/2015 Computer Vision I: Tracking 43
• Calculate weight 𝑤𝑡𝑖= 𝑝(𝑧𝑡|𝑥𝑡 = 𝑥𝑡
𝑖) for every particle 𝑥𝑡𝑖:
![Page 44: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/44.jpg)
Particle Filter / Resampling
11/12/2015 Computer Vision I: Tracking 44
𝑡 = 0
• Draw samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛 from posterior by resamplingfrom 𝑥𝑡
1 , … , 𝑥𝑡𝑛 using the weights 𝑤𝑡
𝑖
• Reducing uncertainty
0 i
i
tw
![Page 45: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/45.jpg)
Particle Filter / Resampling
11/12/2015 Computer Vision I: Tracking 45
𝑡 = 0
random
0 i
i
twrandom random
• Draw samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛 from posterior by resamplingfrom 𝑥𝑡
1 , … , 𝑥𝑡𝑛 using the weights 𝑤𝑡
𝑖
• Reducing uncertainty
![Page 46: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/46.jpg)
Particle Filter / Resampling
11/12/2015 Computer Vision I: Tracking 46
𝑡 = 0
0 i
i
tw
• Draw samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛 from posterior by resamplingfrom 𝑥𝑡
1 , … , 𝑥𝑡𝑛 using the weights 𝑤𝑡
𝑖
• Reducing uncertainty
![Page 47: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/47.jpg)
Particle Filter / Resampling
11/12/2015 Computer Vision I: Tracking 47
Why is this allowed?
Resampling is like multiplication andnormalization:
• Sample density in posterior dependslinearly on
• Density of prior samples 𝑥𝑡1 , … , 𝑥𝑡
𝑛
• Likelihood 𝑤𝑡𝑖
![Page 48: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/48.jpg)
Discrete Bayes Filter vs. Particle Filter
11/12/2015 Computer Vision I: Tracking 48
• Discrete Bayes Filter:
1. Make observation
• Particle Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery sample -> weights
3. Resampling according toweights
4. Randomly move samplesaccording to motion model(Sampling)
5. Go to 1.
![Page 49: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/49.jpg)
Particle Filter / Sampling
11/12/2015 Computer Vision I: Tracking 49
𝑡 = 0
• Obtain samples 𝑥𝑡+11 , … , 𝑥𝑡+1
𝑛 from the new prior by moving eachparticle according to motion model
• Adding uncertainty
![Page 50: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/50.jpg)
The Bayes Filter / Convolute and Multiply
11/12/2015 Computer Vision I: Tracking 50
𝑡 = 0
𝑝 𝑥1 𝑧0 =
𝑥
𝑝 𝑥0 = 𝑥 𝑧0 𝑝 𝑥1 𝑥0 = 𝑥
• Calculate the prior by Convolutionwith motion model
• Adding uncertainty
![Page 51: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/51.jpg)
Particle Filter / Sampling
11/12/2015 Computer Vision I: Tracking 51
𝑝 𝐵 𝐴 = 1 𝑝 𝐵 𝐴 = 2
𝑝 𝐴 = 1 = 𝑝 𝐴 = 2 = 0.5
Why is this allowed?
• We want to sample: Bi~𝑝 𝐵
• Don‘t know 𝑝 𝐵
• We sample first: 𝐴𝑖~𝑝 𝐴
• And then 𝐵𝑖~𝑝 𝐵|𝐴 = 𝐴𝑖
1
1 2 3 4 5 61 2 3 4 5 6
𝐴 𝐵
![Page 52: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/52.jpg)
BA
Particle Filter / Sampling
11/12/2015 Computer Vision I: Tracking 52
1txtx
1tz
1tx
0z
0x...
tz
• We want to sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑧0:𝑡
• Don‘t know 𝑝 𝑥𝑡+1 𝑧0:𝑡
Why is this allowed?
![Page 53: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/53.jpg)
Particle Filter / Sampling
11/12/2015 5311/12/2015 Computer Vision I: Tracking 53
1txtx1tx0x...
1tz0z tz
• We want to sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑧0:𝑡
• Don‘t know 𝑝 𝑥𝑡+1 𝑧0:𝑡
• We use samples 𝑥𝑡𝑖~𝑝 𝑥𝑡 𝑧0:𝑡
Why is this allowed?
![Page 54: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/54.jpg)
Particle Filter / Sampling
11/12/2015 Computer Vision I: Tracking 54
• We want to sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑧0:𝑡
• Don‘t know 𝑝 𝑥𝑡+1 𝑧0:𝑡
• We use samples 𝑥𝑡𝑖~𝑝 𝑥𝑡 𝑧0:𝑡
• We sample 𝑥𝑡+1𝑖 ~𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥𝑡
𝑖 = 𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥𝑡𝑖 , 𝑧0:𝑡
1txtx1tx0x...
1tz0z tz
Why is this allowed?
![Page 55: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/55.jpg)
Particle Filter (Tracking application)
• Tracking an object in a video sequence [Perez at al. 2002]
• States:
• 2D windows (location and size)
• Gaussian motion
• Observations:
• Color histograms
11/12/2015 Computer Vision I: Tracking 55
![Page 56: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/56.jpg)
Particle Filter (Tracking application)
• Pose tracking of an object in a kinect sequence [Krull at al. 2014]
• States:
• 6D Pose• 3D position
• 3D rotation
• Observations:
• Depth images
• Predicted Object coordinates
11/12/2015 Computer Vision I: Tracking 56
![Page 57: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/57.jpg)
Particle Filter (Summary)
• The Particle Filter implements the Bayes Filter
• Prior and posterior are represented as sample sets (particle)
• Likelihood is only evaluated at particles
• Multiplication -> weighted resampling
• Convolution -> random movement according to motion model
11/12/2015 Computer Vision I: Tracking 57
![Page 58: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/58.jpg)
Overview Continuous Approaches
• How to apply it in continuous space?
• Two popular alternatives:
• Particle filter• Represent prior and posterior with samples
• Kalman Filter • Represent prior and posterior distributions as Gaussians
11/12/2015 Computer Vision I: Tracking 58
![Page 59: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/59.jpg)
Kalman Filter
11/12/2015 Computer Vision I: Tracking 59
![Page 60: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/60.jpg)
Important Distributions (Kalman Filter)
11/12/2015 Computer Vision I: Tracking 60
𝑝 𝑧𝑡 𝑥𝑡𝑝 𝑥𝑡+1 𝑥𝑡
Observation Model: Motion Model:
Posterior: Prior:
• Likelihood ofobservation givenstate
• Continuous Gaussianaround real position
• Probability of newstate given old one
• Continuous Gaussian
• Probability of state given previousand current observations
• Continuous, represented asGaussiasn
• Probability of state given onlyprevious observations
• Continuous, represented asGaussiasn
tz
tx
1tz
1tx
0z
0x ...
𝑝 𝑥𝑡 𝑧0:𝑡 𝑝 𝑥𝑡+1 𝑧0:𝑡
1txtx
1tz
1tx
0z
0x ...
tz
tz
tx
![Page 61: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/61.jpg)
Discrete Bayes Filter vs. Kalman Filter
11/12/2015 Computer Vision I: Tracking 61
• Discrete Bayes Filter:
1. Make observation
• Kalman Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery position in closed form
3. Multiply with last priorand normalize in closed form
4. Convolution with motionmodel in closed form
5. Go to 1.
![Page 62: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/62.jpg)
Kalman Filter
11/12/2015 Computer Vision I: Tracking 62
• Calculate likelihood 𝑝(𝑧𝑡|𝑥𝑡) as Gaussian:
![Page 63: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/63.jpg)
Kalman Filter
11/12/2015 Computer Vision I: Tracking 63
• Calculate the posterior by• Multiplying with prior• Normalizing
• Reducing uncertainty
𝑝 𝑥𝑡 𝑧0:𝑡 =𝑝(𝑥𝑡|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡)
𝑝(𝑥𝑡 = 𝑥|𝑧0:𝑡−1)𝑝(𝑧𝑡|𝑥𝑡 = 𝑥) 𝑑 𝑥
• Closed form solution:
• Another Gaussian
𝜎1,2 =𝜎1
2𝜎22
𝜎12 + 𝜎2
2
𝜇1,2 =𝜇1𝜎2
2 + 𝜇2𝜎12
𝜎12 + 𝜎2
2
![Page 64: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/64.jpg)
Discrete Bayes Filter vs. Kalman Filter
11/12/2015 Computer Vision I: Tracking 64
• Discrete Bayes Filter:
1. Make observation
• Kalman Filter:
2. Calculate likelihood forevery position
3. Multiply with last priorand normalize
4. Convolution with motionmodel
5. Go to 1.
1. Make observation
2. Calculate likelihood forevery position in closed form
3. Multiply with last priorand normalize in closed form
4. Convolution with motionmodel in closed form
5. Go to 1.
![Page 65: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/65.jpg)
Kalman Filter
11/12/2015 Computer Vision I: Tracking 65
• Calculate the prior byConvolution with motion model
• Adding uncertainty
𝑝 𝑥𝑡+1 𝑧0:𝑡 = 𝑝 𝑥𝑡 = 𝑥 𝑧0:𝑡 𝑝 𝑥𝑡+1 𝑥𝑡 = 𝑥 𝑑 𝑥
• Closed form solution:
• Another Gaussian
𝜎𝑡+1 = 𝜎𝑡2 + 𝜎𝑡→𝑡+1
2
𝜇𝑡+1 = 𝜇𝑡 + 𝜇𝑡→𝑡+1
![Page 66: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/66.jpg)
Pros and Cons
11/12/2015 Computer Vision I: Tracking 66
Particle Filter:
Observation model can be anything
Kalman Filter:
Multimodal
Likelihood calculation per particlecan be expensive
Easy to implement and parallelize
Problematic in high dimensional statespace (many particles required)
Motion model can be anything
Observation model: linear transformation of stateplus Gaussian noise
Unimodal
Motion model: linear transformation of last stateplus Gaussian noise
Closed form solution in every step
![Page 67: Computer Vision I - Tracking - Heidelberg University · 11/12/2015 Computer Vision I: Tracking 3. Tracking vs Localization •Tracking of objects is closely related to: •camera](https://reader036.vdocuments.net/reader036/viewer/2022081523/5fcb0b834f5e1555df3e8d97/html5/thumbnails/67.jpg)
Pose Estimation vs. Pose Tracking
11/12/2015 Computer Vision I: Tracking 67
One Shot Pose Estimation [1] Pose Tracking
[1] Brachmann, E., Krull, A., Michel, F., Shotton, J., Gumhold, S., Rother, C.: Learning 6d object pose estimation using 3d object coordinates, ECCV (2014)
• Estimate 6D Pose from singleRGB-D image
• Use Object Coordinate Regression
• Stream of RGB-D images
• Use information from previous frames:• Realtime
• Increase robustness, accuracy