lecture: motion - stanford computer vision lab
TRANSCRIPT
![Page 1: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/1.jpg)
Motion
Stanford University
19-Nov-2019
1
Lecture: Motion
Juan Carlos Niebles and Ranjay KrishnaStanford Vision and Learning Lab
![Page 2: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/2.jpg)
Motion
Stanford University
19-Nov-2019
2
CS 131 Roadmap
Pixels Images
ConvolutionsEdgesDescriptors
Segments
ResizingSegmentationClustering
RecognitionDetectionMachine learning
Videos
MotionTracking
Web
Neural networksConvolutional neural networks
![Page 3: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/3.jpg)
Motion
Stanford University
19-Nov-2019
3
What will we learn today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
Reading: [Szeliski] Chapters: 8.4, 8.5[Fleet & Weiss, 2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 4: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/4.jpg)
Motion
Stanford University
19-Nov-2019
4
What will we learn today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
Reading: [Szeliski] Chapters: 8.4, 8.5[Fleet & Weiss, 2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 5: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/5.jpg)
Motion
Stanford University
19-Nov-2019
5
From images to videos
• A video is a sequence of frames captured over time• Now our image data is a function of space (x, y) and time (t)
![Page 6: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/6.jpg)
Motion
Stanford University
19-Nov-2019
6
Why is motion useful?
![Page 7: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/7.jpg)
Motion
Stanford University
19-Nov-2019
7
Why is motion useful?
![Page 8: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/8.jpg)
Motion
Stanford University
19-Nov-2019
8
Optical flow• Definition: optical flow is the apparent motion of brightness
patterns in the image
• Note: apparent motion can be caused by lighting changes without any actual motion– Think of a uniform rotating sphere under fixed lighting vs. a stationary
sphere under moving illumination
GOAL: Recover image motion at each pixel from optical flow
Sour
ce: S
ilvio
Sav
ares
e
![Page 9: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/9.jpg)
Motion
Stanford University
19-Nov-2019
9Picture courtesy of Selim Temizer - Learning and Intelligent Systems (LIS) Group, MIT
Optical flow
Vector field function of the spatio-temporal image brightness variations
![Page 10: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/10.jpg)
Motion
Stanford University
19-Nov-2019
10
Estimating optical flow
• Given two subsequent frames, estimate the apparent motion field u(x,y), v(x,y) between them
• Key assumptions• Brightness constancy: projection of the same point looks the same in every
frame• Small motion: points do not move very far• Spatial coherence: points move like their neighbors
I(x,y,t) I(x,y,t+1)
Sour
ce: S
ilvio
Sav
ares
e
![Page 11: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/11.jpg)
Motion
Stanford University
19-Nov-2019
11
Key Assumptions: small motions
* Slide from Michael Black, CS143 2003
![Page 12: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/12.jpg)
Motion
Stanford University
19-Nov-2019
12
Key Assumptions: spatial coherence
* Slide from Michael Black, CS143 2003
![Page 13: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/13.jpg)
Motion
Stanford University
19-Nov-2019
13
Key Assumptions: brightness Constancy
* Slide from Michael Black, CS143 2003
𝐼 𝑥, 𝑦, 𝑡 = 𝐼(𝑥 + 𝑢 𝑥, 𝑦 , 𝑦 + 𝑣 𝑥, 𝑦 , 𝑡 + 1)
![Page 14: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/14.jpg)
Motion
Stanford University
19-Nov-2019
14
• Brightness Constancy Equation:
Linearizing the right side using Taylor expansion:
I(x,y,t) I(x,y,t+1)
0»+×+× tyx IvIuIHence,
Image derivative along x
→∇I ⋅ u v[ ]T + It = 0
The brightness constancy constraint
Sour
ce: S
ilvio
Sav
ares
e
𝐼 𝑥, 𝑦, 𝑡 = 𝐼(𝑥 + 𝑢, 𝑦 + 𝑣, 𝑡 + 1)
𝐼 𝑥 + 𝑢, 𝑦 + 𝑣, 𝑡 + 1 ≈ 𝐼 𝑥, 𝑦, 𝑡 + 𝐼. / 𝑢 + 𝐼0 / 𝑣 + 𝐼1Image derivative along t
𝐼 𝑥 + 𝑢, 𝑦 + 𝑣, 𝑡 + 1 − 𝐼 𝑥, 𝑦, 𝑡 ≈ 𝐼. / 𝑢 + 𝐼0 / 𝑣 + 𝐼1
![Page 15: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/15.jpg)
Motion
Stanford University
19-Nov-2019
15
Filters used to find the derivatives
𝐼. 𝐼0 𝐼1
![Page 16: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/16.jpg)
Motion
Stanford University
19-Nov-2019
16
The brightness constancy constraint
• How many equations and unknowns per pixel?
The component of the flow perpendicular to the gradient (i.e., parallel to the edge) cannot be measured
edge
(u,v)
(u’,v’)
gradient
(u+u’,v+v’)If (u, v ) satisfies the equation, so does (u+u’, v+v’ ) if
•One equation (this is a scalar equation!), two unknowns (u,v)
∇I ⋅ u ' v '[ ]T = 0
Can we use this equation to recover image motion (u,v) at each pixel?
∇I ⋅ u v[ ]T + It = 0
Sour
ce: S
ilvio
Sav
ares
e
∇𝐼
![Page 17: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/17.jpg)
Motion
Stanford University
19-Nov-2019
17
The aperture problem
Sour
ce: S
ilvio
Sav
ares
e
Actual motion
![Page 18: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/18.jpg)
Motion
Stanford University
19-Nov-2019
18
The aperture problem
Sour
ce: S
ilvio
Sav
ares
e
![Page 19: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/19.jpg)
Motion
Stanford University
19-Nov-2019
19
The aperture problem
Sour
ce: S
ilvio
Sav
ares
e
Perceived motion
![Page 20: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/20.jpg)
Motion
Stanford University
19-Nov-2019
20
The aperture problem
Actual motion
Sour
ce: S
ilvio
Sav
ares
e
![Page 21: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/21.jpg)
Motion
Stanford University
19-Nov-2019
21
The aperture problem
Perceived motion
Sour
ce: S
ilvio
Sav
ares
e
![Page 22: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/22.jpg)
Motion
Stanford University
19-Nov-2019
22
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
![Page 23: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/23.jpg)
Motion
Stanford University
19-Nov-2019
23
The barber pole illusion
http://en.wikipedia.org/wiki/Barberpole_illusion
Sour
ce: S
ilvio
Sav
ares
e
![Page 24: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/24.jpg)
Motion
Stanford University
19-Nov-2019
24
What will we learn today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
Reading: [Szeliski] Chapters: 8.4, 8.5[Fleet & Weiss, 2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 25: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/25.jpg)
Motion
Stanford University
19-Nov-2019
25
Solving the ambiguity…• How to get more equations for a pixel?• Spatial coherence constraint:• Assume the pixel’s neighbors have the same (u,v)
– If we use a 5x5 window, that gives us 25 equations per pixel
B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679, 1981.
Sour
ce: S
ilvio
Sav
ares
e
![Page 26: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/26.jpg)
Motion
Stanford University
19-Nov-2019
26
Lucas-Kanade flow• Overconstrained linear system:
Sour
ce: S
ilvio
Sav
ares
e
![Page 27: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/27.jpg)
Motion
Stanford University
19-Nov-2019
27
Lucas-Kanade flow• Overconstrained linear system
The summations are over all pixels in the K x K window
Least squares solution for d given by
Sour
ce: S
ilvio
Sav
ares
e
![Page 28: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/28.jpg)
Motion
Stanford University
19-Nov-2019
28
Conditions for solvability– Optimal (u, v) satisfies Lucas-Kanade equation
Does this remind anything to you?
When is This Solvable?• ATA should be invertible • ATA should not be too small due to noise
– eigenvalues l1 and l 2 of ATA should not be too small• ATA should be well-conditioned
– l 1/ l 2 should not be too large (l 1 = larger eigenvalue)
Sour
ce: S
ilvio
Sav
ares
e
![Page 29: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/29.jpg)
Motion
Stanford University
19-Nov-2019
29
• Eigenvectors and eigenvalues of ATA relate to edge direction and magnitude • The eigenvector associated with the larger eigenvalue points in
the direction of fastest intensity change• The other eigenvector is orthogonal to it
M = ATA is the second moment matrix !(Harris corner detector…)
Sour
ce: S
ilvio
Sav
ares
e
![Page 30: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/30.jpg)
Motion
Stanford University
19-Nov-2019
30
Interpreting the eigenvalues
l1
l2
“Corner”l1 and l2 are large,l1 ~ l2
l1 and l2 are small “Edge” l1 >> l2
“Edge” l2 >> l1
“Flat” region
Classification of image points using eigenvalues of the second moment matrix:
Sour
ce: S
ilvio
Sav
ares
e
![Page 31: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/31.jpg)
Motion
Stanford University
19-Nov-2019
31
Edge
– gradients very large or very small– large l1, small l2
Sour
ce: S
ilvio
Sav
ares
e
![Page 32: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/32.jpg)
Motion
Stanford University
19-Nov-2019
32
Low-texture region
– gradients have small magnitude– small l1, small l2
Sour
ce: S
ilvio
Sav
ares
e
![Page 33: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/33.jpg)
Motion
Stanford University
19-Nov-2019
33
High-texture region
– gradients are different, large magnitudes– large l1, large l2
Sour
ce: S
ilvio
Sav
ares
e
![Page 34: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/34.jpg)
Motion
Stanford University
19-Nov-2019
34
Errors in Lukas-Kanade
What are the potential causes of errors in this procedure?– Assumed ATA is easily invertible– Assumed there is not much noise in the image
• When our assumptions are violated– Brightness constancy is not satisfied– The motion is not small– A point does not move like its neighbors
• window size is too large• what is the ideal window size?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 35: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/35.jpg)
Motion
Stanford University
19-Nov-2019
35
Improving accuracy• Recall our small motion assumption
– Can solve using Newton’s method (out of scope for this class)
– Lukas-Kanade method does one iteration of Newton’s method• Better results are obtained via more iterations
• This is not exact– To do better, we need to add higher order terms back in:
• This is a polynomial root finding problem
It-1(x,y)
It-1(x,y)
It-1(x,y)
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 36: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/36.jpg)
Motion
Stanford University
19-Nov-2019
36
Iterative Refinement
• Iterative Lukas-Kanade Algorithm1. Estimate velocity at each pixel by solving Lucas-Kanade
equations2. Warp I(t-1) towards I(t) using the estimated flow field
- use image warping techniques3. Repeat until convergence
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 37: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/37.jpg)
Motion
Stanford University
19-Nov-2019
37
When do the optical flow assumptions fail?
In other words, in what situations does the displacement of pixel patches not represent physical movement of points in space?
1. Well, TV is based on illusory motion – the set is stationary yet things seem to move
2. A uniform rotating sphere – nothing seems to move, yet it is rotating
3. Changing directions or intensities of lighting can make things seem to move – for example, if the specular highlight on a rotating sphere moves.
4. Muscle movement can make some spots on a cheetah move opposite direction of motion. – And infinitely more break downs of optical flow.
![Page 38: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/38.jpg)
Motion
Stanford University
19-Nov-2019
38
What will we learn today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
![Page 39: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/39.jpg)
Motion
Stanford University
19-Nov-2019
39
• Key assumptions (Errors in Lucas-Kanade)
• Small motion: points do not move very far
• Brightness constancy: projection of the same point looks the same in every frame
• Spatial coherence: points move like their neighbors
Recap
Sour
ce: S
ilvio
Sav
ares
e
![Page 40: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/40.jpg)
Motion
Stanford University
19-Nov-2019
40
Revisiting the small motion assumption
• Is this motion small enough?– Probably not—it’s much larger than one pixel (2nd order terms dominate)– How might we solve this problem?
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
![Page 41: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/41.jpg)
Motion
Stanford University
19-Nov-2019
41
Reduce the resolution!
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
![Page 42: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/42.jpg)
Motion
Stanford University
19-Nov-2019
42
image Iimage H
Gaussian pyramid of image 1 Gaussian pyramid of image 2
image 2image 1 u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarse-to-fine optical flow estimation
Sour
ce: S
ilvio
Sav
ares
e
![Page 43: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/43.jpg)
Motion
Stanford University
19-Nov-2019
43
image Iimage J
Gaussian pyramid of image 1 (t) Gaussian pyramid of image 2 (t+1)
image 2image 1
Coarse-to-fine optical flow estimation
run iterative L-K
run iterative L-K
warp & upsample
.
.
.
Sour
ce: S
ilvio
Sav
ares
e
![Page 44: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/44.jpg)
Motion
Stanford University
19-Nov-2019
44
Optical Flow Results
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
![Page 45: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/45.jpg)
Motion
Stanford University
19-Nov-2019
45
Optical Flow Results
* Fro
m K
hurra
mH
assa
n-Sh
afiq
ueC
AP54
15 C
ompu
ter V
isio
n 20
03
• http://www.ces.clemson.edu/~stb/klt/• OpenCV
![Page 46: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/46.jpg)
Motion
Stanford University
19-Nov-2019
46
What will we learn today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
Reading: [Szeliski] Chapters: 8.4, 8.5[Fleet & Weiss, 2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
![Page 47: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/47.jpg)
Motion
Stanford University
19-Nov-2019
47
Horn-Schunk method for optical flow
• The flow is formulated as a global energy function which is should be minimized:
![Page 48: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/48.jpg)
Motion
Stanford University
19-Nov-2019
48
Horn-Schunk method for optical flow
• The flow is formulated as a global energy function which is should be minimized:• The first part of the function is the brightness consistency.
![Page 49: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/49.jpg)
Motion
Stanford University
19-Nov-2019
49
Horn-Schunk method for optical flow
• The flow is formulated as a global energy function which is should be minimized:• The second part is the smoothness constraint. It’s trying to make sure that the
changes between pixels are small.
![Page 50: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/50.jpg)
Motion
Stanford University
19-Nov-2019
50
Horn-Schunk method for optical flow
• The flow is formulated as a global energy function which is should be minimized:• 𝛼 is a regularization constant. Larger values of 𝛼 lead to smoother flow.
![Page 51: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/51.jpg)
Motion
Stanford University
19-Nov-2019
51
Horn-Schunk method for optical flow
• The flow is formulated as a global energy function which is should be minimized:
• This minimization can be solved by taking the derivative with respect to u and v, we get the following 2 equations:
![Page 52: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/52.jpg)
Motion
Stanford University
19-Nov-2019
52
Horn-Schunk method for optical flow
• By taking the derivative with respect to u and v, we get the following 2 equations:
• Where is called the Lagrange operator. In practice, it is measured using:
• where is the weighted average of u measured at (x,y).
![Page 53: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/53.jpg)
Motion
Stanford University
19-Nov-2019
53
Horn-Schunk method for optical flow
• Now we substitute in:
• To get:
• Which is linear in u and v and can be solved analytically for each pixel individually.
![Page 54: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/54.jpg)
Motion
Stanford University
19-Nov-2019
54
Iterative Horn-Schunk
• But since the solution depends on the neighboring values of the flow field, it must be repeated once the neighbors have been updated.
• So instead, we can iteratively solve for u and v using:
![Page 55: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/55.jpg)
Motion
Stanford University
19-Nov-2019
55
What does the smoothness regularization do anyway?• It’s a sum of squared terms (a Euclidian distance measure).• We’re putting it in the expression to be minimized.• => In texture free regions, there is no optical flow
Regularized flow
Optical flow
• => On edges, points will flow to nearest points, solving the aperture problem.
Slide credit: Sebastian Thurn
![Page 56: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/56.jpg)
Motion
Stanford University
19-Nov-2019
56
Dense Optical Flow with Michael Black’s method
• Michael Black took Horn-Schunk’s method one step further, starting from the regularization constant:
• Which looks like a quadratic:
• And replaced it with this:
• Why does this regularization work better?
![Page 57: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/57.jpg)
Motion
Stanford University
19-Nov-2019
57
What will we learn today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
![Page 58: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/58.jpg)
Motion
Stanford University
19-Nov-2019
58
• Key assumptions
• Small motion: points do not move very far
• Brightness constancy: projection of the same point looks the same in every frame
• Spatial coherence: points move like their neighbors
Recap
Sour
ce: S
ilvio
Sav
ares
e
![Page 59: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/59.jpg)
Motion
Stanford University
19-Nov-2019
59
Reminder: Gestalt – common fate
![Page 60: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/60.jpg)
Motion
Stanford University
19-Nov-2019
60
Motion segmentation• How do we represent the motion in this scene?
Sour
ce: S
ilvio
Sav
ares
e
![Page 61: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/61.jpg)
Motion
Stanford University
19-Nov-2019
61
Motion segmentation• Break image sequence into “layers” each of which has a coherent (affine) motion
J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993. Sour
ce: S
ilvio
Sav
ares
e
![Page 62: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/62.jpg)
Motion
Stanford University
19-Nov-2019
62
Affine motion
• Substituting into the brightness constancy equation:
yaxaayxvyaxaayxu
654
321
),(),(
++=++=
0»+×+× tyx IvIuI
Sour
ce: S
ilvio
Sav
ares
e
![Page 63: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/63.jpg)
Motion
Stanford University
19-Nov-2019
63
0)()( 654321 »++++++ tyx IyaxaaIyaxaaI
Affine motion
• Substituting into the brightness constancy equation:
yaxaayxvyaxaayxu
654
321
),(),(
++=++=
• Each pixel provides 1 linear constraint in 6 unknowns
[ ] 2å ++++++= tyx IyaxaaIyaxaaIaErr )()()( 654321!
• Least squares minimization:
Sour
ce: S
ilvio
Sav
ares
e
![Page 64: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/64.jpg)
Motion
Stanford University
19-Nov-2019
64
How do we estimate the layers?• 1. Obtain a set of initial affine motion hypotheses
– Divide the image into blocks and estimate affine motion parameters in each block by least squares• Eliminate hypotheses with high residual error
• Map into motion parameter space• Perform k-means clustering on affine motion parameters
–Merge clusters that are close and retain the largest clusters to obtain a smaller set of hypotheses to describe all the motions in the scene
Sour
ce: S
ilvio
Sav
ares
e
![Page 65: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/65.jpg)
Motion
Stanford University
19-Nov-2019
65
How do we estimate the layers?• 1. Obtain a set of initial affine motion hypotheses
– Divide the image into blocks and estimate affine motion parameters in each block by least squares• Eliminate hypotheses with high residual error
• Map into motion parameter space• Perform k-means clustering on affine motion parameters
–Merge clusters that are close and retain the largest clusters to obtain a smaller set of hypotheses to describe all the motions in the scene
Sour
ce: S
ilvio
Sav
ares
e
![Page 66: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/66.jpg)
Motion
Stanford University
19-Nov-2019
66
How do we estimate the layers?• 1. Obtain a set of initial affine motion hypotheses
– Divide the image into blocks and estimate affine motion parameters in each block by least squares• Eliminate hypotheses with high residual error
• Map into motion parameter space• Perform k-means clustering on affine motion parameters
–Merge clusters that are close and retain the largest clusters to obtain a smaller set of hypotheses to describe all the motions in the scene
2. Iterate until convergence:•Assign each pixel to best hypothesis
–Pixels with high residual error remain unassigned•Perform region filtering to enforce spatial constraints•Re-estimate affine motions in each region
Sour
ce: S
ilvio
Sav
ares
e
![Page 67: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/67.jpg)
Motion
Stanford University
19-Nov-2019
67
Example result
J. Wang and E. Adelson. Layered Representation for Motion Analysis. CVPR 1993. Sour
ce: S
ilvio
Sav
ares
e
![Page 68: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/68.jpg)
Motion
Stanford University
19-Nov-2019
68
What will we learn today?
• Optical flow• Lucas-Kanade method• Horn-Schunk method• Pyramids for large motion• Common fate• Applications
![Page 69: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/69.jpg)
Motion
Stanford University
19-Nov-2019
69
Uses of motion
• Tracking features• Segmenting objects based on motion cues• Learning dynamical models• Improving video quality
– Motion stabilization– Super resolution
• Tracking objects• Recognizing events and activities
![Page 70: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/70.jpg)
Motion
Stanford University
19-Nov-2019
70
Estimating 3D structure
Sour
ce: S
ilvio
Sav
ares
e
![Page 71: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/71.jpg)
Motion
Stanford University
19-Nov-2019
71
Segmenting objects based on motion cues
• Background subtraction– A static camera is observing a scene– Goal: separate the static background from the moving foreground
Sour
ce: S
ilvio
Sav
ares
e
![Page 72: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/72.jpg)
Motion
Stanford University
19-Nov-2019
72
Segmenting objects based on motion cues
• Motion segmentation– Segment the video into multiple coherently moving objects
S. J. Pundlik and S. T. Birchfield, Motion Segmentation at Any Speed, Proceedings of the British Machine Vision Conference (BMVC) 2006
Sour
ce: S
ilvio
Sav
ares
e
![Page 73: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/73.jpg)
Motion
Stanford University
19-Nov-2019
73Z.Yin and R.Collins, "On-the-fly Object Modeling while Tracking," IEEE Computer Vision and Pattern Recognition (CVPR '07), Minneapolis, MN, June 2007.
Tracking objects
Sour
ce: S
ilvio
Sav
ares
e
![Page 74: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/74.jpg)
Motion
Stanford University
19-Nov-2019
74
Synthesizing dynamic textures
![Page 75: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/75.jpg)
Motion
Stanford University
19-Nov-2019
75
Super-resolution
Example: A set of low quality images
Sour
ce: S
ilvio
Sav
ares
e
![Page 76: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/76.jpg)
Motion
Stanford University
19-Nov-2019
76
Super-resolution
Each of these images looks like this:
Sour
ce: S
ilvio
Sav
ares
e
![Page 77: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/77.jpg)
Motion
Stanford University
19-Nov-2019
77
Super-resolution
The recovery result:
Sour
ce: S
ilvio
Sav
ares
e
![Page 78: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/78.jpg)
Motion
Stanford University
19-Nov-2019
78
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.
Tracker
Recognizing events and activities
Sour
ce: S
ilvio
Sav
ares
e
![Page 79: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/79.jpg)
Motion
Stanford University
19-Nov-2019
79
Juan Carlos Niebles, Hongcheng Wang and Li Fei-Fei, Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words, (BMVC), Edinburgh, 2006.
Recognizing events and activities
![Page 80: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/80.jpg)
Motion
Stanford University
19-Nov-2019
80
Crossing – Talking – Queuing – Dancing – jogging
W. Choi & K. Shahid & S. Savarese WMC 2010
Recognizing events and activities
Sour
ce: S
ilvio
Sav
ares
e
![Page 81: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/81.jpg)
Motion
Stanford University
19-Nov-2019
81
W. Choi, K. Shahid, S. Savarese, "What are they doing? : Collective Activity Classification Using Spatio-Temporal Relationship Among People", 9th International Workshop on Visual Surveillance (VSWS09) in conjuction with ICCV 09
![Page 82: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/82.jpg)
Motion
Stanford University
19-Nov-2019
82
Human Event Understanding: From Actions to Tasks
• A recent talk:http://tv.vera.com.uy/video/55276
![Page 83: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/83.jpg)
Motion
Stanford University
19-Nov-2019
83
Optical flow without motion!
![Page 84: Lecture: Motion - Stanford Computer Vision Lab](https://reader030.vdocuments.net/reader030/viewer/2022012611/619d8b85d7d9fd41a646664d/html5/thumbnails/84.jpg)
Motion
Stanford University
19-Nov-2019
84
What have we learned today?
• Optical flow• Lucas-Kanade method• Pyramids for large motion• Horn-Schunk method• Common fate• Applications
[Fleet & Weiss, 2005]http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf
Reading: [Szeliski] Chapters: 8.4, 8.5