![Page 1: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/1.jpg)
MSU Fall 2019 1
Computing Motion from Images
Chapter 9 of S&S plus otherwork.
![Page 2: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/2.jpg)
MSU Fall 2019 2
General topicsn Low level change detectionn Region tracking or matching over timen Interpretation of motionn MPEG compressionn Interpretation of scene changes in
videon Understanding human activites
![Page 3: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/3.jpg)
MSU Fall 2019 3
Motion important to human vision
![Page 4: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/4.jpg)
MSU Fall 2019 4
What’s moving: different cases
![Page 5: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/5.jpg)
MSU Fall 2019 5
Image subtraction
Simple method to remove unchanging background from
moving regions.
![Page 6: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/6.jpg)
MSU Fall 2019 6
Change detection for surveillance
![Page 7: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/7.jpg)
MSU Fall 2019 7
Change detection by image subtraction
![Page 8: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/8.jpg)
Closing = dilation+erosion
MSU Fall 2019 8
http://homepages.inf.ed.ac.uk/rbf/HIPR2/close.htm
![Page 9: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/9.jpg)
MSU Fall 2019 9
What to do with regions of change?
n Discard small regionsn Discard regions of non interesting
featuresn Keep track of regions with interesting
featuresn Track in future frames from motion plus
component features
![Page 10: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/10.jpg)
MSU Fall 2019 10
Some effects of camera motion that can cause problems
![Page 11: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/11.jpg)
MSU Fall 2019 11
Motion field
![Page 12: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/12.jpg)
MSU Fall 2019 12
FOE and FOC
Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.
![Page 13: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/13.jpg)
Global Motion Compensation
13Seyed Morteza Safdarnejad, Yousef Atoum, Xiaoming Liu, “Temporally Robust Global Motion Compensation by Keypoint-based Congealing,” in ECCV 2016.
![Page 14: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/14.jpg)
MSU Fall 2019 14
Gaming using a camera to recognize the player’s motion
Decathlete game
![Page 15: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/15.jpg)
MSU Fall 2019 15
Decathlete gameCheap camera replaces usual mouse for input
Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.
![Page 16: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/16.jpg)
MSU Fall 2019 16
Motion detection input device
Running (hands) Jumping (hands)
![Page 17: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/17.jpg)
MSU Fall 2019 17
Motion analysis controls hurdling event (console)
•Top left shows video frame of player
•Middle left shows motion vectors from multiple frames•Center shows jumping patterns
![Page 18: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/18.jpg)
MSU Fall 2019 18
Related workn Motion sensed by crude camerasn Person dances/gestures in spacen Kinect/Leap motion sensorsn System maps movement into musicn Creative environment?n Good exercise room?
![Page 19: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/19.jpg)
MSU Fall 2019 19
Computing motion vectors from corresponding “points”
High energy neighborhoods are used to define points for matching
![Page 20: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/20.jpg)
MSU Fall 2019 20
Match points between framesSuch large motions are unusual. Most systems track small motions.
![Page 21: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/21.jpg)
MSU Fall 2019 21
Requirements for interest points
Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.
![Page 22: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/22.jpg)
MSU Fall 2019 22
Interest = minimum directional variance
Used by Hans Moravec in his robot stereo vision system.
Interest points were used for stereo matching.
![Page 23: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/23.jpg)
MSU Fall 2019 23
Detecting interest points in I1
![Page 24: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/24.jpg)
MSU Fall 2019 24
Match points from I1 in I2
![Page 25: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/25.jpg)
MSU Fall 2019 25
Search for best match of point P1 in nearby window of I2
For both motion and stereo, we have some constraints on where to search for a matching interest point.
![Page 26: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/26.jpg)
MSU Fall 2019 26
Motion vectors clustered to show 3 coherent regions
All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin)
Motion coherence: points of same object tend to move in the same way
![Page 27: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/27.jpg)
MSU Fall 2019 27
Two frames of aerial imagery
Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.
![Page 28: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/28.jpg)
MSU Fall 2019 28
Can code frame N+d with displacments relative to frame N
n for each 16 x 16 block in the 2nd imagen find a closely matching block in the 1st
imagen replace the 16x16 intensities by the
location in the 1st image (dX, dY)n 256 bytes replaced by 2 bytes!n (If blocks differ too much, encode the
differences to be added.)
![Page 29: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/29.jpg)
MSU Fall 2019 29
Frame approximation
Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)
![Page 30: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/30.jpg)
MSU Fall 2019 30
Best matching blocks between video frames N+1 to N (motion vectors)
The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!
Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)
![Page 31: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/31.jpg)
MSU Fall 2019 31
Motion coherence provides redundancy for compression
MPEG “motion compensation”represents motion of 16x16 pixels
blocks, NOT objects
![Page 32: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/32.jpg)
MSU Fall 2019 32
MPEG represents blocks that move by the motion vector
![Page 33: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/33.jpg)
MSU Fall 2019 33
MPEG has ‘I’, ‘P’, and ‘B’frames
![Page 34: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/34.jpg)
MSU Fall 2019 34
Computing Image Flow
![Page 35: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/35.jpg)
MSU Fall 2019 35
![Page 36: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/36.jpg)
Motion Field & Optical Flow Fieldn Motion Field = Real world 3D motion n Optical Flow Field = Projection of the
motion field onto the 2d image3D motion vector
2D optical flow vector
( )vu,u =!
CCD
Slides from Lihi Zelnik-Manor
![Page 37: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/37.jpg)
MSU Fall 2019 37
Assumptions
![Page 38: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/38.jpg)
When does it break?
The screen is stationary yet displays motion
Homogeneous objects generate zero optical flow.
Fixed sphere. Changing light source.
Non-rigid texture motion
Slides from Lihi Zelnik-Manor
![Page 39: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/39.jpg)
MSU Fall 2019 39
Image flow equation 1 of 2
![Page 40: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/40.jpg)
MSU Fall 2019 40
Image flow equation 2 of 2
![Page 41: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/41.jpg)
Estimating Optical Flown Assume the image intensity is constant
( )tyxI ,, ( )dttdyydxxI +++ ,,=
ITime = t Time = t+dt
( )dyydxx ++ ,( )yx,
Slides from Lihi Zelnik-Manor MSU Fall 2019 41
![Page 42: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/42.jpg)
Brightness Constancy Equation( ) ( )dttdyydxxItyxI +++= ,,,,
( ) dttIdy
yIdx
xItyxI
¶¶+
¶¶+
¶¶+= ,,
First order Taylor Expansion
0=++ dtIdyIdxI tyx
Simplify notations:
Divide by dt and denote:
dtdxu =
dtdyv =
tyx IvIuI -=+Problem I: One equation, two unknowns
Slides from Lihi Zelnik-Manor MSU Fall 2019 42
![Page 43: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/43.jpg)
Time t+dt
Problem II: “The Aperture Problem”
Time t
?Time t+dt
Where did the yellow point move to?We need additional constraints
n For points on a line of fixed intensity we can only recover the normal flow
Slides from Lihi Zelnik-Manor MSU Fall 2019 43
![Page 44: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/44.jpg)
Use Local InformationSometimes enlarging the aperture can help
Slides from Lihi Zelnik-Manor MSU Fall 2019 44
![Page 45: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/45.jpg)
Local smoothnessLucas Kanade (1984)
Assume constant (u,v) in small neighborhood
tyx IvIuI -=+ [ ] tyx Ivu
II -=úû
ùêë
é
úúú
û
ù
êêê
ë
é-=úû
ùêë
é
úúú
û
ù
êêê
ë
é
!!
2
1
22
11
t
t
yx
yx
II
vu
IIII
bA =u!Slides from Lihi Zelnik-Manor MSU Fall 2019 45
![Page 46: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/46.jpg)
Lucas Kanade (1984)
bA =u!
( ) bAAA TT 1u
-=!
Goal: Minimize2u bA -!
bAAA TT =u!
2x2 2x1 2x1
Method: Least-Squares
Slides from Lihi Zelnik-Manor MSU Fall 2019 46
![Page 47: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/47.jpg)
Lucas-Kanade Solution
úúû
ù
êêë
é=
åååå
2
2
yyx
yxxT
IIIIII
AA
We want this matrix to be invertible.i.e., no zero eigenvalues
( ) bAAA TT 1u
-=!
Slides from Lihi Zelnik-Manor MSU Fall 2019 47
![Page 48: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/48.jpg)
Break-downsn Brightness constancy is not satisfied
n A point does not move like its neighbors n what is the ideal window size?
n The motion is not small (Taylor expansion doesn’t hold)
Correlation based methods
Regularization based methods
Use multi-scale estimation
Slides from Lihi Zelnik-Manor MSU Fall 2019 48
![Page 49: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/49.jpg)
Multi-Scale Flow Estimation
image It-1 image I
Gaussian pyramid of image It Gaussian pyramid of image It+1
image It+1image Itu=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Slides from Lihi Zelnik-Manor MSU Fall 2019 49
![Page 50: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/50.jpg)
SIFTflow
MSU Fall 2019 50
http://people.csail.mit.edu/celiu/SIFTflow/
Check out the Obstruction Free Photography work in 2016
![Page 51: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/51.jpg)
MSU Fall 2019 51
Tracking several objects
Use assumptions of physics to compute multiple smooth paths.(work of Sethi and R. Jain PAMI)
![Page 52: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/52.jpg)
MSU Fall 2019 52
![Page 53: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/53.jpg)
MSU Fall 2019 53
Tracking in images over time
![Page 54: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/54.jpg)
MSU Fall 2019 54
General constraints from physics
![Page 55: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/55.jpg)
MSU Fall 2019 55
Other possible constraints
n Background statistics stablen Object color/texture/shape might
change slowly over framesn Might have knowledge of objects under
surveillancen Objects appear/disappear at boundary
of the frame
![Page 56: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/56.jpg)
MSU Fall 2019 56
![Page 57: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/57.jpg)
MSU Fall 2019 57
Sethi-Jain algorithm
![Page 58: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/58.jpg)
MSU Fall 2019 58
![Page 59: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/59.jpg)
MSU Fall 2019 59
Total smoothness of m paths
![Page 60: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/60.jpg)
MSU Fall 2019 60
Greedy exchange algorithm
![Page 61: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/61.jpg)
MSU Fall 2019 61
Example data structureTotal smoothness for trajectories of Figure 9.14
![Page 62: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/62.jpg)
MSU Fall 2019 62
Example of domain specific tracking (Vera Bakic)Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback-control loop.
![Page 63: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/63.jpg)
MSU Fall 2019 63
Segmentation of videos/movies
Segment into scenes, shots, specific actions, etc.
![Page 64: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/64.jpg)
MSU Fall 2019 64
Types of changes in videos
![Page 65: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/65.jpg)
MSU Fall 2019 65
Anchor person scene at left
Street scene for news story
Scene break
From Zhang et al 1993
How do we compute the scene change?
![Page 66: Computing Motion from Images - Michigan State Universitycse803/Lectures/week08-motionSlides.pdf · MSU Fall 2019 30 Best matching blocks between video frames N+1 to N (motion vectors)](https://reader034.vdocuments.net/reader034/viewer/2022050113/5f4a65f40737c343504bdaab/html5/thumbnails/66.jpg)
MSU Fall 2019 66
Histograms of frames across the scene change
Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.