computing motion from images

Stockman MSU Fall 2009 1

Computing Motion from Images

Chapter 9 of S&S plus otherwork.

Stockman MSU Fall 2009 2

General topics

Low level change detection Region tracking or matching over

time Interpretation of motion MPEG compression Interpretation of scene changes in

video Understanding human activites

Stockman MSU Fall 2009 3

Motion important to human vision

Stockman MSU Fall 2009 4

What’s moving: different cases

Stockman MSU Fall 2009 5

Image subtraction

Simple method to remove unchanging background from

moving regions.

Stockman MSU Fall 2009 6

Change detection for surveillance

Stockman MSU Fall 2009 7

Change detection by image subtraction

Stockman MSU Fall 2009 8

What to do with regions of change?

Discard small regions Discard regions of non interesting

features Keep track of regions with

interesting features Track in future frames from motion

plus component features

Stockman MSU Fall 2009 9

Some effects of camera motion that can cause problems

Stockman MSU Fall 2009 10

Motion field

Stockman MSU Fall 2009 11

FOE and FOC

Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.

Stockman MSU Fall 2009 12

Gaming using a camera to recognize the player’s motion

Decathlete game

Stockman MSU Fall 2009 13

Decathlete game

Cheap camera replaces usual mouse for input

Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.

Stockman MSU Fall 2009 14

Motion detection input device

Running (hands)Jumping (hands)

Stockman MSU Fall 2009 15

Motion analysis controls hurdling event (console)

•Top left shows video frame of player

•Middle left shows motion vectors from multiple frames

•Center shows jumping patterns

Stockman MSU Fall 2009 16

Related work

Motion sensed by crude cameras Person dances/gestures in space System maps movement into

music Creative environment? Good exercise room?

Stockman MSU Fall 2009 17

Computing motion vectors from corresponding “points”

High energy neighborhoods are used to define points for

matching

Stockman MSU Fall 2009 18

Match points between frames

Such large motions are unusual. Most systems track small motions.

Stockman MSU Fall 2009 19

Requirements for interest points

Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.

Stockman MSU Fall 2009 20

Interest = minimum directional variance

Used by Hans Moravec in his robot stereo vision system.

Interest points were used for stereo matching.

Stockman MSU Fall 2009 21

Detecting interest points in I1

Stockman MSU Fall 2009 22

Match points from I1 in I2

Stockman MSU Fall 2009 23

Search for best match of point P1 in nearby window of I2

For both motion and stereo, we have some constraints on where to search for a matching interest point.

Stockman MSU Fall 2009 24

Motion vectors clustered to show 3 coherent regions

All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin)

Motion coherence: points of same object tend to move in the same way

Stockman MSU Fall 2009 25

Two frames of aerial imagery

Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.

Stockman MSU Fall 2009 26

Can code frame N+d with displacments relative to frame N

for each 16 x 16 block in the 2nd image

find a closely matching block in the 1st image

replace the 16x16 intensities by the location in the 1st image (dX, dY)

256 bytes replaced by 2 bytes! (If blocks differ too much, encode the

differences to be added.)

Stockman MSU Fall 2009 27

Frame approximation

Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)

Stockman MSU Fall 2009 28

Best matching blocks between video frames N+1 to N (motion vectors)

The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!

Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)

Stockman MSU Fall 2009 29

Motion coherence provides redundancy for compression

MPEG “motion compensation” represents motion of 16x16 pixels blocks, NOT objects

Stockman MSU Fall 2009 30

MPEG represents blocks that move by the motion vector

Stockman MSU Fall 2009 31

MPEG has ‘I’, ‘P’, and ‘B’ frames

Stockman MSU Fall 2009 32

Computing Image Flow

Stockman MSU Fall 2009 33

Stockman MSU Fall 2009 34

Assumptions

Stockman MSU Fall 2009 35

IMAGE

FLOW

EQUATION

1 of 2

Stockman MSU Fall 2009 36

Image flow equation 2 of 2

Stockman MSU Fall 2009 37

Aperture problem

Stockman MSU Fall 2009 38

Solving flow by propagation of constraints

Stockman MSU Fall 2009 39

Info at corner constrains the flow along both edges

Solve constraints using contraint propagation or differential equation with boundary conditions.

Stockman MSU Fall 2009 40

Tracking several objects

Use assumptions of physics to compute multiple smooth

paths.(work of Sethi and R. Jain)

Stockman MSU Fall 2009 41

Stockman MSU Fall 2009 42

Tracking in images over time

Stockman MSU Fall 2009 43

General constraints from physics

Stockman MSU Fall 2009 44

Other possible constraints

Background statistics stable Object color/texture/shape might

change slowly over frames Might have knowledge of objects

under survielance Objects appear/disappear at

boundary of the frame

Stockman MSU Fall 2009 45

Stockman MSU Fall 2009 46

Sethi-Jain algorithm

Stockman MSU Fall 2009 47

Stockman MSU Fall 2009 48

Total smoothness of m paths

Stockman MSU Fall 2009 49

Greedy exchange algorithm

Stockman MSU Fall 2009 50

Example data structure

Total smoothness for trajectories of Figure 9.14

Stockman MSU Fall 2009 51

Example of domain specific tracking (Vera Bakic)

Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback-control loop.

Stockman MSU Fall 2009 52

Segmentation of videos/movies

Segment into scenes, shots, specific actions, etc.

Stockman MSU Fall 2009 53

Types of changes in videos

Stockman MSU Fall 2009 54

Anchor person scene at left

Street scene for news story

Scene break

From Zhang et al 1993

How do we compute the scene change?

Stockman MSU Fall 2009 55

Histograms of frames across the scene change

Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.

Stockman MSU Fall 2009 56

Heuristics for ignoring zooms

Stockman MSU Fall 2009 57

American sign language example

Stockman MSU Fall 2009 58

Example from Yang and Ahuja

Stockman MSU Fall 2009 59

Stockman MSU Fall 2009 60