3D IMAGING

Midterm Review

Pinhole cameras

Abstract camera model - box with a small hole in it

Pinhole cameras work in practice

The equation of projection

Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f

y/z, -f) Ignore the third coordinate, and get

(x,y, z) ( fxz

, fyz)

Prove at home

The camera matrix

Homogenous coordinates for 3D point are (X,Y,Z,T) Homogenous coordinates for point in image are

(U,V,W)

U

V

W

1 0 0 0

0 1 0 0

0 0 1f 0

X

Y

Z

T

Properties of “thin” lens (i.e., ideal lens)

Light rays passing through the center are not deviated.

Light rays passing through a point far away from the center are deviated more.

focal point

f

Thin lens equation (cont’d)

fuv

1 1 1u v f+ =

image

Combining the equations(do at home):

Thin lens equation (cont’d)

The thin lens equation implies that only points at distance u from the lens are “in focus” (i.e., focal point lies on image plane).

Other points project to a “blur circle” or “circle of confusion”

in the image (i.e., blurring occurs).

“circle of confusion”

Depth of field

Changing the aperture size or focal length affects depth of field

f / 5.6

f / 32

Basic models of reflection

Specular: light bounces off at the incident angle E.g., mirror

Diffuse: light scatters in all directions E.g., brick, cloth, rough wood

incoming lightspecular reflection

ΘΘ

incoming light

diffuse reflection

Bidirectional Reflectance Distribution Function (BDRF)

);,,,( eeii

surface normal

( , )

( , )e e e

i i i

L

E

• Model of local reflection that tells how bright a surface appears when viewed from one direction when light falls on it from another

The Retina

Cross-section of eye

Ganglion cell layer

Bipolar cell layer

Receptor layer

Pigmentedepithelium

Ganglion axons

Cross section of retina

© Stephen E. Palmer, 2002

Cones cone-shaped less sensitive operate in high light color vision

Two types of light-sensitive receptors

cone

rod

Rods rod-shaped highly sensitive operate at night gray-scale vision

Color Vision.

400 450 500 550 600 650

RE

LAT

IVE

AB

SO

RB

AN

CE

(%

)

WAVELENGTH (nm.)

100

50

440

S

530 560 nm.

M L

The raster image (pixel matrix)

0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.990.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.910.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.920.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.950.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.850.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.330.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.740.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.930.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.990.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.970.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

Color Images: Multi-chip

wavelengthdependent

Color Images: Bayer Grid

Estimate RGBat ‘G’ cells from neighboring values

Slide by Steve Seitz

HSV Color Space

Moravec corner detectorChange of intensity for the shift [u,v]:

2,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

IntensityShifted intensity

Window function

Four shifts: (u,v) = (1,0), (1,1), (0,1), (-1, 1)Look for local maxima in min{E}

Harris Corner Detector

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

],[],[],[,

lnkmflkgnmhlk

[.,.]h[.,.]f

Image filtering

111

111

111

],[g

Practice with linear filters

Original

111

111

111

000

020

000

-

Sharpening filter- Accentuates differences with

local average

Source: D. Lowe

Other filters

-101

-202

-101

Vertical Edge(absolute value)

Sobel

Spatially-weighted average

0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003

5 x 5, = 1

Important filter: Gaussian

Smoothing with Gaussian filter

Smoothing with box filter

A sum of sines

Our building block:

Add enough of them to get any signal f(x) you want!

xAsin(

Fourier analysis in images

Intensity Image

Fourier Image

cos( )A ux vy

Gaussian

Box Filter

Source: S. Marschner

1D example (sinewave):

Aliasing problem

Aliasing in video

Aliasing in graphics

Subsampling without pre-filtering

1/4 (2x zoom) 1/8 (4x zoom)1/2

Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2

Template matching

Goal: find in image

Main challenge: What is a good similarity or distance measure between two patches? Correlation Zero-mean correlation Sum Square Difference Normalized Cross

Correlation

Normalized Cross Correlation

mean image patchmean template

5.0

,

2,

,

2

,,

)],[()],[(

)],[)(],[(

],[

lknm

lk

nmlk

flnkmfglkg

flnkmfglkg

nmh

Matching with filters (Normalized Cross Correlation)

Input Normalized X-Correlation Thresholded Image

True detections

Reducing salt-and-pepper noise by Gaussian smoothing

3x3 5x5 7x7

Alternative idea: Median filtering• A median filter operates over a

window by selecting the median intensity in the window

• Is median filtering linear?

Median filterSalt-and-pepper noise Median filtered

Effects of noise• Consider a single row or column of the image

– Plotting intensity as a function of position gives a signal

Where is the edge?

Solution: smooth first

• To find edges, look for peaks in )( gfdx

d

f

g

f * g

)( gfdx

d

Source: S. Seitz

• Differentiation is convolution, and convolution is associative:

• :

gdx

dfgf

dx

d )(

Derivative theorem of convolution

gdx

df

f

gdx

d

Derivative of Gaussian filter

Final Canny Edges

Alper Yilmaz, CAP5415, Fall 2004

47

Estimating Camera Parameters

11, yx 111 ,, ZYX

222 ,, ZYX

333 ,, ZYX

NNN ZYX ,,

22 , yx

33 , yx

NN yx ,

Ames Room

Julesz: had huge impact because it showed that recognition not needed for stereo.

Epipolar Constraint

Basic Stereo Derivations

Disparity: 21 xxd

d

fBZ

We can always achieve this geometry with image rectification

Image Reprojection reproject image planes onto

common plane parallel to line between optical centers

(Seitz)

Using these constraints we can use matching for stereo

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost• This will never work, so:

Improvement: match windows

Stereo matching as energy minimizationI1 I2 D

• Energy functions of this form can be minimized using graph cuts

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

W1(i ) W2(i+D(i )) D(i )

)(),;( smooth21data DEIIDEE 2

,neighborssmooth )()(

ji

jDiDE 221data ))(()( i

iDiWiWE

Active stereo with structured light

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

Random Dot

Time to Collision

f

L

v

L

Do

l(t)

t=0t

D(t)

And time to collision:

Can be directly measured from image

Can be found, without knowing L or Do or v !!

2D Motion Field

2D Optical Flow

Apparent motion of image brightness pattern

2D Motion Field and 2D Optical Flow Motion field: projection of 3D motion vectors on image plane

• Optical flow field: apparent motion of brightness patterns

• We equate motion field with optical flow field

0 0

00

Object point has velocity , induces in imagei

ii

P

d d

dt dt

v v

r rv v

Brightness Constancy Equation

Taking derivative wrt time:

Normal Motion/Aperture Problem

Barber Pole Illusion

Full 3D Rotation

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

R

• Any rotation can be expressed as combination of three rotations about three axes.

100

010

001TRR

• Rows (and columns) of R are orthonormal vectors.

• R has determinant 1 (not -1).

Velocity Model in 2D

Perspective projection

321

213

132

VXYZ

VZXY

VYZX

Z

Yfy

Z

Xfx

yv

xu

212331

2

2213

32

1

)(

)(

yf

xyf

yZ

Vx

Z

Vfv

xf

xyf

yxZ

V

Z

Vfu