Acquisition

Some slides from: Yung-Yu Chuang (DigiVfx) Jan Neumann, Pat Hanrahan, Alexei Efros

Image Acquisition

Digital Camera

Film

Outline

• Pinhole camera• Lens• Lens aberrations• Exposure• Sensors• Noise• Sensing color

Camera trial #1

scene film

Put a piece of film in front of an object.

source: Yung-Yu Chuang

Pinhole camera

scene film

Add a barrier to block off most of the rays• It reduces blurring• The pinhole is known as the aperture• The image is inverted

barrier

pinhole camera

Modeling projection

• The coordinate system– Put the optical center (Center Of Projection) at the origin– Put the image plane (Projection Plane) in front of the COP– The camera looks down the negative z axis

• we need this if we want right-handed-coordinates

Modeling projection

• Projection equations– Compute intersection with PP of ray from (x,y,z) to COP– Derived using similar triangles (on board)

– We get the projection by throwing out the last coordinate:

In Homogenous Coordinates

• Projection is a matrix multiply using homogeneous coordinates:

Projection

Projection

http://users.skynet.be/J.Beever/pave.htm

Pinhole camera

scene film

Add a barrier to block off most of the rays• It reduces blurring• The pinhole is known as the aperture• The image is inverted

barrier

pinhole camera

Shrinking the Pinhole Aperture

Why make the aperture as small as possible?• Less light gets through• Diffraction effect

Shrinking the Pinhole Aperture

Sharpest image is obtained when:

pinhole diameter

Example: If

f = 50mm,

= 600nm (red),

d = 0.36mm

λfd 2=

λ

High-end commercial pinhole cameras

~$200

Exposure 4 seconds Exposure 96 minutes

Images copyright © 2000 Zero Image Co.

Pinhole Images

Outline

• Pinhole camera

• Lens• Lens Aberrations• Exposure• Sensors• Noise• Sensing color

Adding a lens

scene film

Adding a lens

scene filmlens

“circle of confusion”

A lens focuses light onto the film• There is a specific distance at which objects are “in focus”• Other points project to a “circle of confusion” in the image

(Thin) Lens

• Any object point satisfying this equation is in focus

Thin lens equation:

Circle of Confusion

)'('

iiidb −=

• Blur Circle Diameter b : Derive using similar triangles

d

aperturediameter

io

'i'o

Blur Circle, baperture

Aperture controls Depth of Field

• Changing the aperture affects depth of field– Smaller aperture:

• better DOF• increased exposure

f / 5.6

f / 32

Depth of Field

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm

Thick Lens

• Corrects aberrations• Change zoom

Field of View (Zoom) Field of View (Zoom)

FOV depends on Focal Length

d = image sizeϕ/2 d/2

f

2

f

FOV depends on Focal Length

d = image size

2

FOV depends on Focal Length

d = image size

2

Smaller FOV larger Focal Length

FOV depends on Focal Length

image size

focalpoint

For closer objects: if focal point is larger but image distance and sizeremain unchanged – the objects in focus are more distant.

Simplified Zoom Lens in Operation

From wikipedia

Outline

• Pinhole camera• Lens

• Lens aberrations• Exposure• Sensors• Noise• Sensing color

Radial Distortion

• Radial distortion of the image– Caused by imperfect lenses– Deviations are most noticeable for rays that pass

through the edge of the lens

No distortion Pin cushion Barrel

Correcting radial distortion

from Helmut Dersch

No Distortion Barrel Distortion Pincushion Distortion

ru = rd + k1 rd3

• Radial distance from Image Center:

Radial Distortions

ru = undistorted radius

rd = distorted radius

Before After

http://www.grasshopperonline.com/barrel_distortion_correction_software.html

Correcting Radial Distortions

photo by Robert Johnes

Vignetting

B

A

L3 L1L2

• More light passes through lens L3 for scene point A than scene point B

• Results in spatially non-uniform brightness (in the periphery of the image)

Vignetting

Chromatic Aberration

longitudinal chromatic aberration  transverse chromatic aberration (axial) (lateral)

longitudinal chromatic aberration (axial)

Canon EF 85/1.2 L USM

transverse chromatic aberration (lateral)

Cosina 3.5-4.5/19-35 @ 20 mm

Chromatic Aberration

Good lens

Carl Zeiss Distagon2.8/21

http://www.vanwalree.com/optics/chromatic.html

Chromatic Aberration

Near Lens Center Near Lens Outer Edge • Rays parallel to the axis do not converge

• Outer portions of the lens yield smaller focal lengths

Spherical aberration

Spherical aberration

Spherical lens are free of chromatic aberration but do not focus well.Parabolic lens does.

Spherical aberration

Astigmatism

Different focal length for inclined rays

Astigmatism

Change in size and shape of blur patches

Comapoint off the axis depicted as comet shaped blob

Reading: http://www.dpreview.com

Lens Glare

• Stray inter-reflections of light within the optical lens system• Happens when very bright sources are present in the scene

Outline

• Pinhole camera• Lens• Lens aberrations

• Exposure• Sensors• Noise• Sensing color

Exposure• Two main parameters:

– Aperture (in f stop)

– Shutter speed (in fraction of a second)

Shutter Leaf Shutter

• Advantages– Uniform illumination– Entire frame illuminated at once

• Disadvantages– Illumination not constant over time

– Limitations on shutter speed

Focal Plane Shutter

• Advantages– Cost effective (one shutter needed

for all lenses– Can achieve very fast shutter speeds

(~1/10000 sec)

• Disadvantages– May cause time distortion

Aperture• Aperture is the diameter of the lens opening,

usually specified by f-stop, f/D, a fraction of the focal length.– f/2.0 on a 50mm means that the aperture is 25mm– f/2.0 on a 100mm means that the aperture is 50mm

• When a change in f-stop occurs, the light is either doubled or cut in half.

• Lower f-stop, more light (larger lens opening)

• Higher f-stop, less light (smaller lens opening)

Aperture

Constant speed.

Shutter speed – Rule of Thumb

• 1 step in the shutter speed scale corresponds to 1 stop in the aperture scale.

• Handheld camera: shutter speed = 1 / f

• Stabilized gear: 2-3 shutter speeds slower

• Typical speeds: 1/1000 s, 1/500 s, 1/250 s, 1/125 s, 1/60 s, 1/30 s, 1/15 s, 1/8 s, 1/4 s, 1/2 s 1 s

f/22 f/4

Depth of Field

Small Aperture Large Aperture

(Low speed) (High speed)

Aperture vs. Shutter

1/30 sec. @ f/22

Aperture vs. Shutter

1/6400 sec. @ f/2.5

Small Aperture Large Aperture

(Low speed) (High speed)

Motion Blur

Dynamic Range Short exposure10-6 106

10-6 106

Real worldradiance

Pictureintensity

dynamic range

Pixel value 0 to 255

Long exposure10-6 106

10-6 106

Real worldradiance

Pictureintensity

dynamic range

Pixel value 0 to 255

Varying shutter speeds

HDR – High Dynamic Range Outline

• Pinhole camera• Lens• Lens aberrations• Exposure

• Sensors• Noise• Sensing color

Spatial Sampling

• When a continuous scene is imaged on the sensor, the continuous image is divided into discrete elements - picture elements (pixels)

Spatial Sampling

0x

Sampling

• The density of the sampling denotes the separation capability of the resulting image

• Image resolution defines the finest details that are still visible by the image

• We use a cyclic pattern to test the separation capability of an image

Sampling Frequency

Sampling Frequency

1D Example:0

Nyquist Frequency

• Nyquist Rule: To observe details at frequency f (wavelength d) one must sample at frequency > 2f (sampling intervals < d/2)

•• The Frequency 2f is the NYQUIST frequency.

• Aliasing: If the pattern wavelength is less than 2d erroneous patterns may be produced.

Aliasing - Moiré Patterns Quantization

Digitizers (Quantization) Image Sensors• CCD

– Charge Coupled Device

• CMOS– Complementary Metal Oxide Semiconductor

MOS (Metal Oxide Semiconductor)

• Photosensitive element• Charge acquired depends on the number of

photons which reach the element• CCD devices are arrays of this basic element

Photoelectric Effect

photon phot

on

Hole Electron

Incr

easi

ng e

nerg

y

Valence Band

Conduction Band

1.26eV

• Thermally generated electrons are indistinguishable from photo-generated electrons ‘Dark Current’.

Quantum Efficiency• Not every photon hitting a pixel creates a free electron• Quantum Efficiency (QE) =

– electrons collected / photons hitting the pixel

• QE heavily depends on the wavelength

• QE < 100% degrades the SNR of a camera

• Typical max QE values : 25% (CMOS) … 60% (CCD)

pe SNRQESNR =

QE [%]

lambda [nm]

blue green red

CCD (Charge Coupled Device)

• Boyle and Smith, 1969, Bell Labs• Converts light into electrical signal (pixels)

CCD Readout

•Charge Amplifier

Bucket Brigade• Integration• Charge Shift and

Read-out

CMOS (Complementary Metal-Oxide Semiconductor)

• Each pixel owns its own charge-voltage conversion• No need for external shutter (electronic shutter)• The chip outputs digital bits• Much faster than CCD devices

CCD vs. CMOS• Mature technology• Specific technology• High production cost• High power consumption• Higher fill factor• Blooming• Sequential readout

• Recent technology• Standard IC technology• Cheap• Low power• Less sensitive• Per pixel amplification• Random pixel access• Smart pixels• On chip integration

with other components

Sensor Parameters

• Fill factor– The area in the sensor that is truly sensitive to light– Shift registers and others can reduce it up to a 30%

• Well capacity– The quantity of charge that can be stored in each pixel– Close relation with pixel dimensions

• Integration time:– Exposure time that is required to excite the CCD elements– Depends on the scene brightness

• Acquisition time:– Time needed to transfer the information gathered by the CCD– Depends on the number of pixels in the sensor

Fill Factor

• The ratio between the light sensitive pixel area and the total pixel area.

Total pixel area:5µm x 5µm

Fill factor ≈ 40%

PhotoSensing

Area

Outline

• Pinhole camera• Lens• Lens aberrations• Exposure• Sensors

• Noise• Sensing color

Sensor noise

Noise Sources

• Photon noise / Shot Noise (Poisson)

• Dark Noise (Constant)

•Thermal noise (Poisson)

• Resetting (fixed)

• Read-out noise

• Blooming

(After T. Lomheim, The Aerospace Corporation)http://www.stw.tu-ilmenau.de/~ff/beruf_cc/cmos/cmos_noise.pdf

Photon Shot Noise

• Light is quantum in nature • Noise due to statistics of the detected photons

themselves• The probability distribution for N photons to be

counted in an observation time T is Poisson

• F = fixed average flux (photons/sec)

( ) ( )!

,|N

eFTTFNPFTN −

=

0 2 0 4 0 6 0 8 0 1 0 00

0 .0 5

0 .1

0 .1 5

0 .2

Poisson Distribution, FT = 5

N

Poisson Distribution : std equals sqrt of Mean.

photonsshot NFT ==2σ

0 2 0 4 0 6 0 8 0 1 0 00

0 .0 2

0 .0 4

0 .0 6

0 .0 8

0 .1

0 .1 2

0 .1 4

Poisson Distribution, FT = 10

N

0 2 0 4 0 6 0 8 0 1 0 00

0 .0 2

0 .0 4

0 .0 6

0 .0 8

0 .1

Poisson Distribution, FT = 20

N

Poisson Distribution, FT = 50

0 2 0 4 0 6 0 8 0 1 0 00

0 .0 1

0 .0 2

0 .0 3

0 .0 4

0 .0 5

0 .0 6

N

• As FT grows, Poisson distribution approaches Gaussian distribution.

• Signal To Noise (SNR) Increases with Mean.

NNNNSNR

2

2

2==

σ=

Photon Noise

• More noise in bright parts of the image

• You can identify the white and black regions from the noise image

Photon NoisePhoton Noise more noticeable in dark images.

Dark Current Noise

• Electron emission when no light• Dark current noise is high for long exposures• To remove (some) of it

– Calibrate the camera (make response linear)– Capture the image of the scene as usual– Cover the lens with the lens cap and take another

picture– Subtract the second image from the first image

Original image + Dark Current Noise Image with lens cap on

Result of subtractionCopyright Timo Autiokari,

Dark Current Noise

Sensor noise

ideal relationship betweenelectrons and impinging photons

Photon Noise (QE = 50)Light Signal (QE = 50)

CCD capacity limit

Outline

• Pinhole camera• Lens• Lens aberrations• Exposure• Sensors• Noise

• Sensing color

Sensing Color

beam splitter

light

3 CCD Bayer pattern

Foveon X3TM

Multi-Chip

wavelengthdependent

Field Sequential Field Sequential

Field Sequential

Fuji Corporation

Color Filter Array (CFA)

Color filter array

Color filter arrays (CFAs)/color filter mosaics

Bayer pattern

Bayer’s pattern

Demosaicking CFA’s Color filter array

red green blue output

X3 technology

red green blue output

Foveon X3 sensor

X3 sensorBayer CFA

Cameras with X3

Sigma SD14 Polaroid X530 Hanvision HVDUO5M/10M

Out of production

Color processing

After color values are recorded, more color processing usually happens:

– White balance

– Non-linearity to approximate film response or match TV monitor gamma

White Balance

automatic white balancewarmer +3

Gamma Correction• Gamma correction applied by the converter

redistributes the pixel luminance values so that limited brightness range captured by the sensor is “mapped” to match our eye’s sensitivity. Gamma = 2.2 is a good match to distribute relative brightness in a print or in a video display.

Gamma =1 vs. Gamma = 2.2 Space of response curves

Outline

• Pinhole camera• Lens• Lens aberrations• Exposure• Sensors• Noise• Sensing color

• Summary

Camera pipeline

Sensor Response

• The response of a sensor is proportional to the radiance and the throughput

Measurement Equation

• Scene Radiance L(x,ω,t,λ)• Optics (x’,ω’) = T(x,ω,λ)• Pixel Response P(x,λ)• Shutter S(x,ω,t)

Degradation Due To Sampling

• Sampling in space

• Sampling in intensity

• Sampling in color

• Sampling in time

• Sampling in frequencyExposure

Quantization

Pixels

Lens and pixel PSF (point-spread-function)

Color Filter Array (CFA)