Camera Models and Noise

Stoyan Furnadzhiev

Overview

• Light rays travel from the scene, which is represented by a plenoptic function.

• The plenoptic function gives us the radiance of the scene.

• Radiance - amount of light that is emitted from a scene and falls within a given solid angle in a specified direction

Overview

• Rays travel through the lens system and focus onto the sensor, causing irradiance on a certain pixel.

• Irradiance - power per unit area incident on a surface.

• In discrete sense photons incident on the sensor are collected and converted into measurable voltage.

• This voltage is transformed by analogue-digital-converter (ADC) for the use of the processor.

Overview

� We want to build a physically-based camera model that accurately computes the irradiance on a film given the incoming radiance from the scene.

� Furthermore, we want to calibrate a CCD sensor to remove the effects of fixed pattern nonuniformity and spatial variation in dark current.

Realistic Camera Model

• Current camera models do not compute properly geometry, depth of field or exposure.

• Let's try to build physically-based camera model.

• Simulate the lens system of a camera.

Lens System

• What is a lens system?

– A combination of different spherical/aspherical glass or plastic lenses and stops centred on a common axis.

– Aperture stop - limits the angular spread of rays.

Simulating Lens System

• Computing correct image geometry.

– Ideally one point in object space coresponds to one point in image space.

– Unfortunately the world is not perfect.

– Aberrations in the lens system and/or diffraction effects will spread the point.

– A point in object space is represented on the image space using a point spread function (PSF).

Simulating Lens System

• Computing correct image radiometry.

– Ideally lenses focus light evenly on the image plane.

– Guess what?

– Real lenses suffer from an uneven exposure across the backplane.

– This depends as well on the aperture stop.

Depth of Field

• If object is not focused on the focus plane then it is blurred.

• The radius of the blur depends on the size of the aperture.

• Smaller aperture - smaller blur radius - larger depth of field.

• Larger aperture - more light rays come through the lens system.

Depth of Field

f/4

f/16

Lens Geometry

• Tracing Rays Through Lens System

– A simple algorithm is used:R = RayFor each element Ei from rear to front,If p is outside clear aperture of Ei

ray is blockedElse if medium is on far side of Ei ≠ medium

on near side compute new direction for R using Snell's law

Lens Geometry

• Thick Lens Approximation

– F & F´ - focal points;

– P & P´ - principal planes;

– Distance from P to P´ is the effective thickness (t).

Lens Geometry

• Focusing

– Done by moving the lens along the axis.

– We can estimate the distance T that we should move the lense:T² + T(2f´ + t - z) + f´² = 0, wherez is the distance form the focusing point to the film plane.

Lens Geometry

• The Exit Pupil

– Defined as the image of the aperture stop as viewed from the image space.

Lens Radiometry

• Exposure at a single pixel:

– H(x´) = E(x´)T

– H(x´) is the exposure at x´

– E(x´) is the irradiance at x´

– T is the exposure duration.

• We have to compute E(x´):

– We integrate the radiance at x´ over the solid angle subtended by the exit pupil.

Lens Radiometry

Lens Radiometry

• Z - axial distance from the film plane to the disc of the exit pupil.

• L - the ray coming from the disc at point x´´.

• A - the area of the disc.

• There are 2 approximations of this formula depending on the solid angle.

Sampling

• We estimate E(x´) by sampling radiance -casting rays from the pixel area towards the lens.

• We sample within the solid angle subtended by the exit pupil.

• Choosing a good sampling pattern is essential.

• Uniformly distributed points in the image plane map to uniformly distributed points on the disk.

Sampling

• One mapping takes sub-rectangles [0, x] × [0, 1] to a chord witch are proportional to x.

Results

200mmtelephoto lens

50mmdouble-Gauss lens

35mmwide-angle lens

16mmfisheye lens

Images synthesized with a 35mm wide-angle lens using, in order of decreasing accuracy, the full simulation (up), thick approximation (middle), and the standard model (down).

Same scene as previous slide, but the camera is focused on the picture frame.

An image taken with the fisheye lens. Barel distortion and darkening caused by vignetting are visible.

Camera Calibration and Noise Estimation

• View a CCD (charge-coupled device) sensor from the perspective of machine vision.

• Estimating sensor noise and removing part of it.

• Measuring scene variation, which does not depend on image irradiance.

Overview of CCD Camera

• CCD measures distribution of light on a thin layer of silicon.

• A photon strikes the silicon photoelectron is generated.

• The photoelectrons are collected in a collection site (potential well) representing one pixel.

• The process of charge coupling is used for transferring the stored charge.

• An output amplifier reads out an entire row.

Overview of CCD Camera

Overview of CCD Camera

The photoelectrons (blue) are collected in potential wells (yellow) created by applying positive voltage at the gate electrodes (G). Applying positive voltage to the gate electrode in the correct sequence transfers the charge packets.

The Camera Model

• I - number of electrons at a collection site.

• T - integration time.

• x, y - continuous coordinates on the sensor plane.

• λ - wavelength.

• B - incident spectral irradiance.

• Sr- spatial response of the collection site.

• q - ratio of electrons collected per incident light energy for the device.

Types of Noise

• Fixed pattern noise:

– Spatial nonuniformity caused by processing errors during CCD fabrication.

– KI - electrons collected at a site, where K is a constant associated with the collection site.

• Dark current:

– Free electrons generated by thermal energy.

– Proportional to integration time.

– Temperature dependant.

– NDC - number of dark electrons.

Types of Noise

• Shot noise:

– Characterizes the uncertainty in the number of stored electrons.

– Follows Poisson distribution.

– Cannot be eliminated.

– The variance depends on the number of collected photoelectrons (KI) and dark electrons (NDC).

– NS - zero mean Poisson shot noise.

• The number of electrons per collection site:

– KI + NDC + NS

Types of Noise

• Read noise:

– Generated by the output amplifier.

– Zero mean read noise (NR) is independant of the number of collected electrons.

• Video signal leaving the camera:

– V = (KI + NDC + NS + NR)A,where A is the combined gain of the output amplifier and the camera circuitry.

Types of Noise

• The analogue video signal is quantized to produce digital image, so V is rounded up to a digital value D.

• The quantization process is modeled as the addition of a noise source N

Q

• D = (KI + NDC

+ NS

+ NR)A + N

Q

Modeling Reflectance and Illumination Variation

• Spatial variation in illumination and reflectance leads to spatial variation on the collected charge.

• The collected charge I can be presented like this

– I = S + E, where S holds the mean illumination and reflectance and thus does not depend on the collection cite; E holds the spatial variance of the illumination and of the reflectance of the surface.

Estimating Sensor Noise

• Let's present D in the following way:

– D = µ + N, whereµ = KIA + EDCA, where EDC is the expected value of NDC;N = NI + NC = (NSA) + (NRA + NQ).

– NI is the part of the noise that depends on the number of collected electrons and has Poisson distribution.

– NC does not depend on the number of collected electrons and has Normal distribution.

Camera Calibration

• Estimating Variation in Dark Current.

– We take shots in dark environment so I ≡ 0.

– D = (NDC + NS + NR)A + NQ.

– We average a number of these images.

– We obtain a dark reference image, which we denote by DD.

– This dark image has small variation, which is inverse proportional to the number of taken images.

Camera Calibration

• Estimating Fixed Pattern Variation

– A good estimation for K can be found - K with small variance and high accuracy.

• Let's define a corrected version DC

of an image D.

– DC = (D - DD) ⁄ K

– DC = (I + NS ⁄ K + NR ⁄ K)A + NQ ⁄ K

– Assumption: the errors in the approximations DDand K are small compared to the variance of the remaining noise.

Scene Variation

• Image variance can be separated to camera noise variance and scene variation.

• Scene variation is independent of the image irradiance therefore is useful for scene description and surface identification.

Conclusion

We have managed to build a realistic physically-based lens system, which delivers correct radiance and geometry to the sensor. Furthermore a calibrated camera model can be used to quantify accurately the noise properties of a CCD sensor.

References

• “A realistic Camera Model for Computer Graphics” -Craig Kolb, Don Mitchell, Pat Hanrahan

• “Radiometric CCD Camera Calibration and Noise Estimation” - Glenn E. Healey, Raghava Kondepudy

• Wikipedia

• World Wide Web