Dept. Elect. Eng.Technion – Israel Institute of Technology

Radiometric Nonidealities:

A Unified Framework

Anatoly Litvinov , Yoav Y. Schechner

Support: Ollendorff Foundation (BMBF), GIF

1

Spatial Non-Uniformity

cameraIris

A. Litvinov & Y. Schechner, radiometric nonidealities

2

Spatial Non-Uniformity

iris

Automatic Gain Control (AGC)

time

gain

Temporal Non-Uniformity:

Litvinov, Schechner, radiometric nonidealities

Automatic Gain Control (AGC)

A. Litvinov & Y. Schechner, radiometric nonidealities

AGC

G

Image Mosaicing

Standard seam removal: “feathering”

Levin; Zomet; Peleg; Weiss (04)

Duplaquet (98)Burt; Adelson (83)

Not addressing the root of inconsistencies

Soderblum et. al (78)

Shum; Szeliski (00) Jia; Tang (03)

A. Litvinov & Y. Schechner, radiometric nonidealities

3

electronic readout

light intensity

Fillfactory CMOS chip

Nonlinear Radiometric Response4

Image Intensifier

Photocathode Fiber optic

coupler

CCDAccelerating

potential

A. Litvinov & Y. Schechner, radiometric nonidealities

Phosphor screen:

nonlinear

Nonlinear Radiometric Response5

The Eye

A. Litvinov & Y. Schechner, radiometric nonidealities

Spatial non-uniformity

Auto gainRadiometric

response

42

Prior TechniquesPre-calibrations

Standard target

Integrating sphere

Exposure variations Debevec; Malik (98)

Mitsunaga; Nayar (99)

Mann; Picard (95)

Based on edge detection Lin et al. (04)

AND OTHERS …

Spatial non-uniformity estimationSchechner; Nayar (01)

Only radiometric responseOnly spatial non-uniformityOnly radiometric response

Only radiometric response

Only spatial non-uniformityCombined AGC and radiometric responseMann; Mann (01) Kim; Pollefeys (04)

not includingspatial non-uniformity Candocia; Mandarino (05)

6

Integrating Sphere

A. Litvinov & Y. Schechner, radiometric nonidealities

Spatial non-uniformity pre-calibration

visible mismatch

97 % - not accurate enough

We Achieve :

Blind estimation of: spatial non-uniformity

pixelelectronic readout

light intensity

A. Litvinov & Y. Schechner, radiometric nonidealities

non-uniformity AGC nonlinearity

7

Without “seam-removal” steps

A. Litvinov & Y. Schechner, radiometric nonidealities

Blind estimation of:

pixelelectronic readout

light intensity

non-uniformity AGC nonlinearity

8

Camera Model

GM(x) E

r(I)

A. Litvinov & Y. Schechner, radiometric nonidealities

10

irradiance of pixel xspatial non-uniformity of xradiometric response

grayscale value of x

r[ ]I(x)M(x)v(x) =

log{r [v(x)]} = log(G) + log[M(x)] + log[I(x)]-1

ρ(v) m(x)

G

g

gain

ρ(v) - m(x) – g = i i

A. Litvinov & Y. Schechner, radiometric nonidealities

-1rlog

A. Litvinov & Y. Schechner, radiometric nonidealities

ρ (255)

m (x=1)

m (x=N)

1 -1 -1

=

g1

gF

ρ(v) - m(x) – g = i

1 -1 -1 1 -1 -1

1 -1 -1

i 1

i N

ρ (0)

1 -1 -1

1 -1 -1

1 -1 -1

isB =

12

Bs = i

B γ s = γ i

exp

ρ(v) = γ ρ (v) ^

m(x) = γ m(x) ^

g = γ g^f f

r (v) = [r (v)] ^ -1 -1 γ

M(x) = M(x) γ^

G = G^

f f

γ

A. Litvinov & Y. Schechner, radiometric nonidealities

13

An Image Invariance

log I = γ log I^

A. Litvinov & Y. Schechner, radiometric nonidealities

14

Nevertheless … Mutually Consistent

A. Litvinov & Y. Schechner, radiometric nonidealities

15

ρ(v ) – m(x ) - g = log[I(x)] k k

ρ(v ) – m(x ) - g = log[I(x)] p p

Frame: k

Frame: p

A. Litvinov & Y. Schechner, radiometric nonidealities

k

p

ρ (v ) - ρ(v ) - m(x ) + m(x ) - g + g = 0p p k k k p

m(x) : log of mask at pixel x (v) : log of inverse-radiometric response

g : log of gain at frame f : f

16

A. Litvinov & Y. Schechner, radiometric nonidealities

ρ (0)

ρ (255)

m (x=1)

m (x=N)

1 -1 -1 1 -1 1

1- 1- 1 1- 1 1

-1 1 -1 1 -1 1

=0

g1

gF

1 -1 -1 1 -1 1

1 -1 -1 1 -1 1

ρ (v ) - ρ(v ) - m(x ) + m(x ) - g + g = 0p p k k k p

Rs = 0

Experiment: AGC, Vignetting, Gamma

pixel0 460

1transmittance (vignetting)

frame (time)1

4

150

amplifier gain

1

0 255graylevel

irrad

ianc

e

radiometric response

17

SVD solution

Rs = 0

Null space of R

s = 0

s = [ ρ = const , m = const , g = const ]T

A. Litvinov & Y. Schechner, radiometric nonidealities

1 2 3

Trivial Solutions

ρ(v) = const

light intensity

electronic readout

exp

r -1r

light intensity

electronic readout

A. Litvinov & Y. Schechner, radiometric nonidealities

Entropy Maximization

A. Litvinov & Y. Schechner, radiometric nonidealities

hist histhist

Anatoly Litvinov Yoav Schechner

A Unified Framework

1

4

150

1

0 255

0 460

1

log I = γ log I^

Image invariance

18