Radiometric Self CalibrationRadiometric Self Calibration

Tomoo Mitsunaga Shree K. NayarTomoo Mitsunaga Shree K. NayarHashimoto Signal Processing Lab. Hashimoto Signal Processing Lab. 　 　　 　 Dept. of Computer Dept. of Computer ScienceScience

Sony Corporation Columbia UniversitySony Corporation Columbia University

CVPR ConferenceFt. Collins, Colorado

June 1999

June/1999 CVPR99 2

Problem StatementProblem Statement

How well does the image represent the real world?How well does the image represent the real world?

Image Image MM11

(High exposure)(High exposure)Image Image MM22

(Low exposure)(Low exposure)

• Non-linear responseNon-linear response )( 12 MM )( 12 MM

• Limited dynamic rangeLimited dynamic rangeUsual imaging systems have :Usual imaging systems have :

June/1999 CVPR99 3

Scene Radiance and Image IrradianceScene Radiance and Image Irradiance

Image irradianceImage irradiance : :

RadianceRadiance IrradianceIrradiance

24

2

2cos

4

dkL

h

dLE 2

42

2cos

4

dkL

h

dLE

tEI tEI

td

e

2

2 t

de

2

2

Ideal camera responseIdeal camera response : :

ExposureExposure : :

LLEE

Aperture areaAperture area

ekLI ekLI

June/1999 CVPR99 4

Scene Radiance and Measured Brightness Scene Radiance and Measured Brightness

ImageImageFormationFormation

ImageImageExposureExposure CCDCCD

CameraCameraElectronicsElectronics

DigitizationDigitization

ImageImageFormationFormation

ImageImageExposureExposure FilmFilm

FilmFilmDevelopmentDevelopment ScanningScanning

VideoVideo

PhotoPhoto

SceneSceneradianceradianceLL

ScaledScaledradianceradianceII

MeasuredMeasuredbrightnessbrightness

MM)(

)(1 IfM

MfI

)(

)(1 IfM

MfI

linearlinear

f (M) : The radiometric response function

June/1999 CVPR99 5

Calibration with Reference ObjectsCalibration with Reference Objects

The scene must be controlledThe scene must be controlled• The reflectance of the objects must be knownThe reflectance of the objects must be known• The illumination must be controlledThe illumination must be controlled

June/1999 CVPR99 6

Calibration without Reference ObjectsCalibration without Reference Objects• Differently exposed images from an arbitrary sceneDifferently exposed images from an arbitrary scene• Recover the response function from the imagesRecover the response function from the images• Calibrate the images with the response functionCalibrate the images with the response function

High dynamic range radiance imageHigh dynamic range radiance image

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Response functionResponse functionInput ImagesInput Images

June/1999 CVPR99 7

Previous WorksPrevious Works• Mann and Picard (95) :Mann and Picard (95) :

– Take two images with known exposure ratio Take two images with known exposure ratio RR

– Restrictive model for Restrictive model for ff : :

– Find parameters Find parameters by regression by regression

• Debevec and Malik (97) :Debevec and Malik (97) :– General model for General model for ff : only smoothness constraint : only smoothness constraint

– Take several (say, 10) high quality imagesTake several (say, 10) high quality images

– At precisely measured exposures (shutter speed)At precisely measured exposures (shutter speed)

IM IM

June/1999 CVPR99 8

Obtaining Exposure InformationObtaining Exposure Information

We have only rough estimatesWe have only rough estimates• Mechanical errorMechanical error• Reading error (ex. F-stop number)Reading error (ex. F-stop number)

June/1999 CVPR99 9

Radiometric Self-CalibrationRadiometric Self-Calibration

Works with roughly estimated exposuresWorks with roughly estimated exposures• Inputs :Inputs :

– Differently exposed imagesDifferently exposed images

– Rough estimates of exposure valuesRough estimates of exposure values– ex. F-stop readingex. F-stop reading

• Outputs :Outputs :– Estimated response functionEstimated response function

– Corrected exposure valuesCorrected exposure values

June/1999 CVPR99 10

A Flexible Parametric ModelA Flexible Parametric Model

nN

nnMcMfI

0

)( nN

nnMcMfI

0

)(

High order polynomial model :High order polynomial model :

Parameters to be recovered :Parameters to be recovered :• Coefficients Coefficients ccnn

• Order Order NNff((MM) of some ) of some

popular imaging productspopular imaging products

videovideo

posiposi

neganega

M

f (M)

June/1999 CVPR99 11

Response Function and Exposure RatioResponse Function and Exposure Ratio

Images: Images: qq = 1,2,…. = 1,2,….QQ , Pixels: , Pixels: pp = 1, 2, ….. = 1, 2, …..PP

Exposure ratio:Exposure ratio:)(

)(,

1,

,

1,

,

11,

11,

qp

qp

qp

qp

qpp

qppqq

q

qqq Mf

Mf

I

I

ekL

ekLR

e

eR

)(

)(,

1,

,

1,

,

11,

11,

qp

qp

qp

qp

qpp

qppqq

q

qqq Mf

Mf

I

I

ekL

ekLR

e

eR

1,

0 1,

0 ,

qqN

n

nqpn

N

n

nqpn R

Mc

Mc1,

0 1,

0 ,

qqN

n

nqpn

N

n

nqpn R

Mc

McUsing polynomial model :Using polynomial model :

21

1 1 0 01,1,,

Q

q

P

p

N

n

N

n

nqpnqq

nqpn McRMc

21

1 1 0 01,1,,

Q

q

P

p

N

n

N

n

nqpnqq

nqpn McRMcObjective function : Objective function :

Thus, we obtain ...Thus, we obtain ...

June/1999 CVPR99 12

An Iterative Scheme for OptimizationAn Iterative Scheme for Optimization

OptimizeOptimizefor for ff

OptimizeOptimizefor for ff

OptimizeOptimizefor for RRq,qq,q+1+1

OptimizeOptimizefor for RRq,qq,q+1+1

f f ((ii))

RRq,qq,q+1+1((ii))

Rough estimates Rough estimates RRq,qq,q+1+1

(0)(0)

Rough estimates Rough estimates RRq,qq,q+1+1

(0)(0)

Optimized Optimized f f andand R Rq,qq,q+1+1Optimized Optimized f f andand R Rq,qq,q+1+1

Mff ii )1()( Mff ii )1()(

June/1999 CVPR99 13

Evaluation : Noisy Synthetic ImagesEvaluation : Noisy Synthetic Images

Solid : Computed response functionSolid : Computed response functionDotsDots : Actual response function : Actual response function

MM

f f ((MM))

June/1999 CVPR99 14

Evaluation : Noisy Synthetic Images (cont’d)Evaluation : Noisy Synthetic Images (cont’d)

PercentagePercentage Error inError in

ComputedComputed ResponseResponseFunctionFunction

Trial NumberTrial Number

Maximum Error : 2.7 %Maximum Error : 2.7 %

June/1999 CVPR99 15

• Calibrating by the response functionCalibrating by the response function• Normalizing by corrected exposure valuesNormalizing by corrected exposure values• Averaging with SNR-based weighting Averaging with SNR-based weighting

Computing a High Dynamic Range ImageComputing a High Dynamic Range Image

)(Mf )(Mf 1,pI 1,pI

2,pI 2,pI

QpI ,QpI ,

1,pM 1,pM

2,pM 2,pM

QpM ,QpM ,

1~1e1~1e

Qe~1

Qe~1

2~1e 2~1e

QpI ,

~QpI ,

~

2,

~pI 2,

~pI

Q

q qp

Q

q qpqp

Mw

IMw

1 ,

1 ,,

)(

~)(

Q

q qp

Q

q qpqp

Mw

IMw

1 ,

1 ,,

)(

~)( pI pI

1,

~pI 1,

~pI

)(Mf )(Mf

)(Mf )(Mf

e~e~

)(Mw )(Mw

)(Mf )(Mf

June/1999 CVPR99 16

Results : Low Library (video) Results : Low Library (video)

Captured imagesCaptured images

Calibration chartCalibration chart

Computed responseComputed response functionfunction

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

II

MM

June/1999 CVPR99 17

Results : Low Library (video)Results : Low Library (video)

Captured imagesCaptured images

Computed radiance imageComputed radiance image

June/1999 CVPR99 18

Results : Adobe Room (photograph)Results : Adobe Room (photograph)

II

Captured imagesCaptured images

Computed response functionComputed response function Computed radiance imageComputed radiance image

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1MM

June/1999 CVPR99 19

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Results : Taos Clay Oven (photograph)Results : Taos Clay Oven (photograph)

Captured imagesCaptured images

Computed response functionComputed response function Computed radiance imageComputed radiance image

II

MM

June/1999 CVPR99 20

ConclusionsConclusions

A Practical Radiometric Self-calibration MethodA Practical Radiometric Self-calibration Method• Works withWorks with

– Arbitrary still sceneArbitrary still scene

– Rough estimates of exposureRough estimates of exposure

• Recovers Recovers – Response function of the imaging systemResponse function of the imaging system

– High dynamic range image of the sceneHigh dynamic range image of the scene

Software and DemoSoftware and Demo

http://www.cs.columbia.edu/CAVE/http://www.cs.columbia.edu/CAVE/