motion compensated snr and dr enhancement with motion blur prevention using multicapture

1

Motion Compensated SNR and DR Enhancement With Motion Blur Prevention Using Multicapture

Ali Ercan & Ulrich Barnhoefer

EE392J Project Ali Ercan & Ulrich Barnhoefer 2

Introduction & Motivation

Single exposure trade-off High noise if short

exposure time Motion blur if long

exposure time

EE392J Project Ali Ercan & Ulrich Barnhoefer 3

Introduction & Motivation DR is another problem

Short exposure: Dark areas in the scene cannot be seen

Long exposure: Bright areas saturate If both high DR scene and motion,

with single capture Motion blur free, but noisy and non-

visible dark areas image Less noisy, but motion blurred and

saturated image

EE392J Project Ali Ercan & Ulrich Barnhoefer 4

Introduction & Motivation Our approach to solve these

problems: Use of multicapture combined with

motion estimation High speed is definitely needed Normal video mode can be used –

poorer results due to noise adding CMOS imagers suitable For a better understanding, let us

introduce a simple model of CMOS imagers and describe multicapture

EE392J Project Ali Ercan & Ulrich Barnhoefer 5

Sensor Model Charge Integration

Light on photodiode generates charges

Saturation when well capacity is reached

Noise sources (Reset noise) Shot noise UT

Read noise VT,Vo

(Dark current) (Fixed pattern noise)

0VVUTiQ TTph

EE392J Project Ali Ercan & Ulrich Barnhoefer 6

Multicapture

Nondestructive multiple readout – Single integration

Less noise per capture compared to conventional video sensor – readout noise

k

loklphk VVUkiQ

1

EE392J Project Ali Ercan & Ulrich Barnhoefer 7

Implemented Algorithm

SCENECAMERA

SIMULATOR

MOTIONESTIMATOR

PHOTO-CURRENTESTIMATOR

FINALIMAGE

EE392J Project Ali Ercan & Ulrich Barnhoefer 8

Camera Simulator Multicapture, noise, ADC

implemented – pixel values out

EE392J Project Ali Ercan & Ulrich Barnhoefer 9

Motion Estimator Block based motion estimation on

difference frames Search range ±1 and block size 3x3 Fast imager (e.g. 10,000 fps available) Search range and block size can be increased

in expense of computational load Noise suppression

Known noise levels – characterized CMOS sensor

Error = SSD + xDistance is proportional to noise Thanks to Sebe!

EE392J Project Ali Ercan & Ulrich Barnhoefer 10

Motion Estimator Estimated and perfect motion vectors

EE392J Project Ali Ercan & Ulrich Barnhoefer 11

Photocurrent Estimator

A

B

C

k

l

Co

Ckl

Ck

k

l

Bo

Bkl

Bk

k

l

Ao

Akl

Ak

VVUikQ

VVUikQ

VVUikQ

1

1

1

CCC

BBB

QQQ

QQQ

56'

1

34'

1

CCCC

BBBB

VVUiQ

VVUiQ

566'

1

344'

1

Ao

AAAA

Ao

AAA

VVUUiQ

VVUiQ

2212

111

2

A

CBA

QQ

QQQQ

2

~

2

'1

'11

~

1 )(3

1

EE392J Project Ali Ercan & Ulrich Barnhoefer 12

Photocurrent Estimator

j

l

n

l

Xj

jj Qn

Q1

~ 1

k

okk

ll

k

kn

VVU

nikQ

1

~ 1

kn

VV

kn

Ui

k

QI

k

ok

k

k

l lkk

1

~~

N

jjj IaI

1

~^

2^

2 )( iIE

iIE )(^

11

N

jja

N

j j

vjN

jl l

lu nj

a

ln

a

1

222

22 2

N

l l

jj

j

l l

l

v

ujj

j

jjj

b

ba

nl

bjnb

n

n

j

j

n

njb

b

1

1

12

2

111

1

21

1

EE392J Project Ali Ercan & Ulrich Barnhoefer 13

Results

EE392J Project Ali Ercan & Ulrich Barnhoefer 14

Results

EE392J Project Ali Ercan & Ulrich Barnhoefer 15

Results

EE392J Project Ali Ercan & Ulrich Barnhoefer 16

Results

IMAGE ERRORS(STD OF ERROR IMAGE)

CHECKER

LENA CAMERAMAN

10 ms image 100.9 69.43 71.31

160 ms image 70.79 33.84 37.41

Const. with est. motion vectors 2.587 21.28 12.05

Const. with perfect motion vectors

2.576 17.22 3.092

EE392J Project Ali Ercan & Ulrich Barnhoefer 17

Conclusion

Promising results achieved with this preliminary analysis Motion blur reduced Noise reduced DR increased in dark end and in

bright end in special cases

EE392J Project Ali Ercan & Ulrich Barnhoefer 18

Conclusion Lots of more things to do

Use more sophisticated motion estimation algorithms

Separate motion detection from motion estimation and do estimation when detection occurs

Include extension of DR with sensor saturation

Handle the occlusions

EE392J Project Ali Ercan & Ulrich Barnhoefer 19

Questions