motion deblurring using hybrid imaging
DESCRIPTION
Motion Deblurring Using Hybrid Imaging. Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA. Image Recording Requires Time . Daguerre 1829 1/2 hour exposure . Niépce 1827 8 hours exposure . Motion Blur is Everywhere. Object Motion. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/1.jpg)
Motion Deblurring Using Hybrid Imaging
Moshe Ben-Ezra and Shree K. Nayar Columbia University
IEEE CVPR ConferenceJune 2003, Madison, USA
![Page 2: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/2.jpg)
Image Recording Requires Time
Niépce 18278 hours exposure
Daguerre 18291/2 hour exposure
![Page 3: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/3.jpg)
Motion Blur is Everywhere
Object Motion
Camera Motion
![Page 4: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/4.jpg)
Stabilized Lenses
1/250 second (< -1 stop) Stabilization drifts with time Rotation only
1/15 second (< -5 stops)
Canon Stabilized lens 400mm
![Page 5: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/5.jpg)
Blind Image Deconvolution
Accurate Point Spread Function (PSF) Needed.
![Page 6: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/6.jpg)
Motion Point Spread Function (PSF)
Motion PSF is a Function of:
1. Motion path2. Motion speed
X
Y
Ener
gy ~
1/ s
peed
Spatial spread
H
![Page 7: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/7.jpg)
PSF Detector?
Camera
PSF Detector
Can the PSF detector be a small and simple
imaging device ?
![Page 8: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/8.jpg)
Electron wells
Fundamental Limits of Imaging
Detector’s noise level
Photon flux
Detector
Pixel’s Signal Noise
![Page 9: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/9.jpg)
Fundamental Resolution Tradeoff
Spatial resolution (pixels)
Tem
pora
l res
olut
ion
(fps
)
30
330K720x480
Conventional video camera
130
3
3M2048x1536
Hi-resolution camera
75K320x240
Low-resolution camera
Hybrid imaging system
A Hybrid camera enjoys both worlds
![Page 10: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/10.jpg)
Overview of Approach
PSF Estimation
Low-Res. camera
Hi-Res. camera
Same time period
Deconvolution
Motion Analysis
x
y
![Page 11: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/11.jpg)
Global Motion From Low Resolution Detector
uv1
cos sin x sin cos y
0 0 1
xy1
TranslationRotation
2
),(minarg
tI
yIv
xIu
vu
Objective function (Optical flow constraint)
Lucas Kanade
![Page 12: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/12.jpg)
Simulations: Motion Accuracy from Low- Res. Images
NoiseResolution
= 3 = 9 = 27 = 81
640x640 (1:1) 0.01 0.01 0.02 0.04320x320 (1:4) 0.03 0.04 0.05 0.1
160x160 (1:16) 0.03 0.04 0.07 0.480x80 (1:64) 0.13 0.21 0.39 2.6
Average Motion Error in Pixels
![Page 13: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/13.jpg)
Constraints on Continuous PSF
Energy conservation constraint:
yx
dydxyxh 1),(
h(x(t),y(t))dt ttend tstartt
t t
Path is continuous and twice differentiable
Constant flux assumption:
Smoothness constraint:
![Page 14: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/14.jpg)
PSF Estimation from Computed Motion
x
f1f2 f3 f4
f5
f6y
Frame 2 … Frame 5
y f1f2 f3 f4
f5
f6
x
yh
h2
h3
h4h5
Frame 2 … Frame 5
yh
h2
h3
h4h5
Frame 2 … Frame 5
x
![Page 15: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/15.jpg)
Deconvolution of High Resolution Image Standard iterative ratio-based algorithm*
,ˆ)()()(ˆ)(ˆ
)()(ˆ
)()()1(
)0(
k
kk
OSxIxSxOxO
xIxO
Guaranties non-negative pixel result
* Richardson [72] Lucy [74]
ErrorPSFImage estimate
![Page 16: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/16.jpg)
Designs for Hybrid Imaging
A rig of two cameras
Using a special chip
Using a beam splitter
![Page 17: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/17.jpg)
Our Prototype: Rig of Two Cameras
Primary detector(2048x1536)
Secondary detector(360x240)
Resolution ratio of 1 : 36
![Page 18: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/18.jpg)
Example 1 - Blurred Hi-Res Image
f = 633mm, Exp. Time 1 Sec (> -9 stops)
![Page 19: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/19.jpg)
PSF Estimation from Motion
Low resolution sequence.
X (Pixels)10 130
10
90
Y (P
ixel
s)0.001
0.06
Estimated PSF
f = 633mm, Exp. Time 1 Sec
![Page 20: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/20.jpg)
Deblurred Image
f = 633mm, Exp. Time 1 Sec
![Page 21: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/21.jpg)
Example 1 - Comparison
Deblurred imageBlurred image f = 633mm, Exp. Time 1 Sec
Tripod image (Ground Truth)
![Page 22: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/22.jpg)
Example 2 - Blurred Night Image
f = 884mm, Exp. Time 4 Sec (> -11 stops)
![Page 23: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/23.jpg)
PSF Estimation from Motion
X (Pixels)10 60
10
30
Y (P
ixel
s)0.001
0.003
f = 884mm, Exp. Time 4 Sec
Low resolution sequence.
![Page 24: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/24.jpg)
Deblurred Night Image
f = 884mm, Exp. Time 4 Sec
![Page 25: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/25.jpg)
Example 3 - Comparison
Deblurred imageBlurred image
Tripod image (Ground Truth)
f = 884mm, Exp. Time 4 Sec
![Page 26: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/26.jpg)
Object Deblurring Problem
Moving objects blend into the background
![Page 27: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/27.jpg)
Hybrid Imaging Solution (simulated)
Requires clear high-resolution background image
![Page 28: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/28.jpg)
![Page 29: Motion Deblurring Using Hybrid Imaging](https://reader035.vdocuments.net/reader035/viewer/2022062501/56816843550346895dde18a3/html5/thumbnails/29.jpg)
Quantifying The Affect of Motion Blur
Empirical tests: RMS error. Volume of Solutions (Linear Model):
y Ax z x A 1y A 1z
High-ResolutionImage
Uncertainty (Quantization)
InputImages
Volume of Solutions 1/det(A)
BlurDecimation