![Page 1: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/1.jpg)
Multisampling Compressive Video Spectroscopy
Daniel S. Jeon Inchang Choi Min H. KimKorea Advanced Institute of Science and Technology (KAIST)
![Page 2: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/2.jpg)
Fake or Real?
2
0
0.1
0.2
0.3
0.4
0.5
400 450 500 550 600 650 700
Radianceofpeppers
FakePepper
RealPepper
0
0.1
0.2
0.3
0.4
0.5
400 450 500 550 600 650 700
Radianceofapples
FakeApple
RealApple
Fake Real
[Yasum
aetal.20
08]
![Page 3: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/3.jpg)
Hyperspectral Imaging
RGBImaging MultispectralImaging
HyperspectralImaging
3
3channels <~30channels ≥~30channels
![Page 4: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/4.jpg)
Spectroscopy Imaging
4
Bandpass filter LCTF(liquidcrystaltunablefilter)
Pushbroom CASSI
[Mansouri etal.2007]
[Brusco etal.2006]
[Attas etal.2003]
[Wagadarikar etal.2008]
![Page 5: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/5.jpg)
Multisampling CASSI
5
Maskshiftingusing piezotranslationstage[Kittleetal.2010]
DMD(digital-micromirror-device)
LCoS (liquidcrystalonsilicon)
[Linetal.2014]
[Wuetal.2011]
MultisamplingCASSIsystemsrequiremultiplecaptures
![Page 6: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/6.jpg)
Goal
6
Singlecodedinput Hyperspectralvideo
• Multisamplingcompressiveimagingà Highspectralresolutionà Highspatialresolution
• Singlesnapshothyperspectralimagingà Videospectroscopy
![Page 7: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/7.jpg)
![Page 8: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/8.jpg)
Key Idea
8
• Codedaperturesnapshotspectralcamera
• Multisamplingè Kaleidoscope
![Page 9: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/9.jpg)
System Setup
objective lensimage multiplier relay lens collimating lens relay lenscoded
aperturedetector
prism
9
diffuser
![Page 10: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/10.jpg)
Examine Kaleidoscope
10
![Page 11: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/11.jpg)
View Multiplication
11
1/ 1
bma a f
= =-
1 1 1a b f+ =a
f
b
al bl
: maginificationm
Lightdirection
Diffuser
Detector
![Page 12: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/12.jpg)
Effect of Diffuser
12
Withdiffuser Withoutdiffuser
![Page 13: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/13.jpg)
Raw Input Video
13
![Page 14: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/14.jpg)
Geometric Calibration - Homography
1 2
3 4
2 1
4 3
2 1
4 3
3 4
1 2
3 4
1 2
4 3
2 1
4 3
2 1
4 3
2 1
4 3
2 1
14
![Page 15: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/15.jpg)
Geometric Calibration - Optical Flow
15
Beforeapplyopticalflow Afterapplyopticalflow
Animated5views
![Page 16: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/16.jpg)
Geometric Calibration - Dispersion Direction
Capturedimages Alignedimages
16
Dispersiondirectionflipped
![Page 17: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/17.jpg)
Image Reconstruction
17
( ), ,f x y l
( )0 , ,kf x y l ( ),kT x y ( )2 , ,kf x y l( )1 , ,kf x y l
Prism
( ),kg x yA set of
first-surfacemirrors
⊙
⊙
⊙
( ), ,h x y l
0( , ) ( ( ), , , , ) ( , ) ( , , )k k k kg x y h x x y y T x y f x y dx dy df l l l lL
¢ ¢ ¢ ¢= -ò òòDispersion IncidentMask
![Page 18: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/18.jpg)
Coded Aperture
18
Codedaperturespecs• Randombinarypatterns• correspondstotwo-by-twopixels
,( , ) rect ,k ijk
i j
x yT x y i jæ ö= - -ç ÷D Dè øåT
1pxonsensor
Each9viewpassthroughdifferentcodedaperturepatternsà Enablemultisampling
![Page 19: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/19.jpg)
Prism Dispersion
19
0
10
20
30
40
50
60
450 550 650 750
Pixelshift[p
x]
Wavelength[nm]
Dispersioncalibration
2 1( , , ) ( ( ), , , , ) ( , , )k k kf x y h x x y y f x y dx dyl f l l l¢ ¢ ¢ ¢= -òòDispersion
500nm
600nm
700nm
Dispersedlight Codedlight
![Page 20: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/20.jpg)
Reconstruction
20
• Minimizinganobjectivefunctionwithtotalvariation[Bioucas-DiasandFigueiredo 2007]
2
2Detectorinput
(3D)Lightmodulation
(4D)Hyperspectal
Image(3D)
RegularizingSparsity(3D)
![Page 21: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/21.jpg)
View Multiplication
21
1 view 2 views 3 views
reference9 views5 views(syntheticimages)
![Page 22: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/22.jpg)
Dispersion Direction
22
PSNR:28.20SSIM:0.88
PSNR:30.45SSIM:0.91
5viewswithoutdispersioninversion
5viewswithdispersioninversion
(syntheticimages)
reference
![Page 23: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/23.jpg)
Multiview Tradeoff
23
1(full)
PSNR:27.84SSIM:0.88
1 2 34 5 67 8 9
PSNR:23.42SSIM:0.77
1 2 34 5 67 8 9
PSNR:31.29SSIM:0.92
(syntheticimages)
reference
![Page 24: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/24.jpg)
Comparison
24
0.00.10.20.30.40.50.60.70.80.91.0
450 500 550 600 650
refle
ctan
ce
wavelength [nm]
red patchReference
CASSI
Ours
TraditionalCASSI
OurmultisamplingCASSI
5views
1fullview
![Page 25: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/25.jpg)
Results
25
Input Reconstructedhyperspectral video
sRGB video Wavelengthat600nm
![Page 26: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/26.jpg)
Results
26
Input Reconstructedhyperspectral video
sRGB video Wavelengthat600nm
![Page 27: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/27.jpg)
Results
27
Input Reconstructedhyperspectral video
sRGB video Wavelengthat600nm
![Page 28: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/28.jpg)
Results
28
Input Reconstructedhyperspectral video
sRGB video Wavelengthat600nm
![Page 29: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/29.jpg)
Discussion
• Tradeoff between spatial and spectral resolution
– Significantly enhance spectral resolution
– Sacrifice sensor resolution
• Misalignment of copied views gives a critical
reconstruction problem
• Alternatives for TV-L1 optimization
29
![Page 30: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/30.jpg)
Conclusion
30
• Singlesnapshot-baseddesign
• Hyperspectral videoacquisition
• Highspectralresolution
• Bycouplingmultisamplingandcompressiveimaging
![Page 31: Multisampling Compressive Video Spectroscopyvclab.kaist.ac.kr/eg2016/Eurographics_2016_presentation.pdf · Multisampling CASSI 5 Mask shifting using piezo translation stage [Kittle](https://reader035.vdocuments.net/reader035/viewer/2022071223/60830aad8f7f4a78ed671248/html5/thumbnails/31.jpg)
Acknowledgements
• Korea National Research Foundation (NRF) grants
(2013R1A1A1010165 and 2013-M3A6A6073718)
• Korea ICT R&D program of MSIP/IITP (10041313)
31
Thankyou