high-resolution hyperspectral imaging for cultural heritage

High-resolution Hyperspectral Imaging for Cultural Heritage

Rei Kawakami1 John Wright2 Yu-Wing Tai3 Yasuyuki Matsushita2 Moshe Ben-Ezra2 Katsushi Ikeuchi3

1University of Tokyo, 2Microsoft Research Asia (MSRA), 3Korea Advanced Institute of Science and Technology (KAIST)

2011 Dunhuang Forum

Giga-pixel Camera

M. Ben-Ezra et al.

Giga-pixel Camera

Large-format lens CCD

Spectrum

200 5000[nm]

100.10.001

700 1000600500400300200 [nm]

Ultraviolet InfraredVisible light

1 meter 3100 meter0.00001

CosmicRays

GammaRays

X-raysUltraViolet

InfraredTV AndRadio Waves

Electric Waves

Electromagnetic Spectrum

Vio

let

Blu

e

Gre

en

Yello

w

Ora

ng

e

Red

0

20

40

6080

100

Solar radiationreachingearth’s surface(Relative Energy)

RGB vs. Spectrum

Applications

Light simulation

Layered surface decomposition

Morimoto et al. CVPR2010: Estimating Optical Properties of Layered Surfaces Using the Spider Model

Why difficult?

Approach

Low-reshyperspectral

High-resRGB

High-resolutionHyperspectral image

Combine

Two-step approach

1. Factorize low-res hyperspectral image into basis functions of spectra and coefficients

2. For each pixel in high-res RGB image, estimate coefficients of the basis functions

Problem formulation

W(Image width)

H(Image height)

S

Goal:

Given:

(Spectral wavelength)

Representation: Basis function

W (Image width)

H (Image height)

S

𝒁

= …

01.00…0

= +x 0 x 1.0 x 0 x 0++

Reflectance vectors

1: Matrix factorization

Sparse

For all pixel (i,j)

Sparse matrix

W (Image width)

H (Image height)

S

= …

00.40…

0.6

𝒀 h𝑠

• At each pixel of , only a few () materials are present

Reflectance matrix

2: Reconstruction

W

H

S Sparse

𝒀 𝑟𝑔𝑏

�̂� (𝑖 , 𝑗 )=argmin‖𝒉‖1

Reconstruction

𝒁 (𝒊 , 𝒋 ,∗ )≈ 𝑨𝒉 (𝒊 , 𝒋 )

𝒁

• At each pixel of , materials should be even much fewer

Simulation experiments

Balloons Beads Sponges Oil painting

Flowers CD Peppers Face

Spectral image database F. Yasuma, T. Mitsunaga, D. Iso and S. K. Nayar.Generalized assorted pixel camera: Postcapture control of resolution,Dynamic range, and spectrum. IEEE Trans. IP, 19(9):2241-2253, 2010

460 nm 550 nm RGB/620 nm 460 nm 550 nm RGB/620 nm

Input images: Balloons and Beads examples

Ground truths

Reconstruction using component substitution method

Reconstruction by the proposed method

430 nm 490 nm 550 nm 610 nm 670 nm

Input images: Sponges examples

Ground truths

Reconstruction by the proposed method

Error images of the proposed method

RGBimage

GroundTruth(430 nm)

Estimated430 nm

Method Balloons Beads Sponges Oil painting

Flowers CD Peppers Face

CSM[2] 13.9 28.5 19.9 12.2 14.4 13.3 13.7 13.1

Global 6.9/4.7 10.5/8.8 15.4/12.3

5.4/3.8 9.8/8.9 10.3/10.0

7.1/5.9 4.7/3.8

Local win 7.0/4.9 10.6/8.9 14.0/10.6

5.7/4.1 7.5/6.3 9.6/9.2 8.8/8.0 10.9/10.5

RGB clust 6.6/4.3 9.7/7.9 13.6/10.0

5.5/4.0 7.8/6.5 9.1/8.6 8.5/7.6 4.7/3.8

Proposed 3.0/3.0 9.2/9.2 3.7/3.7 4.7/4.7 5.4/5.4 8.2/8.2 4.7/4.7 3.3/3.3

RMSE

Balloons Beads Sponges Oil painting

Flowers CD Peppers Face

HS camera

Filter

CMOSLens Aperture

Focus

Translational stage

Real data experiment

Input RGB Input (550nm) Input (620nm)Estimated (550nm) Estimated (620nm)

Summary•Method to reconstruct high-resolution

hyperspectral image from ▫Low-res hyperspectral camera▫High-res RGB camera

•Spatial sparsity of hyperspectral input▫Search for a factorization of the input into

basis functions set of maximally sparse coefficients

Acknowledgement

•This work was in part supported by Microsoft CORE 6 project.