compressive sensing of high-dimensional visual signalsas48/talks/ucr_nov13.pdf · digital...
TRANSCRIPT
Compressive Sensing of High-Dimensional Visual Signals
Aswin C Sankaranarayanan Rice University
Interaction of light with objects
Human skin Sub-surface scattering
Electron microscopy Tomography
Reflection
Refraction
Fog Volumetric scattering
The Plenoptic function Collection of all variations of light in a scene
Adelson and Bergen (91)
space (3D)
angle (2D)
spectrum (1D) time
(1D)
Different slices reveal different scene properties
space (3D)
angle (2D)
spectrum (1D)
time (1D)
Lytro light-field camera
High-speed cameras
Plenoptic function
Hyper-spectral imaging
Sensing the plenoptic function
• Extremely high-dimensional – 1000 samples/dim == 10^21 dimensional signal – Greater than all the storage in the world
• Material costs
– Sensing beyond the visible spectrum
• Sensing time can be costly – Medical imaging
• Need low-dimensional methods for sensing!
Signal models
Many signals exhibit concise geometric structures
Signal models
Many signals exhibit concise geometric structures
Subspaces Union of subspaces
Sparse models
Low-dim. non-linear models
GOAL: Tractable low-dimensional models for parsimonious sensing and efficient processing of
high-dimensional visual data
Roadmap
space (3D)
spectrum (1D)
time (1D)
A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging
Roadmap
space (3D)
spectrum (1D)
time (1D)
A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging
• Arbitrary signal
• Measurement matrix of rank N
• Number of measurements
• Least squares recovery
Linear sensing
M measurements
N-dim signal
y = �x+ e
M � N
y � x
• Is it possible to recover when ? • No … for arbitrary
Linear sensing
M measurements
N-dim signal
y = �x+ e
x
M < N
y �
x
x
Simple example: 1-sparse signal
• Signal prior: 1-sparse signal
• Possible with M = 2 measurements!
compressive measurements
signal
1 nonzero entry
y = �x+ e
y1 =
X
k
xk = signal value
y2 =
X
k
kxk = signal value ⇤ location
K-sparse signals
• Signal recovery – K-sparse signal
– Measurement matrix is i.i.d. sub-Gaussian
– Number of measurements
– Recovery via a convex program
compressive measurements
signal
nonzero entries
M ⇠ K log(N/K)
Candes,(Romberg,(Tao((06),(Donoho((06)(
y = �x+ e
• Signal recovery – K-sparse signal
– Measurement matrix is i.i.d. sub-Gaussian
– Number of measurements
– Recovery via a convex program
Compressive sensing
compressive measurements
sparse signal
nonzero entries
M ⇠ K log(N/K)
y = �x+ e
Signal prior Measurement matrix design
Sampling rate
Recovery algorithm
Compressive sensing
• System is under-determined • Under-sampling factor: M/N
– Smaller M/N implies gains when sensing is costly
compressive measurements
sparse signal
nonzero entries
y = �x+ e
�Single-Pixel� CS Camera
random pattern on DMD array
DMD DMD
single photon detector image
reconstruction or processing
Kelly lab
scene
Single pixel camera
• Each configuration of micro-mirrors yield ONE compressive measurement
• A single photo-
detector tuned to the wavelength of interest
• Resolution scalable
Kelly lab, Rice University
Digital micro-mirror
device
Photo-detector
First Image Acquisition
target 65536 pixels
1300 measurements (2%)
11000 measurements (16%)
Roadmap
space (3D)
spectrum (1D)
time (1D)
A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging
SPC on a time-varying scene
• Simple model for recovering video frames from a single-pixel camera (SPC): Group W measurements together to estimate each video frame (image)
• Value of W ~ “aperture time” of the video camera
t=1 t=W measurements
initial estimate
!me$varying$
scene$
SPC Video Uncertainty Principle
Small%W%Less$mo!on$blur$
More$spa!al$blur$
$
Large%W%More$mo!on$blur$
Less$spa!al$blur$
$
t=1 t=W (small) t=W (large)
[Wakin, 2010]
W"
total%M
SE%
SPC Video Uncertainty Principle t=1 t=W (small) t=W (large)
Find$and$exploit$$
op/mal%balance%%between$spa!al$$
and$mo!on$blurs$
CS-MUVI (CS MUltiscale VIdeo recovery)
• Step 1: Given knowledge of the speed of the scene’s motion, choose W to optimize the spatial/temporal resolution tradeoff
measurements
W"total$MSE$
[AS,$Studer,$Baraniuk$2012]$
t=T t=1 t=t0 t=t0+W t=W
CS-MUVI (CS MUltiscale VIdeo recovery)
• Step 1: Choose W to optimize the spatial/temporal resolution tradeoff
• Step 2: Acquire full-rate measurements using “low-pass” codes matched to blur induced by W W"
total$MSE$
measurements t=T t=1 t=t0 t=t0+W t=W
CS-MUVI (CS MUltiscale VIdeo recovery)
• Step 2: Acquire full-rate measurements using low-pass codes matched to blur induced by W
• Step 3: Recover low-resolution video frames from low-pass measurements (can use simple least squares)
b1 bj bF
measurements
low-resolution estimates
t=T t=1 t=t0 t=t0+W t=W
CS-MUVI (CS MUltiscale VIdeo recovery)
b1 bj bF
measurements
low-resolution estimates
t=T t=1 t=t0 t=t0+W t=W
• Step 3: Recover low-resolution video frames from low-pass measurements (can use simple least squares)
• Step 4: Estimate the optical flow between low-resolution frames
CS-MUVI (CS MUltiscale VIdeo recovery)
b1 bj bF
measurements
low-resolution estimates
t=T t=1 t=t0 t=t0+W t=W
• Step 5: Acquire 2nd set of interleaved measurements using “full-resolution” codes
• Step 6: Recover high-resolution video frames from 2nd set of measurements + low-res video frames, low-res optical flow, sparsity
Multiscale Sensing Codes
• Problem: 2x measurement rate (low-res and high-res measurement codes)
• Solution: Design full-res codes that become optimally matched low-res codes when downsampled in space (automatically interleaved)
• Low-res: Hadamard code
• High-res: Upsample Hadamard code using random innovations
Multiscale Sensing Codes
AS, Studer, Baraniuk (12)
1. Start with a row of the Hadamard
matrix
2. Upsample$3. Add
high-freq component
Key Idea: Constructing measurement matrices that have good properties when downsampled
Result
CS-MUVI on SPC
Single pixel camera setup
CS-MUVI: IR spectrum
Joint work with Xu, Studer, Kelly, Baraniuk
InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M
Recovered Video
Preview (initial estimate) Upsampled 4x
CS-MUVI on SPC
Conventional CS-based recovery
CS-MUVI Joint work with Xu, Studer, Kelly, Baraniuk
More results • Number of compressive measurements: 65536 • Total duration of data acquisition: 6 seconds
• Reconstructed video resolution: 128x128x256
Preview (6 different videos) *animated gifs*
CS-MUVI (6 different videos) *animated gifs*
More results • Number of compressive measurements: 65536 • Total duration of data acquisition: 6 seconds
• Reconstructed video resolution: 128x128x256
Preview (6 different videos) *animated gifs*
CS-MUVI (6 different videos) *animated gifs*
Static Image Full res.
Static Image 2x down.
Static Image 4x down.
CS-MUVI Preview
CS-MUVI 64x Comp.
Achievable Spatial resolution • Comparing a recovered frame to a static image of the
scene
CS-MUVI summary
• Key points – Signal prior: Motion model – Measurement matrix: Multi-
scale design
• A practical video recovery
algorithm for the SPC – Scales across wavelengths – Extensions to hyper-spectral
video camera
• Coming soon … to a camera near you
Roadmap
space (3D)
spectrum (1D)
time (1D)
A brief tour of compressive sensing Video sensing beyond the visible spectrum Hyper-spectral imaging
Hyper-spectral data
• Hyper-spectral data provides fingerprints for materials
• Wide range of applications – Agriculture, mineralogy – Microscopy – Surveillance, chemical
imaging
Hyper-spectral image of the Deepwater Horizon oil spill
(Image courtesy SpecTIR.com)
Hyper-spectral data
Signal models for HS data
• Low rank
– Rank depends on number of materials in the scene
– Spectral unmixing problem
• Sparsity – be the image at wavelength i – Sparse in a wavelet basis xi
[x1, x2, . . . , xL]
Low rank (subspace models)
• Individual spectral images lie close to a subspace
• Concise geometric structure: d-dim. space
Subspace
spanned by C
xi = C↵i
C 2 RN⇥d ↵i 2 Rd
d ⌧ N
C
Recovery problem
Given Recover Basis subspace coeff.
Challenges: Bilinearity of unknowns Sparsity of
C↵1,↵2, . . . ,↵T
yi = �ixi + ei
= �iC↵i + ei
C
Given
Structured measurement matrix
Recovery problem
yi = �ixi + ei
= �iC↵i + ei
[y1, y2, . . . yL] = �C [↵1, ↵2, . . . ↵L] + E
Singular value decomposition of [y1, y2, . . . yL] = U⌃V T
yi =
yiyi
�=
��i
�C↵i + ei
yi =
yiyi
�=
��i
�C↵i + ei
Given
Structured measurement matrix 1. SVD to recover subspace coefficients 2. Solve convex program to recover C
Recovery problem
yi = �ixi + ei
= �iC↵i + ei
min kCk1 s.t. 8i, kyi � �iC↵ik2 ✏
CS - Linear dynamical systems
• Measurement matrix is structured
– Alleviates the bilinearity of unknowns – Solving a sequence of linear and convex programs – A sparse prior to estimate subspace basis C
Common measurements
Innovations measurements
Estimate coefficients
Estimate matrix C
Scene
T:x1 Φ
tΦ~
T:y1
T:y~1
T:ˆ 1α
C
tt ˆCx α=T:x1
AS, Turaga, Chellappa and Baraniuk (10)(12)
Hyper-spectral imaging
• Weather data – 2300 Spectral bands in
mid-wave to long-wave IR (3.74 – 15.4 microns)
– Spatial resolution 128 x 64 – Rank 5 Ground
Truth
2% 200x %1%%
M/N = 10% M/N = 2% M/N = 1%
(rank = 20)
Broader applications • Low-rank models are
widely applicable
• Video MRI
Ground truth M/N = 2%
Broader applications • Low-rank models are
widely applicable
• Video MRI
• Classification on compressive data – Purposive sensing
Traffic video
Ground truth Recovered video M/N = 4%
Classification performance in [%] (CS-LDS at 4% under-sampling)
AS(et(al.((10)(
Expt%1% Expt%2% Expt%3% Expt%4%
Oracle$LDS$ 85.71$ 85.93$ 87.5$ 92.06$
CSOLDS$ 84.12$ 87.5$ 89.06$ 85.91$
Summary
• Visual signals are very high-dimensional – Interplay between concise signal models and novel
imaging architectures
• Signal models beyond sparsity – Rich models from computer vision and image processing – Not always easy to bridge the gap – Measurement matrices beyond random iid
Collaborators$
Richard$Baraniuk$
Kevin$Kelly$ Christoph$Studer$ Lina$Xu$ Yun$Li$Pavan$Turaga$
Rama$Chellappa$
References$
Park and Wakin, “A multi-scale framework for compressive sensing of videos,” PCS 2009 Reddy, Veeraraghavan, Chellappa, “P2C2: Programmable pixel compressive cameras for high speed imaging,” CVPR 2011 Sankaranarayanan, Studer, Baraniuk, “CS-MUVI: Video compressive sensing for spatial multiplexing cameras,” ICCP 2012 Sankaranarayanan, Turaga, Chellappa, Baraniuk, “Compressive sensing of dynamic scenes,” SIIMS (under review)
Low resolution estimate t=T t=1 t=t0 t=t0+W t=W measurements
initial estimate
• Initial estimate is called the preview • Fast to compute!!!
• Linear estimation • Fast transform
• Initial estimate is low-resolution
CS-MUVI (CS MUltiscale VIdeo recovery)
• Estimate the optical flow between
low-resolution frames
• Recovery of video at HIGH resolution with sparse models and optical-flow constraints
motion compensation
t=T t=1 t=t0 t=t0+W t=W measurements
initial estimate
CS-MUVI (CS MUltiscale VIdeo recovery)
motion compensation
t=T t=1 t=t0 t=t0+W t=W measurements
initial estimate
minPF
i=1 k xik1s.t
kyt � �txtk ✏
OF constraints between xt and xt�1