Slide 1

CS-MUVI Video compressive sensing for spatial multiplexing cameras Aswin Sankaranarayanan, Christoph Studer, Richard G. Baraniuk Rice University Slide 2 Single pixel camera Digital micro-mirror device Photo-detector Slide 3 Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement Non-visible wavelengths Sensor material costly in IR/UV bands Light throughput Half the light in the scene is directed to the photo-detector Much higher SNR as compared to traditional sensors Digital micro-mirror device Photo-detector Slide 4 Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement static scene assumption Key question: Can we ignore motion in the scene ? Digital micro-mirror device Photo-detector Slide 5 SPC on a time-varying scene Nave approach: Collect W measurements together to compute an estimate of an image what happens ? t=1t=W measurements compressive recovery time varying scene Slide 6 SPC on a time-varying scene Tradeoff Temporal resolution vs. spatial resolution t=1 Small W Less motion blur Lower spatial resolution Large W Higher spatial resolution More motion blur t=W (small) t=W (large) Slide 7 SPC on a time-varying scene Lower spatial res. Higher temporal res. Higher spatial res. Lower temporal res. sweet spot Slide 8 Dealing with Motion Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] State-of-the-art video compression Slide 9 Dealing with Motion Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] State-of-the-art video compression nave reconstruction motion estimates Slide 10 Key points Motion blur and the failure of the sparsity assumption Use least squares recovery ? Recover scene at lower spatial resolution Lower dimensional problem requires lesser number of measurements Tradeoff spatial resolution for temporal resolution Least squares and random matrices Random matrices are ill-conditioned Noise amplification Hadamard matrices Orthogonal (no noise amplification) Maximum light throughput Optimal for least squares recovery [Harwit and Sloane, 1979] Slide 11 Hadamard + least sq. recovery Hadamard Random Slide 12 Hadamard + least sq. recovery Slide 13 Designing measurement matrices Hadamard matrices Higher temporal resolution Poor spatial resolution Random matrices Guarantees successful l 1 recovery Full spatial resolution Can we simultaneously have both properties in the same measurement matrix ? Slide 14 Dual-scale sensing (DSS) matrices 1. Start with a row of the Hadamard matrix 2. Upsample 3. Add high-freq. component Key Idea: Constructing high-resolution measurement matrices that have good properties when downsampled Slide 15 CS-MUVI: Algorithm outline t=T t=1 t=t 0 t=t 0 +W t=W 1. obtain measurements with DSS matrices 1. obtain measurements with DSS matrices 2. low- resolution initial estimate 3. motion estimation 4. compressive recovery with motion constraints Slide 16 Simulation result Slide 17 CS-MUVI on SPC Single pixel camera setup Object InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M Slide 18 CS-MUVI: IR spectrum Joint work with Xu and Kelly Recovered Video initial estimate Upsampled Slide 19 CS-MUVI on SPC Nave frame-to-frame recovery CS-MUVI Joint work with Xu and Kelly Slide 20 CS-MUVI summary Key ingredients Novel Measurement matrix design Exploiting state-of-the-art motion model One of first practical video recovery algorithm for the SMC dsp.rice.edu Slide 21 CS-MUVI summary Limitations Need a priori knowledge of object speed Motion at low-resolution Robustness to errors in motion estimates Future work Dual-scale to multi-scale matrix constructions Multi-frame optical flow Online recovery algorithms dsp.rice.edu