the visual microphone - korea universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf ·...
TRANSCRIPT
![Page 1: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/1.jpg)
The Visual Microphone: Passive Recovery of Sound from Video
Copyright of figures and other materials in the paper belongs original authors.
Presented by Ki-hoon Kim
2015.09.09
Computer Graphics @ Korea University
Abe Davis et al.SIGGRAPH 2014
![Page 2: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/2.jpg)
Ki-hoon Kim | 2016-01-07 | # 2Computer Graphics @ Korea University
3D Position Measurement From Image
Image Data
3D Geometry Data
![Page 3: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/3.jpg)
Ki-hoon Kim | 2016-01-07 | # 3Computer Graphics @ Korea University
Emotion Recognition
Image Data
Sound DataEmotion
Recognition
![Page 4: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/4.jpg)
Ki-hoon Kim | 2016-01-07 | # 4Computer Graphics @ Korea University
Introduction
![Page 5: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/5.jpg)
Ki-hoon Kim | 2016-01-07 | # 5Computer Graphics @ Korea University
• Recording the reflection of a laser pointed at the surface.
Record phase of a reflected laser
Recover high quality audio
Depend on precise positioning(Laser & Receiver)
Surface must be appropriate reflectance
Related Work - Laser microphone
![Page 6: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/6.jpg)
Ki-hoon Kim | 2016-01-07 | # 6Computer Graphics @ Korea University
Using high-speed camera
Record changes in the speckle pattern of laser light
Greater flexibility in the positioning of a receiver
Depend on recording reflected laser light
Related Work - Laser microphone
Heart beat Human Voice
Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern[Zalevsky et al./Opt. Express 2009.]
![Page 7: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/7.jpg)
Ki-hoon Kim | 2016-01-07 | # 7Computer Graphics @ Korea University
Related Work – Extract subtle motions
![Page 8: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/8.jpg)
Ki-hoon Kim | 2016-01-07 | # 8Computer Graphics @ Korea University
Overview
![Page 9: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/9.jpg)
Ki-hoon Kim | 2016-01-07 | # 9Computer Graphics @ Korea University
• 0. Decompose the input video into spatial sub-bands
Corresponding to different orientations and scales
• 1. Computing the Local Motion Signals(Phase Variations)
• 2. Computing the Global Motion Signal
• 3. Denoising
Recovering Sound from Video
![Page 10: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/10.jpg)
Ki-hoon Kim | 2016-01-07 | # 10Computer Graphics @ Korea University
• Complex Image can be expressed(Riesz Pyramid Transform)
𝐴 : Amplitude
𝜑 : Phase
𝜃 : Transform Orientation
𝑟 : Transform Scale
• Compute phase variations
Recovering Sound from VideoLocal Motion Signals
![Page 11: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/11.jpg)
Ki-hoon Kim | 2016-01-07 | # 11Computer Graphics @ Korea University
• Weigh each local signal by its squared amplitude
• Shift in time to align phase signals
• Global motion signal is
Recovering Sound from VideoGlobal Motion Signal
![Page 12: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/12.jpg)
Ki-hoon Kim | 2016-01-07 | # 12Computer Graphics @ Korea University
• Modeling object motion
(a,b) 300 Hz pure tone, [0.1,1] Pa
(c) 20 Hz to 2,200 Hz frequency
Object Response
![Page 13: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/13.jpg)
Ki-hoon Kim | 2016-01-07 | # 13Computer Graphics @ Korea University
• Modeling pixel motion
𝑚 : Magnification of surface in [mm/pixel]
𝜃 : Viewing angle of camera
𝜔 : Frequency
• SNR of recovered sound relationship
𝑛𝑝 : The number of pixels
𝜎𝑛 : Image Noise Standard Deviation
Processing
![Page 14: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/14.jpg)
Ki-hoon Kim | 2016-01-07 | # 14Computer Graphics @ Korea University
Experiments - Frequency Range
Input Sound
ReconstructedSound
![Page 15: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/15.jpg)
Ki-hoon Kim | 2016-01-07 | # 15Computer Graphics @ Korea University
Experiments – Recovery Comparison
• VM – Our Method, LDV – Laser Doppler Vibrometer
• Bold: Better score
• VM is comparable to LDV
• LDV is required the retro-reflective tape
![Page 16: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/16.jpg)
Ki-hoon Kim | 2016-01-07 | # 16Computer Graphics @ Korea University
Recovering Sound with Normal Video Cameras- What is Rolling Shutter?
![Page 17: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/17.jpg)
Ki-hoon Kim | 2016-01-07 | # 17Computer Graphics @ Korea University
Recovering Sound with Normal Video Cameras- How to use Rolling Shutter
• aaaa
![Page 18: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/18.jpg)
Ki-hoon Kim | 2016-01-07 | # 18Computer Graphics @ Korea University
• Assume exposure time 𝐸 ≈ 0,
𝐼𝑛 : Rest State Image
𝐵𝑛 : Image with Rolling Shutter Effect
𝑛 : Frame number
𝑇 : Frame Period
𝑑 : Line Delay
Recovering Sound with Normal Video Cameras- Equation Modeling
![Page 19: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/19.jpg)
Ki-hoon Kim | 2016-01-07 | # 19Computer Graphics @ Korea University
Recovering Sound with Normal Video Cameras- Convert to audio signal
• Missing Sample by Frame Delay
Audio interpolation
![Page 20: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/20.jpg)
Ki-hoon Kim | 2016-01-07 | # 20Computer Graphics @ Korea University
Results
![Page 21: The Visual Microphone - Korea Universitykucg.korea.ac.kr/new/seminar/2015/ppt/ppt-2015-09-09.pdf · 2016-01-07 · Computer Graphics @ Korea University Ki-hoon Kim | 2016-01-07 |](https://reader034.vdocuments.net/reader034/viewer/2022042110/5e8b001fa33ebd22225cb16b/html5/thumbnails/21.jpg)
Ki-hoon Kim | 2016-01-07 | # 21Computer Graphics @ Korea University
Q&A