stereo 2d to 3d on an fpga - mitweb.mit.edu/6.111/www/f2015/projects/asbiswas_project... ·...
TRANSCRIPT
Stereo — 2D to 3D on an FPGA
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas
Massachusetts Institute of Technology
baxelrod, xnie, asbiswas
November 10, 2015
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 1 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera
4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels
4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondences
Even smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
Overall System Layout
cam1
cam2
preprocessing 1
preprocessing 2 ddr buffer sgm ddr buffer rendering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 3 / 23
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 4 / 23
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 5 / 23
Why Rectification?
Intrinsic optical distortion
Improper alignment of the cameras (rotation, translation)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 6 / 23
Why Rectification?
Intrinsic optical distortion
Improper alignment of the cameras (rotation, translation)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 6 / 23
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
Rectification: Example
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 8 / 23
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 9 / 23
Gaussian Filtering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 10 / 23
Gaussian Filtering
Reduce noise in the image
Convolution: weighted sum ofsurrounding pixels
Separate horizontal and verticalpasses
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 11 / 23
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 12 / 23
Disparity
A shift to the left of an imagefeature when viewed in the rightimage
Disparity cost: associatedmatching cost between twopixels
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 13 / 23
Census Transform
Used to compute disparity matching costs
5x5 window for each pixel to represent the information from thesurroundings of the pixel
produces a bit stream
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 14 / 23
Left Image Right Image
Left Image Right Image
Depth/Disparity Map
Minimize Global Energy
Minimize Global Energy
Minimize Global Energy
D is a disparity image
Minimize Global Energy
Similarity Costs for Each Pixel
Minimize Global Energy
Small Disparity Change (Smoothness)
Minimize Global Energy
Small Disparity Change (Smoothness)
Small Penalty P1
Minimize Global Energy
Object Boundary
Minimize Global Energy
Object Boundary : Large Penalty P2
2D Global Minimization
2D Global Minimization
NPComplete Problem
1D Optimization● Dynamic Programming
1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines
1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines
1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines
Streaking Issues
Left Image Right Image
Streaking Issues
Use Multiple Directions
Use Multiple Directions
Use Multiple Directions
r varies over 8 directions
Use Multiple Directions
r varies over 8 directions
Total Cost :
Semi Global Matching
Left Image Right Image
No Streaking!!!
Streaming Design
Total Cost :
Depth Map Rendering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 1 / 3
Point Cloud Rendering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 3
Timeline
WEEKS: 0 1 2 3 4
Pre-Processing
Camera Capture
Rectification Parameters
Rectification
Census Transform
Gaussian Filtering
Pre-Processing Complete
Semi-Global Matching
CPU Testing
Streaming CPU Test
HLS First Pass
HLS Second Pass
HLS Testing
Memory Architecture
DMA
Reverse DMA
Integration
Integration/Testing
Post-Processing
Project Complete
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 3 / 3