ece532 final project demo disparity map generation on a fpga using stereoscopic cameras ece532 final...

ECE532 Final Project Demo

Disparity Map Generation on a FPGA Using Stereoscopic Cameras

ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting


Depth Detection

• Depth information applied in automotive collision avoidance, XBOX Kinect, robot vision

• Input stereoscopic image pair compared to produce disparity map

• Disparity map contains depth information at each pixel location


Disparity Map to Depth

• Disparity value is (a – b)

• Given:f – focal length

d – camera distance

• Depth = z+f = fd/(a-b)


Disparity Map to Depth

• Goal:

– Produce a disparity map for depth sensing using stereoscopic cameras

– Implement in hardware:• Adaptive Window Sizing• Sum of Absolute Differences Pixel Matching• Disparity Map Generation• Output Disparity Map Filtering


Disparity Map Algorithm

Variation Window Sizing• Areas of less pixel variation receives larger

window• Areas of more pixel variations receives smaller

window• Window size information to be used for S.A.D

phase

Variation Window Sizing

S.A.D Pixel Matching

Disparity Map

Generation

Disparity Map Filter



Sum of Absolute Differences• Simple cost function for pixel matching between

reference and search frames• Other cost functions such as sum of square

differences



Disparity Map

Generation




Disparity Map Generation• Determines disparity value, pixel coordinate

difference of matching pixels in reference and search frames

• Stores in memory as disparity map



Disparity Map

Generation




Disparity Map Filter• A filter can be applied to remove outliers



Disparity Map

Generation



Final Hardware System

• Video In IP

• Disparity Map Frontend

• Disparity Map Backend

• HDMI Out IP

• MicroBlaze

Spartan 6 FPGA

disparity_map_frontend

asynch fifo (reference image)

asynch fifo (search image)

pixel conversion

pixel conversion

BRAM BRAM

disparity calculation

block

disparity_map_backend

async fifo (disparity as grayscale pixel)

address calculation for AXI burst write

Master Burst Read Interface

Microblaze softprocessor

AXI Interconnect

Memory Controller

Master Burst Write Interface

HDMI out IP

video in IP

VmodCam Reference IP

async fifo (CAM A input)

async fifo (CAM B input)

Master Burst Write Interface

DDR2 SDRAM

VModCam Stereoscopic

Camera

Monitor

Legend

Custom IP

Existing IP

Peripheral Hardware


Design Process

• MATLAB implementation

• MicroBlaze C Program

• Hardware modules divided among team members


Results

• MATLAB produced disparity map


Results

• Hardware produced disparity map


Performance Comparison

System Frame Size Time

MATLAB 100x200 ~5 minutes(>1hr for 640x480)

MicroBlaze C Program

640x480 ~20 minutes

Hardware System 640x480 2 - 8.5 seconds


What We Learned…

• Simulation is faster than synthesis

• How to use EDK/SDK Tools and ModelSim

• Communicating about interfaces between blocks is key


Demo!


Thank You!

• Questions/Comments

ece532 final project demo disparity map generation on a fpga using stereoscopic cameras ece532 final...

Documents

disparity map disparity

pixel variations

backendasync fifo disparity

axi burst writemaster

stereoscopic camerasimplement

search framesstores

depth sensing

depthdisparity value