Detailed 3D Reconstruction of Static Scenes from a Multi-camera SystemQIANG DUAN, COMP8755

SUPERVISOR: DR. YUCHAO DAI, DR. BEN SWIFT

THE RESEARCH SCHOOL OF COMPUTER SCIENCE

AUSTRALIAN NATIONAL UNIVERSITY

OCTOBER 2017

Introduction

* The image shown here is indicative only. The actual equipment may differ.[1] Image achieved from: https://www.bhphotovideo.com/c/buy/professional-360-video-cameras/ci/36033/N/3527143157

Reconstruct 3D scenes from 360 degree multi-camera system

Development tool:OPENCV and MATLAB

MotivationDevelop research and learning abilities for advanced technologies in computer science.

Have better understanding of multi-view geometry.

Design, implement an algorithm for 3D reconstruction of static scenes captured from multi-camera system.

1. Camera calibration

Estimates the parameters of a lens and image sensor of an image or video camera.

Three parameters:◦ Intrinsic parameter

◦Extrinsic parameter

◦Distortion coefficients

[2] image achieved from https://au.mathworks.com/help/vision/ug/camera-calibration.html

Pinhole camera model:intrinsic parameters

[2] image achieved from https://au.mathworks.com/help/vision/ug/camera-calibration.html

Extrinsic parametersdenote the coordinate system transformations from 3D world coordinates to 3D camera coordinates

[3] image achieved from: https://www.researchgate.net/figure/312376910_fig20_Figure-42-The-extrinsic-parameters-consisting-of-a-translation-vector-t-and-a-rotation

2. Image undistortRefer to OpenCV: undistort() function.

3. Feature matching using SIFT descriptor

4. Epipolar geometry and Structure-from-motion (SFM) algorithm

Epipolar constraint:1. If the projection point xL is known,

then the epipolar line eR––xR is knownand the point X projects into the rightimage, on a point xR which must lie onthis particular epipolar line

2. If the points xL and xR are known, theirprojection lines are also known. If thetwo image points correspond to thesame 3D point X the projection linesmust intersect precisely at X.

[4] image achieved from :https://en.wikipedia.org/wiki/File:Epipolar_geometry.svg

Result: Sparse 3D points

[5] triangulation process refers to MATLAB triangulate() function

5. Superpixel

[6] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels Compared to State-of-the-art Superpixel Methods, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, num. 11, p. 2274 - 2282, May 2012.

Applying SLIC algorithm. Assume every superpixel is flat and pixels located on same plane.

6. HomographyDecompose homography matrix for each superpixel to obtain the normal vector and distance of its plane

Plane equation: 𝑛𝑇 ∗ 𝑋 + 𝑑 = 0

[7] image and equations are achieved from :Dingfu ZHOU, Yuchao Dai and Hongdong Li. Reliable Scale Estimation and Correction for Monocular Visual Odometry. In Intelligent Vehicles Symposium, 2016 Proceedings (IV 2016)

Overall process

Feature matching using SIFT sparse correspondence

Triangulation sparse 3D point cloud

Apply SLIC algorithm superpixel

Calculate and decompose homography matrix normal vector and distance

Construct plane according to normal vector, matched points and distance 3D planes

Optimizing assemble and glue planes to obotain dense 3D scene

Future works

Applying to panoramic scene reconstruction

A better feature matching algorithm can help

Apply dense feature matching instead of sparse matching

Thanks!

Any Question?

Intrinsic parameters illustration