1

Exploiting Inertial Planes for

Multi-sensor 3D Data Registration

PhD dissertation

Hadi Aliakbarpour

Faculty of Science and Technology

October 2012, University of Coimbra

2

Introduction

3

Introduction: Problem Statement

This dissertation investigate the problem of multi-sensor 3D data registration using a network of IS-camera pairs.

Target applications: Surveillance, human behaviour modelling, virtual-reality, smart-room, health-care, games, teleconferencing, human-robot interaction, medical industries, and scene and object understanding

4

Introduction: Motivation

Performing 3D data registration and scene reconstruction using a set of planar images is still one of the key challenges of computer vision.

A network of cameras, whose usage and ubiquitousness have been increasing in the last decade, can provide such planar images from different views of the scene.

Recently, IS has been becoming much cheaper and more available so that nowadays most smart-phones are equipped in both IS and camera sensors. 3D earth cardinal orientation (North-East-Down) is one of the outputs of an IS.

How can we benefit from having a network of IS and camera couples, for the purpose of 3D data registration?

5

Introduction: Overall View

6

Main Contributions

A homographic framework is developed for 3D data registration using a network of cameras and inertial sensors. Geometric relations among different projective image planes and Euclidean inertial planes involved in the framework are explored. [AD12a] [AD11c] [AD10b] [AD11b] [AD10a] [AFKD10] [AFQ+11].

A real-time prototype of the framework is developed which is able to perform fully reconstruction of human body (and objects) in a large scene. The real-time characteristic is achieved by using a parallel processing architecture on a CUDA-enabled GP-GPU [AAMD11].

A two-point-based method to estimate translations among virtual cameras in the framework is proposed and verified [AD12a] [AD11a] [AD10a] [AFQ+11].

The uncertainties of the homography transformations involved in the framework and their error propagations on the image planes and Euclidean planes have been modelized using statistical geometry.

7

Additional Contributions

Within the context of the proposed framework, a genetic algorithm is developed to provide an optimal coverage of the camera network to a polygonal object (or a scene).

A method to estimate extrinsic parameters among camera and laser range finder is developed [ANP+09]. A related SLaRF; available to download at http://isr.uc.pt/~hadi is prepared.

8

Methodologies on

Main Contributions

9

Virtual Camera: Concept & Geometry

Mapping: From Scene onto Inertial Planes

10A 3D point X is registered on different Euclidean planes using homographies

11

Extension to a Network of IS-Camera Pairs

A network of IS-camera couples is used to observe the scene from different views

3D Reconstruction Using Inertial Planes: Illustrative example

13

Off-the-plane point Y is subject to parallax and on-the-plane point X with no parralax

An exemplary case: a person is observed by three cameras

Top-view of the registration plane. Area in white is the intersection to the person.

14

Translation Among Virtual Cameras

Knowing the heights of two 3D points (X1 and X2) is sufficient to recover the translation (t) among two cameras

15

Parametric Homography RelationsAmong Different Planes in the Framework

16

Real-time implementation using GP-GPU

17

Parallelized Operations: Virtual Planes Generations

18

Modelization of Uncertainties

The uncertainties of the homography transformations involved in the framework and their error propagations on the image planes and Euclidean planes have been modelized using statistical geometry.

19

Modelization of Uncertainties: examples

• Uncertainty of point μ’X , where mapped from μref to μ’ :

• Uncertainty of point μ(k)X , where mapped from μk-1 to μk :

20

Methodologies on

Additional Contributions

21

Optimization for Sensor Configuration (Camera Placement)

The quality of reconstruction using a camera network depends to mainly three parameters:

1. Number of cameras

2. The quality of the applied background subtraction technique

3. The cameras configurations (e.g. positions)

22

Optimization for Sensor Configuration: Problem Statement

C1C2

X

Y

{W} refπ

e1 e2

e3

e4

e5

An exemplary convex polygon with 5 edges are observed by two camera. The problem is how to arrange cameras to have optimum registration of the polygon with most completeness.

After registering with the present camera configuration: An extra part colored in red is registered as a part of the object!

23

Optimization for Sensor Configuration: Solution

Solution: To use geometry (e.g. normal of the edges etc. ), define some cost functions and applying GA.

24

Integration of a Laser Range Finder

11 Camera LRF

Sensitivity to Illumination

Very high NA

Occlusion handling

Weak Fair

Sensitivity to texture

High NA

Precision in range sensing

Fair Very good

Color sensing Very good NA

LRF is an active sensor which can be used as a complementary sensor to the cameras:

Comparison table

10=

31

)()()(

x

LC

LC

LC tRT

Estimation of the rigid transformation, C T L(α) , among a stereo camera and a LRF

25

Experiments

26

Volumetric 3D Reconstruction: Offline

First virtual plane

Second virtual plane

47’th virtual plane

Sta

tue

Set

up a

nd s

cene

27

Analysis on Translation Estimation: Certainties w.r.t. Noise in IS

Empirical analysis the effects of IS noise to the translation estimation method

Input noise in degrees (roll, pitch and yaw of inertial sensor)

Output uncertainty in cm (on three elements of the estimated

translation vector)

28

Output uncertainty (cm)

Input noise (cm)

Analysis on Translation Estimation: Certainties w.r.t. other Noise

Output uncertainty (cm)

Input noise (pixels)

29

Volumetric 3D Reconstruction using Real-time using GPU: video

30

Uncertainty of Virtual Image’s Points

The uncertainties for pixels of the virtual camera’s image plane are demonstrated by covariance ellipses, where they are scaled 1000 times for clarity.

31

Uncertainty of Inertial plane’s Points

The uncertainties for different registered points on the Euclidean inertial plane, demonstrated by covariance ellipses. The blue and red ellipses stand for points registered by the first and second camera, respectively. For the sake of clarity the covariance values are scaled 500 and 600 times, respectively for the first and second cameras

32

Changes of Uncertainty in Virtual Image Plane

Uncertainties for an exemplary pixel x = [ 450 450 1 ]T where s = [ π/2 -π/2 0 ]T

33

Optimization for Sensor Configuration (Camera Placement)

1200x1200 cm2

34

Estimate of Extrinsic Parameters Among Camera And Laser Range Finder

10=

31

)()()(

x

LC

LC

LC tRT

Reprojection of LRF data on the image (blue points)

+Result

Imag

eR

ange

dat

a

α = 2o

α = 12o

α = 23.2o

(during 6 months)

35

Conclusion & Future Work

36

Conclusion

• We investigated the use of IS for 3D data registration by using a network of cameras and inertial sensors.

• A volumetric data registration algorithm was proposed.

• Normally the volumetric reconstruction of a scene is time consuming due to the huge amount of data to be processed. In order to achieve a real-time processing, a prototype was built using GP-GPU and CUDA

• A method to estimate the translation among cameras within the network was proposed. The certainty of the method has been evaluated in the presence of different noise.

• The issue of sensor configuration, particularly the cameras’ positions in the scene was investigated and a geometric method to find an optimal configuration was proposed using genetic algorithm.

• A method to estimate the extrinsic parameters among camera and LRF was proposed as a step towards applying range data in the framework.

37

Future Work

Integration of range data within the proposed inertial-based data registration framework.

To develop a probabilistic algorithm for fusion of heterogeneous data, capable of dealing with the uncertainty of each sensor node.

To investigate a multi-layer 3D tracking of human/objects. In this future investigation, we will provide contribution to model and recognize the state of scene and to analyse the behavior of small group using probabilistic approaches.

38

Thank you!