A Three-DimensionalRepresentation method for

Noisy Point Clouds based onGrowing Self-Organizing Maps

accelerated on GPUs

Author:

Supervisors:

Sergio Orts Escolano

Dr. José García Rodríguez

Dr. Miguel Ángel Cazorla Quevedo

Doctoral programme in technologies for the information society

2/79

Outline

Introduction

3D Representation using Growing Self-Organizing Maps

Improving keypoint detection from noisy 3Dobservations

GPGPU parallel implementations

Applications

Conclusions

3/79

Index Introduction

• Motivation

• Framework

• Goals

• Proposal

3D representation using growing self-organizing maps

Improving keypoint detection from noisy 3D observations

GPGPU parallel Implementations

Applications

Conclusions

4/79

Motivation Most computer vision problems require the use of an

effective way of representation

• Graphs, regions of interest (ROI), B-Splines, Octrees, histograms, …

• Key step for later processing stages: feature extraction, feature matching, classification, keypoint detection, …

Introduction Motivation

5/79

Motivation (II) 3D data captured from the real world

• Implicitly is comprised of complex structures and nonlinear models

The advent of low cost 3D sensors

• E.g. Microsoft Kinect, Asus Xtion, PrimeSense Carmine, …

• RGB and Depth (RGB-D) streams (25 frames per second (fps))

• High levels of noise and outliers

Only few works in 3D computer vision deal with real-time constraints

• 3D data processing algorithms are computationally expensive

Finding a 3D model with different features:

• Rapid adaptation

• Good quality of representation (topology preservation)

• Flexibility (non-stationary data)

• Robust to noisy data and outliers

Introduction Motivation

6/79

Motivation (III) 3D models of objects and scenes have been extensively used in

computer graphics

• Suitable structure for rendering and display

• Common graphics representations include quadric surfaces [Gotardo et al., 2004], B-spline surfaces [Gregorski et al., 2000], and subdivision surfaces

• Not general enough to handle such a variety of features: flexibility, adaption, noise-aware, … that are present in computer vision problems

(Left) Point cloud captured from a manufactured object (builder helmet). (Right) 3D mesh generated from the captured point cloud (post-processed)

Introduction Motivation

7/79

Framework Regional research project (GV/2011/034)

• Title: “Visual surveillance systems for the identification andcharacterization of anomalous behaviour” Project financed by theValencian Government in Spain

Regional research project (GRE09-16)

• Title: “Visual surveillance systems for the identification andcharacterization of anomalous behaviour in restricted environmentsunder temporal constraints” Project financed by the University ofAlicante in Spain

National research project (DPI2009-07144)

• Title: “Cooperative Simultaneous Localization and Mapping (SLAM) inlarge scale environments”

Research stay at IPAB – University of Edinburgh (BEFPI/2012/056)

• Title: “Real-time 3D feature estimation and keypoint detection ofscenes using GPGPUs”

Introduction Framework

8/79

Goals Proposal and validation of a 3D representation

model and a data fitting algorithm for noisy point clouds

• Deals with noisy data and outliers

• Flexible

• Dynamic (non stationary data)

• Topology preserving

An accelerated hardware implementation of the proposed technique

• Considerable speed-up regard CPU implementations

• Real-time frame rates

Introduction Goals

9/79

Goals (II) Validation of the proposed method on different

real computer vision problems handling 3D data:

• Robot vision: 6DoF Egomotion

• 3D Object recognition

• Computed-aided design/manufacturing (CAD/CAM)

Integration of 3D data processing algorithms in complex computer vision systems

• Filtering, downsampling, normal estimation, feature extraction, keypoint detection, matching, …

• The use of a GPU as a general purpose processor

Introduction Goals

10/79

Proposal Growing Self-Organizing Maps (GSOM) for 3D

data representation

• Low cost 3D sensors: noisy data

• Time-constrained conditions

• Applications: 3D computer vision problems

Hardware-based implementation of the proposed GSOM method

• General Purpose computing on Graphics Processing Units (GPGPU) paradigm

Integrate the entire pipeline of 3D computer vision systems using GPGPU paradigm

Introduction Proposal

11/79

Index Introduction

3D Representation using Growing Self-Organizing Maps

• Review

• 3D Growing Neural Gas network

• Experiments: Input space adaptation & normal estimation

• Extensions of the GNG algorithm

Improving keypoint detection from noisy 3D observations

GPGPU Parallel Implementations

Applications

Conclusions

12/79

Review SOMs were originally proposed for data clustering and pattern

recognition purposes [Kohonen, 1982, Vesanto and Alhoniemi, 2000, Dittenbach et al., 2001]

As the original model had some drawbacks due to the pre-established topology of the network, growing approaches were proposed in order to deal with this problem

Growing Neural Gas network has been successfully applied to the representation of 2D shapes in many computer vision problems [Stergiopoulou and Papamarkos, 2006, García-Rodríguez et al., 2010, Baena et al., 2013]

Already exist approaches that use traditional SOMs for 3D data representation: [Yu, 1999, Junior et al., 2004]

• Difficulties to correctly approximate concave structures

• High computational cost

• Synthetic data

3D Representation GSOM Review

13/79

Review (II) Moreover, there exist some limitations and unexplored

topics in the application of SOM-based methods to 3Drepresentation:

• Majority of these works do not consider the high computational cost of the learning step

• Do not guarantee response within strict time constraints

• Assumed perfect point clouds that were noise-free

• Data fusion (Geometric information + colour information) has not been considered

• Not dealing with point cloud sequences, only single-shot data

3D Representation GSOM Review

14/79

3D Growing Neural Gas Network Obtaining a reduced and compact representation of 3D data

• Self Organizing Maps – Growing Neural Gas

Growing Neural Gas Algorithm (GNG) [Fritzke, 1995]

• Incremental training algorithm

• Links between the units in the network are established through Competitive Hebbian Learning (CHL)

• Topology Preserving Graph

• Flexibility, growth, rapid adaption and good quality of representation.

GNG representation is comprised of nodes (neurons) and connections (edges)

• Wire-frame model

Initial, intermediate and final states of the GNG learning algorithm

3D Representation GSOM GNG network

15/79Highly Parallelizable

GNG algorithm Input data is defined in ℝ𝒅

• For 3D representation d=3

Adaption: Reconfiguration module

• Random patterns are presented to the network

Growth: It starts with two neurons and new neurons are inserted

Flexibility: neurons and connections may be removed during the learning stage

This process is repeated until an ending condition is fulfilled:

• Number of neurons/patterns

• Adaptation error threshold

3D Representation GSOM GNG algorithm

16/79

Experiments: Data acquisition Algorithm independent of the data source

We managed 3D data that can come from different sensors

• Laser unit, a LMS-200 Sick mounted on a sweeping unit.

o Outdoor environments. Its range is 80 metres with an error of 1millimetre per metre

• Time-of-Flight camera

o SR4000 camera. It has a range of 5-10 metres

o The accuracy varies depending on the characteristics of the observed scene, such as objects reflectivity and ambient lighting conditions

o Generation of point clouds during real time acquisition

• Range camera: structured light, Microsot Kinect device

o RGB-D information. Indoor environments. Its range is from 0.8 to 6metres

o Generation of point clouds during real time acquisition

3D Representation GSOM Experiments: Data

17/79

Experiments: Data acquisition

Mobile robots used for experiments.Left: Magellan Pro unit usedfor indoors. Right: PowerBot used for outdoors.

3D sensors used for experiments. From left to right: Sick laser unitLMS-200, Time-Of-Flight SR4000 camera and Microsoft Kinect

3D Representation GSOM Experiments: Data

18/79

Experiments: Data Sets

Some public data sets have been used to validate the proposed method:

• Well known Stanford 3D scanning repository. It contains complete models that have been previously processed (noise free)

• Dataset captured using the Kinect sensor. Released by the Computer Vision Laboratory of the University of Bologna [Tombari et al., 2010a]

• Own dataset obtained using three previously mentioned 3D sensors

3D Representation GSOM Experiments: Data

19/79

Blensor software: It allowed us to generate synthetic scenes and to obtain partial views of the generated scene as if a Kinect device was used

• It provided us with ground truth information for experiments

Experiments: Data Sets

3D Representation GSOM Experiments: Data

Simulated scene Simulated scene + Gaussian noise

20/79

Experiments

GNG method has been applied to 3D data representation

• Input space adaptation

• Noise removal properties

Extensions of the GNG based algorithm

• Colour-GNG

• Sequences management

• 3D Surface reconstruction

3D Representation GSOM Experiments

21/79

Experiments: GNG 3D representation

Applying GNG to Kinect data Applying GNG to SR4000 data

Applying GNG to laser data

3D Representation GSOM Exp: GNG 3D representation

22/79

Applying GNG to Kinect data

3D Representation GSOM Exp: GNG 3D representation

Experiments: GNG 3D representation

23/79

GNG method obtains better adaptation to the input space than other filtering methods like Voxel Grid technique

• Obtains lower adaptation (Mean Squared Error (MSE) )

• Tested on CAD models and simulated scenes

Input space adaptation MSE for different models (metres). Voxel Grid versusGNG. Numbers in bold provide the best results.

3D Representation GSOM Exp: Input space Adaptation

Experiments: Input space adaptation

Lower error

24/79

Noisy model σ = 0.4 Original CAD model

GNG representation Voxel grid representation

Filtering quality using 10,000 nodes. GNG vs Voxel Grid comparison

3D Representation GSOM Exp: Input space Adaptation

Experiments: Input space adaptation (II)

25/79

Normal estimation over noisy input data

Surface normals are important properties of a geometric surface, and are heavily used in many areas such as computer vision and computer graphics

Normal or curvature estimation can be affected by the presence of noise

Representation obtained using the GNG method allows to compute more accurate normal information

3D Representation GSOM Exp: Normal estimation

Experiments: Normal estimation

26/79

Experiments: Normal estimation (II)

Top:. Normal estimation on a filtered point cloud produced by theGNG method. Bottom: Normal estimation on a raw point cloud.

Normals are considered more stable as their distribution is smooth and also they have less abrupt changes in their directions

3D Representation GSOM Exp: Input space Adaptation

27/79

Extension: Colour-GNG

Modern 3D sensors provide colour information (e.g. Kinect, Carmine, Asus Xtion, … )

GNG is extended considering colour information during the learning step

• Input data is defined in ℝ𝒅 where d = 6

• Colour information is considered during the weight adaptation step but it was not included in the CHL (winning neurons) process

o We are still focus on topology preservation

• Winning neuron step only compute Euclidean distance using x,y,z components

• No post-processing steps are required as neurons’ colour is obtained during the learning process

3D Representation GSOM Extensions: Colour-GNG

28/79

Extension: Colour-GNG (II)

(a),(b),(c) show original point clouds. (d),(e),(f) show downsampled pointclouds using the proposed method

3D Representation GSOM Extensions: Colour-GNG

29/79

Extension: Colour-GNG (III)

Mario figure is down-sampled using the Colour-GNG method

Results are similar to those obtained with the colour interpolation post-processing step

3D Representation GSOM Extensions: Colour-GNG

30/79

Extension: Sequences management

An improved workflow to manage point cloud sequences using the GNG algorithm

Extension of the GNG for processing sequences of point clouds

It is not required to restartthe learning

It provides a speed-up in the runtime as neurons are kept between point clouds

This extension was applied in a mobile robotics application

3D Representation GSOM Extensions: Sequences

31/79

Extension: 3D Surface Reconstruction

Three-dimensional surface reconstruction is not considered in the original GNG algorithm as it only generates wire-frame models

[Holdstein and Fischer, 2008, Do Rego et al., 2010, Barhak, 2002] have already considered the creation of 3D triangular faces modifying the original GNG algorithm

• Post-processing steps are required for avoid gaps and holes in the final mesh

We extended the CHL developing a method able to produce full 3D meshes during the learning stage

• No post-processing steps are required

• A new learning scheme was developed

3D Representation GSOM Extension: 3D Reconstruction

32/79

Extension: 3D Surface Reconstruction (II)

Considered situations for edge and face creationduring the extended CHL

Avoid non-manifold andoverlapping edges

• More than 2 neighbours

• it is checked if the face tobe created already exist

A face is created whenever the already existing edges or the new ones form a triangle

The neuron insertion process was also modified

3D Representation GSOM Extension: 3D Reconstruction

33/79

Edge removal constraint based on the Tales sphere

Left: The triangle formed by these 3neurons is close to a right triangle

Right: The edge connecting s1 and ni

is removed as the angle formed

Face creation during the insertion of new neurons

Left: neuron insertion between the neuron q with highest error and its neighbour f with highest error.

Right: four new triangles and two edges are created considering r, qand f.

3D Representation GSOM Extension: 3D Reconstruction

Extension: 3D Surface Reconstruction (III)

34/79

Extension: 3D Surface Reconstruction (IV)

Post-processing steps were avoided causing gaps and holes inthe final 3D reconstructed models

3D Representation GSOM Extension: 3D Reconstruction

Different views of reconstructed models using an existing GNG-based method [Do Rego et al., 2010] for surface reconstruction

35/79

Extension: 3D Surface Reconstruction (V)

Reconstructed models using our extended GNG method for facereconstruction and without applying post-processing steps

3D Representation GSOM Extension: 3D Reconstruction

36/79

Extension: 3D Surface Reconstruction (VI)

Top: 3D model of a person(Kinect sensor).

Bottom: digitized foot. (footdigitizer)

Left: Noisy point clouds captured using the Kinect sensor.

Right: 3D reconstruction using the proposed method.

3D Representation GSOM Extension: 3D Reconstruction

37/79

Index Introduction

3D Representation using Growing Self-Organizing Maps

Improving keypoint detection from noisy 3D observations

• Review

• Improving keypoint detection

• Correspondences matching

• Results

GPGPU Parallel Implementations

Applications

Conclusions

38/79

Review

General System Overview

Motivation: dealing with noisydata obtained from 3D sensorsas the Microsoft Kinect or lasers

Result: Improving 3Dkeypoint detection and therefore registrationproblem

Filtering and down-sampling have become essential steps in3D data processing

We propose the use of the GNG algorithm for downsamplingand filtering 3D data

Beneficial attributes will be demonstrated through the 3Dregistration problem

Improving Keypoint detection Review

39/79

Review (II)Registration: Aligning various 3D point cloud data views into a complete model

Pairwise matching

Improving Keypoint detection Review

40/79

3D Keypoint detection Applying keypoint detection algorithms to filtered point

clouds

State-of-the-art 3D keypoint detectors

• Different techniques are used to test and measure theimprovement achieved using GNG method to filterand downsample input data

Improving Keypoint detection Keypoint detection

41/79

3D Keypoint detection (II)

3D Keypoint detectors

• SIFT3D: using depth as the intensity value in theoriginal SIFT algorithm

• Harris3D: use surface normals of 3D points

• Tomasi3D: performs eigenvalue decomposition overcovariance matrix

• Noble3D: evalutes the ratio between the determinantand the trace of the covariance matrix

Improving Keypoint detection Keypoint detection

42/79

3D Feature descriptors

Feature descriptors are calculated over detected keypointsto perform feature matching

• FPH and FPFH: based on an histogram of the differences of angles between the normals of the neighbour points

• SHOT and CSHOT: a spherical grid centered on the point divides the neighbourhood so that in each grid bin a weighted histogram of normals is obtained

FPFH CSHOT

Improving Keypoint detection Feature descriptors

43/79

Feature matching (II) Correspondences between keypoints are validated

through RANSAC algorithm, rejecting thoseinconsistent correspondences

Improving Keypoint detection Feature matching

44/79

Results: Feature matching

Correspondences matching computed on different input data

Top: raw point clouds

Middle: reduced representation using the GNG (20, 000 neurons)

Bottom: reduced representation using the GNG (10, 000 neurons)

RANSAC is used to reject wrong matches

Improving Keypoint detection Results

Raw 3D data

GNG 20,000 nodes

GNG 10,000 nodes

45/79

Results: Transformation errors

Mean, median, minimum and maximum RMS*2 errors of the estimated transformations using different keypoint detectors. (metres).

Lowest transformation errorLowest max errors

*2 Root Mean Square - Transformation error

Improving Keypoint detection Results

*1 Uniform Sampling

*1

46/79

Index Introduction

3D Representation using Growing Self-Organizing Maps

Improving keypoint detection from noisy 3D observations

GPGPU Parallel Implementations

• Graphics Processing Unit

• GPU-based implementation of the GNG algorithm

• GPU-based tensor extraction algorithm

• Conclusions

Applications

Conclusions

47/79

Graphics Processing Unit

GPUs have democratized High Performance Computing (HPC)• Massively parallel processors on a commodity PC

• Great ratio FLOP/€ compared with other solutions

However, this is not for free• New programming model

• Algorithms need to be re-thought and re-implemented

Growing Neural Gas algorithm is computationally expensive• Most computer vision applications are time-constrained

• A GPGPU implementation is proposed

GPGPU Implementations GPUs

48/79

Graphics Processing Unit (II) More transistors for data processing

GPU are comprised of streaming multi-processors

High GPU Memory bandwidth

GPGPU: General Purpose computing on Graphics Processors Units

Key hardware feature is that the cores are SIMT

• Single instruction multiple threads

G80CUDA NVIDIAs Architecture

GPGPU Implementations GPUs

49/79

GPU Implementation GNG Stages of the GNG algorithm

that are highly parallelizable

• Calculate distance to neurons for every pattern

• Search winning neurons

• Delete neurons and edges

• Search neuron with max error

Other improvements

• Avoid memory transfers between CPU and GPU

• Hierarchy of memories

Highly parallelizable stages

GPGPU Implementations GPU Implementation GNG

50/79

Parallel Min/Max Reduction A parallel Min/Max reduction

that computes the Min/Max of large arrays of values (Neurons)

Strategy used to find Min/Max winning neurons

Reduce linear computational cost n of the sequential version to the logarithmic cost log(n)

Provides better performance for a large number of neurons

Proposed version: 2MinParallelReduction

• Extended version to obtain 2minimum values in the same number of steps

Example of Parallel Reduction Algorithm

GPGPU Implementations GPU Implementation GNG

51/79

Experimental setup Main GNG Parameters:

• ~0-20,000 neurons and a maximum λ (entries per iteration) of 1,000-2,000

• Others parameters have been fixed based on previous works [García-Rodriguez et al., 2012]

o єw = 0.1 , єn = 0.001

o amax = 250, α = 0.5 , β = 0.0005

Hardware

• GPUs: CUDA capable devices used in experiments

• CPU: single thread and multiple thread implementations were tested

o Intel Core i3 540 3.07Ghz

GPGPU Implementations Experimental Setup

52/79

Experiments: 2MinParallelReduction

Runtime and speed-up using CPU and GPU implementations

GPGPU Implementations Exp: 2MinParallelReduction

53/79

Experiments: GNG learning runtime

GPU and CPU GNG runtime, and speed-up for different devices

GPGPU Implementations Exp: GNG Runtime

54/79

Experiments: Hybrid version

Example of CPU and Hybrid GNG runtime for different devices

CPU implementation was faster for small network sizes

• We developed an hybrid implementation

• GPU version automatically starts computing when it is detected that computing time is lower than the one obtained by the CPU

GPGPU Implementations Exp: Hybrid version

55/79

GPU-based Tensor extraction algorithm

Time-constrained 3D feature extraction

• Most feature descriptors cannot be computed online due to their high computational complexity

o 3D Tensor - [Mian et al., 2006b]

o Geometric Histogram - [Hetzel et al., 2001]

o Spin Images - [Andrew Johnson, 1997]

An accelerated GPU-based implementation of an existing 3D feature extraction algorithm is proposed

• Accelerate entire pipeline of RGB-D based computer vision systems

• Highly parallelizable• Geometrical properties• Invariant to linear

transformations

GPGPU Implementations GPU Feature extraction

56/79

GPU-based Tensor extraction algorithm (II)

The surface area of the mesh intersecting each bin of the grid is the value of the tensor element

As many threads as voxels are launched in parallel where each GPU thread represent a voxel (bin) of the grid

Each thread computes the area of intersection between the mesh and its corresponding voxel using Sutherland Hodgman’s polygon clipping algorithm. [Foley et al., 1990]

GPGPU Implementations GPU Feature extraction

57/79

Experiments: Performance

Runtime comparison and speed-up obtained for proposed methods using different graphics boards

GPGPU Implementations Exp: performance

58/79

Index Introduction

3D Representation using Growing Self-Organizing Maps

Improving keypoint detection from noisy 3D observations

GPGPU Parallel Implementations

Applications

• Robotics

• Computer Vision

• CAD/CAM

Conclusions

59/79

Applications

Different cases of study where the GNG-based method proposed in this PhD thesis was applied to different areas

• Robot Vision

o 6DoF Pose Registration

• Computer Vision

o 3D object recognition under cluttered conditions

• CAD/CAM

o Rapid Prototyping in Shoe Last Manufacturing

Applications Exp: performance

60/79

Robotics: 6DoF pose registration

The main goal of this application is to perform six degrees of freedom (6DoF) pose registration in semi-structured environments

• Man-made indoor and outdoor environments

We combined our accelerated GNG-based algorithm with the method proposed in [Viejo and Cazorla, 2013]

• Planar patches extraction

It provides a good starting point for Simultaneous Location and Mapping (SLAM)

GNG was applied directly to raw 3D data

Applications Robotics

61/79

Robotics: 6DoF pose registration (II)

Left: planar patches extracted from SR4000 camera

Right: filtered data using the GNG network: more planar patches areextracted

Applications Robotics

Without GNG GNG

62/79

Robot trajectory

Robotics: 6DoF pose registration (III)

Planar based 6DoF pose registration results

Left image shows map building results without using GNG while theresults shown on the Right are obtained after computing a GNG mesh

Applications Robotics

Without GNG GNG

63/79

3D Object Recognition

The main goal of this application is the recognition of objects under time constraints and cluttered conditions

The GPU-based of the semi-local surface feature (tensor)is successfully used to recognize objects in cluttered scenes

A library of models is constructed offline, storing all extracted 3D tensors in an efficient way using a hash table

• Multiple views

Applications 3D Object recognition

64/79

3D Object Recognition (II)

Object recognition is performed on sceneswith different level of occlusion

Objects are occluded by other objects stored and non-stored in the library

The averaged recognition rate was 84%, wrong matches 16% and false negatives0%

Applications 3D Object recognition

65/79

Scene 2

3D Object Recognition (III) GPU-based 3D feature implementation is successfully used in a 3D object

recognition application

Parallel matching is performed on the GPU: correlation function

Implemented prototype took around 800 ms with a GPU implementation to perform 3D object recognition of the entire scene

Applications 3D Object recognition

66/79

Rapid Prototyping in Shoe Last Manufacturing

With the advent of CAD/CAM and rapid acquisitiondevices it is possible to digitize old raised shoe lasts forreusing them in the shoe last design software

Applications CAD: Rapid Prototyping

Process to reconstruct existing shoe lasts and computing topology preservation error regard the original CAD design

67/79

Rapid Prototyping in Shoe Last Manufacturing (II)

The main goal of this research is to obtain a grid of points that isadapted to the topology of the footwear shoe last from a sequenceof sections with disorganized points acquired by sweeping anoptical laser digitizer

Typical sequence of sections of a shoe last. Noisy data obtained from the digitizer

Applications CAD: Rapid Prototyping

68/79

Rapid Prototyping in Shoe Last Manufacturing (III)

Voxel Grid versus GNG: Mean erroralong different sections of the shoe last

Applications CAD: Rapid Prototyping

69/79

Rapid Prototyping in Shoe Last Manufacturing (IV)

GNG vs VG topological preservation comparison

Applications CAD: Rapid Prototyping

3D Reconstruction GNG 3D Reconstruction VG

GNG nodes

VG nodes

Input space

70/79

Index Introduction

3D Representation using Growing Self-Organizing Maps

Improving keypoint detection from noisy 3D observations

GPGPU Parallel Implementations

Applications

Conclusions

• Contributions

• Future work

• Publications

71/79

Contributions Contributions made in the topic of research:• Proposal of a new method to create compact, reduced and

efficient 3D representations from noisy data

‐ Development of a GNG-based method capable to deal with different sensors

‐ Extension of the GNG algorithm to consider colour information

‐ Extension of the GNG algorithm for 3D surface reconstruction

‐ Sequences management

‐ Integration of the proposed method in 3D keypoint detection algorithms improving their performance

‐ GPU-based implementation to accelerate the learning process of the GNG and NG algorithms.

‐ An hybrid implementation of the GNG algorithm that takes advantage of the CPU and GPU processors

Conclusions Contributions

72/79

Contributions (II)• Integration of 3D data processing algorithms in complex

computer vision systems:

‐ Normal estimation has been ported to the GPU considerably decreasing its runtime

‐ Point cloud triangulation has been ported to the GPU accelerating its runtime

‐ A GPU time-constrained implementation of a 3D feature extraction algorithm

• Application of the proposed method in various real computer vision applications:

‐ Robotics: Localization and mapping: 6DoF pose registration

‐ Computer vision: 3D object recognition under cluttered conditionsCAD/CAM: rapid prototyping in shoe last manufacturing

Conclusions Contributions

73/79

Future work Other improvements on the GPU implementation of the

GNG algorithm:

• Using multi-GPU to manage several neural networks simultaneously

• Distributed computing

• Testing new architectures: Intel Xeon Phi [Fang et al., 2013a]

• Generating random patterns using GPU

More applications of the accelerated GNG algorithm will be studied in the future

• Clustering multi-dimensional data: Big Data

• Medical Image Reconstruction

Extension of the real-time implementation of the 3D tensor

• Visual features extracted from RGB information

• Improve implicit keypoint detector used by the 3D tensor

Conclusions Future work

74/79

Publications• 4 JCR Journal paperso “Real-time 3D semi-local surface patch extraction using GPGPU”

S. Orts-Escolano, V. Morell, J. Garcia-Rodriguez, M. Cazorla, R.B. Fisher; Journal of Real-Time Image Processing. December 2013; ISSN: 1861-8219; Impact Factor: 1.156 (JCR 2012)

o “GPGPU implementation of growing neural gas: Application to 3Dscene reconstruction”S. Orts, J. García Rodríguez, D. Viejo, M. Cazorla, V. Morell; J.Parallel Distrib. Comput. 72(10); pp: 1361-1372 (2012); ISSN: 0743-7315; ImpactFactor: 1.135 (JCR 2011)

o “3D-based reconstruction using growing neural gas landmark: application to rapid prototyping in shoe last manufacturing”A. Jimeno-Morenilla, J. García-Rodriguez, S. Orts-Escolano, M. Davia-Aracil; The International Journal of Advanced Manufacturing Technology: May 2013. Vol 69. pp: 657-668; ISSN: 0268-3768; Impact Factor: 1.205 (JCR 2012)

o “Autonomous Growing Neural Gas for applications with time constraint: Optimal parameter estimation” J. García Rodríguez, A. Angelopoulou, J. M. García Chamizo, A. Psarrou, S. Orts-Escolano, V. Morell-Giménez; Neural Networks 32: pp: 196-208 (2012), ISSN: 0893-6080; Impact Factor: 1.927 (JCR 2012)

Conclusions Publications

75/79

Publications (II)• International conferenceso “Point Light Source Estimation based on Scenes Recorded by a RGB-D

camera”B. Boom, S. Orts-Escolano, X. Ning, S. McDonagh, P. Sandilands, R.B. Fisher; British Machine Vision Conference, BMVC 2013, Bristol, UK. Rank B

o “Point Cloud Data Filtering and Downsampling using Growing Neural Gas”S. Orts-Escolano, V. Morell, J. Garcia-Rodriguez and M. Cazorla; International Joint Conference on Neural Networks, IJCNN 2013, Dallas, Texas. Rank A

o “Natural User Interfaces in Volume Visualisation Using Microsoft Kinect”A. Angelopoulou, J. García Rodríguez, A.Psarrou, M. Mentzelopoulos, B. Reddy, S. Orts-Escolano, J.A. Serra. International Conference on Image Analysis and Processing, ICIAP2013, Naples, Italy: 11-19. Rank B

o “Improving Drug Discovery using a neural networks based parallel scoring functions”H. Perez-Sanchez, G. D. Guerrero, J. M. Garcia, J. Pena, J. M. Cecilia, G. Cano, S. Orts-Escolano and J. Garcia-Rodriguez. International Joint Conference on Neural Networks, IJCNN 2013, Dallas, Texas. Rank A

Conclusions Publications

76/79

Publications (III)• International conferenceso “Improving 3D Keypoint Detection from Noisy Data Using Growing

Neural Gas”J. García Rodríguez, M. Cazorla, S. Orts-Escolano, V. Morell. International Work-Conference on Artificial Neural Networks, IWANN 2013, Puerto de la Cruz, Tenerife, Spain: 480-487. Rank B

o “3D Hand Pose Estimation with Neural Networks”J. A. Serra, J. García Rodríguez, S. Orts-Escolano, J. M. García Chamizo, A. Angelopoulou, A. Psarrou, M. Mentzelopoulos, J. Montoyo-Bojo, E. Domínguez. International Work-Conference on Artificial Neural Networks, IWANN 2013, Puerto de la Cruz, Tenerife, Spain: 504-512. Rank B

o “3D Gesture Recognition with Growing Neural Gas” J. A. Serra-Perez, J. Garcia-Rodriguez, S. Orts-Escolano, J. M. Garcia-Chamizo, A. Angelopoulou, A. Psarrou, M. Mentzeopoulos, J. Montoyo Bojo. International Joint Conference on Neural Networks. IJCNN 2013, Dallas, Texas. Rank A

o “Multi-GPU based camera network system keeps privacy using Growing Neural Gas”S. Orts-Escolano, J. García Rodríguez, V. Morell, J.Azorín López, J. M. García Chamizo. International Joint Conference on Neural Networks (IJCNN) 2012, Brisbane, Australia, June: 1-8. Rank A

Conclusions Publications

77/79

Publications (IV)• International conferenceso “A study of registration techniques for 6DoF SLAM”

V. Morell, M.Cazorla, D. Viejo, S. Orts-Escolano, J. García Rodríguez. International Conference of the Catalan Association for Artificial Intelligence, CCIA 2012, University of Alacant, Spain: 111-120. Rank B

o “Fast Autonomous Growing Neural Gas”J. García Rodríguez, A. Angelopoulou, J. M. García Chamizo, A. Psarrou, S. Orts, V. Morell. International Joint Conference on Neural Networks, IJCNN 2011, San Jose, California: 725-732. Rank A

o “Fast Image Representation with GPU-Based Growing Neural Gas”J.García Rodríguez, A. Angelopoulou, V. Morell, S. Orts, A. Psarrou, J. M. García Chamizo. International Work-Conference on Artificial Neural Networks, IWANN 2011, Torremolinos-Málaga, Spain: 58-65. Rank B

o “Video and Image Processing with Self-Organizing Neural Networks”J. García Rodríguez, E. Domínguez, A. Angelopoulou, A. Psarrou, F. J. Mora-Gimeno, S. Orts, J. M. García Chamizo. International Work-Conference on Artificial Neural Networks, IWANN 2011, Torremolinos-Málaga, Spain: 98-104. Rank B

Conclusions Publications

78/79

Publications (V)• National conferenceso “Procesamiento de múltiples flujos de datos con Growing Neural Gas

sobre Multi-GPU” S. Orts-Escolano, J. García-Rodríguez, V. Morell-Giménez. Jornadas de Paralelismo JP, Elche, España, 2012

• Book chapterso “A Review of Registration Methods on Mobile Robots”

V. Morell-Gimenez, S. Orts-Escolano, J. García Rodríguez, M. Cazorla, D. Viejo. Robotic Vision: Technologies for Machine Learning and Vision Applications. IGI GLOBAL

o “Computer Vision Applications of Self-Organizing Neural Networks”J. García-Rodríguez, J. M. García-Chamizo, S. Orts-Escolano, V. Morell-Gimenez, J.Serra-Perez, A. Angelolopoulou, M. Cazorla, D. Viejo. Robotic Vision: Technologies for Machine Learning and Vision Applications. IGI GLOBAL

• Poster presentationso “6DoF pose estimation using Growing Neural Gas Network”

S.Orts, J. Garcia-Rodriguez, D. Viejo, M. Cazorla, V. Morell, J. Serra. 5th International Conference on Cognitive Systems, Cogsys 2012, TU Vienna, Austria

o “GPU Accelerated Growing Neural Gas Network”S. Orts, J. Garcia, V.Morell. Programming and Tuning Massively Parallel Systems, PUMPS 2011,Barcelona, Spain. (Honorable Mention by NVIDIA)

Conclusions Publications

79/79

This presentation is licensed under a Creative Commons Attribution-

NonCommercial-ShareAlike 4.0 International License.

A Three-DimensionalRepresentation method for

Noisy Point Clouds based onGrowing Self-Organizing Maps

accelerated on GPUs

Author:

Supervisors:

Sergio Orts Escolano

Dr. José García Rodríguez

Dr. Miguel Ángel Cazorla Quevedo

Doctoral programme in technologies for the information society