automatic 3d reconstruction and modeling - ros.org ... · automatic 3d reconstruction and modeling...
TRANSCRIPT
![Page 1: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/1.jpg)
1
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Title Slide
� Nice Picture of 3D Models
� Nice Picture of SegwayAutomatic 3D Reconstruction
and Modeling
![Page 2: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/2.jpg)
2
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Spot the difference!
![Page 3: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/3.jpg)
3
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
3D Model inspection
![Page 4: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/4.jpg)
4
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Goal and Motivation
� Growing demand for digitizing indoor and outdoor environments
� Applications
� 3D content generation for
� mapping service applications, e.g. google maps, bing maps
� video games based on real locations
� interactive museum exhibits
� Pre-visualization and virtual sets for the film industry
� Virtual Reality Systems
� Realistic simulators for robotic algorithms (e.g. ROS/gazebo)
� Goal: autonomous system for efficient creation of photo
realistic 3D models of indoor and outdoor environments
![Page 5: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/5.jpg)
5
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Overview
Data Acquisition
Registration
SurfaceReconstruction
TextureReconstruction
![Page 6: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/6.jpg)
6
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
3D Object Representations
� Raw Data
� Point Cloud
� Rang Image
� Polygon Soup
� Surfaces
� Meshes
� Parametric
� Implicit
� Solids
� Voxels
� BSP Trees
� CSG
� High Level Structures
� Scene Graphs
� Skeleton
� etc.
![Page 7: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/7.jpg)
7
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Data Acquisition (1)
� Robot hardware including range and imaging sensors
� Frontier based exploration for active environment exploration
� 2D mapping and localization (Grisetti et. al, 2005)
prev. 3D scans
next scan
trajectory
frontier
Current location
ScanningLidar SLR Camera
Pan-tilt unit
ExplorationLidar
Bosch’s 3D Mapping Robot Willow Garage’s PR2
5MP Camera
Tilting Lidar
Base Lidar
Stereo Cameras
![Page 8: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/8.jpg)
8
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Data Acquisition (2)
![Page 9: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/9.jpg)
9
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Data Acquisition (3)
![Page 10: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/10.jpg)
10
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Pairwise Registration
� Iterative Closest Point (Besl & McKay ’92)
� Optimization loop
� Finding point correspondences
(distance, normal, color)
� Minimizing error metric
� Apply transform and iterate
� Fast convergence
� Easy to implement
� Errors accumulate for multiple scans
![Page 11: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/11.jpg)
11
� SLAM: estimating the robot’s pose and
a map at the same time.
� Full SLAM posterior:
� Finding the most probable solution:
� Maximum a-posteriori estimation!
Measurement ModelMotion ModelMap Prior
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Global Registration
A vector of features representing the environment.
The robot’s position and orientation.
A feature observation.
The control vector applied a time t-1.
Motion links (wheel odometry)
Measurement links (scan matching)
![Page 12: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/12.jpg)
12
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Global Registration – Example
2D Floorplan 3D Map
![Page 13: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/13.jpg)
13
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Global Registration with Map Prior
� Full SLAM posterior:
� Use a local surface model to
approximate global map prior:
Motion Model Sensor ModelMap Prior
∏ ⋅⋅⋅⋅= −
i
iiiiiimxzpuxxpmpxpzuxmp ),|(),|()()(),|,(
10η
� Intuition: observation of surfaces belong to smooth manifolds
� Result: Probabilistic Non-rigid Registration (Pitzer et al. ICRA 2010)
![Page 14: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/14.jpg)
14
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Global Registration with Map Prior – Example
Registration without map priors Registration with map priors
![Page 15: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/15.jpg)
15
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Surface Reconstruction
![Page 16: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/16.jpg)
16
� Gradient of the smoothed indicator function is equal to the
smoothed surface normal field (Kazhdan et al., 2006)
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Volumetric Surface Reconstruction
χoriented points indicator function iso-surface
χ∇indicator gradient
� Surface reconstruction using a indicator function
� Formulation as a Poisson problem:
� Bilateral filter as smoothing function.
Point Cloud
Surface Model
![Page 17: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/17.jpg)
17
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Appearance Reconstruction
� Reconstruction of material properties (e.g. color, specularity, temperature)
� Inference of material types (e.g. wood, plastic, vegetation)
� Enhancing visual impression by adding more realism
Surface Model Textured Surface Model
![Page 18: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/18.jpg)
18
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Color Reconstruction (1)
1. Surface partitioning
� Surface segmentation into nearly planar regions
� Merging of similar regions using adjacency graphs
2. Unfolding
� Mapping of surface into texture domain
� Finding a conformal mapping which
preserves the angular distortion
� Minimizing the complex conformal
energy (Levy et al., 2002):
3. Color Reconstruction
� Unprojection of 3D points into camera images
![Page 19: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/19.jpg)
19
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Color Reconstruction (2)
� Global optimization of texture color to remove
discontinuities.
� Multi-view blending algorithm:
� Poisson formulation with Dirichlet boundary
conditions (Perez et al., 2003)
before blending
averaging
our algorithm
![Page 20: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/20.jpg)
20
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
![Page 21: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/21.jpg)
21
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Handling Large 3D Models (1)
� Office model (50m x 140m)
� 28M Vertices, 55M Faces
� 772 MByte Geometry
� 600 MByte Texture
� Generation of scene graphs by
iteratively dividing 3D model
� Model simplification of scene graph
nodes to limit number of elements
per node
� View-dependent culling and loading
![Page 22: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/22.jpg)
22
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
![Page 23: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/23.jpg)
23
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
![Page 24: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/24.jpg)
24
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
![Page 25: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/25.jpg)
25
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Failure cases:
Future work:
� Overcome limitation to static environments
� Online integration of new data vs. batch processing
Reconstruction of fine structures (plants, furniture, etc.)
Texture reconstruction failures due to registration errors
![Page 26: Automatic 3D Reconstruction and Modeling - ROS.org ... · Automatic 3D Reconstruction and Modeling Benjamin Pitzer ... Title Slide Nice Picture of 3D ... Finding a conformal mapping](https://reader031.vdocuments.net/reader031/viewer/2022022010/5b05e3157f8b9ad1768c072c/html5/thumbnails/26.jpg)
26
Automatic 3D Reconstruction and Modeling
Benjamin Pitzer | 10/18/2010 | © 2010 Robert Bosch LLC and affiliates. All rights reserved.
Research and Technology Center North America
Thank you for your attention!
http://bosch-ros-pkg.sourceforge.net/