automated 3d model construction for urban environments
DESCRIPTION
Next Generation 4D Distributed Modeling and Visualization. Automated 3D Model Construction for Urban Environments. Christian Frueh John Flynn Avideh Zakhor. University of California, Berkeley. June 13, 2002. Presentation Overview. Introduction Ground based modeling Mesh processing - PowerPoint PPT PresentationTRANSCRIPT
Automated 3D Model Construction for Urban
Environments
Christian Frueh John Flynn Avideh Zakhor
Next Generation 4D Distributed Modeling and Visualization
University of California, Berkeley
June 13, 2002
Presentation Overview
Introduction
Ground based modeling
- Mesh processing
Airborne modeling
- Aerial photos
- Airborne laser scans
3D Model Fusion
Rendering
Conclusion and Future Work
Introduction
Goal: Generate 3D model of a city for virtual walk/drive/fly-thrus and simulations
• Fast• Automated• Photorealistic
• 3D model of street scenery & building façades
• highly detailed
needed:
• 3D Model of terrain and buildings tops & sides
• coarse resolution
For Fly-Thru: For Walk/Drive-Thru:
Introduction
3D model of building façades
3D Model of terrain and building tops
Airborne Modeling•Laser scans/images from
plane
Ground Based Modeling• Laser scans & images from
acquisition vehicle
Complete 3D City Model
Fusion
Airborne Modeling
Available Data:
• Aerial Photos
• Airborne laser scans
Acquisition of terrain shape and top-view building geometry
Goal:
Geometry: 2 approaches:
I) stereo matching of photos
Texture: from aerial photos
II) airborne laser scans
Airborne Modeling
Stereo photo pairs from city/urban areas, ~ 60% overlap
Approach I : Stereo Matching
• Camera parameter computation,
• Matching,• Distortion reduction, • Model generation
• Segmentation
Manual: Automated:
Semi-Automatic
(last year)
Stereo Matching
Stereo pair from downtown Berkeley and the estimated disparity after removing perspective distortions
Stereo Matching Results
Downtown Oakland
Airborne Modeling
Scanning city from plane
point cloud
• Resolution 1 scan point/m2
• Berkeley: 40 Million scan points
Approach II: Airborne Laser Scans
Airborne Laser Scans
• Re-sampling point cloud
• Sorting into grid
• Filling holes
usable for:
Map-like height field
• Mesh Generation• Monte Carlo Localization
Textured Mesh Generation
1. Connecting grid vertices to mesh
2. Applying Q-slim simplification
3. Texture mapping:
• Semi-automatic• Manual selection of few correspondence
points: 10 mins/entire Berkeley• Automated camera pose estimation• Automated computation of texture for mesh
Airborne Model
East Berkeley campus with campanile
Airborne Model
http://www-video.eecs.berkeley.edu/~frueh/3d/airborne/
Downtown Berkeley
Ground Based Modeling
buildings
truck
x
z
y
2D laser v
u
Acquisition vehicleTruck with rack:• 2 fast 2D laser scanners• digital camera
Acquisition of highly detailed 3D building façade models
Scanning setup• vertical 2D laser scanner for
geometry capture • horizontal scanner for pose
estimation
Goal:
Scan Matching & Initial Path Computation
Horizontal laser scans:• Continuously captured during vehicle motion
t = t0
t = t1
• Overlap
Relative position estimation by scan-to-scan matching
Scan matching
Translation (u,v)Rotation
(u, v)
Adding relative steps (ui, vi, i)
path (xi,yi,i)
(u1, v1, 1)(u2, v2, 2)
…
(ui, vi, i)
(ui-1, vi-1, i-1)
3 DOF pose (x, y, yaw)
6 DOF Pose Estimation From Images
Scan matching cannot estimate vertical motion- Small bumps and rolls - Slopes in hill areas
Full 6 DOF pose of the vehicle is important; affects:- Future processing of the 3D and intensity data - Texture mapping of the resulting 3D models
Extend initial 3 DOF pose by deriving missing 3 DOF (z, pitch, roll) from images
6 DOF Pose Estimation From Images
Central idea: photo-consistency
• Each 3D scan point can be projected into images using initial 3 DOF pose
• If pose estimate is correct, point should appear the same in all images
• Use discrepancies in projected position of 3D points within multiple images to solve for the full pose
6 DOF Pose Estimation – Algorithm
3DOF of laser as initial estimate Project scan points into both images If not consistent, use image correlation to
find correct projection Ransac used for robustness
6 DOF Pose Estimation – Results
with 3 DOF pose
with 6 DOF pose
6 DOF Pose Estimation – Results
Model
Number
Original
Number of
Seams
Completely
Removed
Less
notic able No Change Worse
1 48 42 0 4 2
2 15 8 0 7 0
3 29 25 2 2 0
4 24 18 1 5 0
5 33 24 0 9 0
6 18 15 0 3 0
7 34 28 1 3 2
Total 201 160 4 33 4
Perc entage 80 2 16 2
Monte Carlo Localization (1)
Previously: Global 3 DOF pose correction using aerial photography
b) path after MCL correctiona) path before MCL correction
After correction, points fit to edges of aerial image
Monte Carlo Localization (2)
Extend MCL to work with airborne laser data and 6 DOF pose
No perspective shifts of building tops, no shadow lines
Use terrain shape to estimate z coordinate of truck
• Fewer particles necessary, increased computation speed
• Significantly higher accuracy near high buildings and tree areas
• Correct additional DOF for vehicle pose (z, pitch, roll)
• Modeling not restricted to flat areas
Now:
Monte Carlo Localization (3)
Resulting corrected path overlaid with airborne laser height field
Track global 3D position of vehicle to correct relative 6 DOF motion estimates
Path Segmentation
vertical scans: 107,082scan points: ~ 15 million
24 mins, 6769 meters
Too large to process as one block!
• Segment path into quasi-linear pieces
• Cut path at curves and empty areas
• Remove redundant segments
Path Segmentation
Resulting path segments overlaid with edges of airborne laser height map
Simple Mesh Generation
Simple Mesh Generation
Triangulate
Point cloud Mesh
Problem:• Partially captured
foreground objects • erroneous scan points
due to glass reflection
Side views look “noisy”
Remove foreground: extract facades
Façade Extraction and Processing (1)
1. Transform path segment into depth image
2. Histogram analysis over vertical scans
scan n
r
depth
0
5
10
15
20
25
1 6 11 16 21 26 31 36
split depth
depth
local minimum
main depth
scanner
split depth
ground points
depth value sn,υ for
a scan point Pn,υ
main depth
Façade Extraction and Processing (2)
3. Separate depth image into 2 layers:
background =building facades
foreground =trees, cars etc.
Façade Extraction and Processing (3)
4. Process background layer:
• Fill areas occluded by foreground objects by extending geometry from boundaries
•Horizontal, vertical, planar interpolation, RANSAC
• Detect and remove invalid scan points
• Apply segmentation
• Fill remaining holes in large segments
• Final result: “clean” background layer
• Remove isolated segments
Façade Extraction – Examples (1)
without processing with processing
Façade Extraction – Examples (2)
with processing
without processing
Façade Extraction – Examples (3)
without processing
with processing
Facade Processing
Foreground Removal
Mesh Generation
Downtown Berkeley
Automatic Texture Mapping (1)
Camera calibrated and synchronized with laser scanners
Transformation matrix between camera image and laser scan vertices can be computed
1. Project geometry into images
2. Mark occluding foreground objects in image
3. For each background triangle:
Search pictures in which triangle is not occluded, and texture with corresponding picture area
Automatic Texture Mapping (2)
Efficient representation: texture atlas
Copy texture of all triangles into “mosaic” image
Typical texture reduction: factor 8..12
Automatic Texture Mapping (3)
Large foreground objects: Some of the filled-in triangles are not visible in any image!
“texture holes” in the atlas
Texture synthesis: preliminary
• Mark holes corresponding to non-textured triangles in the atlas
• Search the image for areas matching the hole boundaries
• Fill the hole by copying missing pixels from these image
Automatic Texture Mapping (4)
Texture holes
marked
Texture holes filled
Automatic Texture Mapping (5)
Ground Based Modeling - Results
Façade models of downtown Berkeley
Ground Based Modeling - Results
Façade models of downtown Berkeley
Model Fusion
Goal:Fusion of ground based and airborne model to one single model
1. Registration of models
2. Combining the registered meshes
Model Fusion:
Façade model
Airborne model
Registration of Models
Models are already registered with each via Monte-Carlo-Localization !
Which model to use where?
Preparing Ground Based Models
Intersect path segments with each other Remove degenerated, redundant triangles in overlapping areas
original mesh
redundant triangles removed
Preparing Airborne Model
Remove facades in airborne model where ground based geometry is available
Add ground based façades Fill remaining gaps with a “blend
mesh” to hide model transitions
Ground based model has 5-10 times higher resolution
Preparing Airborne Model
Initial airborne model
Preparing Airborne Model
Remove facades where ground based geometry is available
Combining Models
Add ground based façade models
Combining Models
Fill remaining gaps with a “blend mesh” to hide model transitions
Model Fusion - Results
Rendering
Subdivide model and create multiple level-of-details (LOD)
Generate scene graph, decide which LOD to render when
• Up to 270,000 triangles, 20 MB texture per path segment
• 4.28 million triangles, 348 MB texture for 4 downtown blocks
Ground based models:
• Difficult to render interactively!
Multiple LODs for façade meshes
Highest LOD
• Geometry: 10%
• Texture: 25%
Lower LOD
of original mesh
•Qslim mesh simplification
•Texture subsampling
Façade Model Subdivision for Rendering
Subdivide 2 highest LODs of façade meshes along cut planes
Sub-scene
LOD 2
Path segment
Global scene
LOD 0LOD 1
LOD 0LOD 1
LOD 0LOD 1
Submesh
Submesh
Submesh
Interactive Rendering
Downtown blocks: Interactive rendering with web-based browser!
Future Work
Resolution enhancement and post-processing of LIDAR data
Devise new data acquisition system and algorithms to capture - both sides of street simultaneously- texture for upper parts of tall buildings
Include foreground objects in model Add temporal component to dynamically
update models Compact representation Interactive rendering