geometric algorithms in 3d real-time rendering and facial expression modeling prof. marina l....
TRANSCRIPT
Geometric Algorithms in 3D Real-Time Rendering and
facial Expression Modeling
Prof. Marina L. Gavrilova
University of Calgary
Quote:"Perseverance is not a long race;
it is many short races one after another."-Walter Elliott
• Brief Overview of the Area
• Geometric Approaches for mesh reconstruction and image morphing
• Real-Time Optimally Adapting Mesh Algorithms
• Non Photo-realistic Facial Expression Modeling and Morphing Algorithms
• Swarm Real-time Visualization System
Presentation Outline
Objective:To showcase the power of the geometric and
topology-based approaches in 3D visualization and modeling
Relevant Areas
• Computer-Aided Design• Geographical Information Systems• Biometrics• 3D Animation• Computer Games• Strategic Planning• Defense• Other areas
State of the Art
• TERRAIN MODELING
• Multi-resolution dynamic mesh• Viewer-dependent refinement• Culling techniques• View frustum algorithms• Triangle-count preservation• Subdivision refinement• Controlled LOD
• Visual continuity and smooth transition are crucial
• Lindstrom, Hoppe, Duchaineauy, Hesse, Silva, Skala, Kolingerova
• EXPRESSION MODELING
• Multi-resolution adaptable mesh• Light source and view point
dependent rendering• Control points and areas• Tracking algorithms for transforming • Fitting algorithms for modeling• Wavelets to extract specific
characteristics• Visual continuity and feature
preservation are crucial
• Jain, Salesin, Gooch, Gordon, Thalmann, Plemenos, Sourin
Our Approach TERRAIN MODELING
Challenge: massive amount of data, smooth rendering
Methodology:• Multi-resolution dynamic triangle
mesh with new refinement strategy• New split and merge queue operations• Frame-to-frame coherence algorithm
for local updates• New geo-morph smoothing algorithm Outcome:• Visual continuity, rendering
efficiency, frame-rate constancy for massive terrain
EXPRESSION MODELING
Challenge: massive amount of detail, tracking changes
Methodology:• NPR approach to feature extraction and
modeling• New Distance-Transform algorithm for
morphing• Delaunay triangulation for mesh
representation• Image segmentation and feature-line
extraction algorithm
Outcome: Expressive detail preservation, efficient
storage, fast processing, 3D controllable deformations
Our Focus
• Topology preservation and geometry plays an important role in many algorithms, allowing for rendering efficiency, visual continuity and feature preservation.
Part I:3D Real-Time Massive Terrain Visualization
Sourcing data from real world
Understanding Height FieldsDEM: Digital Elevation Model
• Contains only relative Height
• Regular interval
• Pixel color determine height
•Discrete resolution
Converting Height field data into 3D topological mesh
200 255 150 100
100 255 255 200
200 150 200 100
• Pixel value (z) is used as Height Map
• Vertices are generated as points in 3D
• A Mesh is triangulated
Terminology
DEM: Digital Elevation ModelVDR: Viewer Dependent RefinementROAM: Realtime Optimally Adapting MeshCLOD: Continuous Level of Detail
What is mesh?
• It is often required to convert geometric objects in any other form to polygonal meshes of geometric simplexes (i.e. triangles) before rendering
• Meshes are used for computer modeling, biometric, GIS, cartographic, and animation applications
• Mesh optimization techniques are used to reduce the mesh to an acceptable size (user defined), minimizing the loss of quality (subject to some form of error analysis).
Mesh definition
• A mesh is defined as a pair (V,K) where V is the set of vertices and K is a simplicial complex specifying the connectivity of the mesh simplices (the adjacency of the vertices, edges and faces).
• The 0-simplices are called vertices, the 1-simpices are called edges, and the 2-simplices are called faces.
• Triangulation is an example of mesh.
Example: 3D Terrain Rendering
• Uses DEM as input of the application
•Generates dynamic mesh to achieve frame coherent animated view in real-time
Contribution• In our work we used the ROAM definition introduced in
[Duchaineauy], and developed a number of algorithms that rely heavily on topological properties of the modeled terrain in order to handle massive data sets.
• The observation of hardware performance reacting to strip length allowed us to design a fast and effective greedy approach to dynamically maintain triangle stripping. The method resulted in achieving rendering efficiency and frame rate constancy.
• We further improved the rendering efficiency while preserving important terrain features utilizing topological information.
• We eliminated the priority queue for merge operation, amortized update cost by allowing split queue to degenerate, took advantage of frame-to-frame coherence for local update and devised an efficient method for mesh-smoothing using geo-morph while preserving important terrain features.
Methodology applications
• Massive terrain rendering
• Realistic deformation modeling using mass spring system
• Adaptive spatial memory subdivision in robot motion planning simulation
Basic Approach
• The entire optimization process can be viewed as an optimal spatial subdivision problem. There exist only two legal moves to achieve optimality: subdivide and merge.
• Models that are based on this concept do not require computationally extensive optimization process. Instead, the mesh progressively adapts to an optimal solution.
• These models usually define a multi-criteria based error metric to assess the energy of the mesh.
Conventional Approach
• Regular Grid and TIN: Triangulated Irregular Network• Problems:
– Significant preprocessing– Generates too many triangles– the visible area might constitute a tiny fraction of that detail– Viewpoint dependent refinement is not possible with static
meshes. – Have a bad triangle to screen pixel ration.– Unable to obtain a steady performance.
• Unable to Render Massive Terrain
Problems with static mesh
d
h h
21 unit
a2
a2h
h1
1d
d
2p
2a p
2 tan2
h da
1screen pixel ratio 1:r h h
Problems with static mesh …
Detail
98.5%
1.5%
Screen
10%
90%
A Static Mesh (40,000 tri)
Dynamic mesh
• Dynamic mesh generation is based on real-time adaptive models that allow local transformation of a given mesh to different levels of details and adapt progressively to the changes in the rendering environment.
• Dynamic models use some form of hierarchical data structure to store a mesh in multiple LOD.
• There are many different versions of dynamic mesh currently under research and development. Several patented technologies have been released in the industry (i.e. Intels MRM). The two most prominent approaches are Progressive Meshes (PM) and Real-time Optimally Adapting Mesh (ROAM).
ROAM :•Can Subdivide when more details necessary
•Merge & Split Queue
•Tree Structured
ROAM (25,000 Triangles)
The ROAM Advantage
ROAM advantages
• An efficient top-down dynamic mesh model. • Display of complex surfaces can be provided by
dynamically computing a multi-resolution triangular mesh for each view.
• One can minimize geometric distortions on the screen while maintaining a fixed triangle count.
• Visual discontinuities could be minimized, and efficient mesh corrections can ensure selected lines of site, or object proximity are correctly represented.
• An incremental priority-queue based approach in ROAM allows exploiting frame-to-frame coherence for fast efficient computation of the adapting mesh.
Geometric subdivision
Problems with Geometric Subdivisions
ROAM principle
The basic operating principle of ROAM
Quadtree and Bintree for ROAM
Comparison between Quadtree (top) and Bintree (bottom)
ROAM Model
DEM
Inte
rpola
tion
En
gin
e Triangle Bin Tree Data structure
Render
Ref
ine
Err
orM
etri
c
Speed: 15.0 FPS
Size: 15000 Tri
Our ROAM Model
Speed: 45.0 FPS
Size: 19000 Tri
DEM
Inte
rpola
tion
En
gin
e Triangle Quad Tree Data structure
Err
orM
etri
c
Ref
ine
Render
G Q
Wavelet
ErrorAnalysis C
lean
up
Op
era
tion
ROAM Implementation and Improvements
• Pre-process DEM• Reduce overhead for maintaining Split queue• Use degenerated split queue• Replace priority merge queue (HEAP) with a
simple FIFO queue.• Merge arbitrary obscured regions (outside visible
frustum) only • Improve smoothness, reduce popping effect
Proposed Unique Geometry-based methodology
• DEM Analysis
• Adaptive Loop Subdivision
• Geo-Morphing
• Visible frustum computation
• CCM - Cascading Coupling March method for massive terrain
Preprocessing of DEM input
• Convolute the surface (2nd Difference)
• Efficient LOD distribution
• Used in Advanced Shading (feature preservation)
Differentiating the Surface: Convolution
Original DEM of Kluane Park First differentiation. Blue area Shows steep mountains.
Final LOD distribution image Shows rocky areas require more details
Second Differentiation. Blue areas indicate steep mountain peaks.
Differentiating the Surface (cont’d)…
• Take Weighted Average of neighboring pixel Gradients
• 2 passes gives us the rate of gradient changes
Statistical Analysis of ROAM
• Bottleneck: Split and Merge Operation and visibility test• Only 2.7% triangles are merged within the visible frustum• 75% of split operations occurs on closest 5% region of the
visible frustum• Taking above into account, special priority function for
split queue was developed based on U=Uniformity • F=Frustum priority• D=Curvature priority• P=Distance priority• DET=Curvature (preprocessed)• Lev(f)=Subdivision level of face f
Methodology: Speeding up the process - Amortizing refinement
• Updating the Split queue over multiple frames
• Allow split queue (heap) to degenerate• Refine integrity using error threshold test• Re-generate queue after constant number of
frames• Dramatic gain in speed
Methodology: Maintaining Degenerated S-Queue
Methodology: Geo-Morph
• Smooth interpolation of vertex over subdivision
• Requires maintenance of one additional queue (G-queue)
Methodology: Smoothing
• Compute face normal • Compute vertex normal• Local computation of normal
during refinement (Geo-Morph)
• Apply vertex normal when rendering
• Advanced shading: Linear combination of face and vertex normal (Hybrid)
Example of smoothing
Result with Texturing
Massive Terrain Model: ALS
Adaptive Loop Subdivision and Cascading Coupling March
Quote:“Map of a region was a size of a city, map of an
Empire was a size of a Province."Jorge Luis Borges
Loop Scheme
Patching alignment
A
B CA
BCFaces:
•Regular
•Tface
Regular Tface
Adaptive Loop Subdivision
Recursive Split
A
A
A
A
BB
B
B
C C
C
C
AB CA
BC
Split
•Prefer: Tface
•Retract Tface
Patch Stitching: CCM
“CASCADING COUPLING MARCH”
Method Description
• We develop CCM to stitch two quadrants of the adaptive mesh patch, and to ensure seamless connectivity by progressively refining and connecting the subdivision tree (quadtree) data structure.
• We ensure integrity of the data structure among multiple patches so that subdivision and simplification are not limited to the boundaries of the patches.
• CCM allows to consider the entire data structure as a number of strongly connected sets of quad-trees. As a result we can refine the mesh as one seamless continuous structure (not patches stitched together).
CCM idea
• We perform a coupled recursive march of the two quad-trees from the top to the bottom of the tree, subdividing or connecting border triangles.
• To increase geometric regularity we used non-orthogonal axis and used space warps to convert the scheme back to orthogonal axis for rendering. This is crucial when we are performing viewer dependent refinements. Patch locating and coordinate transform algorithms are designed to maintain a list of visible patches. Additional patches are added and connected using CCM as the simulation advances. Obscured patches are deleted using recursive march.
• This new method allows a greater flexibility, efficiency and control over smoothness
Two Adjacent Patches
Snapshot 3 … Level 2
Results on Mesh Rendering
- Strategies evolved, Runtime and other aspects
Results (Output obtained)
(a) 0, 1, 4curv dist ur r r= = = (b) 0, 1, 6curv dist ur r r= = = (c) 0, 1, 8curv dist ur r r= = =
(d) 1, 0, 4curv dist ur r r= = = (e) 1, 0, 6curv dist ur r r= = = (f) 1, 0, 8curv dist ur r r= = =
Results (Runtime) (Cont’d)Frame Per Second - Modified ROAM Scheme
0
20
40
60
80
0 10 20 30 40 50 60Time (sec)
Fra
me/
Sec
ond
Kluane 25000 Alaska 35000Fractal 55000 Kluane AveAlaska Ave Fractal AveKluane Alaska Fractal
55.15 39.12 27.87
Average Frame/ Second
Results (Runtime) (Cont’d)
Rendering Statistics for Modified ROAM Scheme
0
5000
10000
15000
20000
25000
0 50 100 150 200
Time of Simulation (Seconds)
Num
ber
o T
rian
gles
Total T riangles Visible T riangles
Results (Runtime) (Cont’d)Average Load of The Geomorph Queue
in ALS Based Models
0
500
1000
1500
2000
2500
3000
3500
Number of Triangles
Aver
age
num
ber
of
ver
tice
s
in t
he
Geo
morp
h q
ueu
e
20 frames 286 427 567 717 805
60 frames 791 1097 1446 1781 2012
100 frames 1019 1746 2382 2842 3276
20000 25000 30000 35000 40000
Summary of Results
• High Speed (45 FPS average) for 20000 faces!!• Static mesh upper limit: 55 FPS• Continuity is achieved (Geo-Morph) – 300
vertices on average for 45 FPS –very small overhead
• Visual appeal (smoothing & Texture)• Less variant Frame rate• Efficient Understructure (amortized update)
More Results on Mesh Rendering
See Animation Demos