Visibility Sorting and Compositing Visibility Sorting and Compositing without Splitting without Splitting
for Image Layer Decompositionsfor Image Layer Decompositions
Visibility Sorting and Compositing Visibility Sorting and Compositing without Splitting without Splitting
for Image Layer Decompositionsfor Image Layer Decompositions
John Snyder, Jed LengyelJohn Snyder, Jed LengyelJohn Snyder, Jed LengyelJohn Snyder, Jed Lengyel
Layered Decomposition ProblemLayered Decomposition ProblemLayered Decomposition ProblemLayered Decomposition Problem
Given set of moving geometric Given set of moving geometric partsparts, ,
and moving camera,and moving camera,
find visibility ordering for every frame.find visibility ordering for every frame.
Given set of moving geometric Given set of moving geometric partsparts, ,
and moving camera,and moving camera,
find visibility ordering for every frame.find visibility ordering for every frame.
partsparts sorted imagessorted images composited resultcomposited result
sortsort compositecomposite
Occlusion CyclesOcclusion CyclesOcclusion CyclesOcclusion Cycles
C
B
A C
B
A
When no order exists, can split or group parts.When no order exists, can split or group parts.
C2
C1
B
A C2
C1
B
A
G
G
G G
G
G
splitsplit
groupgroup
Non-SplittingNon-Splitting Layered Decomposition Layered DecompositionNon-SplittingNon-Splitting Layered Decomposition Layered Decomposition
Don’t split parts that form occlusion cycles:Don’t split parts that form occlusion cycles:
• run-time splitting is slowrun-time splitting is slow
• splitting is often unnecessarysplitting is often unnecessary
• splitting destroys coherencesplitting destroys coherence
Don’t split parts that form occlusion cycles:Don’t split parts that form occlusion cycles:
• run-time splitting is slowrun-time splitting is slow
• splitting is often unnecessarysplitting is often unnecessary
• splitting destroys coherencesplitting destroys coherence
Sorting requires no global separating plane.Sorting requires no global separating plane.
Image-Based Rendering AccelerationImage-Based Rendering Acceleration[Lengyel97][Lengyel97]Image-Based Rendering AccelerationImage-Based Rendering Acceleration[Lengyel97][Lengyel97]
spritessprites composited imagecomposited image
Other ApplicationsOther ApplicationsOther ApplicationsOther Applications
• fast special effects fast special effects
• animation playback with selective displayanimation playback with selective display
• incorporation of external image streamsincorporation of external image streams
• image stream compressionimage stream compression
• fast hidden line renderingfast hidden line rendering
• z resolution targetingz resolution targeting
• fast special effects fast special effects
• animation playback with selective displayanimation playback with selective display
• incorporation of external image streamsincorporation of external image streams
• image stream compressionimage stream compression
• fast hidden line renderingfast hidden line rendering
• z resolution targetingz resolution targeting
Previous WorkPrevious WorkPrevious WorkPrevious Work
• invisibility cullinginvisibility culling [Greene93,Zhang97] [Greene93,Zhang97]
• analytic hidden surface removalanalytic hidden surface removal [Mulmuley89] [Mulmuley89]
• global visibilityglobal visibility [Teller93,Durand97] [Teller93,Durand97]
• image layer decompositionsimage layer decompositions [Schumacker69,Newell72,Fuchs80] [Schumacker69,Newell72,Fuchs80]
• dynamic visibilitydynamic visibility [Torres90,Sudarsky96] [Torres90,Sudarsky96]
• depth sorting for special effectsdepth sorting for special effects [Max85] [Max85]
• invisibility cullinginvisibility culling [Greene93,Zhang97] [Greene93,Zhang97]
• analytic hidden surface removalanalytic hidden surface removal [Mulmuley89] [Mulmuley89]
• global visibilityglobal visibility [Teller93,Durand97] [Teller93,Durand97]
• image layer decompositionsimage layer decompositions [Schumacker69,Newell72,Fuchs80] [Schumacker69,Newell72,Fuchs80]
• dynamic visibilitydynamic visibility [Torres90,Sudarsky96] [Torres90,Sudarsky96]
• depth sorting for special effectsdepth sorting for special effects [Max85] [Max85]
Occlusion GraphsOcclusion GraphsOcclusion GraphsOcclusion Graphs
•pairwisepairwise occlusion relation: A occlusion relation: AEE B B
•relations on parts forms directed graph: relations on parts forms directed graph: occlusion occlusion graphgraph
•pairwisepairwise occlusion relation: A occlusion relation: AEE B B
•relations on parts forms directed graph: relations on parts forms directed graph: occlusion occlusion graphgraph
AB
view from Eview from E
AA BB
side viewside view
EE
Occlusion Graphs (Ex. 1)Occlusion Graphs (Ex. 1)Occlusion Graphs (Ex. 1)Occlusion Graphs (Ex. 1)
C
B
A A B
C
Occlusion Graphs (Ex. 2)Occlusion Graphs (Ex. 2)Occlusion Graphs (Ex. 2)Occlusion Graphs (Ex. 2)
C
B
A A B
C
Occlusion Graphs (Ex. 3)Occlusion Graphs (Ex. 3)Occlusion Graphs (Ex. 3)Occlusion Graphs (Ex. 3)
C
B
A C
B
A
A B
C
Forms strongly connected component (SCC).Forms strongly connected component (SCC).
Resolving Non-Binary CyclesResolving Non-Binary CyclesResolving Non-Binary CyclesResolving Non-Binary Cycles
SCCs need not be rendered as an aggregate!SCCs need not be rendered as an aggregate!
A B C
sum
A out C B out A C out B
A B C
sum
A out C B out A C out B
Occlusion Testing via CollisionOcclusion Testing via CollisionOcclusion Testing via CollisionOcclusion Testing via Collision
EE
B A
For convex hulls of objects A and B:
BE A ch(A E) B For convex hulls of objects A and B:
BE A ch(A E) B
Occlusion Testing via Collision:Occlusion Testing via Collision:Occluding ExampleOccluding ExampleOcclusion Testing via Collision:Occlusion Testing via Collision:Occluding ExampleOccluding Example
EE
ch(A E) B BE Ach(A E) B BE A
ch(A E)B
Occlusion Testing via Collision:Occlusion Testing via Collision:Non-Occluding ExampleNon-Occluding ExampleOcclusion Testing via Collision:Occlusion Testing via Collision:Non-Occluding ExampleNon-Occluding Example
EEch(A E) B
////ch(A E) B BE Ach(A E) B BE A
Trivial Sorting AlgorithmTrivial Sorting AlgorithmTrivial Sorting AlgorithmTrivial Sorting Algorithm
• compute occlusion graphcompute occlusion graph
• detect and topologically sort SCCsdetect and topologically sort SCCs
• at least quadratic in number of parts!at least quadratic in number of parts!
• compute occlusion graphcompute occlusion graph
• detect and topologically sort SCCsdetect and topologically sort SCCs
• at least quadratic in number of parts!at least quadratic in number of parts!
EEA B C D F
Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm
EEA
B CD F
A B,C,D,F but nothing occludes A. Process A first.A B,C,D,F but nothing occludes A. Process A first.
Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm
EE B CD F
Sorting Output: ASorting Output: A
A
Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm
EEC
D F
Sorting Output: A, BSorting Output: A, B
B
Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm
EE D F
Sorting Output: A, B, CSorting Output: A, B, C
C
Improving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial AlgorithmImproving the Trivial Algorithm
EE
Sorting Output: A, B, C, D, FSorting Output: A, B, C, D, F
D F
Incremental Visibility Sort (IVS)Incremental Visibility Sort (IVS)Incremental Visibility Sort (IVS)Incremental Visibility Sort (IVS)
•similar to Newell, Newell, & Sancha algorithm, 1972similar to Newell, Newell, & Sancha algorithm, 1972
•sorts parts, not polygonssorts parts, not polygons
•detects SCCs, doesn’t split detects SCCs, doesn’t split
•uses order from last query, not depth orderuses order from last query, not depth order
•culls more efficiently using kd-treeculls more efficiently using kd-tree
•similar to Newell, Newell, & Sancha algorithm, 1972similar to Newell, Newell, & Sancha algorithm, 1972
•sorts parts, not polygonssorts parts, not polygons
•detects SCCs, doesn’t split detects SCCs, doesn’t split
•uses order from last query, not depth orderuses order from last query, not depth order
•culls more efficiently using kd-treeculls more efficiently using kd-tree
Depth order doesn’t Depth order doesn’t indicate visibility order.indicate visibility order.Depth order doesn’t Depth order doesn’t indicate visibility order.indicate visibility order.
EE
A
B
ZZ
IVS AlgorithmIVS AlgorithmIVS AlgorithmIVS Algorithm
While L is nonempty, pop off top element AWhile L is nonempty, pop off top element A
if A is unmarkedif A is unmarked
if nothing remaining on L occludes Aif nothing remaining on L occludes Asend A to outputsend A to output
elseelsemark and reinsert A into Lmark and reinsert A into L
else // A is markedelse // A is markedcheck for occlusion cyclecheck for occlusion cycle
if found, group cycle objects and reinsertif found, group cycle objects and reinsertelse reinsert A into Lelse reinsert A into L
While L is nonempty, pop off top element AWhile L is nonempty, pop off top element A
if A is unmarkedif A is unmarked
if nothing remaining on L occludes Aif nothing remaining on L occludes Asend A to outputsend A to output
elseelsemark and reinsert A into Lmark and reinsert A into L
else // A is markedelse // A is markedcheck for occlusion cyclecheck for occlusion cycle
if found, group cycle objects and reinsertif found, group cycle objects and reinsertelse reinsert A into Lelse reinsert A into L
Occlusion CullingOcclusion CullingOcclusion CullingOcclusion CullingFundamental IVS query: Fundamental IVS query:
Which parts remaining in L occlude AWhich parts remaining in L occlude A??
• use convex hulls around partsuse convex hulls around parts
• bound hulls with spatial and angular extentsbound hulls with spatial and angular extents
• occlusion cull reduces to 1D interval intersectionocclusion cull reduces to 1D interval intersection
• bounded parts organized in dynamic kd-treebounded parts organized in dynamic kd-tree
Fundamental IVS query: Fundamental IVS query:
Which parts remaining in L occlude AWhich parts remaining in L occlude A??
• use convex hulls around partsuse convex hulls around parts
• bound hulls with spatial and angular extentsbound hulls with spatial and angular extents
• occlusion cull reduces to 1D interval intersectionocclusion cull reduces to 1D interval intersection
• bounded parts organized in dynamic kd-treebounded parts organized in dynamic kd-tree
objectobject convex hullconvex hull kd extentkd extent
Spatial ExtentsSpatial ExtentsSpatial ExtentsSpatial Extents
////
D
EA
B
D
EA
BD
EB A
D
EB A
B B A AB B A A B B A AB B A A
Angular ExtentsAngular ExtentsAngular ExtentsAngular Extents
////
EA
B
Z
X
Y
EA
B
Z
X
Y
EA
B
Z
X
Y
EA
B
Z
X
Y
B B A AB B A A B B A AB B A A
Kd-Tree Occlusion CullingKd-Tree Occlusion CullingKd-Tree Occlusion CullingKd-Tree Occlusion Culling
• rebalance kd-tree every framerebalance kd-tree every frame
• kd-tree supports part deactivationkd-tree supports part deactivation
• rebalance kd-tree every framerebalance kd-tree every frame
• kd-tree supports part deactivationkd-tree supports part deactivation
B
DC
A
xx11
yy11 yy22
xx11
yy11yy22
A CBD
yyxx
Exact Occlusion TestingExact Occlusion TestingExact Occlusion TestingExact Occlusion Testing
Does Does B B A? A?
• uses hybrid Chung/Gilbert collision algorithmuses hybrid Chung/Gilbert collision algorithm
• requires extremal vertex query on A and Brequires extremal vertex query on A and B
• no need to create no need to create ch(Ach(A E) E) dynamically dynamically
• exploits coherence in object motionexploits coherence in object motion
Does Does B B A? A?
• uses hybrid Chung/Gilbert collision algorithmuses hybrid Chung/Gilbert collision algorithm
• requires extremal vertex query on A and Brequires extremal vertex query on A and B
• no need to create no need to create ch(Ach(A E) E) dynamically dynamically
• exploits coherence in object motionexploits coherence in object motion
DDextremal vertex minimizes D•v on Aextremal vertex minimizes D•v on AA vv
IVS ComplexityIVS ComplexityIVS ComplexityIVS Complexity
(n+r)(s + c o + c )2 (n+r)(s + c o + c )2
variable “typical”case
worstcase
n - number of parts n n
r - number of reinsertions O(n) O(n2)
s - occluder search complexity O(log n) O(n)
c - maximum size of SCC O(1) O(n)
o - maximum num. of occluders O(1) O(n)
total O(n log n) O(n4)
variable “typical”case
worstcase
n - number of parts n n
r - number of reinsertions O(n) O(n2)
s - occluder search complexity O(log n) O(n)
c - maximum size of SCC O(1) O(n)
o - maximum num. of occluders O(1) O(n)
total O(n log n) O(n4)
Performance with Increasing ObjectsPerformance with Increasing ObjectsPerformance with Increasing ObjectsPerformance with Increasing Objects
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0 500 1000 1500 2000
number of objects
cpu
sec
on
ds
maxavg
Future WorkFuture WorkFuture WorkFuture Work
• animation previeweranimation previewer
• continuous time queriescontinuous time queries
• adaptive splitting, especially for terrainadaptive splitting, especially for terrain
• faster, less conservative visibility testsfaster, less conservative visibility tests
• automatic part decompositionautomatic part decomposition
• visibility sorting with minimal splittingvisibility sorting with minimal splitting
• animation previeweranimation previewer
• continuous time queriescontinuous time queries
• adaptive splitting, especially for terrainadaptive splitting, especially for terrain
• faster, less conservative visibility testsfaster, less conservative visibility tests
• automatic part decompositionautomatic part decomposition
• visibility sorting with minimal splittingvisibility sorting with minimal splitting
ConclusionsConclusionsConclusionsConclusions
• non-splitting layered decomposition useful non-splitting layered decomposition useful
• software visibility sorting is practical software visibility sorting is practical
• basic ideas of approach:basic ideas of approach:
• exploit temporal and spatial coherenceexploit temporal and spatial coherence
• exploit properties of convex objectsexploit properties of convex objects
• exploit incremental collision detection algorithmsexploit incremental collision detection algorithms
• non-splitting layered decomposition useful non-splitting layered decomposition useful
• software visibility sorting is practical software visibility sorting is practical
• basic ideas of approach:basic ideas of approach:
• exploit temporal and spatial coherenceexploit temporal and spatial coherence
• exploit properties of convex objectsexploit properties of convex objects
• exploit incremental collision detection algorithmsexploit incremental collision detection algorithms
Extent TrackingExtent TrackingExtent TrackingExtent Tracking
• use “vertex descent”use “vertex descent”
• on convex polytope, local minimizer of extent is also on convex polytope, local minimizer of extent is also global minimizer global minimizer
• use “vertex descent”use “vertex descent”
• on convex polytope, local minimizer of extent is also on convex polytope, local minimizer of extent is also global minimizer global minimizer
DD
frame t0frame t0
v0v0
frame t1frame t1
v1v1
v0v0
Results: Tumbling ToothpicksResults: Tumbling ToothpicksResults: Tumbling ToothpicksResults: Tumbling Toothpicks
• uniform scale (“us”): add more of same sizeuniform scale (“us”): add more of same size
• biases occlusion complexity superlinearlybiases occlusion complexity superlinearly
• uniform density (“ud”): add more of scaled sizeuniform density (“ud”): add more of scaled size
• occlusion complexity increases linearlyocclusion complexity increases linearly
• uniform scale (“us”): add more of same sizeuniform scale (“us”): add more of same size
• biases occlusion complexity superlinearlybiases occlusion complexity superlinearly
• uniform density (“ud”): add more of scaled sizeuniform density (“ud”): add more of scaled size
• occlusion complexity increases linearlyocclusion complexity increases linearly
• eccentric ellipsoids moving in eccentric ellipsoids moving in cubical volume cubical volume
• eccentric ellipsoids moving in eccentric ellipsoids moving in cubical volume cubical volume
Performance with Increasing VelocityPerformance with Increasing VelocityPerformance with Increasing VelocityPerformance with Increasing Velocity
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0 10 20 30 40 50 60 70
velocity scale
cp
u s
ec
on
ds
ud max ud avg
0
0.5
1
1.5
2
2.5
3
3.5
0 500 1000 1500 2000
number of objects
cp
u s
ec
on
ds
us max us avg ud max ud avg
Performance with Increasing ObjectsPerformance with Increasing ObjectsUniform Density & Uniform ScaleUniform Density & Uniform ScalePerformance with Increasing ObjectsPerformance with Increasing ObjectsUniform Density & Uniform ScaleUniform Density & Uniform Scale
Kd-Tree Culling PerformanceKd-Tree Culling PerformanceKd-Tree Culling PerformanceKd-Tree Culling Performance
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0 500 1000 1500 2000
number of objects
cp
u s
ec
on
ds
k=1 ud k=3 ud
• six aircraft flying through valleysix aircraft flying through valley
• investigates rendering acceleration at various investigates rendering acceleration at various levels of terrain splitting levels of terrain splitting
• update rate assumptions:update rate assumptions:• aircraft parts: 20%aircraft parts: 20%• terrain parts: 70%terrain parts: 70%• sky 40%sky 40%
• six aircraft flying through valleysix aircraft flying through valley
• investigates rendering acceleration at various investigates rendering acceleration at various levels of terrain splitting levels of terrain splitting
• update rate assumptions:update rate assumptions:• aircraft parts: 20%aircraft parts: 20%• terrain parts: 70%terrain parts: 70%• sky 40%sky 40%
Results: Canyon FlybyResults: Canyon FlybyResults: Canyon FlybyResults: Canyon Flyby
• aggregation penalty: aggregation penalty: render every frame render every frame
• aggregation penalty: aggregation penalty: render every frame render every frame
Results: Canyon FlybyResults: Canyon FlybyResults: Canyon FlybyResults: Canyon Flyby
split cpu avg cpu max exp. factor parts agg. update rate
20x20 17.33 29.82 5.04 0.2% 41.8%14x14 8.08 14.59 3.78 1.0% 38.0%10x10 5.17 9.88 3.09 6.4% 40.4%7x7 4.51 9.68 2.58 22.5% 42.4%5x5 5.37 11.01 2.28 53.1% 72.0%
split cpu avg cpu max exp. factor parts agg. update rate
20x20 17.33 29.82 5.04 0.2% 41.8%14x14 8.08 14.59 3.78 1.0% 38.0%10x10 5.17 9.88 3.09 6.4% 40.4%7x7 4.51 9.68 2.58 22.5% 42.4%5x5 5.37 11.01 2.28 53.1% 72.0%
• cpu times are in ms per framecpu times are in ms per frame
• update rate is poly-weightedupdate rate is poly-weighted
• cpu times are in ms per framecpu times are in ms per frame
• update rate is poly-weightedupdate rate is poly-weighted
Newell, Newell, & Sancha (1972)Newell, Newell, & Sancha (1972)Newell, Newell, & Sancha (1972)Newell, Newell, & Sancha (1972)• traverse depth-sorted list of polygonstraverse depth-sorted list of polygons• if next polygon doesn’t overlap, send to outputif next polygon doesn’t overlap, send to output• otherwise use tests of increasing complexity:otherwise use tests of increasing complexity:• screen bounding box, vertex/plane, 2d intersectionscreen bounding box, vertex/plane, 2d intersection
• if unoccluded, polygon is sent to outputif unoccluded, polygon is sent to output• if occluded, polygon is marked and reinsertedif occluded, polygon is marked and reinserted• marked polygons are split to remove cyclemarked polygons are split to remove cycle
• traverse depth-sorted list of polygonstraverse depth-sorted list of polygons• if next polygon doesn’t overlap, send to outputif next polygon doesn’t overlap, send to output• otherwise use tests of increasing complexity:otherwise use tests of increasing complexity:• screen bounding box, vertex/plane, 2d intersectionscreen bounding box, vertex/plane, 2d intersection
• if unoccluded, polygon is sent to outputif unoccluded, polygon is sent to output• if occluded, polygon is marked and reinsertedif occluded, polygon is marked and reinserted• marked polygons are split to remove cyclemarked polygons are split to remove cycle
EE
AA
ZZ
CC
BBFF
HH
GG