stereoscopy, from xy to z
TRANSCRIPT
![Page 1: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/1.jpg)
Samuel Gateau, NVIDIA Robert Neuman, DISNEY Marc Salvati, OLM SIGGRAPH | 2011 | Vancouver
Stereoscopy, From XY to Z http://developer.nvidia.com/siggraph-2011-stereoscopy-course
![Page 2: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/2.jpg)
Agenda
9:00 Welcome
9:05 Stereoscopy fundamentals and depth perception Samuel Gateau
9:40 Stereopsis and 3D hints Marc Salvati
10:10 Depth as storytelling tool Robert Neuman
10:30 Real-time & gaming techniques Samuel Gateau
11:00 Cartoon Authoring for 3D Marc Salvati
11:30 Creative choices for 3D Robert Salvati
11:50 Managing a depth budget Robert Salvati
12:10 Questions
Course available at
http://developer.nvidia.com/siggraph-2011-stereoscopy-course
![Page 3: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/3.jpg)
Special Thanksβ¦
![Page 4: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/4.jpg)
![Page 5: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/5.jpg)
![Page 6: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/6.jpg)
How does it work ?
![Page 7: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/7.jpg)
TWO EYES, ONE SCREEN, TWO IMAGES
Changes to the rendering pipe
![Page 8: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/8.jpg)
In Mono
Eye space
Z Y
X
Near plane
Scene is viewed from one eye
and projected with a perspective
projection along eye direction on
Near plane in Viewport Mono Frustum
Scene
Viewport
![Page 9: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/9.jpg)
In Stereo
Eye space
Z Y
X
Scene
Near plane
![Page 10: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/10.jpg)
In Stereo:
Two eyes
Eye space
Z Y
X
Left and Right eyes Shifting the mono eye along
the X axis
Scene
Near plane
![Page 11: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/11.jpg)
In Stereo:
Two eyes
Eye space
Z Y
X
Left and Right eyes Shifting the mono eye along
the X axis
Eye directions are parallels
Scene
Near plane
![Page 12: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/12.jpg)
Virtual Screen
In Stereo: Two Eyes,
One Screen
Eye space
Z Y
X
One βvirtualβ screen
Left and Right eyes Shifting the mono eye along
the X axis
Eye directions are parallels
Scene
Near plane
![Page 13: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/13.jpg)
Scene
In Stereo: Two Eyes,
One Screen
Virtual Screen
Eye space
Z Y
X
Left Frustum Right Frustum
One βvirtualβ screen Where the left and right
frustums converge
Left and Right eyes Shifting the mono eye along
the X axis
Eye directions are parallels
Near plane
![Page 14: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/14.jpg)
In Stereo: Two Eyes, One Screen,
Two Images
Virtual Screen
Eye space
Z Y
X
Two images 2 images are generated at
the near plane in each views
Scene
Left Image
Right Image
Left and Right eyes Shifting the mono eye along
the X axis
Eye directions are parallels
One βvirtualβ screen Where the left and right
frustums converge
Near plane
![Page 15: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/15.jpg)
In Stereo: Two Eyes, One Screen,
Two Images
Virtual Screen
Eye space
Z Y
X
Scene
Left Image
Right Image
Left Image
Right Image
Real Screen
Near plane
Two images 2 images are generated at
the near plane in each views
Presented independently to
each eyes of the user on the
real screen
![Page 16: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/16.jpg)
Stereoscopic Rendering
Render geometry twice
From left and right eyes
Into left and right images
![Page 17: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/17.jpg)
DEFINING STEREO PROJECTION
Basic definitions so we all speak English
![Page 18: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/18.jpg)
Stereo Projection
Human vision is really like 2 eyes looking at a parallel direction
Left Eye
Right Eye
Z
Y
X
Eye space
![Page 19: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/19.jpg)
Stereo Projection
Stereo projection matrix is a horizontally offset version of regular mono projection
matrix
Offset Left / Right eyes along X axis
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Screen
Mono Frustum
![Page 20: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/20.jpg)
Stereo Projection
Projection Direction is parallel to mono direction (NOT toed in)
Left and Right frustums converge at virtual screen
Left Eye
Right Eye
Mono Eye
Left Frustum
Right Frustum
Virtual Screen
Z
Y
X
Eye space
![Page 21: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/21.jpg)
Parallel, NOT Toed in!
Historically, live camera mounted in parallel stereo would waste a lot of the view field
Waste view field is wasted film area
Left Eye
Right Eye
Mono Eye Z
Y
X
Eye space
Virtual Screen
![Page 22: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/22.jpg)
Parallel, NOT Toed in!
Hence the Toed-in camera solution
But Toed in frustum introduces deformation which is really painful
Image Planes are not parallel to the screen plane
This can be corrected in post production but not perfect
Left Eye
Right Eye
Mono Eye Z
Y
X
Eye space
![Page 23: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/23.jpg)
Interaxial
Distance between the 2 virtual eyes in eye space
The mono, left & right eyes directions are all parallels
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye Interaxial
![Page 24: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/24.jpg)
Separation
The normalized version of interaxial by the virtual screen width
More details in a few slidesβ¦.
πππππππ‘πππ = πΌππ‘ππππ₯πππ
ππππππ π€πππ‘β
Screen width Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye Interaxial
Virtual Screen
![Page 25: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/25.jpg)
Convergence
Virtual Screenβs depth in eye space (βScreen Depthβ)
Plane where Left and Right Frustums intersect
Convergence
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Left Frustum
Right Frustum
Virtual Screen
![Page 26: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/26.jpg)
Parallax
Signed Distance on the virtual screen between the projected
positions of one vertex in left and right image
Parallax is function of the depth of the vertex
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Virtual Screen
Parallax
Vertex depth
Convergence
Inte
raxi
al
![Page 27: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/27.jpg)
Depth Perception
![Page 28: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/28.jpg)
DEPTH PERCEPTION
Where the magic happens
![Page 29: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/29.jpg)
Virtual vs. Real Screen
Virtual Screen
Virtual Space
Z Y
X
Scene
The virtual screen is
perceived AS the real screen
Left Image
Right Image
Real Screen
![Page 30: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/30.jpg)
Parallax is Depth
Virtual Screen
Virtual Space
Z Y
X
Scene
Left Image
Right Image
Real Screen
![Page 31: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/31.jpg)
Parallax is Depth
Virtual Screen
Virtual Space
Z Y
X
Scene
Parallax creates the depth
perception for the user
looking at the real screen
presenting left and right
images
Left Image
Right Image
Real Screen
![Page 32: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/32.jpg)
In / Out of the Screen
Z
Y
X
Eye space
Left Eye
Right Eye
Mono Eye
Screen Out of the Screen In the Screen
Convergence
Vertex Depth Parallax Vertex Appears
Further than Convergence Positive In the Screen
Vertex Depth Parallax Vertex Appears
Further than Convergence Positive In the Screen
Equal Convergence Zero At the Screen
Vertex Depth Parallax Vertex Appears
Further than Convergence Positive In the Screen
Equal Convergence Zero At the Screen
Closer than Convergence Negative Out of the Screen
![Page 33: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/33.jpg)
COMPUTING PARALLAX & PROJECTION MATRIX
Equations !!!
![Page 34: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/34.jpg)
Computing Parallax Thank you Thales
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Convergence
Inte
raxi
al
Parallax
In eye space: ππππππππ₯ππ¦π
πΌππ‘ππππ₯πππ=
π·πππ‘ββπΆπππ£πππππππ
π·πππ‘β
ππππππππ₯ππ¦π = πΌππ‘ππππ₯πππ Γ 1 βπΆπππ£πππππππ
π·πππ‘β
Scr
een
Depth
![Page 35: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/35.jpg)
Scr
een
Computing Parallax In image space (not pixels but in range [0,1])
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Depth
Convergence
Inte
raxi
al
Parallax
![Page 36: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/36.jpg)
Scr
een
Computing Parallax And clip space for free
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Depth
Convergence
Inte
raxi
al
Parallax
![Page 37: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/37.jpg)
Parallax in normalized image space
Para
llax
in n
orm
aliz
ed im
age
spac
e
Depth
Separation
Co
nve
rgen
ce
Parallax diverges quickly to negative infinity for object closer to the eye
Parallax is 0 at screen depth
Maximum Parallax at infinity is separation distance between the eyes
ππππππππ₯ = πππππππ‘πππ Γ 1 βπΆπππ£πππππππ
π·πππ‘β
![Page 38: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/38.jpg)
REAL EYE SEPARATION
Take care of your audience
![Page 39: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/39.jpg)
Real Eye Separation
Interocular (distance between the eyes) is on
average 2.5β 6.5 cm
Equivalent to the visible parallax on screen for
objects at infinity
Depending on the screen width, we define a
normalized βReal Eye Separationβ
Different for each screen model
A reference maximum value for
the Separation used in the stereo
projection for a comfortable experience
Real Screen
Screen Width
Interocular
Parallax at infinity
π πππ πΈπ¦π πππππππ‘πππ = πΌππ‘πππππ’πππ
π πππ ππππππ ππππ‘β
![Page 40: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/40.jpg)
Real Eye Separation is infinity
The maximum Parallax at infinity is
Separation
Real Eye Separation should be used as
the very maximum Separation value
Real Screen
πππππππ‘πππ < π πππ πΈπ¦π πππππππ‘πππ
![Page 41: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/41.jpg)
Separation must be Comfortable
Never make the viewer look diverge
People donβt have the same eyes
For Animation movie, separation must
be very conservative because of the
variety of the screen formats
IMAX vs Home theatre
For Interactive application, let the user
adjust Separation
When the screen is close to the user (PC
scenario) most of the users cannot handle
more than 50% of the Real Eye Separation
Real Screen
![Page 42: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/42.jpg)
Real Eye Separation is the Maximum Parallax
Real Screen Real Screen
πππ ( ππππππππ₯) < π πππ πΈπ¦π πππππππ‘πππ
![Page 43: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/43.jpg)
Safe Parallax Range
Para
llax
Depth
Separation 1
Co
nve
rgen
ce
Separation 2
Real Eye Separation
-Real Eye Separation
πππ ( ππππππππ₯) < π πππ πΈπ¦π πππππππ‘πππ
![Page 44: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/44.jpg)
PARALLAX BUDGET
Convergence and Separation working together
![Page 45: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/45.jpg)
Nearest pixel
Farthest pixel
Parallax budget
Para
llax
Depth
Separation C
on
verg
ence
Parallax Budget How much parallax variation is used in the frame
![Page 46: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/46.jpg)
In Screen : Farthest Pixel
At 100 * Convergence, Parallax is 99% of the Separation
For pixels further than 100 * Convergence,
Elements looks flat on the far distance with no depth differentiation
Between 10 to 100 * Convergence, Parallax vary of only 9%
Objects in that range have a subtle depth differentiation
Para
llax
Depth
Separation
Co
nve
rgen
ce
![Page 47: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/47.jpg)
Out of the Screen : Nearest pixel
At Convergence / 2, Parallax is equal to -Separation, out of the screen
Parallax is very large (> Separation) and can cause eye strains
Para
llax
Depth
Separation C
on
verg
ence
![Page 48: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/48.jpg)
Convergence sets the scene in the screen
Defines the window into the virtual space
Defines the style of stereo effect achieved (in / out of the screen)
Para
llax
Depth
Separation
Co
nve
rgen
ce 1
Far pixel Near pixel
Co
nve
rgen
ce 2
Parallax budget 1
Parallax budget 2
![Page 49: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/49.jpg)
Separation scales the parallax budget Scales the depth perception of the frame
Para
llax
Depth
Separation 1
Co
nve
rgen
ce
Separation 2
Parallax budget 1
Far pixel Near pixel
Parallax budget 2
![Page 50: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/50.jpg)
Adjust Convergence
Convergence is a Camera parameter driven by the look of the
frame
Artistic / Gameplay decision
Should adjust for each camera shot / mode
Make sure the scene elements are in the range
[ Convergence / 2, 100 * Convergence ]
Adjust it to use the Parallax Budget properly
Dynamic Convergence is a bad idea
Except for specific transition cases
![Page 51: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/51.jpg)
Stereopsis and 3D Hints
![Page 52: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/52.jpg)
Depth as storytelling tool
![Page 53: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/53.jpg)
Agenda
9:00 Welcome
9:05 Stereoscopy fundamentals and depth perception Samuel Gateau
9:40 Stereopsis and 3D hints Marc Salvati
10:10 Depth as storytelling tool Robert Neuman
10:30 Real-time & gaming techniques Samuel Gateau
11:00 Cartoon Authoring for 3D Marc Salvati
11:30 Creative choices for 3D Robert Salvati
11:50 Managing a depth budget Robert Salvati
12:10 Questions
Course available at
http://developer.nvidia.com/siggraph-2011-stereoscopy-course
![Page 54: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/54.jpg)
Real-time & Gaming Techniques
![Page 55: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/55.jpg)
RENDERING IN STEREO
Letβs do it
![Page 56: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/56.jpg)
Stereoscopic Rendering
Render geometry twice Do stereo drawcalls Duplicate drawcalls
From left and right eyes Apply stereo projection Modify projection matrix
Into left and right images Use stereo surfaces Duplicate render surfaces
![Page 57: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/57.jpg)
How to implement stereo projection ?
Start from the mono transformation stack
Inject the side, separation and convergence to get a stereo transformation
stack
Pixel
Shader
Rasterization Vertex Shader
World
space
Eye
space
Clip
space
Normalized
space
Image
space
View
Transform
Projection
Transform
Perspective
Divide
Viewport
Transform
Stereo Projection Matrix
Pixel
Shader
Rasterization Vertex Shader
Eye
Space
Stereo Clip
space
Stereo
Normalized
space
Stereo
Image
space
Stereo
Projection
Transform
Perspective
Divide
Viewport
Transform β¦ β¦
Pixel
Shader
Rasterization Vertex Shader
Clip
space
Stereo Clip
space
Stereo
Normalized
space
Stereo
Image
space
Stereo
Separation
Perspective
Divide
Viewport
Transform β¦ Eye
space
Projection
Transform β¦
Stereo shift on clip position
![Page 58: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/58.jpg)
Stereo Projection Matrix Right handed column major matrix ( OpenGL style )
Modified version of the Projection matrix for stereo to
transform geometry position from eye space to stereo clip
space
πππ ππππ π π‘ππππ = π·ππππππππππ π‘ππππ Γ πππ ππ¦π
Right handed column major matrix ( OpenGL style )
π·ππππππππππ π‘ππππ =
π11 0 π13 β π πππ β π ππππππ‘πππ βπ πππ β π ππππππ‘πππ β ππππ£πππππππ 0 π22 π23 00 0 π33 π340 0 β1 0
Side is -1 for left , +1 for right pij are the coefficients of the standard mono perspective projection
![Page 59: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/59.jpg)
Stereo Projection Matrix Left handed row major matrix ( D3D9 style )
πππ ππππ π π‘ππππ = πππ ππ¦π Γ π·ππππππππππ π‘ππππ
Left handed row major matrix ( D3D9 style )
π·ππππππππππ π‘ππππ =
π11 0 0 00 π22 π32 0
π13 + π πππ β π ππππππ‘πππ 0 π33 1βπ πππ β π ππππππ‘πππ β ππππ£πππππππ 0 π34 0
Side is -1 for left , +1 for right pij are the coefficients of the standard mono perspective projection
![Page 60: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/60.jpg)
Stereo shift on clip position
Just before rasterization in the vertex shader, offset the clip position by the
parallax amount
πππππππ . π₯ += ππππ β πππππππ‘πππ β ( πππππππ . π€ β πΆπππ£πππππππ )
Side is -1 for left, +1 for right
Pixel
Shader
Rasterization Vertex Shader
Clip
space
Stereo Clip
space
Stereo
Normalized
space
Stereo
Image
space
Stereo
Separation
Perspective
Divide
Viewport
Transform β¦ Eye
space
Projection
Transform β¦
![Page 61: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/61.jpg)
Screen Left Image
Right Image
Stereo rendering surfaces
View dependent render targets
must be duplicated
Back buffer
Depth Stencil buffer
Intermediate full screen render
targets used to process final
image
High dynamic range, Blur, Bloom
Screen Space Ambient Occlusion
Right Image
Left Image
![Page 62: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/62.jpg)
Mono rendering surfaces
View independent render targets DONβT need to be
duplicated
Shadow map
Spot light maps projected in the scene
Screen
![Page 63: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/63.jpg)
How to do the stereo drawcalls ?
Simply draw the geometries twice, in left and right versions of stereo surfaces
Can be executed per scene pass
Draw left frame completely
Then Draw right frame completely
Need to modify the rendering loop
Or for each individual objects
Bind Left Render target, Setup state for left projection, Draw geometry
Bind Right render target, Setup state for right projection, Draw Geometry
Might be less intrusive in an engine
Not everything in the scene needs to be drawn
Just depends on the render target type
![Page 64: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/64.jpg)
When to do what?
Use Case Render Target
Type Stereo Projection Stereo Drawcalls
Shadow maps Mono No
Use Shadow projection Draw Once
Main frame
Any Forward rendering pass Stereo Yes Draw Twice
Reflection maps Stereo
Yes
Generate a stereo
reflection projection
Draw Twice
Post processing effect
(Drawing a full screen quad) Stereo
No
No Projection needed at all Draw Twice
Deferred shading lighting
pass
(Drawing a full screen quad)
Stereo
G-buffers
Yes
Be careful of the
Unprojection
Should be stereo
Draw twice
![Page 65: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/65.jpg)
Real-time technique
![Page 66: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/66.jpg)
EVERYTHING IS UNDER CONTROL
What could go possibly wrong ?
![Page 67: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/67.jpg)
3D Objects
All the 3D objects in the scene should be rendered using a unique
Perspective Projection in a given frame
All the 3D objects must have a coherent depth relative to the
scene
Lighting effects are visible in 3D so should be computed correctly
Highlight and specular are probably best looking evaluated with mono eye
origin
Reflection and Refraction should be evaluated with stereo eyes
![Page 68: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/68.jpg)
Pseudo 3D objects : Sky box, Billboardsβ¦
Sky box should be drawn with a valid depth further than the
regular scene
Must be Stereo Projected
Best is at a very Far distance so Parallax is maximum
And cover the full screen
Billboard elements (Particles, leaves ) should be rendered in a
plane parallel to the viewing plane
Doesnβt look perfect
Relief mapping cannot be abused
![Page 69: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/69.jpg)
Several 3D scenes
Different 3D scenes rendered in the same frame using different
scales
Portrait viewport of selected character
Split screen
Since scale of the scene is different, Must use a different
Convergence to render each scene
![Page 70: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/70.jpg)
Out of the screen objects
The userβs brain is fighting against the perception of hovering
objects out of the screen
Extra care must be taken to achieve a convincing effect
Objects should not be clipped by the edges of the window
Be aware of the extra horizontal guard bands
Move object slowly from inside the screen to the outside area to
give eyes time to adapt
Make smooth visibility transitions
No blinking
Realistic rendering helps
![Page 71: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/71.jpg)
2D Objects 2D object in depth attached to 3D anchor point
Starcraft2 screenshot , Courtesy of Blizzard
2D objects presenting User interface at screen
Billboards in depth Particles with 3D positions
![Page 72: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/72.jpg)
2D Objects must be drawn at a valid Depth
With no stereo projection
Head Up Display interface
UI elements
Either draw with no stereo projection or with stereo projection at Convergence
At the correct depth when interacting with the 3D scene
Labels or billboards in the scene
Must be drawn with stereo projection
Use the depth of the 3D anchor point used to define the position in 2D window
space
Needs to modify the 2D ortho projection to take into account Stereo
![Page 73: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/73.jpg)
2D to 3D conversion shader function
float4 2Dto3DclipPosition(
in float2 posClip : POSITION, // Input position in clip space
uniform float depth // Depth where to draw the 2D object
) : POSITION // Output the position in clip space
{
return float4(
posClip.xy * depth, // Simply scale the posClip by the depth
// to compensate for the division by W
// performed before rasterization
0, // Z is not used if the depth buffer is not used
// If needed Z = ( depth * f β nf )/(f β n);
// ( For DirectX )
depth ); // W is the Z in eye space
}
![Page 74: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/74.jpg)
Selection, Pointing in S3D
Selection or pointing UI interacting with the 3D scene donβt work if drawn
mono
Mouse Cursor at the pointed objectβs depth
Can not use the HW cursor
Crosshair
Needs to modify the projection to take into account depth of pointed
elements
Draw the UI as a 2D element in depth at the depth of the scene where pointed
Compute the depth from the Graphics Engine or eval on the fly from the depth
buffer (Contact me for more info)
Selection Rectangle is not perfect, could be improved
Cf nvidia talk at GDC 2008
![Page 75: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/75.jpg)
Performance considerations
At worse the frame rate is divided by 2
But applications are rarely GPU bound so less expensive in practice
Since using Vsynch when running in stereo, you see the standard Vsync frequence
jumps
Not all the rendering is executed twice (Shadow maps)
Memory is allocated twice for all the stereo surfaces
Try to reuse render targets when possible to save memory
Get another GPU
![Page 76: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/76.jpg)
STEREO CULLING
![Page 77: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/77.jpg)
3D Objects Culling
When culling is done against the mono frustumβ¦
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Screen Left Frustum
Right Frustum
Mono Frustum
![Page 78: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/78.jpg)
3D Objects Culling
β¦ Some in screen regions are missing in the right and left frustum β¦
They should be visible
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Screen Left Frustum
Right Frustum
Mono Frustum
![Page 79: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/79.jpg)
3D Objects Culling
β¦ And we donβt want to see out of the screen objects only in one eye β¦
It disturbs the stereo perception
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Screen Left Frustum
Right Frustum
Mono Frustum
![Page 80: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/80.jpg)
3D Objects Culling
Here is the frustum we want to use for culling
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Screen Left Frustum
Right Frustum
Mono Frustum
![Page 81: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/81.jpg)
Z
Y
X
Eye space Left Eye
Right Eye
Mono Eye
Screen Left Frustum
Right Frustum
Mono Frustum
3D Objects Culling Computing Stereo Frustum origin offset
Z = Convergence / ( 1 + 1 / Separation )
Z
Interaxial
Convergence
Screen Width
![Page 82: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/82.jpg)
3D Objects Culling
Culling this area is not always a good idea
Blacking out pixels in this area is better
Through a shader
Equivalent to the βFloating windowβ used
in movies
Left Eye
Right Eye
Mono Eye
Screen Left Frustum
Right Frustum
Mono Frustum
![Page 83: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/83.jpg)
STEREO TRANSFORM STACK TRICKS
![Page 84: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/84.jpg)
Fetching Stereo Render Target
When fetching from a stereo render target use the good texture coordinate
Render target is addressed in STEREO IMAGE SPACE
Use the pixel position provided in the pixel shader
Or use a texture coordinate computed in the vertex shader correctly
Pixel Shader
β¦ Stereo Image
Space POSITION.xy
Fetch Texel at
POSITION.xy
Do something with it
Stereo Render Target
Pixel Shader
β¦ Mono Image
Space uv
Fetch Texel at uv
Do something with it
Stereo Render Target
![Page 85: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/85.jpg)
Unprojection in pixel shader When doing deferred shading technique, Pixel shader fetch the depth buffer
(beware of the texcoord used, cf previous slide)
And evaluate a 3D clip position from the Depth fetched and XY viewport position
Make sure to use a Stereo Unprojection Inverse transformation to go to Mono Eye
space
Otherwise you will be in a Stereo Eye Space !
Pixel Shader
Stereo Image Space
POSITION.xy
Image space
Normalized space
Clip space
Mono Eye space
Fetch Depth at
POSITION.xy
Viewport Inverse
Transform
Perspective Multiply
Stereo Projection
Inverse Transform
Evaluate Image Space
Position
Stereo Depth Buffer
Pixel Shader
Stereo Image Space
POSITION.xy
Image space
Normalized space
Clip space
Stereo Eye space
Fetch Depth at
POSITION.xy
Viewport Inverse
Transform
Perspective Multiply
Mono Projection
Inverse Transform
Evaluate Image Space
Position
Stereo Depth Buffer
![Page 86: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/86.jpg)
SELECTION IN S3D
Whatβs under this screen pixel ?
![Page 87: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/87.jpg)
Right Depth buffer
From stereo depth buffers to parallax Aka, Whatβs under that cursor ?
Given the left and
right depth buffers
A pixel position in the screen
(Cursor)
How to find the unique
fragment of the scene under
that pixel like we would do in
the mono case ?
Left Depth Buffer
Right Depth buffer
Left Depth Buffer
screen
![Page 88: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/88.jpg)
From stereo depth buffers to parallax
There is a unique solution in
mono
which is not trivial in stereoβ¦
Virtual Screen
Eye Space
Z Y
X
Scene
Right Depth buffer
Left Depth Buffer
screen
![Page 89: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/89.jpg)
From stereo depth buffers to parallax
The fragments are different at
the Cursor position in left and
right buffer
Virtual Screen
Eye Space
Z Y
X
Scene
Right Depth buffer
Left Depth Buffer
screen
![Page 90: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/90.jpg)
From stereo depth buffers to parallax
Correct left and right cursor
locations
Are pointing at the same scene
fragment
Are shifted away from the mono
position from Parallax
Virtual Screen
Eye Space
Z Y
X
Scene
Right Depth buffer
Left Depth Buffer
screen
![Page 91: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/91.jpg)
Left Depth Buffer
From stereo depth buffers to parallax
Parallax is bounded in a given range of
pixels [MinParallax, MaxParallax]
Deduced From the range [near, far]
So we know where to look in the depth
buffers
Correct location for the left & right pixels
is in the neighborhood of the mono pixel
Now we need a technique to find the
correct solution in left and right depth
buffers in this area
Right Depth buffer
screen
Max parallax
Min parallax
Search area
![Page 92: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/92.jpg)
From stereo depth buffers to parallax
Search area in each buffer is only half of the total parallax
range and symmetrical around the mono pixel
Look into pixel segment from the depth buffers
Left Depth Buffer Right Depth Buffer Right depth buffer pixel row Left depth buffer pixel row
![Page 93: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/93.jpg)
From stereo depth buffers to parallax
The left and right pixels over the same scene fragment
Are horizontally at the same distance away from the mono pixel
because they should be shifted by the same half parallax
And the 2 depths found should be equal and evaluate to the
correct half parallax
Right depth buffer pixel row Left depth buffer pixel row Left Depth Buffer Right Depth Buffer
![Page 94: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/94.jpg)
From stereo depth buffers to parallax
Start search from the mono pixel
Progress on both sides pixel by pixel to find the one where
Right depth buffer pixel row Left depth buffer pixel row Left Depth Buffer Right Depth Buffer
X offset (Left) X offset (Right)
ππππππππ₯ ππππ‘β = πππππ ππ‘ πππππ ππ‘ππππ‘ = πππππ ππ‘πππβπ‘
![Page 95: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/95.jpg)
From stereo depth buffers to parallax
Min parallax could be negative
Scene out of the screen
Look into both directions around the mono pixel
Right depth buffer pixel row Left depth buffer pixel row Left Depth Buffer Right Depth Buffer
X offset (Left) X offset (Right)
![Page 96: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/96.jpg)
From stereo depth buffers to parallax
Start search from the mono pixel
Progress on both side pixel by pixel to find the one where
Parallax( Depth ) / 2 = Xoffset
Xoffset Right
Right depth buffer pixel row Left depth buffer pixel row Left Depth Buffer Right Depth Buffer
X offset (Left) X offset (Right)
![Page 97: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/97.jpg)
From stereo depth buffers to parallax
Start search from the mono pixel
Right depth buffer pixel row Left depth buffer pixel row Left Depth Buffer Right Depth Buffer
![Page 98: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/98.jpg)
From stereo depth buffers to parallax
Progress pixel by pixel to find the one where
Parallax( depth ) / 2 = X offset
Right depth buffer pixel row Left depth buffer pixel row
X offset (Left) X offset (Right)
Parallax( Depth )
![Page 99: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/99.jpg)
Cartoon authoring for 3D
![Page 100: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/100.jpg)
Creative choices for 3D
![Page 101: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/101.jpg)
Managing a depth budget
![Page 102: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/102.jpg)
Special Thanksβ¦
![Page 103: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/103.jpg)
![Page 104: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/104.jpg)
![Page 105: Stereoscopy, From XY to Z](https://reader031.vdocuments.net/reader031/viewer/2022020703/61fb469f2e268c58cd5c3faf/html5/thumbnails/105.jpg)
Course available at http://developer.nvidia.com/siggraph-2011-stereoscopy-course Ping us for any question at [email protected]
Questions