Rafał MantiukDept. of Computer Science and Technology, University of Cambridge
High dynamic range
in VR
These slides are a part of the tutorial
Cutting-edge VR/AR Display Technologies (Gaze-, Accommodation-,
Motion-aware and HDR-enabled)
Presented at IEEE VR in Reutlingen on the 18th of March 2018
The latest version of the slides and the slides for the remaining
part of the tutorial can be found at:
https://vrdisplays.github.io/ieeevr2018/
Material is copyright © Rafał Mantiuk, 2018, except where
otherwise noted.
Rafał Mantiuk, Univ. of Cambridge3
HDR & VR ?
Do we have HDR VR headsets?
OLED contrast 1,000,000:1
Rafał Mantiuk, Univ. of Cambridge5
http://www.oculusvr.com/
ToC & Benefits
HDR in a nutshell
Display technologies in VR
Perception & image quality
Rafał Mantiuk, Univ. of Cambridge6
Slide 7
Dynamic range
max L
min L
(for SNR>3)
Luminance
Rafał Mantiuk, Univ. of Cambridge
Dynamic range (contrast)
As ratio:
Usually written as C:1, for example 1000:1.
As “orders of magnitude”
or log10 units:
As stops:
C =Lmax
Lmin
C10 = log10
Lmax
Lmin
C2 = log2
Lmax
Lmin
One stop is doubling
of halving the amount of light
8 Rafał Mantiuk, Univ. of Cambridge
High dynamic range (HDR)
Luminance [cd/m2]
10-6 10-4 10-2 100 102 104 106 108 Dynamic
Range
1000:1
1500:1
30:1
9 Rafał Mantiuk, Univ. of Cambridge
Visible colour gamut
The eye can perceive more colours
and brightness levels than
a display can produce
a JPEG file can store
The premise of HDR:
Visual perception and not the
technology should define accuracy
and the range of colours
The current standards not fully
follow to this principle
10 Rafał Mantiuk, Univ. of Cambridge
Luminous efficiency function
(weighting)
Light spectrum (radiance)
Luminance
Luminance – how bright the surface will appear
regardless of its colour. Units: cd/m2
Luminance
11
𝐿𝑉 = න
350
700
𝑘𝐿 𝜆 𝑉 𝜆 𝑑𝜆 𝑘 =1
683.002
Rafał Mantiuk, Univ. of Cambridge
Luminance and Luma
Luminance
Photometric quantity
defined by the spectral
luminous efficiency function
L ≈ 0.2126 R + 0.7152 G +
0.0722 B
Units: cd/m2
Luma
Gray-scale value computed
from LDR (gamma
corrected) image
Y = 0.2126 R’ + 0.7152 G’+ 0.0722 B’
R’ – prime denotes gamma
correction
Unitless
R ' = R1/g
12 Rafał Mantiuk, Univ. of Cambridge
Linear vs. gamma-corrected values
Rafał Mantiuk, Univ. of Cambridge13
Sensitivity to luminance
Weber-law – the just-noticeable difference
is proportional to the magnitude of a
stimulus
The smallest
detectable
luminance
difference
Background
(adapting)
luminance
Constant
L
ΔLTypical stimuli:
Ernst Heinrich Weber[From wikipedia]
14 Rafał Mantiuk, Univ. of Cambridge
Consequence of the Weber-law
Smallest detectable difference in luminance
Adding or subtracting luminance will have different visual
impact depending on the background luminance
Unlike LDR luma values, luminance values are not
perceptually uniform!
L ΔL
100 cd/m2 1 cd/m2
1 cd/m2 0.01 cd/m2
15
For k=1%
Rafał Mantiuk, Univ. of Cambridge
How to make luminance (more)
perceptually uniform?
Using “Fechnerian” integration
luminance - L
response
-R
1
ΔL
dR
dl(L) =
1
DL(L)Derivative of
responseDetection
threshold
R(L) 1
L(l)dl
0
L
Luminance
transducer:
16 Rafał Mantiuk, Univ. of Cambridge
Assuming the Weber law
and given the luminance transducer
the response of the visual system to light is:
R(L) 1
L(l)dl
0
L
17 Rafał Mantiuk, Univ. of Cambridge
Fechner law
Response of the visual system to luminance
is approximately logarithmic
The values of HDR pixel values are much
more intuitive when they are plotted /
considered / processed in the logarithmic
domain
Gustav Fechner[From Wikipedia]
R(L) aln(L)
18 Rafał Mantiuk, Univ. of Cambridge
But…the Fechner law does not hold for
the full luminance range
Because the Weber law does not hold either
Threshold vs. intensity function:
L
ΔL
The Weber law
region
19 Rafał Mantiuk, Univ. of Cambridge
Weber-law revisited
If we allow detection threshold to vary with luminance
according to the t.v.i. function:
we can get more accurate estimate of the “response”:
R(L) =1
tvi(l)dl
0
L
ò
L
ΔL tvi(L)
20 Rafał Mantiuk, Univ. of Cambridge
Fechnerian integration and Stevens’ law
21
R(L) - function
derived from the
t.v.i. function
R(L) =1
tvi(l)dl
0
L
ò
Rafał Mantiuk, Univ. of Cambridge
Tone-mapping problem
luminance range [cd/m2]
conventional display
simultaneouslyhuman vision
adapted
Tone mapping
22 Rafał Mantiuk, Univ. of Cambridge
Why do we need tone mapping?
To reduce excessive dynamic range
To customize the look (colour grading)
To simulate human vision
for example night vision
To adapt displayed images to a display and viewing
conditions
To make rendered images look more realistic
Different tone mapping techniques achieve different goals
Rafał Mantiuk, Univ. of Cambridge23
Tone-mapping in rendering
Any physically-based
rendering requires tone-
mapping
“HDR rendering” in games is
pseudo-physically-based
rendering
Goal: to simulate a camera or
the eye
Greatly enhances realism
24
LDR illumination
No tone-mapping
HDR illumination
Tone-mapping
Half-Life 2: Lost coast
Rendering
engine
Simulate
Linear RGB
Map
Rafał Mantiuk, Univ. of Cambridge
Tone-curve
Image histogram
Best tone-
mapping is the
one which does
not do anything,
i.e. slope of the
tone-mapping
curves is equal
to 1.
25 Rafał Mantiuk, Univ. of Cambridge
Tone-curve
But in practice
contrast (slope)
must be limited
due to display
limitations.
26 Rafał Mantiuk, Univ. of Cambridge
Tone-curve
Global tone-
mapping is a
compromise
between clipping
and contrast
compression.
27 Rafał Mantiuk, Univ. of Cambridge
Sigmoidal tone-curves
Very common in
digital cameras
Mimic the response
of analog film
Analog film has been
engineered over many
years to produce
good tone-reproduction
Fast to compute
Rafał Mantiuk, Univ. of Cambridge28
Tone-curve as an optimization problemGoal: Minimize the visual difference between
the input and displayed images
29 Rafał Mantiuk, Univ. of Cambridge
Illumination &
reflectance separation
Input
Illumination
Reflectance
30
Y = I ×RImage
Illumination Reflectance
Illumination and reflectance
Reflectance Illumination
White ≈ 90%
Black ≈ 3%
Dynamic range < 100:1
Reflectance critical for
object & shape detection
Sun ≈ 109 cd/m2
Lowest perceivable
luminance ≈ 10-6 cd/m2
Dynamic range 10,000:1 or
more
Visual system partially
discounts illumination
31 Rafał Mantiuk, Univ. of Cambridge
Reflectance & Illumination TMO
Hypothesis: Distortions in reflectance are more apparent
than the distortions in illumination
Tone mapping could preserve reflectance but compress
illumination
for example:
Tone-mapped image
Reflectance
Illumination
Tone-mapping
white
c
whited LLIRL )/(
32 Rafał Mantiuk, Univ. of Cambridge
𝐿𝑑 = 𝑅 ∙ 𝑇(𝐼)
How to separate the two?
(Incoming) illumination – slowly changing
except very abrupt transitions on shadow boundaries
Reflectance – low contrast and high frequency variations
33 Rafał Mantiuk, Univ. of Cambridge
Bilateral filter
Better preserves sharp edges
Still some blurring on the
edges
Reflectance is not perfectly
separated from illumination
near edges
Tone mapping result
[Durand & Dorsey, SIGGRAPH 2002]34 Rafał Mantiuk, Univ. of Cambridge
Glare
“Alan Wake” © Remedy Entertainment
35 Rafał Mantiuk, Univ. of Cambridge
Glare Illusion
36
PaintingPhotography
Computer Graphics
HDR rendering in gamesRafał Mantiuk, Univ. of Cambridge
Scattering of the light in the eye
From: Sekuler, R., and Blake, R. Perception, second ed. McGraw- Hill, New York, 1990
37 Rafał Mantiuk, Univ. of Cambridge
Ciliary corona and lenticular halo
*
=
=+ From: Spencer, G. et al.
1995. Proc. of
SIGGRAPH. (1995)
38 Rafał Mantiuk, Univ. of Cambridge
Examples of simulated glare
[From Ritschel et al, Eurographics 2009]
39 Rafał Mantiuk, Univ. of Cambridge
Glare (or bloom) in games
Convolution with large, non-separable filters is too slow
The effect is approximated by a combination of Gaussian
filters
Each filter with different “sigma”
The effect is meant to look good, not be be accurate
model of light scattering
Some games simulate
camera rather than the eye
40 Rafał Mantiuk, Univ. of Cambridge
41 Rafał Mantiuk, Univ. of Cambridge
42 Rafał Mantiuk, Univ. of Cambridge
Simulation of night vision [Wanat 2014]
Age-adaptive night vision
43 Rafał Mantiuk, Univ. of Cambridge
ToC & Benefits
HDR in a nutshell
Display technologies in VR
Perception & image quality
Rafał Mantiuk, Univ. of Cambridge44
VR display technologies
TFT-LCD AMOLED
Contrast: <3000:1
Transmissive
Complex temporal
response
Arbitrary bright
Constant power at
constant backlight
Contrast: >10,000:1
Emmisive
Rapid response
Brightness affects longevity
Power varies with image
content
Rafał Mantiuk, Univ. of Cambridge45
TN, STN, MVA,
PVA, IPS
LCD
color may change with the viewing angle
contrast up to 3000:1
higher resolution results in smaller fill-factor
color LCD transmits only up to 8% (more often close to
3-5%) light when set to full white
TN LCD
Rafał Mantiuk, Univ. of Cambridge46
LCD temporal response
Experiment on an IPS LCD screen
We rapidly switched between two
intensity levels at 120Hz
Measured luminance integrated
over 1s
The top plot shows the difference
between expected (𝐼𝑡−1+𝐼𝑡
2) and
measured luminance
The bottom plot: intensity
measurement for the full
brightness and half-brightness
display settings
Rafał Mantiuk, Univ. of Cambridge47
OLED
based on
electrophosphorescence
large viewing angle
the power consumption
varies with the brightness of
the image
fast (< 1 microsec)
arbitrary sizes
life-span is a concern
more difficult to produce
Rafał Mantiuk, Univ. of Cambridge48
Low persistence displays
Most VR displays flash an
image for a fraction of
frame duration
This reduces hold-type
blur
And also reduces the
perceived lag of the
rendering
Rafał Mantiuk, Univ. of Cambridge49
HT
C V
ive
Ma
te 9
Pro
+ D
ayD
ream
Lens in VR displays
Aberrations when viewing off-center
Chromatic aberration
Loss of resolution
Difficult to eliminate if the exact eye
position is unknown
Glare
Scattering of the light in the lens
From Fresnel fringes
Reduces dynamic range
Rafał Mantiuk, Univ. of Cambridge50
Exa
mp
les fro
m: h
ttp://d
oc-o
k.o
rg/?
p=
14
14
High resolution
Colour Image
High Dynamic
Range Display
HDR Display
• Modulated LED array
• Conventional LCD
• Image compensationLow resolution
LED Arrayx =
Rafał Mantiuk, Univ. of Cambridge51
HDR display
Desired
image
LCD imageDLP image
DLP blur
(PSF)Subject to:
52
Resolution
Relevant units: pixels per visual degree [ppd]
Nyquist frequency in cycles per degree = ½ of ppd
PC & mobile resolution
1981: 12” 320x200 monitor @50cm: 10.9 ppd
1990: 12” 1024x768 monitor @50cm: 37 ppd
2011: 3.5” 960x640 iPhone @30cm: 68 ppd
2016: 31” 4K monitor @50cm: 50 ppd
2018: 6” phone @30cm: 117 ppd
VR resolution
2016 HTC Vive: 10 ppd
2018 HTC Vive Pro: 13 ppd
Rafał Mantiuk, Univ. of Cambridge53
ToC & Benefits
HDR in a nutshell
Display technologies in VR
Perception & image quality
Rafał Mantiuk, Univ. of Cambridge54
Contrast Sensitivity Function
Detection of barely
noticeable contrast on a
uniform background
Varies with luminance
CSF is NOT MTF of
visual system
Contrast constancy
There is little variation in
magnitude of perceived
contrast above the
detection threshold
Rafał Mantiuk, Univ. of Cambridge55
Ge
org
eson
an
d S
ulli
va
n.
19
75
. J. P
hsysio
iPhone 4
Retina display
HTC Vive
Retinal velocity
Sensitivity drops rapidly once
images start to move
The eye tracks moving objects
Smooth Pursuit Eye Motion
(SPEM)
Stabilizes images on the retina
But tracking is not perfect
Loss of sensitivity mostly caused
by imperfect SPEM
SPEM worse at high velocities
Motion sharpenning
Relatively small effect
Rafał Mantiuk, Univ. of Cambridge56
Spatio-velocity contrast sensitivity
Kelly’s model [1979]
Hold-on blur
The eye smoothly follows a moving object
But the image on the display is “frozen” for 1/60th of a
second
Rafał Mantiuk, Univ. of Cambridge57
Real-w
orld
Perf
ect
motion
Physical image + eye motion + temporal integration
Hold-on blur
The eye smoothly follows a moving object
But the image on the display is “frozen” for 1/60th of a
second
Rafał Mantiuk, Univ. of Cambridge58
60 H
z d
ispla
y
Physical image + eye motion + temporal integration
Hold-on blur
The eye smoothly follows a moving object
But the image on the display is “frozen” for 1/60th of a
second
Rafał Mantiuk, Univ. of Cambridge59
Bla
ck f
ram
e insert
ion
Physical image + eye motion + temporal integration
Flicker
Critical Flicker Frequency
Strongly depends on
luminance – big issue for
HDR VR headsets
Increases with eccentricity
and stimulus size
It is possible to detect
flicker even at 2kHz
For saccadic eye motion
Rafał Mantiuk, Univ. of Cambridge60
[Hartmann et al. 1979]
Simulation sickness
Conflict between vestibular
and visual systems
When camera motion
inconsistent with head motion
Frame of reference (e.g.
cockpit) helps
Worse with larger FOV
Worse with high luminance
and flicker
Rafał Mantiuk, Univ. of Cambridge61
Summary
HDR for VR is a great idea because
It gives more realistic experience
Better quality with the same number of pixels
Additional depth cues
HDR for VR is bad idea because
Increased flicker visibility
Increased simulation sickness
Lens glare will reduce effective dynamic range
In both cases
Tone-mapping will become an important part of VR rendering
Rafał Mantiuk, Univ. of Cambridge62
References Concise overview of high dynamic range imaging
Mantiuk, R. K., Myszkowski, K., & Seidel, H. (2015). High Dynamic Range Imaging. In Wiley Encyclopedia of
Electrical and Electronics Engineering (pp. 1–42). Hoboken, NJ, USA: John Wiley & Sons, Inc.
https://doi.org/10.1002/047134608X.W8265
Downloadable PDF: http://www.cl.cam.ac.uk/~rkm38/pdfs/mantiuk15hdri.pdf
Comprehensive book on display technologies
Hainich, R. R., & Bimber, O. (2011). Displays: Fundamentals and Applications. CRC Press.
https://goo.gl/RLe8nA
Book on HDR Imaging
Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., & Myszkowski, K. (2010). High Dynamic
Range Imaging: Acquisition, Display, and Image-Based Lighting (2nd editio). Morgan Kaufmann.
Lecture on tone-mapping (Advanced Graphics, Univ. of Cambridge)
http://www.cl.cam.ac.uk/teaching/1718/AdvGraph/06_HDR_and_tone_mapping.pdf
http://www.cl.cam.ac.uk/teaching/1718/AdvGraph/materials.html
Simulation of night vision
Wanat, R., & Mantiuk, R. K. (2014). Simulating and compensating changes in appearance between day and
night vision. ACM Transactions on Graphics (Proc. of SIGGRAPH), 33(4), 147.
https://doi.org/10.1145/2601097.2601150
Rafał Mantiuk, Univ. of Cambridge63