motion perception in displays sfu...
TRANSCRIPT
Motion Perception in Displays
Motion Perception in Displays
Scott Daly
Dolby Laboratories
Scott Daly
Dolby Laboratories
Sim
on
Fra
ser
Un
iver
sity
Oct
. 2
011
��LCTV Basics
LCTV Basics
�Transm
ission m
odulation, Spatial , Color, etc.
��Basics of Spatiotemporal vision
Basics of Spatiotemporal vision
�Motion
�Eye m
ovements
�Eccentricity
��LCD Temporal Issues
LCD Temporal Issues
�Overdrive
�
Outline
�Dynamic Gamma
�Display Temporal Rendering Function
�Analysis of Temporal LCD approaches
��Perceptual Appearance: Motion Sharpness Effect
Perceptual Appearance: Motion Sharpness Effect
��Standardized Metrics
Standardized Metrics
��Conclusions/Summary
Conclusions/Summary
��What’s next:
What’s next:
�Other Temporal Artifacts
�What Does Motion Really Look Like?
LCTV Basics
LCTV Basics
Light Modulation via Liquid Crystals
�LCD is a “transm
issive” display
�Light is not created by the liquid crystals themselves
�A light source behind the panel shines through the display (CCFL, LED)
�Diffusion panel behind the LCD scatters and re-directs the light evenly
Glass Substrate
Transparent
Electrode
Alignment Layer
Backlight direction
Polarizing Filter
with Retardation Film
�Driving the Display
�2 polarizing transparent panels (One Vertical , One Horizontal)
�Liquid crystal solution sandwiched in betw
een
�Liquid crystals are rod-shaped m
olecules
-Bend light in response to an electric current
-Act like a shutter –allow light to pass through or block (or attenuate)
Alignment Layer
Liquid Crystals
Color Filter
Glass Substrate
RG
B
RG
BR
GB
Polarizing Filter w/ret
Pixels to resolution
45" LCD “full HDTV”
6.22 million pixels
�Physical resolution vs.
#Pixel Dim
ensions
�Full HD (1920 x 1080 progressive)
achieved in 2003
�Full HD now shown up to 108” for LCTV
0.5135mm
0.5135mm
1,920(H)×
RGB
×1,080(V) Progressive�4k x 2k pixel resolution shown by several
manufacturers (65”; 24 m
illion pixels)
�Usually, pixel physical resolution for
LCTV is near 45 ppi
Salient Characteristics of LCD: MTF & PSF
Width
Horizontal gap
Length Vertical gap
�LCD MTF does not vary with gray level or
spatial neighbors
�Rigid pixel via fixed aperture + steady
Backlight
�MTF is sinc function based on subpixel
dim
ensions
�color crosstalk correction sometimes needed Vertical gap
�CRT spatially nonlinear � ���
MTF hard to assess, use
�Spatial superadditivity in H direction :
�Spatial sub-additivity if power supply not
powerful enough
Salient Characteristics of LCD: MTF details
Sinc MTF for MI-600 (G plane)
MI-600 MTF and CSF
Width
Horizontal gap
��Comparison to visual system “MTF” = CSF
Comparison to visual system “MTF” = CSF
•Sinc is only a gradual LPF within HVS CSF “window”
•Viewing distance = 2000 pix
•~2H for HDTV, ~4H for VGA
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
0
0.2
0.4
0.6
0.81 H SF (cy/pix)
V SF (cy/pix)
MTF
Sinc MTF for MI-600 (G plane)
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0
0.2
0.4
0.6
0.81
H SF (cy/pix)
V SF (cy/pix)
MTF
Fig
ure
13A
: S
inc
MT
F f
or
MI-
600 F
igure
13B
: M
I-600 a
nd C
SF
Length Vertical gap
Current Challenges for LC TV
�High Dynamic Range at consumer cost
�Wide Color Gamut at consumer cost
�Ultra high resolution 4k x 2k and up
�Achieving perfect motion fidelity :
1. Speeding up response tim
e for pixels
•How fast a pixel can change color without blurring
•Currently <= 2-4 m
illiseconds cites, but not for all gray level transitions
2. Hold-response blur (problem with Plasm
a also)
3. Judder (frame rate issue, problem with CRTs and Plasm
a also)
�Human visual perception plays a role in perform
ance
Some Basics in Spatiotemporal Vision
Some Basics in Spatiotemporal Vision
Properties of the Visual System
Properties generally dissected along
Properties generally dissected along
these dim
ensions:
these dim
ensions:
•Luminance Level
•Spatial Frequency
•Local Spatial Content
•Temporal Frequency
•Temporal Frequency
•Motion
•Global Color
•Eccentricity
•Depth
Properties of the Visual System
Properties generally dissected along
Properties generally dissected along
these dim
ensions:
these dim
ensions:
•Spatial Frequency
• •Temporal Frequency
•Temporal Frequency
•Motion
• •
Engineering vs. Physiological Models of the Visual System
��Engineering Models of
Engineering Models of
visual behavior aim
visual behavior aim
for mathematical
for mathematical
descriptions of key
descriptions of key
functionality
functionality
��Psychophysics and
Psychophysics and
black
black--box m
odeling
box m
odeling
have gotten the m
ost
have gotten the m
ost
mileage for practical
mileage for practical
mileage for practical
mileage for practical
applications
applications
��While physiological
While physiological
plausibility is helpful,
plausibility is helpful,
simplification is
simplification is
desired
desired
��No need to m
odel
No need to m
odel
down to the
down to the
neurotransm
itter
neurotransm
itter
��How is more
How is more
important than where
important than where
Spatial Frequency
��Spatial behavior constant with visual
Spatial behavior constant with visual
angle (degrees)
angle (degrees)
��Spatial frequencies specified in
Spatial frequencies specified in
cycles/degree (cpd, cy/deg)
cycles/degree (cpd, cy/deg)
��Spatial frequency behavior
Spatial frequency behavior
described with
described with CSF
CSF(contrast
(contrast
sensitivity function)
sensitivity function)
•Sim
ilar to OTF of optics, MTF of
•Sim
ilar to OTF of optics, MTF of
electrical systems, but it is
nonlinear and adaptive
•Measured with psychophysics
��One of the m
ost useful, and widely
One of the m
ost useful, and widely
used properties of visual system
used properties of visual system
Campbell and Robson ’66
Campbell and Robson ’66
Spatial Frequency
��Spatial behavior constant with visual
Spatial behavior constant with visual
angle (degrees)
angle (degrees)
��Spatial frequencies specified in
Spatial frequencies specified in
cycles/degree (cpd, cy/deg)
cycles/degree (cpd, cy/deg)
��Spatial frequency behavior
Spatial frequency behavior
described with
described with CSF
CSF
(contrast sensitivity function)
(contrast sensitivity function)
•Sim
ilar to OTF of optics, MTF of
•Sim
ilar to OTF of optics, MTF of
electrical systems, but it is
nonlinear and adaptive
•Measured with psychophysics
��One of the m
ost useful, and widely
One of the m
ost useful, and widely
used properties of visual system
used properties of visual system
Campbell and Robson ’66
Campbell and Robson ’66
Spatial Frequency Sensitivity
Spatial Frequency Sensitivity
>100cd/m
2
0.0001 cd/m
2
Spatial Frequency Sensitivity
Log Contrast Sensitivity
Log Spatial Frequency
Log Contrast Sensitivity
Spatial Frequencyin Application
��The m
ax spatial frequency that can be displayed digitally is the Nyquist frequency
The m
ax spatial frequency that can be displayed digitally is the Nyquist frequency
��It is ½ the sampling frequency (e.g., 500 pixels can display at most 250 cycles/pixel)
It is ½ the sampling frequency (e.g., 500 pixels can display at most 250 cycles/pixel)
��Common m
ax frequency seen by humans (I.e, CSF) is 30 cy/deg for medium brightness
Common m
ax frequency seen by humans (I.e, CSF) is 30 cy/deg for medium brightness
•Highest m
ax ever seen is 60 cy/deg (very high brightness, Carlson @ RCA)
��Examples of visual Nyquist frequencies and viewing distances for common displays:
Examples of visual Nyquist frequencies and viewing distances for common displays:
�NTSC (425 lines) at 6H
(2550 pixels):
22 cy/deg
�NTSC (425 lines) at 6H
(2550 pixels):
22 cy/deg
�NTSC (425 lines) at 3H
(1275 pixels):
11
�XGA (1024x768) at 3H
(2304 pixels): 20
�SXGA (1280x1024) at 1H
(1024 pixels):
9
�1366 x 720 HDTV at 3H
(2160 pixels):
19
�Full HDTV (1920x1080) at 6H
(6480 pixels):
57
�Full HDTV (1920x1080) at 3H
(3240 pixels):
28
�Full HDTV (1920x1080) at 2H
(2160 pixels): 19
2D Spatial Frequency
��2D frequencies im
portant for im
ages
2D frequencies im
portant for im
ages
��2D CSF is
2D CSF is not
notrotationally symmetric (isotropic)
rotationally symmetric (isotropic)
��Lack of sensitivity near 45 degrees, called the oblique effect
Lack of sensitivity near 45 degrees, called the oblique effect
200
250
30
35
40
V spatial frequency (cy/deg)
2D Spatial CSF
10
20
30
10
20
30
0
50
100
150
200
H SF cpd
V SF cpd
S
510
15
20
25
30
35
40
5
10
15
20
25
H spatial frequency (cy/deg)
V spatial frequency (cy/deg)
Temporal Frequency
��CSF for temporal frequencies also
CSF for temporal frequencies also
has been m
easured and m
odeled
has been m
easured and m
odeled
��To right is shown temporal CSF for
To right is shown temporal CSF for
different light adaptation levels for
different light adaptation levels for
luminance
luminance
•Top curve is best for mid-bright
display applications
display applications
��Opponent Color
Opponent Colorsignals temporal
signals temporal
CSF also has about 1/2 the
CSF also has about 1/2 the
bandwidth and sensitivity of the
bandwidth and sensitivity of the
luminance
luminance
��DeLange 52, Kelly 60s
DeLange 52, Kelly 60s--70s, W
atson
70s, W
atson
80s
80s
Spatiotemporal Frequency
��Psychophysical data m
easurement of
Psychophysical data m
easurement of
spatio
spatio--temporal CSF is common
temporal CSF is common
•Robson 66
•Van Nes, Koenderinck, Bouman 67
•Kelly 79
•Kelly and Burbeck 80
0.51
Standing W
ave
•Kelly and Burbeck 80
��Test signal is product of spatial and
Test signal is product of spatial and
temporal frequency m
odulation
temporal frequency m
odulation
•Standing W
ave
•Counterphase flicker
0
200
400
0
2
4
6-1.5-1
-0.50
0.5 spatial position
time
Spatiotemporal CSF
��Spatiotemporal CSF (measured with counterphase flicker)
Spatiotemporal CSF (measured with counterphase flicker)
��Window of visibility
Window of visibility
��Data shows max visible temporal frequency (CFF) near 50 cy/sec
Data shows max visible temporal frequency (CFF) near 50 cy/sec
•CFF = Critical Fusion Frequency = m
ax temporal frequency that can be seen
1.52
2.53
log temporal frequency (cy/sec)
Spatiotemporal CSF
2.53
log Sensitivity
Spatiotemporal CSF
•D.H Kelly 79
•Koenderinck &
van Doorn 79
(bim
odal)
-1-0.5
00.5
11.5
2-1
-0.50
0.51
1.5
log spatial frequency(cy/deg)
log temporal frequency (cy/sec)
-1
0
1
2-1
0
1
20
0.51
1.52
log SF (cy/deg)
log TF (cy/sec)
log Sensitivity
(bim
odal)
•Burbeck &
Kelly 80
(excitatory-
inhibitory
separable
version)
��Thus 60 fps usually causes no visible flicker (foveal)
Thus 60 fps usually causes no visible flicker (foveal)
��Movie film at 24 fps causes visible flicker, so projectors shutter each
Movie film at 24 fps causes visible flicker, so projectors shutter each
frame 2 or 3 tim
es to increase fundamental temporal frequency
frame 2 or 3 tim
es to increase fundamental temporal frequency
•Before the 1920s , movies were called “the flickers’
Brightness and Light Adaptation effects on T-CSF
��Higher brightness
Higher brightness ��
Increase in peak sensitivity of temporal CSF
Increase in peak sensitivity of temporal CSF
��Higher brightness
Higher brightness ��
Increase in bandwidth of temporal CSF
Increase in bandwidth of temporal CSF
��CFF = Critical Fusion Frequency (CSF bandwidth cut
CFF = Critical Fusion Frequency (CSF bandwidth cut--off)
off)
Fer
ry-P
ort
er L
aw
Fer
ry-P
ort
er L
aw -
>
CFF and Eccentricity
��CFF= critical fusion frequency.
CFF= critical fusion frequency.
•Defined as frequency when 100% m
odulation signal looks identical to flat-field
•Viewer does not see any flicker
��For fovea and typical display light levels, CFF around 55 Hz
For fovea and typical display light levels, CFF around 55 Hz
��For periphery at same light levels, it can increase to over 80Hz
For periphery at same light levels, it can increase to over 80Hz
Motion
Motion and Retinal Velocity
��For objects in real
For objects in real
world, Velocity
world, Velocity
more important
more important
than flicker
than flicker
��Standing waves can
Standing waves can
be de
be de--composed
composed
into traveling waves
into traveling waves
��Smooth tracking eye
Smooth tracking eye
movements can
movements can
0100
200
300
400
500
-3-2-10123Standing W
ave = Two Opposing Travelling W
aves
spatial position
amplitude
SF=6/512 cy/pix TF=1/8 cy/sec V=(1/8)/(6/512)= 11 pix/sec
t=0 ; 0.5++ ; 4 secs
--> <--
movements can
movements can
reduce image
reduce image
velocity on the
velocity on the
retina
retina
��Spatiovelocity CSF
Spatiovelocity CSF
by W
atanabe ‘68
by W
atanabe ‘68
��Retinal Velocities &
Retinal Velocities &
stabilization
stabilization
��Retinal Velocity
Retinal Velocity
CSFs by Kelly from
CSFs by Kelly from
Motion & Vision
Motion & Vision
series 79
series 79
-1-0.5
00.5
11.5
20
0.51
1.52
2.53
log spatial frequency (cy/deg)
log visual sensitivity
CSFs for Different Retinal Velocities
0.001
0.012
0.15 deg/sec
3
11
32
64 128
200
Sampled Motion and the W
indow of Visibility
Watson, Ahumada, Farrell 86
Watson, Ahumada, Farrell 86
Sampled Motion and the W
indow of Visibility
Watson, Ahumada, Farrell 86
Watson, Ahumada, Farrell 86
��Rectangular support shown is window of visibility (idealized separable version)
Rectangular support shown is window of visibility (idealized separable version)
•Max spatial = 50 cy/deg (depending on conditions, well studied)
•Max temporal = 30 cy/sec (depending on conditions and visual eccentricity, well studied)
��Undersampled m
otion
Undersampled m
otion
��Replications due to sampling = temporal aliases
Replications due to sampling = temporal aliases
��Note: this would look awful
Note: this would look awful
Sampled Motion and the W
indow of Visibility
��Camera constrained window of visibility (not HVS)
Camera constrained window of visibility (not HVS)
��Aliasing vs. Blur tradeoffs at im
age capture via Temporal LPF prefilter
Aliasing vs. Blur tradeoffs at im
age capture via Temporal LPF prefilter
via exposure aperture length
via exposure aperture length
via illumination duration
via illumination duration
Andrew Davidhazy @ RIT
Andrew Davidhazy @ RIT
Sampled Motion and the W
indow of Visibility
Watson, Ahumada, Farrell 86
Watson, Ahumada, Farrell 86
��Example of sm
oothly perceived m
otion
Example of sm
oothly perceived m
otion
��Sampling rate increases spreads out replications
Sampling rate increases spreads out replications
��Preventing aliases in window of visibility results in smooth true m
otion
Preventing aliases in window of visibility results in smooth true m
otion
��Sampling rate depends on object speed and spatial content
Sampling rate depends on object speed and spatial content
•(I.e., bandwidth)
Sampled Motion and the W
indow of Visibility
��Watson, Ahumada, Farrell 86
Watson, Ahumada, Farrell 86
��Now that we have smooth m
otion by keeping aliases out of the window of
Now that we have smooth m
otion by keeping aliases out of the window of
visibility……
visibility……
��We still need to worry about motion blur due to capture aperture
We still need to worry about motion blur due to capture aperture
•Thus the use of shorter capture tim
e than the frame duration
Relations betw
een Temporal, Spatial, and Motion
)0,
,(
),
,(
ty
tx
lt
yx
ly
xυ
υ−
−=
)(
),
()
,,
(t
yy
xx
yx
ty
xf
ff
ff
Lf
ff
L+
+=
υυ
δ
�Its 3D Fourier spectrum is given by
�Translational motion can be defined as
),
,(
ty
xf
ff
L
0=
++
ty
yx
xf
ff
υυ
is n
on z
ero o
nly
on t
he
pla
ne
def
ined
by
�The m
otion of an object causes temporal component in the
spatiotemporal spectrum.
�The temporal component is proportional to spatial frequency and velocity
Relations betw
een Temporal, Spatial, and Motion and MTF
)(
).(
)(
)(
),
(t
xt
xx
xt
xs
fT
fT
ff
fL
ff
L+
=υ
δ
Sp
atio
-tem
pora
l sp
ectr
um
is
low
pas
s fi
lter
ed b
y t
he
ST
CS
F ,
as w
ell
as d
isp
lay M
TF
(co
mb
ined
ST
syst
em M
TF
: T
)
When
eye
accu
rate
ly t
rack
s th
e m
oti
on, th
e re
tinal
im
age
is
pure
ly s
pat
ial
)(
)(
)(
)(
xx
xx
xs
fT
fT
fL
fL
υ=
)(
)(
)(
xx
xx
df
Tf
Tf
Tυ
=
Sp
atia
l tr
ansf
er f
unct
ion d
ue
to d
isp
lay s
pat
io-t
emp
ora
l M
TF
Sp
atia
l M
TF
Tem
pora
l M
TF
Advanced Issues in Spatiotemporal Vision
Advanced Issues in Spatiotemporal Vision
Properties of Visual System: Motion:retinal velocity
��Retinal Velocities
Retinal Velocities
��No Eye Movements Occur
No Eye Movements Occur
��Image velocity = retinal velocity
Image velocity = retinal velocity
��Spatiovelocity CSF
Spatiovelocity CSF
(stabilized retina)
(stabilized retina)
-1-0.5
00.5
11.5
20
0.51
1.52
2.53
log spatial frequency (cy/deg)
log visual sensitivity
CSFs for Different Retinal Velocities
0.001
0.012
0.15 deg/sec
3
11
32
64 128
200
log spatial frequency (cy/deg)
-1
0
1
2-1
0
1
2-10123
log SF (cy/deg)
log V (deg/sec)
log visual sensitivity
Spatiovelocity CSF
-10
12
3-1
-0.50
0.51
1.52
2.53
log spatial frequency (cy/deg)
log velocity (deg/sec)
Spatiovelocity CSF
Properties of Visual System: Motion:Eye Movements
��Eye m
ovement’s tracking changes the window of visibility
Eye m
ovement’s tracking changes the window of visibility
��B. Girod 93, Perfect (and m
andatory) object tracking
B. Girod 93, Perfect (and m
andatory) object tracking
Properties of Visual System: Motion:Eye Movements
Types of eye m
ovements:
Types of eye m
ovements:
��Saccadic Eye m
ovements (jumps)
Saccadic Eye m
ovements (jumps)
•Usually > 160-300 deg/sec
•With larger display, larger saccades will still fit on
screen, giving m
ore of a feeling of being in real
world
��Smooth Pursuit Eye Movements
Smooth Pursuit Eye Movements
(tracking )
(tracking )
•80 deg/sec for 90 degree field of view (+-45 deg)
•30 deg/sec for 30 deg field of view
050
100
150
0
20
40
60
80
100
120
140
160
Target Velocity (deg/sec)
Eye Tracking Velocity (deg/sec)
Smooth Pursuit Eye Tracking
o - Observer KC
•Some retinal slippage (slope= 0.9)
��Drift Eye m
ovements (very small )
Drift Eye m
ovements (very small )
•responsible for the prevention of im
age fading due
to low S of spatial & temporal CSFs
•No expected consequences of large screen on these
•Approx 0.10 to 0.15 deg/sec
•Other sm
all eye m
ovements: Tremor,
Microsaccades
��Data from Meyer 85 : Smooth tracking data
Data from Meyer 85 : Smooth tracking data
•Red line is model we use for eye m
ovements
•-smooth pursuit + baseline drift as minim
um
Target Velocity (deg/sec)
10-1
100
101
102
10-1
100
101
102
Target Velocity (deg/sec)
Eye Tracking Velocity (deg/sec)
Smooth Pursuit Eye Tracking
o - Observer KC
Properties of Visual System: Motion:
��Problem: spatial CSFs vs. velocity are narrower than usual CSF
Problem: spatial CSFs vs. velocity are narrower than usual CSF
��Static CSF viewing does not result in stabilized image on retina
Static CSF viewing does not result in stabilized image on retina
��EEye drifts and sm
all pursuit m
ovements cause retinal velocities during CSF
ye drifts and sm
all pursuit m
ovements cause retinal velocities during CSF
examination
examination
103
CSF M
odels: Drift Range vs. Natural Static
103
CSF M
odels: Drift Range vs. Natural Static
10-1
100
101
102
100
101
102
Spatial Frequency (cy/deg)
Sensitivity
<--Modified Kelly Model w/ Drift Range
(0.1-2.0 deg/s)
<- Static CSF
10-1
100
101
102
100
101
102
Spatial Frequency (cy/deg)
Sensitivity
<--Envelope of Drift Range
(0.1-2.0 deg/s)
<- Static CSF
��This gives us more confidence in the m
odel for spatial attributes
This gives us more confidence in the m
odel for spatial attributes
Eye Movement Model Spatiovelocity CSF
��Use best case eye
Use best case eye
movements for
movements for
detection of moving
detection of moving
targets
targets
��Eye Movement
Eye Movement
Model
Model
��Shifts image
Shifts image
-1
0
1
2-1
0
1
2-10123
log SF (cy/deg)
log V (deg/sec)
log visual sensitivity
Spatiovelocity CSF
-10
12
3-1
-0.50
0.51
1.52
2.53
log spatial frequency (cy/deg)
log velocity (deg/sec)
Spatiovelocity CSF
��Shifts image
Shifts image
velocities to retinal
velocities to retinal
velocities that are
velocities that are
low
low
��Daly ’98 (SPIE HVEI)
Daly ’98 (SPIE HVEI)
-1
0
1
2-2
0
2-10123
log SF (cy/deg)
log V (deg/sec)
log visual sensitivityEye Movement Model Spatiovelocity CSF
-10
12
3-1
-0.50
0.51
1.52
2.53
log spatial frequency (cy/deg)
log velocity (deg/sec)
Eye Movement Model Spatiovelocity CSF
Eye Movement Model Spatiovelocity CSF
��Spatiovelocity CSF
Spatiovelocity CSF
using Eye m
ovement
using Eye m
ovement
model
model
��ω ωωωω ωωω=
= vvρ ρρρρ ρρρ
(cy/sec) = (deg/sec)(cy/deg)
(cy/sec) = (deg/sec)(cy/deg)
ωω=
= temporal frequency
temporal frequency
v =
v = velocity
velocity
ρρ=
= spatial frequency
spatial frequency
-1
0
1
2-2
0
2-10123
log SF (cy/deg)
log V (deg/sec)
log visual sensitivityEye Movement Model Spatiovelocity CSF
-10
12
3-1
-0.50
0.51
1.52
2.53
log spatial frequency (cy/deg)
log velocity (deg/sec)
Eye Movement Model Spatiovelocity CSF
ρρ=
= spatial frequency
spatial frequency
��Rotation back into
Rotation back into
spatiotemporal CSF
spatiotemporal CSF
including effects of
including effects of
eye m
ovements
eye m
ovements
•Can be used to
assess smoothness
of motion
-1
0
1
2-1
0
1
20123
log SF (cy/deg)
log TF (cy/sec)
log visual sensitivity
Eye Movement Model Spatiotemporal CSF
-1-0.5
00.5
11.5
2-1
-0.50
0.51
1.52
2.53
log spatial frequency(cy/deg)
log temporal frequency (cy/sec)Eye Movement Model Spatiotemporal CSF
Spatiotemporal visibility demos
Application of SV EMM m
odel : Analysis of Digital Video Form
ats
��Analysis of interlace, flicker and resolution issues
Analysis of interlace, flicker and resolution issues
��Use spatiotemporal CSF to analyze progressive and interlace parameters
Use spatiotemporal CSF to analyze progressive and interlace parameters
•720 lines progressive @
60 fps, 1080 lines progressive @
30 fps, 1080 lines
interlace @
60 fps
•all have sim
ilar uncompressed data rates
��Viewing distance = 3H
Viewing distance = 3H
2.5
Video Nyquist Boundaries + ST CSF
2.5Video Nyquist Boundaries + EMM ST CSF
-1-0.5
00.5
11.5
2-0.50
0.51
1.52
spatial frequency (cy/deg)
temporal frequency (cy/sec)
720P @
3H
1080I @ 3H
1080P @
3H
-1-0.5
00.5
11.5
2-0.50
0.51
1.52
spatial frequency (cy/deg)
temporal frequency (cy/sec)
720P @
3H
1080I @ 3H
1080P @
3H
Different Viewing Distances
2.5
Video Form
at Nyquist Boundaries (6H)
2.5
Video Form
at Nyquist Boundaries (9H)
��Analysis of interlace, flicker and resolution issues
Analysis of interlace, flicker and resolution issues
��Use spatiotemporal CSF to analyze progressive and interlace parameters
Use spatiotemporal CSF to analyze progressive and interlace parameters
•720 lines progressive @
60 fps, 1080 lines progressive @
30 fps, 1080 lines
interlace @
60 fps
•SD signal of 480P also considered (some DVDs)
��Increase Viewing distance to 6H and 9H
Increase Viewing distance to 6H and 9H --> Interlace advantage lost
> Interlace advantage lost
-1-0.5
00.5
11.5
2-0.50
0.51
1.52
spatial frequency (cy/deg)
temporal frequency (cy/sec)
720P @
6H
1080I @ 6H
480P @
6H -->
-1-0.5
00.5
11.5
2-0.50
0.51
1.52
spatial frequency (cy/deg)
temporal frequency (cy/sec)
720P @
9H
1080I @ 9H -->
480P @
9H ->
Speculative Video Form
at
0.51
1.52
2.5
temporal frequency (cy/sec)
360Hz 1080I @ 3H
Speculative Video Form
at
360 Hz 1080I @ 3H
360 Hz 1080I @ 3H
-1-0.5
00.5
11.5
2-0.50
spatial frequency (cy/deg)
temporal frequency (cy/sec)
��Auxiliary issues:
Auxiliary issues:
•Interlace is more difficult to compress
•2H becoming m
ore common w
ith large displays, so 1080 not enough
•Cost
•Trumbull’s Showscan (explored up to 100 Hz): some considered too realistic
and not cinematic
Closer Examination of Spatiovelocity CSF via
Closer Examination of Spatiovelocity CSF via
Eye Tracking
Eye Tracking
Verification of Eye Movement Model & SV CSF
��Laird, Pelz, Rosen, Montag and Daly (2006)
Laird, Pelz, Rosen, Montag and Daly (2006)
��Spatiovelocity Model based on Kelly’s experiments
Spatiovelocity Model based on Kelly’s experiments
•Using retinal stabilization to control velocities on the
retina
•No directed eye m
ovements
��However, in real im
age viewing applications,
However, in real im
age viewing applications,
•eyes will actually be in m
otion,
•And generally be directed as well
•And generally be directed as well
��The Spatiovelocity m
odel may not be valid when the
The Spatiovelocity m
odel may not be valid when the
eyes are actually in m
otion…
eyes are actually in m
otion…
•… if auxiliary signals from eye control circuitry to V5,
the m
otion area, affect …..??
��Build/optimize 2D spatio
Build/optimize 2D spatio--velocity CSF m
odel
velocity CSF m
odel
•Further refine Daly (Kelly+EMM) model
•Incorporate calculated retinal velocities
•Study effects of eye m
ovements on retinal velocity
sensitivity
��Equipment & Methodology:
Equipment & Methodology:
•Sony Trinitron MultiScan G420
CRT
•ASL Series 504 Remote
Eyetracker
•2IFC
Mean Lum. of screen
Mean Lum. of screen
60 cd/m
60 cd/m
22
Dist Obs. from Screen
Dist Obs. from Screen
84 cm
84 cm
Horiz. Deg span of
Horiz. Deg span of
23.95
23.95oo
Experimental Setup
Horiz. Deg span of
Horiz. Deg span of
screen
screen
23.95
23.95oo
Size of stim
ulus
Size of stim
ulus
2.46
2.46oox 2.46
x 2.46oo
��Stimuli:
Stimuli:
•Gabor
(contrast, frequency, velocity)
•Disembodied Edge
(contrast, velocity)
Eye tracking velocity calculations
Eye tracking velocity calculations
Conversion from position to degrees:
Conversion from degrees to velocity:
Tested spatiotemporal frequencies
Tested spatiotemporal frequencies
ων=
Spat Freq
Spat Freq
(Cyc/Deg)
(Cyc/Deg)
Temporal Freq (Hz)
Temporal Freq (Hz)
10
10
20
20
30
30
442.5
2.5
5.0
5.0
7.5
7.5
881.25
1.25
2.5
2.5
3.75
3.75
16
16
0.625
0.625
1.25
1.25
ρν=
deg
ree
cycl
esse
ccy
cles
sec
deg
=
Retinal velocities with and without directed eye m
ovements
��Experiment tests 4 cases:
Experiment tests 4 cases:
•mixtures of Gabor velocity, fixation points, and envelope:
Time
Trad
CSF
Time
ST
CSF
Gabor does
Gabor does
not move
not move
Sine m
oves, but
Sine m
oves, but
envelope & fixation
envelope & fixation
do not:
do not:
Retinal velocity
Retinal velocity
Time
Time
Time
SV CSF
No eye
tracking
Time
SV CSF
eye tracking
Fixation m
oves
Fixation m
oves
with Gabor:
with Gabor:
Retinal velocity
Retinal velocity
depends on eye
depends on eye
tracking
tracking
Gabor moves,
Gabor moves,
but fixation
but fixation
does not:
does not:
Retinal velocity if
Retinal velocity if
observer can ignore
observer can ignore
envelope?
envelope?
��Eye fixation is good (able to ignore m
oving object, if requested)
Eye fixation is good (able to ignore m
oving object, if requested)
��Moving sines (fixed envelope) = m
oving gabor (m
oving envelope)
Moving sines (fixed envelope) = m
oving gabor (m
oving envelope)
Retinal velocities without directed eye m
ovements
��Eye tracking is good, results similar to static
Eye tracking is good, results similar to static
��No signals related to eye m
otion affect neural processing (no intercedent)
No signals related to eye m
otion affect neural processing (no intercedent)
Retinal ‘stasis’ with and without directed eye m
ovements
��Data shifted horizontal for separability, since they superimpose closely
Data shifted horizontal for separability, since they superimpose closely
��Eye tracking removes the decrease in Sensitivity with increasing
Eye tracking removes the decrease in Sensitivity with increasing
temporal frequency , for all tested spatial frequencies
temporal frequency , for all tested spatial frequencies
��Maybe m
otion sharpening at 4 cpd?
Maybe m
otion sharpening at 4 cpd?
Retinal velocities with and without directed eye m
ovements
�The red dots correspond
to the points in the table
Sensitivity results on Spatiovelocity CSF m
odel
10Hz
20Hz
30Hz
4 cpd
2.5 deg/sec
5 deg/sec
7.5 deg/sec
8 cpd
1.25 deg/sec
2.5 deg/sec
3.75 deg/sec
16 cpd
0.625 deg/sec
1.25 deg/sec
�The velocities result from the
particular spatial and temporal
frequency combination.
Fine tuning parameters of SV m
odel
()
()
−
⋅⋅
⋅⋅
⋅=
max
12
12
10
4ex
p2
,ρπρ
πρρ
cc
vc
cc
kv
CSF
RR
3
2
3lo
g
⋅
+=
Rv
cs
sk
Wh
ere:
�S
V C
SF
in r
etin
al v
eloci
ties
, v
r, a
nd s
pat
ial
freq
uen
cy ρ
22
13
log
⋅+
=R
vc
ss
k
()
22
1m
ax+
=R
vc
pρ
�K
elly
model
modif
ied t
o f
it d
ata
•(K
elly
mo
del
on
ly a
t lo
w L
A l
evel
, an
d n
ois
ier
dis
pla
ys
of
the
pas
t)
�N
on-l
inea
r le
ast
squar
es r
outi
ne:
•
Sen
siti
vit
y v
alu
es f
rom
mo
del
fit
to
ex
per
imen
tal
resu
lts
Test of model on combined frequencies
Test of model on combined frequencies
��Experiment 5
Experiment 5
•Moving edge results
•Sensitivity to blurring of edge
•As a function of edge contrast
Test of the SV m
odel
Test of the SV m
odel
��Revised SV CSF m
odel based on new parameters (inset)
Revised SV CSF m
odel based on new parameters (inset)
��Prediction results of moving edges via m
odel :
Prediction results of moving edges via m
odel :
•Based on W
atson
& Ahumada JOV
2005
•Use 2D integral of
CSF x signal
spectrum to
spectrum to
model sensitivity
•Perfect eye
tracking assumed
•Channels not
needed since no
masking ??
•OK, but could be
better
(facilitation?)
�S
ensi
tiv
ity
det
erm
ined
by
ret
inal
vel
oci
ty
•N
ot
affe
cted
by e
ye
movem
ents
�S
ensi
tiv
ity
sim
ilar
fo
r 2
ty
pes
of
mo
tio
n
•M
ovin
g s
inuso
ids
wit
hin
Gab
or
Verification of Eye Movement Model & SV CSF -Summary
•G
abor
movin
g a
cross
fie
ld o
f vie
w
�O
pti
miz
ed 2
D s
pat
io-v
elo
city
CS
F m
od
el
•M
ore
app
lica
ble
to T
V i
mag
ery
•U
se o
f re
tinal
vel
oci
ty a
nd u
nst
abil
ized
sti
muli
LCD Temporal Basics
LCD Temporal Basics
�W
hy
do
es L
CD
mo
tio
n b
lur
hap
pen
?
�L
CD
Tem
po
ral
MT
F c
om
po
nen
ts�
Tem
po
ral-
resp
on
se b
lur
&
�H
old
-ty
pe
blu
r (
tem
po
ral
ren
der
ing
fu
nct
ion
)�
Ho
ld-t
yp
e b
lur
(te
mp
ora
l re
nd
erin
g f
un
ctio
n)
�L
CD
mo
tio
n b
lur
mo
del
ing
�L
CD
mo
tio
n b
lur
anal
ysi
s�
Slo
w-r
esp
on
se b
lur
vs.
ho
ld-t
yp
e b
lur
�A
nal
ysi
s o
f P
rop
ose
d s
olu
tio
ns
Slow-response blur : LCD Temporal Characteristics
�Inputvs.
Outputtemporalresponsesshown
�Overallspeedandasymmetryare
important
Level 0
Level 0
010
20
30
40
50
60
7080
90
100
120
140
160
�Slowerresponseshavemore
temporalLPF
andleadto
motionblur
�Asymmetric
responsesleadto
HSFflicker
�Overdrive
Level 0
Level 2
Level 4
Level 6
Level 8
Level 0
Level 2
Level 4
Level 6
Level 8
Output
Input
Out 0
Out 2
Out 4
Out 6
Out 8
In 0In 1
In 2In 3
In 4 In 5
In 6In 7
In 8
0
20
40
60
80
Response
Time in ms
Output
Input
Level 0
Level 2
Level 4
Level 6
Level 8
Level 0
Level 2
Level 4
Level 6
Level 8
0102030405060
Response Time
(ms)
Output
Input
Spatial consequences of Temporal LPF Spectrum after
temporal lowpass
f t
f x
f t
f x
Spectrum at the
retina with eye
Spectrum of
motion sequence
Temporal
low-pass
of the
f t
f x
020
40
60
80
100
120
0
0.2
0.4
0.6
0.81
Temporal Frequency
Temporal MTF
f t
f x
Original image
spectrum
Klo
mp
enh
ou
wer
20
04
temporal lowpass
of display
retina with eye
tracking
motion sequence
of the
display
spectrum
�T
he
moti
on o
f an
ob
ject
cau
ses
tem
pora
l co
mp
onen
t in
the
spat
ial/
tem
pora
l sp
ectr
um
.
�T
his
sp
ectr
um
is
low
-pas
s fi
lter
ed b
y t
he
dis
pla
y s
pat
ial/
tem
pora
l tr
ansf
er f
unct
ion.
�T
he
eye
trac
kin
g c
ause
s th
e re
tina
imag
e to
hav
e p
ure
sp
atia
l co
mp
onen
t of
spec
trum
wit
hout
any
tem
pora
l co
mp
onen
t.
�B
ut
the
tem
pora
l lo
w-p
ass
filt
erin
g i
n t
he
dis
pla
y r
educe
s th
e sp
atia
l b
andw
idth
of
the
reti
na
imag
e, w
hic
h c
ause
s th
e p
erce
pti
on o
f m
oti
on b
lur.
Overdrive
Overdrive
ImprovingLCD Temporal Characteristics with Overdrive
�Slowerresponsesleadto
motionblur
�OverdriveLUT–from
graylevelto
graylevel(intendedto
necessary
map)
Targ
et
Over
dri
ve
No O
ver
dri
ve
Sta
rt
t
Sta
rt
Targ
etOverdrive Value
�Okumura
01,Sekiya02
LCD Temporal Response and its Temporal MTF
0.5
0.6
0.7
0.8
0.91
LCD temporal response
32 m
s
8 m
s
2 m
s
OD
0.5
0.6
0.7
0.8
0.91
Temporal MTF
32 m
s
8 m
s
2 m
s
OD
�T
emp
ora
l over
dri
ve
can e
ffec
tive
imp
rove
the
tem
pora
l M
TF
�
thus
reduci
ng t
he
moti
on b
lur
�A
t p
eak o
f H
VS
tem
pora
l C
SF
(8H
z), over
dri
ve
can e
ven
exce
ed a
2m
s te
mp
ora
l re
sponse
05
10
15
20
25
30
35
40
0
0.1
0.2
0.3
0.4
t in m
s
LCD temporal response
0
10
20
30
40
50
60
0
0.1
0.2
0.3
0.4
Temporal Frequency (Hz)
Temporal MTF
Designing a temporal overdrive algorithm
Target value
Y1
Y2
LC
D
Pre
vio
us
Val
ue
Y2
Y3
Fra
me
bu
ffer
Tar
get
val
ues
dn
-3d
n-2
dn
-1d
n
LC
D m
od
el
Overdrive algorithm results
8
8
72
01020
30405060
8
104
010
2030405060
8 40
72
104
136
168
200
232
255
No Overdrive
With Overdrive
72
136
200
255
72
136
200
255
Output
Input
8
72
136
200
255
200
0
Output
Input
�Note that overdrive m
akes temporal responses more
symmetrical: this essentially eliminates the flickering
artifacts
Temporal responses w/ OD are generally in range of 3-5ms
Dynamic Gamma Method
for Overdrive Analysis
for Overdrive Analysis
Dynamic Gamma Approach
�(Static) Display Gamma:
0.4
0.6
0.81
1.2
1.4
Display Output
10-2
10-1
100
Display Output
050
100
150
200
250
300
0
0.2
0.4
Input Digital Counts
100
101
102
103
10-3
Input Digital Counts
γd
cy=
Feng et al 04 & 05
Definition of first order Dynamic Gamma
90
% p
oin
t
targ
et
Outp
ut
DG
val
ue
Zn
-1
Zn
-2
Zn
-2
Zn=
25
5
Z=
0
Dri
vin
g W
avef
orm
Res
po
nse
tim
e
t
Fra
me
per
iod
�T
he
LC
D i
np
ut/
ou
tpu
t re
lati
on
ship
ch
ang
es w
ith
tim
e w
hen
dis
pla
yin
g m
oti
on
�D
yn
amic
gam
ma
val
ue:
th
e o
utp
ut
val
ue
mea
sure
d a
t th
e en
d o
f t
he
firs
t fr
ame
�C
an u
se t
he
sam
e eq
uip
men
t as
res
po
nse
tim
e m
easu
rem
ent
�A
dv
anta
ges
ov
er u
se o
f re
spo
nse
tim
es
Fra
me
No
.
0
1
Zn=
0
150
200
250
300
Output Level
0
32
64
96
128
160
192
224
255
Representation: Table vs. Figure
0032
32
64
64
96
96
128
128
160
160
196
196
224
224
255
255
0000
22
22
39
39
61
61
82
82
111
111
140
140
155
155
220
220
……
…
25
5
Mea
sure
d v
alues
()
nn
nz
df
d,
1−=
dn
-1:
25
5
050
100
150
200
250
300
0
50
100
150
Driving Value
Output Level
25
25
55135
135
141
141
15
15
55175
175
186
186
203
203
214
214
236
236
255
255
Sta
rtin
g v
alu
e
Tar
get
val
ue
……
…
0
Mea
sure
d v
alues
Eac
h c
urv
e re
pre
sents
dif
fere
nt
pre
vio
us
val
ue
�F
irst
ord
er Dγ
mo
del
s th
e ed
ge
mo
tio
n
0
Derivation of overdrive lookup table
200
250
300
0
16
32
48
64
80
96
112
050
100
150
200
250
300
0
50
100
150
Target Level
Output Level
112
128
144
160
176
192
208
224
240
255
�To go from 32 (previous frame) to 64: needs OD value of 130
Application for comparing LCD systems
100
150
200
250
300
Output Level
0
32
64
96
128
160
192
224
255
100
150
200
250
300
Output Level
0
32
64
96
128
160
192
224
255
Slo
w L
CD
No
OD
Slo
w L
CD
wit
h O
D
Dynamic Gamma useful for comparing LCD systems
(overdrive + inherent temporal response)
Assessment of overdrive perform
ance with dynamic gamma:
050
100
150
200
250
300
0
50
Driving Value
050
100
150
200
250
300
0
50
Driving Value
050
100
150
200
250
300
-500
50
100
150
200
250
300
Target Level
Output Level
050
100
150
200
250
300
0
50
100
150
200
250
300
1st
ord
er �
Edge
2
nd
advan
tage �
real
vid
eo
Fas
t L
CD
wit
h O
DF
ast
LC
D w
ith
OD
(2
nd-o
rder
dyn
amic
γ)
Current Overdrive algorithm results
40.0
60.0
80.0
100.0
120.0
Response time ms
response time ms
40.0
60.0
80.0
100.0
120.0
Without Overdrive
With Overdrive
0**
64**
128**
192**
255**
0*
32*
64*
96*
128*
160*
192*
224*255*
0.0
20.0
Input
Response time ms
Output
input
Output
response time ms
0**
64**
128**
192**
255**
0*
32*
64*
96*
128*
160*
192*
224*255*
0.0
20.0
Current Overdrive algorithm results
Example of visual consequences:
Conventional driver
High Perform
ance Overdrive
Display Temporal Rendering Function
Display Temporal Rendering Function
Comparative Display Basics of Temporal Aperture
tR
eal
worl
d t
emp
ora
l w
avef
orm
tC
RT
tem
pora
l outp
ut
(im
puls
e-ty
pe)
LC
D t
emp
ora
l outp
ut
(hold
-ty
pe)
LCD Motion Blur
��LCD
LCD’’s slow response:
s slow response: slow
slow--resp
onse
blur
resp
onse
blur
•physical;
•can be captured by a fixed-position camera
tt
Idea
l L
CD
tem
pora
l
outp
ut
(0-r
esp
onse
)
Pra
ctic
al L
CD
tem
pora
l
outp
ut
(slo
w r
esp
onse
)
��LCD’s hold
LCD’s hold--type rendition + HVS’ sm
ooth pursuit & lowpass
type rendition + HVS’ sm
ooth pursuit & lowpass
filtering:
filtering: hold
hold--type blur
type blur
•perceptual;
•only happen when human eyes are tracking;
•can NOT be captured by a fixed-position camera
•Lindholm
96 , Parker 97, Kurita 98, Kurita 01
Role of eye tracking in LCD hold-type blur
��Eye integrates along
Eye integrates along
tracking path (10
tracking path (10--50ms, LA)
50ms, LA)
��For CRT display, integration
For CRT display, integration
of eye tracking path causes
of eye tracking path causes
no m
ixing of black and
no m
ixing of black and
white displayed elements
white displayed elements
along path
along path
��Result for CRT is sharp
Result for CRT is sharp
CRT: impulse
FRAME 0
FRAME 1
FRAME 2
PERCEIVED
t
��Result for CRT is sharp
Result for CRT is sharp
moving edge
moving edge
��For LCD display, eye track
For LCD display, eye track
path goes through regions of
path goes through regions of
white and black displayed
white and black displayed
elements, so that mixing of
elements, so that mixing of
signals occurs due to
signals occurs due to
temporal integration of the
temporal integration of the
eye
eye
��Result for LCD (hold) is a
Result for LCD (hold) is a
blurred m
oving edge
blurred m
oving edge
FRAME 0
FRAME 1
FRAME 2
SHARP EDGE
BLURRED EDGE
PERCEIVED
LCD: hold
EYE M
OVEMENT:
TRACKING EDGE
Eye tracking Demo
Eye tracking Demo
Image examples
t
Ret
ina
imag
e o
f a
mo
vin
g e
dg
e
on
ho
ld d
isp
lay w
ith
eye
trac
kin
g
�h
old
blu
r
Ret
ina
imag
e of
a m
ovin
g e
dge
on
imp
uls
e dis
pla
y w
ith e
ye
trac
kin
g
�N
om
oti
on b
lur
Quan
tita
tive
Sim
ula
tion
)(
sin
)(
Tf
cf
Tt
th
=
The
effe
ct o
f te
mp
ora
l hold
is
ver
y s
imil
ar t
o s
pat
ial
aper
ture
eff
ect
of
CC
D
senso
r. T
he
tem
pora
l M
TF
is
giv
en b
y a
sin
cfu
nct
ion.
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.91
Temporal MTF
32 m
s
8 m
s
2 m
s
OD
Motion Blur Due to Hold: Temporal MTF
Motion Blur Due to Hold: Temporal MTF
)(
sin
)(
Tf
cf
Tt
th
=
�F
or
a fa
ster
LC
D p
anel
, th
e te
mp
ora
l M
TF
is
lim
ited
by t
he
ho
ld e
ffec
t. T
her
e is
dim
inis
hin
g g
ain
in
fu
rth
er i
mp
rov
ing
LC
D t
emp
ora
l re
spo
nse
.
�F
or
a g
iven
tem
po
ral
sam
pli
ng
, th
e o
nly
way
to
red
uce
ho
ld b
lur
is t
o r
edu
ce
tem
po
ral
aper
ture
.
�P
lot
incl
ud
es L
CD
tem
po
ral
resp
on
se +
ho
ld a
per
ture
010
20
30
40
50
60
0
0.1
Temporal Frequency (Hz)
Analysis of Temporal LCD System Issues
Analysis of Temporal LCD System Issues
Pan et al 05
Pan et al 05
Sim
plified display-perception chain
(,
,)
(,
,)
Ix
yt
Ix
vt
yv
tt
=+
+
Inp
ut:
ass
um
ing
th
e d
yn
amic
dis
cret
e co
nte
nt
I d(x
,y,t
) i
s an
im
age
I c(x
,y,t
)
mo
vin
g a
t a
con
stan
t sp
eed
HV
S
S&
H
(ht(
t))
Sm
oo
th
pu
rsu
it
Lo
wp
ass
filt
er
LC
D/C
RT
I d(x
,y,t
)I s
(x,y
,t)
I m(x
,y,t
)I o
(x,y
,t)
(,
,)
(,
,)
dc
xy
Ix
yt
Ix
vt
yv
tt
=+
+
(,
,)
(,
,)*
()
sd
tI
xy
tI
xy
th
t=
(2)
Sm
oo
th p
urs
uit
eye
mo
vem
ent:
()
(,
,)
(,
,)*
(,
)*
()
om
xyt
Ix
yt
Ix
yt
xy
t=
ΛΛ
(,
,)
(,
,)
ms
xy
Ix
yt
Ix
vt
yv
tt
=−
−
(1)
Sa
mp
le-a
nd
-ho
ld :
(3)
Lo
wp
ass
fil
ter:
ht (t
) is
th
e te
mp
ora
l re
con
stru
ctio
n f
un
ctio
n o
f an
LC
D o
r C
RT
Λxy
(x,y
) &
Λt(
t) a
re s
pat
ial
& t
emp
ora
l im
pu
lse
resp
on
se f
un
ctio
ns
of
the
filt
er
com
pen
sate
s m
oti
on
to
mak
e th
e o
bje
ct s
till
on
ret
ina.
The general LCD m
otion blur model
'
(,
,)
('
,'
,'
")(
,)
(")
"(
')'
oc
xy
xy
tt
t
Ix
yt
Ix
vt
py
vt
qt
tt
pq
td
pd
qd
th
tdt
∞ =−∞
=−
−−
−−
−Λ
Λ∫∫∫∫
•T
he
spat
ial
and
tem
po
ral
low
pas
s im
pu
lse
fun
ctio
ns
(Λxy
(x,y
) an
d Λ
t(t)
) a
re
un
kn
ow
n
•A
ssu
min
g t
hat
HV
S h
as t
he
sam
e lo
wp
ass
imp
uls
e fu
nct
ion
s fo
r L
CD
an
d
CR
T, an
d u
sin
g i
mag
e p
erce
ived
on
CR
T a
s a
refe
ren
ce
•T
he
inp
ut-
ou
tput
rela
tio
nsh
ip o
f th
e ch
ain
:
CR
T, an
d u
sin
g i
mag
e p
erce
ived
on
CR
T a
s a
refe
ren
ce
'
(,
,)
(',
',')
(')
'oL
CD
CR
TL
CD
ox
yt
t
Ix
yt
Ix
vt
yv
tt
th
tdt
∞ =−∞
=−
−−
∫
The general LCD m
otion blur model
�T
he
LC
D t
emp
ora
l re
con
stru
ctio
n f
un
ctio
n h
t L
CD
(t)
affe
cts
spat
iall
y a
nd
tem
po
rall
y.
�T
he
reco
nst
ruct
ion f
un
ctio
n h
tLC
D(t
) ca
n b
e m
easu
red
dir
ectl
y o
r d
eriv
ed
fro
m t
he
tem
po
ral
wav
efo
rm.
'
(,
,)
(',
',')
(')
'oL
CD
CR
TL
CD
ox
yt
t
Ix
yt
Ix
vt
yv
tt
th
tdt
∞ =−∞
=−
−−
∫
fro
m t
he
tem
po
ral
wav
efo
rm.
�W
hen
htL
CD
(t)
is δ
-fu
nct
ion, th
en L
CD
an
d C
RT
hav
e th
e sa
me
resu
lt
(mo
tio
n b
lur
do
es n
ot
exis
t)
�G
ener
ally
, th
e m
od
el i
s n
ot
in t
he
form
of
con
vo
luti
on.
�T
he
mo
tio
n s
pee
d v
xan
d t
he
LC
D r
eco
nst
ruct
ion f
un
ctio
n j
oin
tly d
eter
min
e
mo
tio
n b
lur.
�F
aste
r th
e m
oti
on
is,
mo
re b
lurr
ed t
he
per
ceiv
ed i
mag
es a
re.
�W
ider
th
e re
con
stru
ctio
n f
un
ctio
n i
s, m
ore
blu
rred
th
e p
erce
ived
im
ages
are.
Th
e re
con
stru
ctio
n f
un
ctio
n i
s th
e k
ey
Blur width calculated by the m
odel
Blur width calculated by the m
odel
1. A
ssu
me
a v
irtu
al h
ori
zon
tall
y
mo
vin
g s
har
p e
dg
e p
erce
ived
on
CR
T
2. C
alcu
late
th
e p
erce
ived
ed
ge
0x
L+
D L
inte
nsi
ty
⊗ ⊗⊗⊗
3. C
alcu
late
th
e b
lur
wid
th o
f th
e
calc
ula
ted
per
ceiv
ed e
dg
e
()
()*
(/
)L
CD
CR
TL
CD
oo
tx
Ix
Ix
hx
v=
2. C
alcu
late
th
e p
erce
ived
ed
ge
usi
ng
th
e re
con
stru
ctio
n
fun
ctio
n o
f th
e L
CD
BW
0x
L+
D L
inte
nsi
ty
L+
10%
D
L+
90%
D
Poin
t 1
Poin
t 2
Th/2
1 0
t
Th
Traditional LCD (slow response blur + hold-type blur)
t
The
tem
pora
l w
avef
orm
(lin
ear
tran
siti
on b
etw
een
fram
es)
Th
1 0
t
2T
h
The
reco
nst
ruct
ion
funct
ion (
linea
r
tran
siti
on b
etw
een
fram
es)
t0
0.2T
0.4T
0.6T
0.8T
T1.2T
1.4T
1.6T
1.8T
2T
0
0.2
0.4
0.6
0.81
tT
h
2T
hT
he
tem
pora
l w
avef
orm
(sin
e tr
ansi
tion b
etw
een
fram
es)
The
reco
nst
ruct
ion
funct
ion (
linea
r
tran
siti
on b
etw
een
fram
es)
Blu
r w
idth
: 1.1
vT
x
-15
-10
-5
05
10
15
20
0
0.2
0.4
0.6
0.81
1.2
Position (pixel)
Intensity
Perceived Edge on CRT
Perceived Edge on LCD (the first-ordre hold model)
Perceived Edge on LCD (the sine hold model)
CR
TL
CD
(fi
rst-
ord
er)
LC
D
(sin
e)
intensity
t
(a)
Th=
T
1/T 0
t
(b)
The ideal LCD temporal response (hold-type blur only)
Blu
r w
idth
: 0.8
vT
-15
-10
-50
510
15
20
0
0.2
0.4
0.6
0.81
1.2
Position (pixel)
Intensity
x
intensity
Cu
rren
t L
CD
Idea
l L
CD
�H
old
-ty
pe
+sl
ow
res
po
nse
blu
r: 1
.1vT
�H
old
-ty
pe
blu
r: 0
.8vT
�so
slo
w r
esp
on
se b
lur:
1.1
vT-0
.8vT
=0
.3vT
Hold-type blur vs. slow-response blur
�so
slo
w r
esp
on
se b
lur:
1.1
vT-0
.8vT
=0
.3vT
�7
0%
vs.
30
%
�S
o, h
old
-ty
pe
blu
r is
th
e m
ajo
r fa
cto
r
1)Black Data Insertion (BDI)
•Hong 04, Kim
ura 05
2)Backlight Flashing and Scrolling (BF)
•Fisekovic 01, Sluyterm
an 05
•Adaptive backlight flashing (60 and 120 Hz), Feng 06
Four key proposed solutions
•Adaptive backlight flashing (60 and 120 Hz), Feng 06
3)Frame Rate Doubling (FRD)
•Sekiya 02, Kurita 05
4)Motion-Compensated Inversing Filtering (MCIF)
•Klompenhauer 01, 05
Reconstruction functions of the four proposals
BD
I tT
/2T
2T
1 0
tT
/2T
2T
1 0
t
t
FR
D
tt
t
T
1/T
0
t
2T
T-T
12T-T
1T
2
T
1 0
t
BF
MC
IF
tt
Frame Rate Conversion based on Motion Compensation
Frame Rate Conversion based on Motion Compensation
temporal
Input Im
age
(60 frame/sec)
Converted sequence
(120 frame/sec)
1/60 sec
Images created
by frame rate
conversion
1/120 sec
1/60 sec
1/120 sec
Input
60p
Pre-
processing
Motion vector
estim
ation
Interpolation
Up-converted
output 120p
Motion vector estimation
Frame interpolation
Input
60p
Pre-
processing
Motion vector
estim
ation
Interpolation
Up-converted
output 120p
Motion vector estimation
Frame interpolation
Detect motion vector
Estimate object's motion in-
between Frame #1 and #2
as direction and speed
From frame#1 and
#2, create new
picture that shift its
position by
1/120sec
Interpolated picture
1/60 sec
Fra
me#
1F
ram
e#
1F
ram
e#
1F
ram
e#
1F
ram
e#
2F
ram
e#
2F
ram
e#
2F
ram
e#
2
BDI
BDI
BF
BF
FRD
FRD
MCIF
MCIF
Requirement on LCD
Requirement on LCD
temporal response
temporal response
High
High
Medium
Medium
High
High
No
No
Requirement on
Requirement on
backlight temporal
backlight temporal
response
response
No
No
High
High
No
No
No
No
Other Requirement
Other Requirement
No
No
Sync betw
een
Sync betw
een
LCD and
LCD and
backlight
backlight
Accurate
Accurate
motion
motion
estim
ation
estim
ation
Motion
Motion
estim
ation
estim
ation
Comparison betw
een different approaches
backlight
backlight
estim
ation
estim
ation
The ghosting artifact
The ghosting artifact
Likely
Likely
Likely
Likely
No
No
No
No
The luminance
The luminance
reduction artifact
reduction artifact
Yes
Yes
Yes
Yes
No
No
No
No
flickering artifact
flickering artifact
Yes
Yes
Yes
Yes
No
No
No
No
Reduction of motion
Reduction of motion
blur (smaller the
blur (smaller the
number is, the
number is, the
better)
better)
50%
50% (limited
(lim
ited
by LCD
by LCD
temporal
temporal
response)
response)
25% or less
25% or less
(lim
ited by
(lim
ited by
backlight
backlight
temporal
temporal
response)
response)
50%
50% (limited
(lim
ited
by LCD
by LCD
temporal
temporal
response)
response)
??
Perceptual Motion Sharpening
Perceptual Motion Sharpening
Motion Sharpening
Motion Sharpening
��Ramachandran ’74 (observations on blurred m
ovie frames)
Ramachandran ’74 (observations on blurred m
ovie frames)
•things tend to look blurred when they are m
oving fast-----but---
•blurred edges look sharper when they are m
oving than w
hen stationary
��Poor tracking
Poor tracking � ���� ���
blurred retina image
blurred retina image
��Motion sharpening effect
Motion sharpening effect � ���� ���
perceived m
otion is sharper than the still
perceived m
otion is sharper than the still
images, which suggests that the
images, which suggests that the
-1-0.5
00.5
11.5
20
0.51
1.52
2.53
log spatial frequency (cy/deg)
log visual sensitivity
CSFs for Different Retinal Velocities
0.001
0.012
0.15 deg/sec
3
11
32
64 128
200
perceived m
otion is sharper than the still
perceived m
otion is sharper than the still
images, which suggests that the
images, which suggests that the
perception of sm
ooth pursuit is different
perception of sm
ooth pursuit is different
from still image.
from still image.
��Less understood , higher order effect
Less understood , higher order effect
•Sharpness constancy
•Deblurring
��If m
otion sharpening effect is involved,
If m
otion sharpening effect is involved,
previous analysis based on retinal im
age
previous analysis based on retinal im
age
blur is insufficient
blur is insufficient
Motion induced Blur and Sharpening
Motion induced Blur and Sharpening
��Westerinck 90
Westerinck 90
��Studied perceived sharpness of im
ages
Studied perceived sharpness of im
ages
�with varying degrees of blur
�As a function of translational motion speeds
Conclusion: Evaluation of Motion Blur Reduction
Conclusion: Evaluation of Motion Blur Reduction
��Motion blur characterization
Motion blur characterization
•Objective m
ethod: measured retina image using a
simulated tracking camera –assuming perfect tracking
•Subjective m
ethod: Compared the perceived blur with
blurred edge of a still image
��Motion blur perception
Motion blur perception (
))
(*
)(
)/
()
()
(/
σxb
vx
gaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
��Motion blur perception
Motion blur perception
•The subjective m
ethod agrees with the objective
derived m
otion blur �
Perception of motion blur is
similar to perception of still im
age blur
•Backlight flashing can significantly reduce the
perception of motion blur.
Other Key Studies
Other Key Studies
20
30
40
50
60
頻度(%) Frequency (%)
20
30
40
50
60
頻度(%) Frequency (%)
Distribution of Moving Object Velocity
Average; ;;;16.5 deg/sec
No.1
~No.32
The region where blur caused
by camera can be clearly seen.
Analysis of ITU-R BT-1210-3 test material for HDTV sequence
0
10
20
0~
10
10~
20
20~
30
30~
40
40~
50
50~
60
視覚
速度
(deg/s)
Frequency (%)
Velo
cit
y i
n v
iew
ing
an
gle
(d
eg
/sec)
0
10
20
0~
10
10~
20
20~
30
30~
40
40~
50
50~
60
視覚
速度
(deg/s)
Frequency (%)
Velo
cit
y i
n v
iew
ing
an
gle
(d
eg
/sec)
[3H
]
In term
s of the fastest motion within every sequence,
70% of the sequence distributed below 20[deg/sec]
T.Fujine et.al.;”Real-Life In-Home Viewing Conditions for FPDs and Statistical C
haracteristics of Broadcast Video
Signal,” Digest AM-FPD’06
345
Quality Scale ����
50% aperture
double-rate
Acceptance
limit
Threshold of
perception
Observer Study of Picture Quality Improvement
Observer Study of Picture Quality Improvement
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
12
05
10
15
20
25
30
Velocity in Viewing Angle (deg/sec)
Quality Scale
100% aperture
(hold type)
50% aperture
T.Kurita;
”Moving Picture Quality Improvement for Hold-type AM-LCDs,”SID
’01, 35.1, pp.986-989 (2001)
()
)(
*)
()
/(
)(
)(
/σx
bv
xg
au
sre
ctv
xtr
ix
step
xb
lurE
dg
e∗
=��
Basic
Basic
Analysis of Methods to Overcome Hold
Analysis of Methods to Overcome Hold--Blur
Blur
Analysis of Methods to Overcome Hold
Analysis of Methods to Overcome Hold--Blur
Blur
BDI
BDI (black data insertion)
BF
BF (backlight flashing)
FRD
FRD (frame doubling)
(frame doubling)
Pan
, F
eng
, &
Dal
y:
"Qu
anti
tati
ve
An
alysi
s o
f L
CD
Mo
tio
n B
lur
and
Per
form
ance
of
Ex
isti
ng A
pp
roac
hes
“ I
CIP
20
05
ASV1.0
Response time : 15ms
Response time 12ms
ASV2.2
≪ ≪≪≪Hold type drive
≫ ≫≫≫
7ms + Pseudo impulse drive
ASV3.0
≪ ≪≪≪Pseudo impulse drive
≫ ≫≫≫
5ms double rate LCD
w/ MC double rate drive
≪ ≪≪≪Motion Compensated 120Hz≫ ≫≫≫
BDI
BDI
BF
BF
FRD
FRD
Requirement on LCD temporal response
Requirement on LCD temporal response
High
High
Median
Median
High
High
Requirement on backlight temporal
Requirement on backlight temporal
response
response
No
No
High
High
No
No
Other Requirement
Other Requirement
No
No
Sync betw
een LCD
Sync betw
een LCD
and backlight
and backlight
Accurate m
otion
Accurate m
otion
estim
ation
estim
ation
The ghosting artifact
The ghosting artifact
Likely
Likely
Likely
Likely
No
No
The luminance reduction artifact
The luminance reduction artifact
Yes
Yes
Yes
Yes
No
No
flickering artifact
flickering artifact
Yes
Yes
Yes
Yes
No
No
Reduction of motion blur (smaller the
Reduction of motion blur (smaller the
number is, the better)
number is, the better)
50% (limited by LCD
50% (limited by LCD
temporal response)
temporal response)
25% or less (limited
25% or less (limited
by backlight
by backlight
temporal response)
temporal response)
50% (limited by
50% (limited by
LCD temporal
LCD temporal
response)
response)
Standardized Metrics
Standardized Metrics
Motion Picture Response Tim
e (MPRT)
Motion Picture Response Tim
e (MPRT)
Steps to Measure MPRT
1.Move an edge cross screen. The edge is made of a transition from one
gray level to another level. Total of 30 transitions (6 levels in digital
counts or L* space)
2.Using the pursuit camera to m
easure the blur width
3.MPRT is average blur edge width (BEW) norm
alized by the m
oving speed
-also referred to as E-BET (extended blurred edge tim
e)
MPRT basics and issues
MPRT basics and issues
��Motion blur can be characterized by m
otion picture response tim
e (MPRT)
Motion blur can be characterized by m
otion picture response tim
e (MPRT)
metric, which is measured with a tracking camera that simulates the eye
metric, which is measured with a tracking camera that simulates the eye
tracking of a m
oving edge
tracking of a m
oving edge
��The system is expensive and tim
e consuming
The system is expensive and tim
e consuming
��Theoretically, motion blur is a pure temporal issue that can be uniquely
Theoretically, motion blur is a pure temporal issue that can be uniquely
determ
ined by the temporal response function (via LTI: linear systems
determ
ined by the temporal response function (via LTI: linear systems
theory)
theory)
•Nonlinearities (of both LCD and HVS) are only reasons for failure of LTI
•Small amplitude signals m
ay be within linear region approxim
ation of
•Small amplitude signals m
ay be within linear region approxim
ation of
both
•Still, MPRT does not consider HVS effects (too m
uch norm
alization)
��In Q&A with Someya (Mitsubishi) at IDW05, he thought that MPRT from
In Q&A with Someya (Mitsubishi) at IDW05, he thought that MPRT from
temporal measurement is only accurate for hold displays, but not for
temporal measurement is only accurate for hold displays, but not for
impulse displays such as displays using backlight flashing and black data
impulse displays such as displays using backlight flashing and black data
insertion
insertion
��At SID 06, Klompenhouwer described advanced m
otion blur measurement
At SID 06, Klompenhouwer described advanced m
otion blur measurement
schemes as “inventing a complex system to m
easure a sim
ple temporal
schemes as “inventing a complex system to m
easure a sim
ple temporal
response”
response”
Motion Blur Measurement with Sim
ulated Tracking Camera
Motion Blur Measurement with Sim
ulated Tracking Camera
∫−
=L
CD
edt
vtx
Ex
E)
()
(
The simulated retina image is the integration of a sequence of
temporal captured frames in the m
otion tracking trajectory
∑∫ =
∆∆
−=
−=
: i
CC
De
LC
De
ti
tiv
xE
xE
dt
vtx
Ex
E
1
),
()
(
)(
)(
Captured Frames in one Display Frame Period
Captured Frames in one Display Frame Period
Frame 1
Frame 2
Frame 3
Frame 4
Frame 5
Via high speed digital camera (900 fps)
Frame 6
Frame 7
Frame 8
Frame 9
Frame 10
Summary
Summary
��Basic Spatiotemporal Vision
Basic Spatiotemporal Vision
��Spatiotemporal Vision with Eye Movements
Spatiotemporal Vision with Eye Movements
��LCD Motion Issues
LCD Motion Issues
•Temporal Response
•Overdrive
Summary
Summary
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
•Overdrive
•Temporal Rendering Function
��Observer study of LCTV m
otion sharpness
Observer study of LCTV m
otion sharpness
matching
matching
What’s next :
What’s next :
Other Temporal Artifacts
Other Temporal Artifacts
��Motion Blur and Sharpness … as discussed
Motion Blur and Sharpness … as discussed
��Flicker … m
entioned
Flicker … m
entioned
•Asymmetrical temporal response
•Periphery & Brightness issues
��Judder
Judder
•Stepper-like m
otion, seen with slower steady m
otions
Other Temporal Artifacts
Other Temporal Artifacts (
))
(*
)(
)/
()
()
(/
σxb
vx
gaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
•Stepper-like m
otion, seen with slower steady m
otions
•CRT’s fast temporal response is not desired
��Multiple Edges
Multiple Edges
•Examples from Backlight Flashing + m
ismatched Overdrive
•Hollywood is happy with 24 fps ! (looks cinematic)
oDCI
oAliasing control via cameraman & editors
��Human eye is poor for seeing m
otion
Human eye is poor for seeing m
otion
What does real
What does real--world Movement really look like?
world Movement really look like?
What does real
What does real--world Movement really look like?
world Movement really look like?
��Multiples edges
Multiples edges can
can be
be
seen with some types of
seen with some types of
tracking & saccades
tracking & saccades
combinations ??
combinations ??
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
Marcel Duchamp (1912) Nude descending a staircase
Marcel Duchamp (1912) Nude descending a staircase
What does real
What does real--world Movement really look like?
world Movement really look like?
��Multiples edges
Multiples edges can
can be
be
seen with some types of
seen with some types of
tracking & saccades
tracking & saccades
combinations ??
combinations ??
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
��Motion blur can be seen if attentive ??
Motion blur can be seen if attentive ??
What does real
What does real--world Movement really look like?
world Movement really look like?
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
��Motion blur can be seen if attentive ??
Motion blur can be seen if attentive ??
What does real
What does real--world Movement really look like?
world Movement really look like?
Csuri (1984) @ OSU
Csuri (1984) @ OSU
��Human eye is poor for seeing m
otion
Human eye is poor for seeing m
otion
•Muybridge 1870s->
��Multiples edges can be seen with some types of
Multiples edges can be seen with some types of
tracking & saccades combinations ??
tracking & saccades combinations ??
What does real
What does real--world Movement really look like?
world Movement really look like?
��Motion blur can be seen if attentive ??
Motion blur can be seen if attentive ??
��Experience and attention have large effects on
Experience and attention have large effects on
motion perception
motion perception
References:
References:
Spatiotemporal analysis of displaying perceived object motion:
Spatiotemporal analysis of displaying perceived object motion:
•Frequency domain analysis, (Watson 85, Girod 93, Klompenhouwer 04)
•Spatiovelocity analysis (W
atanabe 68 Kelly 79, Adelson & Bergen 85, Daly 98,
Laird 06)
•Tim
e domain analysis (Adelson & Bergen 85, Pan 05)
•Motion blur perception (Ramachandran 74, Parker ’81, Westerink 90, Bex 95,
Takeuchi 05, Laird 06)
Motion blur in LCD:
Motion blur in LCD:
Understanding Motion Blur & LCD TV
Motion blur in LCD:
Motion blur in LCD:
•Caused by the hold-type temporal rendering m
ethod of LCDs combined with
the smooth pursuit eye m
ovement of human visual system (HVS)
–(Lindholm
96, Parker 97, Kurita 98, 01, Klompenhouwer 05, Pan 05)
Motion blur reduction approaches:
Motion blur reduction approaches:
•Temporal overdrive (Okumura 01, Sekiya, 02)
•Temporal aperture reduction: black data insertion -BDI (Hong 04, Kim
ura
05), backlight flashing (Fisekovic 01, Sluyterm
an 05)
•Frame rate doubling –
FRD (Sekiya 02, Kurita 05)
•Motion compensated inverse filtering –MCIF ( Klompenhouwer 01)
Thank you for your interest and patience
Thank you for your interest and patience
Reference Capability of Conference Projector
Reference Capability of Conference Projector
8bitramp
Bit Depth, Contrast, and Spatial CSF
�Displaywith:10bits,Cmax=0.95,Lmax=1000cd/m
^2(CSFpeak=500;upper50%)
Active Matrix Drive
Active element
(Transistor)
X electrode
Y electrode
Structure
�Transistors are attached to
each subpixel for precise and
X1
X2
Pixel
Y1
Y2
Circuitry
each subpixel for precise and
faster sw
itching of its gray
value
�X and Y electrodes are
form
ed on the same substrate
as the TFT array.
�Switching signals are
applied to the Y electrode.
Video is applied to the X
electrode
LCD Tonescale
20
40
60
80
10
0
12
0Transparency and derivative
of the S-curve function
0
020
40
60
80
100
Vo
ltag
e ap
plied
Transparency and derivative
of the S-curve function
VT
-cu
rve
der
ivat
ive
of
the
S c
urv
e fu
nct
ion
06
41
28
19
22
55
Corr
esp
on
din
g c
od
e val
ue
Fig
.5 S
-cu
rve
of
the
LC
D a
nd
its
der
iva
tive
.
�Voltage Transmission curve (device level tonescale)
�Inherent LCD tonescale is S-shaped, like film
LCD System Tonescale
�Gamma-correction via LUT usually performed
�for quality desktop LCD monitor, for LCTV other goals
Salient Characteristics of LCD: Brightness
You can see the difference when playing images whose entire area is bright.
Larg
e A
rea
Brightn
ess
CRT &
Plasm
a T
VAQUOS L
CD T
V
�No brightness dependence on area
�Independent Backlight + Transm
issive Modulation
230 cd/m
2
58 cd/m
2
Peak b
rightn
ess
470 cd/m
2
470 cd/m
2
Assessment along Image Quality Dim
ensions
2002
2005
0
0.2
0.4
0.6
0.81
Response Time
(msec)
Contrast
[Dark]
Power
Consumption
(W) ※20”
Set Price
(yen/inch)
Life
(hour)
Under
7 10
20
30
10000
10000
over
100%
80
60
40
20
200
300
200
100000
10000
1000
100
Under 30
50
70
90
20000
15000
10000
5000
Under
1000
80
40
2800
600
400
80
200
0.00
0.20
0.40
0.60
0.80
1.00
Response Time
(msec)
Power
Consumption
(W) ※20”
Set Price
(yen/inch)
Color Gamut
(vs.NTSC)
Contrast
[Bright
(300lx)]
Life
(hour)
Contrast
[Dark]
100000
10000
1000
100
Under 30
50
70
90
20000
15000
10000
5000
Under
1000
Under
7 10
20
30
10000
2800
1200
600
10000
80
500
over
100%
80
60
40
20
200
300
200
80
40
Color Gamut
(vs.NTSC)
Contrast
[Bright (300lx)]
High Speed Response Technology
Wide Color B/L System / High Transmissivity Panel
Optical Optimized Panel (Higher Contrast)
Brightness
[All screen (cd/㎡
)
Size
(inch)
Brightness
[Peak]
(cd/㎡
)
Pixel Density
(PPI)
100%
300
400
500
600
300 400 500
60
600
40
20
over
200
160
120
80
Brightness
[All screen]
(cd/㎡
)
Size
(inch)
Brightness
[Peak]
(cd/㎡
)
Pixel Density
(PPI)
400
500
600
300 400 500
60
600
4020
over
200
160
120
LCD (ASV)
CRT
PDP
OLED (EL)
Aside: Spatial Superadditivity of CRT
�Superadditivity is a nonlinearity that makes it hard to determ
ine MTF
(as well as apply linear systems theory in image processing applications)
�These m
easurements on b/w
CRT without shadow m
ask, which
introduces phase complications as well.
�Data from Naim
an ’92 (SPIE HVEI)
Salient Characteristics of LCD: Low Reflection
�
Comparison of Reflection
When the room brightness is 300 LUX (normal brightness), liquid
crystal's low reflection rate greatly reduces mirror effects.
LCD TV
Plasma or CRT TV
LC TV Status 2006
0 000
0.2
0.20.2
0.20.4
0.40.4
0.40.6
0.60.6
0.60.8
0.80.8
0.81 111
Resp
onse
Tim
eR
esp
onse
Tim
eR
esp
onse
Tim
eR
esp
onse
Tim
e(R
oom
Tem
p.)
(Room
Tem
p.)
(Room
Tem
p.)
(Room
Tem
p.)
Resp
onse
Tim
eR
esp
onse
Tim
eR
esp
onse
Tim
eR
esp
onse
Tim
e(L
ow
Tem
p)
(Low
Tem
p)
(Low
Tem
p)
(Low
Tem
p)
Contr
ast
Contr
ast
Contr
ast
Contr
ast
(Brigh
t)(B
righ
t)(B
righ
t)(B
righ
t)
Contr
ast
Contr
ast
Contr
ast
Contr
ast
Brigh
tness
Brigh
tness
Brigh
tness
Brigh
tness
Pow
er
Cons.
Pow
er
Cons.
Pow
er
Cons.
Pow
er
Cons.
Vie
win
g A
ngl
eV
iew
ing
Angl
eV
iew
ing
Angl
eV
iew
ing
Angl
e
2004
2004
2004
2004
NOW
NOW
NOW
NOW
( usual “sales” caveat)
�Other Remaining Challenges
�High Dynamic Range, Wide Color Gamut, High-frame rate, Cost
0 000C
ontr
ast
Contr
ast
Contr
ast
Contr
ast
(Dar
k)(D
ark)
(Dar
k)(D
ark)
Colo
r G
amut
Colo
r G
amut
Colo
r G
amut
Colo
r G
amut
(EB
U)
(EB
U)
(EB
U)
(EB
U)
Colo
r D
epth
(10 b
it)
Colo
r D
epth
(10 b
it)
Colo
r D
epth
(10 b
it)
Colo
r D
epth
(10 b
it)
Bla
ck
Shift
B
lack
Shift
B
lack
Shift
B
lack
Shift
Brigh
tness
Brigh
tness
Brigh
tness
Brigh
tness
[Peak
][P
eak
][P
eak
][P
eak
]
Brigh
tness
Brigh
tness
Brigh
tness
Brigh
tness
[Avg
.][A
vg.]
[Avg
.][A
vg.]
2004
2004
2004
2004
CR
TC
RT
CR
TC
RT
Eccentricity and Periphery
Eccentricity : Position in visual field
Eccentricity : Position in visual field
•0 degrees eccentricity refers to w
here your eyes are pointed, corresponds to
fovea in retina
•90 degrees eccentricity is near edges of visual field (periphery)
Spatial Bandwidth of eye reduces in periphery
Spatial Bandwidth of eye reduces in periphery
Cones are densely packed in fovea : high spatial sampling
Cones are densely packed in fovea : high spatial sampling --> high bandwidth
> high bandwidth
They become less dense as eccentricity increases
They become less dense as eccentricity increases
100
101
100
101
102
103
CSFs for Eccentricity Range 0-25 deg
Spatial Frequency (cy/deg)
Contrast Sensitivity
Properties of Visual System: Eccentricity
��How eccentricity changes across image as viewing distance changes
How eccentricity changes across image as viewing distance changes
(left)
(left)
•Assuming viewer looking at center of im
age (pixel = 320)
��Eccentricity m
odel predictions of how visual spatial sensitivity varies across image
Eccentricity m
odel predictions of how visual spatial sensitivity varies across image
(right)
(right)
1H at pixel=320
2H ""
4H ""
30
Eccentricity vs. Location (640x480 image)
1H at center
2H ""
4H ""
1
Visual Sensitivity vs. Location
6H ""
0100
200
300
400
500
600
05
10
15
20
25
Image location (pixel)
Eccentricity (deg)
6H ""
2H at 160
0100
200
300
400
500
600
0
0.2
0.4
0.6
0.8
Image location
Sensitivity
Eye Movement Model –maxim
um smooth pursuit velocity
��Rotation into Spatiotemporal CSF with Eye Movement model
Rotation into Spatiotemporal CSF with Eye Movement model
��Max smooth pursuit velocity condition dependent, some unknowns
Max smooth pursuit velocity condition dependent, some unknowns
•Predictive vs. non-predictive m
otion
•Oscillation : eye m
ovements can actually lead
•Field of view
2
2.5
log temporal frequency (cy/sec)
EMM (max 2 deg/sec) ST CSF
2
2.5
log temporal frequency (cy/sec)
EMM (max 8 deg/sec) ST CSF
2
2.5
log temporal frequency (cy/sec)EMM (max 30 deg/sec, Westheimer) ST CSF
-1-0.5
00.5
11.5
2-0.50
0.51
1.5
log spatial frequency (cy/deg)
log temporal frequency (cy/sec)
-1-0.5
00.5
11.5
2-0.50
0.51
1.5
log spatial frequency (cy/deg)
log temporal frequency (cy/sec)
-1-0.5
00.5
11.5
2-0.50
0.51
1.5
log spatial frequency (cy/deg)
log temporal frequency (cy/sec)
Eye Movement Model –maxim
um smooth pursuit velocity
��Rotation into Spatiotemporal CSF with Eye Movement model
Rotation into Spatiotemporal CSF with Eye Movement model
��Max smooth pursuit velocity condition dependent
Max smooth pursuit velocity condition dependent
•Predictive vs. non-predictive m
otion
•Oscillation : eye m
ovements can actually lead
•Field of view
��If pursuit could be as high as 1200
If pursuit could be as high as 1200 deg
deg/sec , looks like traditional ST
/sec , looks like traditional ST
CSF, but with higher flicker fusion
CSF, but with higher flicker fusion
CSF, but with higher flicker fusion
CSF, but with higher flicker fusion
-1-0.5
00.5
11.5
2-0.50
0.51
1.52
2.5
log spatial frequency (cy/deg)
log temporal frequency (cy/sec)
EMM (max 1200 deg/sec) ST CSF
Measured eyetracks
Measured eyetracks
��Transfer relative eye positions to degrees:
Transfer relative eye positions to degrees:
Eye tracking data of a person tracking a m
oving Gabor
�velocity variations due to instrumentation or physiology ?
LCD Flickering
�Asymmetric
temporalresponsesleadto
HSFflicker(HighSpatialFrequency)
INPUT SIGNAL (slow))
DISPLAY RESPONSE
INPUT MEAN
MOVING IMAGE PATTERN SEEN AT SINGLE PIXEL
(Fast)
DISPLAY MEAN
TIM
E
LUMINANCE
DISPLAY MEAN
DISPLAY MEAN
(Faster)
SLA’s Hi-speed Image Measurement System
��The camera captures im
age sequences at at least 4
The camera captures im
age sequences at at least 4
times of the LC TV frame rate (4x60). {Dalsa, Uniq}
times of the LC TV frame rate (4x60). {Dalsa, Uniq}
Co
mp
ute
r
��The camera captures im
age sequences at at least 4
The camera captures im
age sequences at at least 4
times of the LC TV frame rate (4x60). {Dalsa, Uniq}
times of the LC TV frame rate (4x60). {Dalsa, Uniq}
��One computer drives both the camera and LC TV to
One computer drives both the camera and LC TV to
synchronize the data capture and processing.
synchronize the data capture and processing.
��The algorithm & software developed by SLA m
akes
The algorithm & software developed by SLA m
akes
data capture and processing 100% automatic and
data capture and processing 100% automatic and
real
real--time.
time.
��The m
easurement is
The m
easurement is fast
fast(about 20 m
inutes for each
(about 20 m
inutes for each
temperature) and the hardware system is
temperature) and the hardware system is compact
compact..
Application for comparing types of Overdrive
50
100
150
200
250
300
20
30
40
50
60
70
80
90
100
Fir
st O
rder
Dyn
amic
Gam
ma
No
n-m
od
el B
ased
OD
Mo
del
Bas
ed O
D
050
100
150
200
250
300
00
10
20
30
40
50
60
70
80
90
100
0
10
050
100
150
200
250
300
-500
50
100
150
200
250
300
Target Level
Output Level
050
100
150
200
250
300
-500
50
100
150
200
250
300
Target Level
Output Level
Sec
on
d O
rder
Dyn
amic
Gam
ma
150
200
250
300
Temporal Response
Definition of second order dynamic gamma
Zn+
1
Zn
Fir
st o
rder
Dyn
amic
gam
ma
Sec
on
d o
rder
Dyn
amic
gam
ma
dn=
f(d
n-2
, z n
, z n
+1)
-6-4
-20
24
68
-500
50
100
150
Time in Frame
Temporal Response
Fra
me
No.
0
1
Zn-1
Z
n-2
Z
n-2
2
�S
econd o
rder
DG
model
s m
oti
on o
f m
ore
com
ple
x i
mag
e, t
hus
it i
s m
ore
rea
list
ic m
easu
re o
f m
oti
on b
lur
of
real
vid
eo
Examples: second order responses
150
200
250
300
Temporal Response
150
200
250
300
Temporal Response
-5-4
-3-2
-10
12
34
-500
50
100
Time in Frame
Temporal Response
-5-4
-3-2
-10
12
34
-500
50
100
150
Tim
e in Frame
Temporal Response
Analysis of Backlight Flashing
Analysis of Backlight Flashing
Further analysis for backlight flashing
Further analysis for backlight flashing
��With backlight flashing, the m
otion blur is greatly
With backlight flashing, the m
otion blur is greatly
reduced. The m
otion image looks clear and shows
reduced. The m
otion image looks clear and shows
more realism
more realism
��Object Evaluation
Object Evaluation
•The display output was captured w
ith a 240 Hz
camera
camera
•Retina image was derived from integration along
the m
otion trajectory (assuming perfect eye
tracking, a la Laird)
��Subjective Evaluation
Subjective Evaluation
•to quantify the amount of motion blur relative to
the perception of still im
age blur at various
backlight flashing w
idths
Der
ive
reti
na
imag
e fr
om
cap
ture
d f
ram
e b
y i
nte
gra
tion a
long
the
moti
on t
raje
ctory
Predicting Retinal Image
No
Ov
erd
riv
e, n
o f
lash
ing
No
ov
erd
rive,
fla
shin
g a
t 1
/8
du
ty c
ycl
e
No Overdrive
No Overdrive
Mu
ltip
le e
dg
es !
Using Overdrive
Using Overdrive
Ov
erd
rive,
No
fla
shin
gO
ver
dri
ve,
fla
shin
g a
t 1
/8
du
ty c
ycl
e
Motion Sharpening Study
Motion Sharpening Study
��Hammet & Bex ’95, tried to parse out:
Hammet & Bex ’95, tried to parse out:
��Sharpness constancy : don’t see blur until
Sharpness constancy : don’t see blur until
error signal of blur is visible
error signal of blur is visible
��Deblurring: neural processing generates
Deblurring: neural processing generates
high spatial frequencies
high spatial frequencies
��Blur matching experiment:
Blur matching experiment:
•1cy/deg Moving sine vs. blurrable square w
ave
•Unadapted vs. adapted to high SF
•Unadapted vs. adapted to high SF
•Adapted assumed to knockout or at least
reduce high SFs
•Decrease on plot means more Motion
Sharpening
•Sharpness constancy (top-down version) should
not be affected by high SF adaptation
•Data indicates some Deblurring processing
��Experiment flaw : sine waves don’t blur
Experiment flaw : sine waves don’t blur
with LPF
with LPF
Observer Study on Perceived Motion Blur
Observer Study on Perceived Motion Blur
with Backlight Flashing
with Backlight Flashing
Observer study for Motion blur matching with edge
Observer study for Motion blur matching with edge
��Subjective experiment
Subjective experiment
•Compare the perceived m
otion blur with still image blur
•Characterizing the effectiveness of backlight flashing in reducing
motion blur
•Can determ
ine if “motion sharpening effect” occurs
��Experiment conditions
Experiment conditions
Experiment conditions
Experiment conditions
•4 m
oving speeds: 27, 53, 80, 94 degrees per second
•4 flashing duty cycle: 1 (hold), ½
, ¼, and 1/8
•6 observers
��Feng HVEI 06 (SPIE Electronic Imaging Conference)
Feng HVEI 06 (SPIE Electronic Imaging Conference)
Observer study task
Observer study task
Per
fect
shar
p
edge
movin
gS
imu
late
d e
dg
e ≈
Mea
sure
d
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
�G
auss
ian e
dge
sim
ula
ted w
ith e
dge
wid
th p
aram
eter
σ
�O
bse
rver
adju
sts σ
to i
ncr
ease
the
blu
r w
idth
and/o
r ad
just
vto
reduce
the
blu
r w
idth
.
�M
ovin
g e
dge
com
par
ed a
gai
nst
sti
ll e
dge
edge
movin
gS
imu
late
d e
dg
e ≈
Mea
sure
d
reti
na
blu
r as
sum
ing
per
fect
trac
kin
g
y = 0.9286x - 0.9576
R2 = 0.9919
0.00
5.00
10.00
15.00
20.00
25.00
30.00
Subjective Matched Width
Results of Motion Blur Matching
Results of Motion Blur Matching
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
-5.00
0.00 0
510
15
20
25
30
Modeled Width
�Modeled width = sim
ulated retinal blur, assuming perfect eye tracking
�Overdrive with different combinations of speed and flashing duty cycles
(including no flashing)
�The regression line shows good correlation with a slope of 0.93, which is
very close to the unit slope. This indicates the perceived blur closely m
atches
the retina blur.
�Motion sharpening effects did not occur (too m
uch m
asking due to edge?)
��The LCD m
odel blur is m
odeled using Fourier
The LCD m
odel blur is m
odeled using Fourier
analysis: It is the combination of display
analysis: It is the combination of display
temporal low pass filtering and eye tracking
temporal low pass filtering and eye tracking
��To reduce m
otion blur
To reduce m
otion blur � ���� ���
improve the display
improve the display
temporal MTF
temporal MTF
Summary of motion blur study
Summary of motion blur study
()
)(
*)
()
/(
)(
)(
/σx
bv
xgaus
rect
vx
tri
xst
epx
blu
rEdge
∗=
•Temporal overdrive to improve LCD temporal response
•Reduce the temporal aperture function to reduce
motion blur due to hold
��Implemented overdrive and backlight flashing
Implemented overdrive and backlight flashing
on a LCD with LED backlight.
on a LCD with LED backlight.
•The LED can be flashed at various duty cycle in sync
with LCD driving
Distribution of Text Scroll Velocity in TV Broadcasting
Average; ;;;13.8 [deg/sec]
Aver
age;
13.8
Max
;35.9
From 71 programs of BS-digital broadcasting
20
30
40
50
60
頻度(%) Frequency (%)
“Telop”, which doesn't contain camera blur, mostly
distributes below 20[deg/sec]
Fujine et.al.;”Real-Life In-Home Viewing Conditions for FPDs and Statistical Characteristics of
Broadcast Video Signal,” Digest AM-FPD’06
0
10
20
0~
10
10~
20
20~
30
30~
40
40~
50
50~
60
視角
速度
(deg/sec)
頻度(%)
Vel
oci
tyin
vie
win
gan
gle
(deg
/sec
)[3
H]
Frequency (%)
MPRT m
easurement equipment
MPRT m
easurement equipment
�Photal –Otsuka electronics MPRT-2000
�Photal –Otsuka electronics MPRT-2000
-Equipping a pursuit CCD camera enables the evaluation close to that by eye perception
-A unique algorithm as a tim
e-based norm
alization perm
its the comparison among the
different types of displays
-Processes for measurements and analyses are computer-controlled including for moving
picture displays on the sample FPD
-Moving picture characteristic estim
ation for optional bitmap pictures
-Expansion of lineup of selections for many purposes・
The system w
ith a color CCD camera, the addition to the conventional system w
ith a
luminance filtered CCD camera ・
The pursuit color camera for a small display, the
addition to the conventional system for a m
edium and large display
Motion Blur and Temporal Response
Motion Blur and Temporal Response
Without Hold Effect (LCD is driven at infinite high frame rate),
the retina image is the temporal response function of a step
function Et(t)
)/
()
(v
xE
xE
t−
=With finite frame rate, motion blur is the
)(
)/
()
(
)(
)(
)(
)(
)(
0
0
/
vtx
dtd
t
vx
ttt
rect
vx
Ex
E
tE
th
dt
rect
th
xE
⊗−
==
⊗=∫
−
∞−
h(t
)
With finite frame rate, motion blur is the
convolution of the temporal response and the
temporal aperture function
x