video course: image and resolution enhancement 1dehaan/slides/2_moderndisplaysnew.pdf28 consequences...

1Video course: Image and Resolution Enhancement

Centre for Technical Training © Koninklijke Philips Electronics N.V. 2003

technische universiteit eindhoven

Video processing

G. de Haan


Schedule lectures 5P5302

Week 1 Week 2 Week 3 Week 4

Basics

(Ch 2, 3)

Video displays

(Ch 9)

Filtering

(Ch 4)

Picture-Rate

Conversion (Ch 7)

Week 5 Week 6 Week 7 Week 8

De-interlacing

(Ch 8)

Motion Estimation

(Ch 10)

Object Detection

(Ch 11) X


3

Short

recapitulation


4 3D frequencies in video

time

fh (c/pw)

fv (c/ph)

ft (c/pp)


5 The sampling theorem

time

We have a continuous signal

fsfs

2

0 frequency

Sampling theorem: we can reconstruct the continuous signal

from its discrete representation, provided it contained no

frequencies above half the sampling frequency

We sample it to obtain a discrete representation

Ts


6 What is the required sampling grid? (vertically)

• Limit eye: 30 cycles/degree

• At 6x picture height, viewing angle: ~10 degrees

• finest pattern on screen: 10*30=300 cycles

• Sampling theorem: we need >600 samples (lines)




7 What about the temporal sampling density?

Temporal response of average viewer + sampling theorem 100Hz?

Unfortunately, there is a complication…


8 Temporal frequency above Nyquist limit imaging system

time

Trackingeye

Temporal sampling

with e.g. 50 Hz

Temporal frequency of

e.g. 120 Hz!


9 Conclusion for the picture-rate

• Sampling causes ambiguity for frequencies >fs/2

• If high T-frequency due to motion, correct eye-tracking can transform it to DC

• For correct eye-tracking coarser patterns (below Nyquist) must be part of the same moving object.

• In practice little ambiguity, and minimal picture rate determined by flicker

demo


10

Colour and the

eye


11 The human eye

•Two type of receptors: rods and cones

• Colour perception only with cones, (at higher brightness levels)

• Rods and cones are at the back of the retina, the nerves are at front


12

Consequences

of colour vision

for technology




13Every wavelength between 400nm and 700nm

corresponds to a colour


Sun-light contains a continuous spectrum of

wavelengths, which appears white14


An incandescent lamp imitates the sun15

Continuous spectrum (black-body radiator), lower T


But we can make white light much more efficiently16

Spectrum fluorescent lamp


17

• Used in displays and in fluorescent lamps

• Based on red, green and blue primary =

RGB-model, sometimes white added for

improved efficiency (RGBY)

• Primaries defined by emission of phosphors

or LEDs (lamps, CRT, PDP, OLED), or

colour filters (LCD)

Additive colour mixing


18

Colour vision

and display

technology




19 Colour synthesis – Optical superposition (projection)


20 Colour synthesis – temporal synthesis (colour

sequential)

Circle

colseq


21 Colour synthesis – spatial synthesis (CRT, LCD, PDP)

Shadow-mask (colour CRT) Matrix display panel


22

More

consequences of

motion tracking


23 The retina of the eye has a slow response (integration

effect)


24 If we do not track the motion, objects get blurred

Same thing as motion blur

with long shutter time




25 Object tracking with the eye

Position on screen


26 A moving ball on the retina of the tracking eye

Position on screen

picture number

Po

sit

ion

on

reti

na

n-1 n n+1 n+2


27 With correct tracking, sharp object


28 Consequences of motion tracking of the eye

• A display image is projected on a moving target (retina of the eye)

• If the eye moves substantially while the pixel is shown, the pixel size increases (blur)

• A pixel is only valid at a single point in space AND time, spatial or temporal spreading or mis-positioning degrades quality

• If pixels are shown with different delays, their relative position is modified by the eye movement

• A single TV-image need not be complete, if soon enough complementary information is shown (interlace, colour sequential display)

• The quality of individual images is relevant for photography – Video is different!


29 Demo of difference video and photography

• A single image may contain alias

• But either the pixel-grid should move…

• Or the image-content should move…

• And the image must be stationary on the retina of

the eye, so:

• Stationary on the screen, fixed eye…; or

• Moving over the screen, tracking eye…; but

• Tracking only possible with predictable motion…


30 Why problems with displays?

• No artifacts as long as the delay from input to output is the same for all pixels

• Same scanning order not guaranteed with diverging imaging and display technologies

x

y

x

y

x

y

MuxDe-

mux x

y

Memoryre-order




31

Popular display

concepts


32 A revolution started in 2000!

Source : Sony + Nikkei Electronics 2002-09-09

0

100

200

300

400

500

600

700

2000 2001 2002 2003 2004 2005

PDP TV

LCD TV

PJ TV

CRT TV

15%

8%

39%

38%

94%

28%

48%65%

82%

TV Shipment in Japan by Technologies (billions yen)


And a new winner seems to emerge…33

After the break, we shall see that video needs to be adapted to display


Video processing

G. de Haan


35

The cathode-Ray

Tube


36 Cathode ray tube

• 100 year old workhorse

• Beam scan device

• Now almost obsolete…

• Pixels are shown one-after-the-

other, i.e. not simultaneously,

and are visible for a very short

time (~0.1ms)!






• So for CRT no problem if also camera scans the pixels one-after-the-other

x

y

x

y

x

y

MuxDe-

mux x

y

Memoryre-order


38

The Liquid Crystal

Display


39

• A voltage across the Liquid

Crystal material changes the

polarization of the light

• With two polarizing filters,

around the crystal, the

intensity of the light traveling

through the panel can be

modulated

LCD principle


40

• Most mature flat panel technology

• Major share of FPD market

• Pixels are shown line-by-line, and

are visible the entire picture-period,

unless the backlight is non-

continuous

Liquid Crystal Display




• So for LCD ‘no problem’ if also camera scans the pixels line-by-line

x

y

x

y

x

y

MuxDe-

mux x

y

Memoryre-order


42

The Plasma

Display Panel




Plasma display panels; when it must be really large43


44PDP principlean array of small fluorescent lamps, i.e. an emissive matrix-type display


45 Sub-field addressed displays (grey with on/off display)

time

Po

sit

ion

(p

ixe

ls)

sf01 2 3 4 5 6 7

sf0...

Current frame Next frame

255

128

54

75

159

32

64

1 2 4 8 16 32 64 128


46 Plasma Display Panel

• Large Displays, up to 150”

• High Resolution

• High Brightness

• Good Contrast

• Pixels are shown

simultaneously and are

either on-or-off, i.e. making

intermediate values requires

sub-fields, so pixels are

shown with different

duration!




• So, rather unclear how the camera should operate to achieve this….

x

y

x

y

x

y

MuxDe-

mux x

y

Memoryre-order


48

The Digital Micro-

Mirror Device

(DLPTM)




49 Digital Mirror Device

• The DMD is an array of

several aluminium

mirrors, each of which

acts as a light switch.


50 DMD principle

• Each mirror can rotate

in one of two direction:

• +10 degrees

• -10 degrees

• + 10 degrees is ON

• - 10 degrees is OFF


51 Mirrors can only switch pixel on and off


52 DMD is also an on/off display (binary)

time

Po

sit

ion

(p

ixe

ls)

sf01 2 3 4 5 6 7

sf0...


255

128

54

75

159

32

64

1 2 4 8 16 32 64 128


53 DMD

• High resolutions are possible and available on the market

• No polarizers required higher efficiency than LCD

• Excellent contrast

• No electrodes visible on screen

• Pixels are shown simultaneously and are either on-or-off, i.e.

making intermediate values requires sub-fields, so pixels are

shown with different duration!


54 So what problems do we have with displays?


• If pixels are shown with different delays, their relative position is modified by the eye movement

• CRT flashes pixels one-after-the-other

• LCD hold pixels complete picture-period

• DMD and PDP use PWM, i.e. pixel timing depends on brightness…

Motion




55

Motion blur and

LCDs


56 Motion blur on display and camera

Camera

The object moves relative to a

stationary camera that integrates the

image during the shutter time

Hold time

Shutter time

The eye smoothly follows a moving

object that remains stationary during

the display hold time

Display


57 Dynamic resolution – Sharpness of moving images

Object moves while

shutter of photo-

camera is open…

Eye moves over TV-screen, while

image remains the same during

picture period…

analogy

Two phenomena with the same result, i.e. motion blur:


58 Motion blur on S&H displays: NOT because LCD too slow


59 It is NOT because the LCD is too slow

Frame store

Video in Video out

start

desired (over)drive

Overdrive LUT

THAT is solved with a technique called “overdrive”

time (frames)

Lum

-1 0 1 2 3 4

time (frames)

Lum

-1 0 1 2 3 4


60 Consequences of motion tracking ability of viewer

• We should NOT (temporarily) display objects at other

places than where the viewer expects them

• A picture element (pixel) is only valid at ONE point in time!

• The relative timing of images, or image parts, in an image

sequence may NOT be altered




Video processing

G. de Haan


62 Why problems, and what are they?


• Same scanning order not guaranteed with diverging imaging and display technologies

x

y

x

y

x

y

MuxDe-

mux x

y

Memoryre-order


63

• Delay is a linear function of the spatial position:

• Motion translates time differences into space differences:

• Therefore, motion causes geometrical distortions:

ycxcct 321

First category of artifacts due to technology divergence

tvx

vycvxcvcx

321


64

Scanning camera

and display


Tube-to-tube (the original video chain)65

Scanning order registrationScanning order display


66 Tube to tube

0

0

0

0

0x

Circle

motion




67

FT-CCD camera

and LCD


Frame-transfer CCD to LCD68

Simultaneous registration Line-sequential display


69 FT-CCD to tube or LCD

0

20

0

20

vycx

3


70 Explanation of the trapezium distortion

1. The image is shown on the

screen as captured by

the camera …

2.The eye tracks the average

motion

3. The bottom line is shown

20ms later than the top line4. And will, therefore, land on a

shifted position on the retina _____


71

Scanning camera

and PDP


Tube-to-PDP72

Scanning order registrationSimultaneous display




73 Tube to plasma panel

0

-20

0

-20

vycx

3



The image moves

to the right …

The bottom line is registered

20ms later than the top lineTherefore, the image is

distorted at the display, when it

shows all pixels simultaneously ____


75 Intermediate conclusion

• Category 1 imaging and display combinations yield

geometrical distortion with motion

• This degradation is usually seen as mild, and totally

acceptable for consumer vision


76

• Delay is a linear function of the spatial position, and motion. Therefore causes geometrical distortions:

• but the parameters cn are no constants, but vary as a function of:

• The position on the screen• Tiled displays, e.g. vidiwall and large size LCDs

• The picture number • Film on TV, TV on PC, 100 Hz TV

• The colour channel • Colour sequential display

• The bit slice of the digitised video• Plasma display panels

Second category of artifacts

vycvxcvcx

321


77

Tiled displays


Example tiled display78




79 FT-CCD to multi-CRT/LCD vidiwall

0

20

0

20

20 20

0 0

)(,

)(,

23

21

bottomvycvc

topvycvc

x


80 Explanation of the distortion

1. The image moves

to the right …

2. The bottom line of every tile

is written 20ms later than

the top line

3. So, the bottom line of the top tiles don’t

match with the top line of the bottom tiles


81

Picture repetition

displays


25 Hz film on 50 Hz CRT82

50Hz Line-sequential display25Hz simultaneous

registration


83 25 Hz film to 50Hz CRT

0/20

20/40

0/20

20/40

)_(,

)_(,

23

21

imageevenvycvc

imageoddvycvc

x



1. The image is shown on the

screen as captured by

the camera …

2. The eye tracks the average

motion

3. The bottom line is shown

10ms later than the top line4. and 10 ms later, the image is repeated




85 Motion rendering, picture repetition

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

n n+1 n+2


86 Motion rendering, picture repetition

frame period, T = hold time, ti

time

Po

sit

ion

(p

ixe

ls)

n n+1 n+2

Result

seen by

viewer

tracking

the

motion

Renata


87

n-1

MC-picture interpolation

n

n-1/2

picture number

D

2

1

D

2

1


88

Colour sequential

displays


89 Single-panel, colour sequential, projection system

Circle

colseq

Color

wheel

Lens

Lens

Lamp

ScreenLight

valve


90 Advantages and drawbacks of colour sequential

• Cost advantage:

• Only one light valve device (LCD, DMD) required

• Spatial resolution advantage:

• No division of pixels over RGB, i.e. resolution loss

• Higher speed drawback:

• In one picture period 3 (R, G, B) images must be shown

• Motion portrayal drawback:

• R, G, and B images NOT shown at the same time




Video processing

G. de Haan


92 Motion rendering, color sequential

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

n n+1 n+2


93 However, if objects move…

The eye will track the average motion

And red, green and blue land at

different locations of retina


94 Motion rendering, color sequential

time

Po

sit

ion

(p

ixe

ls)

n n+1 n+2

Result

seen by

viewer

tracking

the

motion


95 FT-CCD to colour sequential display

0

7

14

0

7

14

0

7

14

0

7

14

)(,

)(,

)(,

3

2

1

bluevc

greenvc

redvc

x

Colseq

moving


96

Bit-sequential

displays




97 What if the display can only be “on” or “off”

• Plasma Display Panels (PDPs) are basically arrays of

small fluorescent lamps in three colours that can be

individually switched on or off.

• Digital Micro-mirror Devices (DMDs) are basically arrays

of mirrors that can be slightly tilted to either totally reflect

light to the screen, or deflect it in a direction where it will

not pass through the projection lens


98 PDP is an array of miniature fluorescent lamps


99 DMD is an array of miniature mirrors that can move


100 Sub-field addressed displays, like PDP and DMD

time

Po

sit

ion

(p

ixe

ls)

sf01 2 3 4 5 6 7

sf0...


255

128

54

75

159

32

64

1 2 4 8 16 32 64 128


101 Motion artifact, moving bars

127

128

time

po

sit

ion

sf01 2 3 4 5 6 7

sf0...

255

0

127

128

?

current

frame

next

frame


102 Motion rendering, PDP

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

n n+1 n+2yvonne




103 Motion rendering, PDP

time

Po

sit

ion

(p

ixe

ls)

n n+1 n+2

Result

seen by

viewer

tracking

the

motion


104 Understanding the artifacts

1. The image moves

to the right …

2. The eye tracks the average

motion

3. The bit-slices appear sequentially in time,…

and therefore land at different locations on the retina


105 FT-CCD to Plasma Display Panel (PDP)

)(,

......................

......................

)2(,

)(,

3

2

1

LSBvc

MSBvc

MSBvc

x

nd


106 Motion compensation: Shift bits

time

po

sit

ion

(pix

els

)

sf01 2 3 4 5 6 7

sf0...

current

frame

next

frame

011 1 1 1 11

motion vector

0

1

1 1

1 1

1

no

MC

shift

bit


107 Motion compensation: Shift bits

time

po

sit

ion

(pix

els

)

sf01 2 3 4 5 6 7

sf0...

t

x

hole

double

assignment

current

frame

next

frame

011 1 1 1 11

motion vector

0

1

1 1

1 1

1


108 Alternative approach: Do not shift but fetch values

timesf01 2 3 4 5 6 7

sf0...

po

sit

ion

(pix

els

)

original

frame

current

sub-frame

fetch1

135

123shift

?0?

125

99




109 Results comparison

no MC

shift bit

fetch bit

SA



• Category 2 imaging and display combinations yield spatial

discontinuities, false colours, false contours and echoes

• These degradations are usually very annoying, and repair

is almost necessary for consumer vision

• Motion compensated temporal interpolation

• True-motion estimation required

• Interpolation of bit planes not trivial


111

Sample & hold

displays


112 Third category of artifacts

• Non-impulsive displays

• Light results as the integral over a

• fixed amount of time (LCD)

• a data dependent amount of time (PDP)

• No single delay can be defined between input and output

pixels


113 FT-CCD to LCD (fixed integration time)

0-20

0-20

0-20

0-20


114 Motion rendering LCD

Previous

frame

time

Po

sit

ion

(p

ixe

ls)

Current

frame

tracking

the

motion

Next

frame

Result

seen by

viewer

intensityn n+1 n+2

frame period, T, hold time, ti




115

First solution:

Change backlight


116 Motion rendering, flash backlight when LCD stable

frame period, T

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

tracking

the

motion

n n+1 n+2


117

The scanning speed is synchronized with the speed of

entering information to the screen.

Flash the backlight when LC reaches desired value

Scanning Backlight, practical implementation

Frame time

Address Flash


118




Frame time

Address Flash

Scanning Backlight


119




Frame time

Address Flash

Scanning Backlight


120




Frame time

Address Flash

Scanning Backlight




121




Frame time

Address Flash

Scanning Backlight


122




Frame time

Address Flash

Scanning Backlight


123




Frame time

Address Flash

Scanning Backlight


124 Motion rendering, scanning backlight LCD

hold time, ti < frame period, T

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

n n+1 n+2

Backlight: off on


125 Motion rendering, scanning backlight LCD

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

tracking

the

motion

n n+1 n+2

hold time, ti < frame period, T

Backlight: off on


126 This is what it looks like




127

Lamp

driver

8 light

sources

Reflective

polarizer

diffuser

collimator

see through view of an Aptura scanning backlight

This is what it looks like

See through view of an Aptura scanning backlight


128 Alternative: Scanning Highlighting Backlight

stripe in scattering mode

is scanned through

the light guide

light output

common electrode

Glass substrates row electrodesLight source


129

Second solution:

Change picture-rate


130 Simplest option: Black frame insertion

Original imageintermediate image:

black

Combined image as

seen by viewer

input frame periodtime

(frames)

n

n+1

Shortening the light emission time of each frame decreases motion blur


131 Motion rendering standard LCD

Previous

frame

time

Po

sit

ion

(p

ixe

ls)

Current

frame

tracking

the

motion

Next

frame

Result

seen by

viewer

intensityn n+1 n+2

frame period, T, hold time, ti


132 Motion rendering, black frame insertion

frame period, T/2 = hold time, ti

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

tracking

the

motion

n n+1 n+2

Insert black

frame




133 Improved option: Motion compensated picture rate

doubling

frame period, T = hold time, ti

time

Po

sit

ion

(p

ixe

ls)

n n+1 n+2

Result

seen by

viewer

tracking

the

motion

motion

compensated

interpolation


134 Improvement with MC picture-rate doubling

frame period, T/2

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

tracking

the

motion

n n+1 n+2


135 Blur of BFI same as MC picture-rate doubling

frame period, T/2

time

Po

sit

ion

(p

ixe

ls)

Result

seen by

viewer

tracking

the

motion

n n+1 n+2

• But with half intensity and reduced contrast!


136

Third solution:

Video pre-

processing


137 This is our problem:

LCD & EYE (Spatial low-pass filter)

Compensation

(Spatial high-pass filter)

H(f)1

H(f)


138 A video processing solution: Inverse filtering

LCD & EYE (Spatial low-pass filter)

Compensation

(Spatial high-pass filter)

H(f)1

H(f)




139 Motion compensated inverse filtering MCIF

LCD&EYE

Spatial Low-Pass Filter

video

input

Simple filter few taps, rotate with motion direction, tap-distance

varies with motion vector length

Motion Compensated Inverse Filter

Motion

estimation

High-pass

filter

+

Orientation &

Tap-distance


140 Results: (simulation)

MCIF

Display + eye-

tracking (simulation):

Video on the screen:

Original


141 Options to improve motion on LCD

1. Pulsing backlight

2. Shorten picture time (more pictures/second)

3. Sharpening dependent on object velocity

50 pictures/

second

100 pictures/

second


142

Motion sharpness

Slow LCD Fast LCD 100Hz scan +MCIF

LCD motion portrayal improvement



• Category 3 display artifacts yield spatial blur proportional

to motion (and depending on brightness)

• This degradation is usually annoying, and repair is

required for consumer vision

• Flashing or scrolling back light

• Or black field insertion (inferior)

• Picture rate increase with motion compensation

• Inverse filtering (velocity dependent spatial filtering)


144 Display model and remedial video processing

(Variable)

spatial filter

Model for light

integration

Variable

delay

Model for scan

raster

mismatch

Display model

V-steered

“inverse”

filter

Integration

artifact

reduction

MC Picture

interpolation

Scan

mismatch

artifact

elimination

In Out

Remedial processing




145 Consequences of motion tracking of the eye

• A display image is projected on a moving target (retina of the eye)

• If the eye moves substantially while the pixel is shown, the pixel size increases (blur)


• If pixels are not shown simultaneously, there relative position is modified by the eye movement

• A single TV-image need not be complete, if soon enough complementary information is shown (interlace, colour sequential display)

• The quality of individual images is relevant for photography – Video is different!

video course: image and resolution enhancement 1dehaan/slides/2_moderndisplaysnew.pdf28 consequences...

Documents