Current Work on Video Coding

Raj Kumar

Overview Fine-Grained Scalability (FGS)

What is it and why do we need it What are its drawbacks

Our new scheme (FGS+) B-Frame only scheme All-Frame Scheme

Future Directions

Fine Grained Scalability What is it What are the pros and cons

I PB B B Base Layer

Enhancement Layer

Traditional Vs Fine-grained Encoding

I BPBB

I PB B B Base Layer

Enhancement Layer

Temporal FGS

I P Base Layer

Spatial Enhancement

Layer

Temporal Enhancement

Layer

Problem with Temporal FGS Reference Frames are poor quality Doesn’t use motion-prediction well Compression ratio suffers

compared to traditional encoders (approx 2-2.5 DB)

Insight FGS provides two

degrees of freedom SNR Temporal

Is there an optimal path that will maximize overall quality. How do we

determine it How do we take

advantage of it

Equal-bandwidth lines

Non-optimal paths Max overall quality

Equal-quality lines

(A) The Video SNR-Temporal Plane

SNR bandwidth

Temporal bandwidth

Idea Can we use a part of the enhancement

layer, in addition to the base layer, to predict motion?

We can if we know how much of the base layer will be present at the decoder when the temporal layers are introduced

So if we build in rules about when temporal layers are introduced relative to spatial quality, we can!

Implementation Increase SNR

quality to a predetermined point

Then improve temporal quality

Further improve SNR quality at the new frame-rate

And so on….

Enhanced-reference scheme

Scheme using fixed reference rate

(B) SNR quality of enhanced-scheme

Bandwidth

SNR dB 2T fps

T fps

4T fps

3T fps

FGS+ Scheme (B-frame only)

I P Base Layer

Spatial Enhancement

Layer

Temporal Enhancement

Layer

Two issues When (at what SNR quality) do we

introduce new temporal-frames? How much extended-reference do

we use

Introducing Temporal Frames We conducted a

study where users chose preferred frame-rate at different bit-rates:

As the bit-rate goes up, people prefer better frame-rates

High-motion videos (Stefan, Coastguard) require a quicker transition

0102030405060708090

100

0 5

200300500

Spatio-Temporal Preferences

0

2

4

6

8

10

12

0 500 1000 1500

Bitrate (kbps)

Fram

e-R

ate

(fps) Foreman

Stefan

Coastguard

Mobile

Determining the size of the reference

Plot SNR performance as a function of enhancement at various bit-rates

Choose enhancement corresponding to best quality

SNR as function of enhancement

27

29

31

33

35

37

39

0 1 2 3 4 5 6bitplanes

SN

R (

dB

)

300 kbps 500 kbps1000 kbps 1500 kbps

 

Performance Improvement over

FGS varies from 0.19 dB to 1.28 dB

At low bit-rates simple videos benefit (Coastguard)

At high bit-rates complex videos benefit (Mobile)

Improvement of FGS+ (B-frames) over FGS

0

0.2

0.4

0.6

0.8

1

1.2

1.4

300 800 1300kbps

SN

R (

dB)

Foreman StefanCoastguard Mobile

FGS+ for All-Frames We can improve

performance further by using enhancement for base-layer P-frames also

But P-frames are present at all bit-rates. How much enhancement do we use?

(B) Effect of Reference Bandwidth T on Performance

T

SNR (dB)

P(r) = probability(R)

Transmission bandwidth R

Original Compression

New Compression Negative effect

of choice of T

Positive effect of choice of T

FGS+ Scheme (All-frames)

I P Base Layer

Spatial Enhancement

Layer

Temporal Enhancement

Layer

Improvement in Performance At 3 bit-planes Performance

varies from –0.03 dB to 0.59 dB

Performance will degrade at low bit-rates, due to incomplete references

Improvement of All-frames FGS+ over B-Frames only FGS+

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

300 800 1300

SNR (dB)

kbps

Foreman StefanCoastguard Mobile

Conclusion FGS+ uses a spatio-temporal

notion of video-quality Improves performance of

traditional FGS from 0.16 to 1.61 dB

Current Issues Spatio-Temporal

preference curves vary for different videos

How do we choose the correct curve automatically

Use Motion-Vectors, Image-Complexity

0102030405060708090

100

0 5

200300500

Spatio-Temporal Preferences

0

2

4

6

8

10

12

0 500 1000 1500

Bitrate (kbps)

Fra

me-

Rat

e (f

ps) Foreman

Stefan

Coastguard

Mobile

Future Directions Factors other than video-characteristics

do affect spatio-temporal preferences Environment (resolution, display-type,

lighting) User preferences (moods, some prefer

spatial-quality always) How do we account for these other

dimensions ? Large dimensional space requires many

(thousands) of sample points

Future Directions Build online experiment

Interactive web-based tool that lets user watch many videos

Allow user to change spatio-temporal settings

Record data corresponding to changes (video-characteristics, display resolution and type, user information)