perceptual quality driven 3-d video over...

Perceptual Quality Driven 3-D Video over Networks

C.T.E.R. Hewage

Supervisors: Dr. Stewart Worrall and Dr. Safak Dogan

2

Outline Introduction

Aim and Objectives

Efficient Coding Approaches for Stereoscopic Video

Objective Quality Measures for Stereoscopic Video

Efficient Transmission Strategies for Backward Compatible Stereoscopic Video Applications

Error Concealment Techniques for 3-D Video

Conclusion

Future Works

List of Publications

3

Introduction

3-D Video – Provides the sense of “being there” 3-D Video communications – Provides more natural

conditions for human interaction Tele-presence, Tele-immersion, 3D TV

NodeB

RNC

LTE

WiMAX

3DTV

Mobile 3DTV

4

Stereoscopic Video

The left and right views are fused in the visual cortex of the brain to perceive the depth of a scene

•Advantages of stereoscopic video over other representations of 3-D video

• Simple representation (easy camera arrangement)

• Cost effective display systems

• Easy to adapt for existing audio-visual communication technologies

5

Stereoscopic Video Capture

Left and Right View Camera Simple

Cost effective

Colour plus Depth Camera

128

255

0

Znear

Zfar(a) (b)

(a)Colour image

(b)Per-pixel depth image.

6

Problem Definition

3-D Video delivery beyond conventional video requires enormous system resources such as storage and bandwidth.

Development of 3-D video technologies start from the beginning would not be a feasible and cost effective solution.

The initial development of 3-D video technologies should be backward compatible for existing 2-D video applications.

Quality evaluation of 3-D video can be done accurately using subjective evaluation tests.

The transmission aspects of 3-D video has not received much attention to date.

7

Aim and Objectives Aim: To enable backward compatible 3-D

video services over bandwidth limited and unreliable communication networks using a perceptual quality driven approach.

Objectives: Explore efficient methods of encoding 3-D video using existing

method of 2-D video compression

Propose objective quality metrics for coded 3-D video in a range of compression ratios and packet loss rates.

Propose, develop and test methods of error resilience over errorprone wireless channel conditions.

Propose, develop and test error concealment methods for 3-D video data transmitted over unreliable communication networks

8


Aim and Objectives





Conclusion

Future Works


9

Efficient Encoding Approaches for Stereoscopic Video

Exploration of effcient encoding configurations for 3-D video using existing video coding standards Compression efficiency

Compatibility with end-to-end communication technologies

Asymmetric coding of colour plus depth video Coarsely quantized depth images

Temporarly down-sampled depth images

Performance analysis of stereoscopic video transmission over IP

10

Exploration of efficient encoding configurations

Parallel encoding configurations

Single-output encoding configurations

H.264/AVC

CODEC

H.264/AVC

CODECColour

Depth

Colour

Base Layer

EnhancementLayerDepth

MultiLayer

CODEC

Side by Side Image

Colour plus Depth

H.264/AVC

Encoder

11

Stereoscopic video encoding with MPEG4-MAC

Global MAC Bit-

Stream

MPEG-4

MAC

EncoderColour

Depth

Shape

•A single output bit-stream•A monochrome depth map•The binary shape of the video object need to be transmitted•Backward compatibility

H.264/SVC

EncoderBase Layer

EnhancementLayer

Colour

Depth

Global SVC

Bit-stream

•A single output bit-stream•Backward compatibility•Asymmetric coding support•Possibility of layered depth images (LDI)•Inter layer prediction can be used depending on the correlation of the stereo image pair

12

“Orbi” Colour (Left) and Depth (Right) video sequences

R-D Performance of SSV coding using MPEG-4 MAC, H.264/AVC and scalable H.264/AVC

13







14

Asymmetric coding of colour plus depth stereoscopic video

Mixed-resolution encoding concept for stereoscopic video is based on the response of Human Visual System (HVS).

Asymmetric coding of colour and depth

map video??? Coarsely quantized depth maps

Temporally down-sampled depth maps

15

Coarsely Quantized Depth Map Sequences

514843383328231813Mode 5

514641363126211611Mode 4

49443934292419149Mode 3

47423732272217127Mode 2

45403530252015105Mode 1

Depth

45403530252015105Colour

QPType of video

For depth image coding at the enhancement layer, five Quantization Parameter (QP) modes are used. A similar set of QP values are used to obtain R-D results for the colour image coding at the base layer.

Coarse Grain Scalability (CGS) of scalable H.264/AVC is utilized to obtain different R-D performances for colour and depth sequences

•JSVM Version 8.13

•IPPP…IPP… sequence

•I frame every 75 frames

16

Image Quality and Percentage Depth Bitrate @ overall bitrate of 1Mbps

38.4436.24Depth using Mode 5

39.5337.15Depth using Mode 440.6438.08Depth using Mode 3

41.6238.77Depth using Mode 2 42.6439.25Depth using Mode 1 37.4837.01Colour

InterviewOrbiY-PSNR (dB)

Image Sequence

0.1436.8736.81Mode 50.1836.9436.87Mode 40.2436.9536.90Mode 30.3136.7436.70Mode 20.4036.3236.30Mode 1

RightLeft

Percentage depth bitrate

(%)

Image Quality: PSNR (dB)Mode

0.1737.1136.98Mode 50.2237.1937.14Mode 40.2837.0736.99Mode 30.3436.8736.86Mode 20.4136.7636.70Mode 1

RightLeft

Percentage depth bitrate

(%)

Image Quality: PSNR (dB)Mode

17







18

Stereoscopic Video Performance Over IP

The error sensitivities of different components of immersive media may be diverse in nature.

The losses in colour image sequence may have greater impact on the perceived quality than the damage caused by a corrupted depth image sequence.

The error performance analysis can be utilized to propose novel UPA and UEP mechanisms for 3-D video communication applications.

19

Stereoscopic Video Performance Over IP

Orbi Interview

20


Aim and Objectives





Conclusion

Future Works


21


3-D video quality = {image quality, depth perception, presence, naturalness, eye strain, etc.}

Subjective evaluation tests are widely used to obtain human response for 3-D perception. Time consuming.

Enormous effort and large number of subjects are required.

Controlled test environments are required.

Does objective quality measures of 3-D video represent the true quality as perceived by human observers?

Objective quality metrics for 3-D video enable researchers, developers to evaluate the quality more efficiently.

22


The 3-D video quality is measured using subjective and objective quality metrics for a range of compression rates and packet loss rates. Subjective tests are conducted according to the ITU-

Recommendation BT.1438 (using DSCQS method).• Two perceptual attributes namely, overall image quality and depth

perception are measured subjectively. PSNR, SSIM and VQM measures of colour, depth and rendered left

and right image sequences are considered as objective quality measures.

Then the relationships (correlations) between objective and subjective quality ratings are derived using a symmetrical logistic function as described in the following.

p = 1 / [1 + exp ( D - DM ) G ]

where, p is the normalized opinion score, D is the distortion parameter, DM and G are constants and G may be positive or negative.

23

Quality of Asymmetrically Coded Colour Plus Depth Map Video

Orbi Breakdance

24

Correlation Between Objective and Subjective Quality Measures

25


0.21730.06590.71950.55500.10540.7528Average SSIM of the Rendered Left and Right Views

0.76020.12330.01880.79100.21170.0015Depth SSIM0.21120.06500.72740.53470.10340.7618Colour SSIM

0.12150.04930.84320.27740.07450.8764Average VQM of the Rendered Left and Right

0.12240.04900.84210.24600.07010.8904Colour VQM

0.17290.05880.77680.59360.10900.7356Average PSNR of the Rendered Left and Right views

0.77500.21190.000010.77140.12420.0043Depth PSNR0.14010.05290.81920.42650.09240.8100Colour PSNR

SSERMSECCSSERMSECC

Depth PerceptionOverall Image QualityObjective Quality Model

26

Perceived Quality under Transmission Errors

Orbi Breakdance

27


0.142500.08900.63100.28270.12530.5099Average SSIM of Rendered Left and Right Views

0.14740.09050.61830.29270.12750.4926Colour SSIM

0.065540.06030.83030.11280.07920.8045Average VQM of Rendered Left and Right

0.080390.06680.79180.12410.08300.7848Colour VQM

0.043100.04890.88840.11230.07890.8054Average PSNR of Rendered Left and Right views

0.044260.04960.88540.11580.08020.7992Colour PSNR

SSERMSECCSSERMSECC

Depth PerceptionOverall Image QualityObjective Quality Model

28


Aim and Objectives





Conclusion

Future Works


29


3-D video should be protected over unreliable and error

prone communication channels

UEP is an effective method to provide unbalanced

protection for video delivery over networks.

However adding redundant information to 3-D video

stream is not a feasible approach.

Hence in this study, UEP is implemented by

varying the transmission power for colour and depth map streams

mapping colour and depth map bit-streams into different medium

access priority classes.

30


In this study colour bit-stream is assigned higher protection compared to the depth bit-stream

Why?

31

Unequal Error Protection for Backward Compatible 3-D video Transmission over WiMAX

The UPA module dynamically allocates differentiated transmissionpower for colour and depth map streams based on the generated lookup tables.

The colour bit-stream will be survived under transmission errors than the depth map stream due to the high power associated with the colour video packets.

3-D video input

WiMAX

TransmissionSystem

Decoder

Depth video Packets

Colour video Packets3-D Video

Encoder based on

SVC

Extractor Module

Power control

UPAModule

Feedback

32

Performance of the UEP Scheme

8 8.5 9 9.5 10 10.5 11 11.5 12 12.5 13 13.517

19

21

23

25

27

29

31

33

35

37

39

40.5

SNR (dB)

Y−

PS

NR

(dB

)

Colour without UEPColour with UEPDepth without UEPDepth with UEP

8 9 10 11 12 1322

24

26

28

30

32

34

36

38

40

42

SNR (dB)Y

−P

SN

R (

dB)

Colour without UEPColour with UEPDepth without UEPDepth with UEP

Orbi Interview

33


Without UEP With UEP

Without UEP With UEP

34


Perception of Overall Image Quality Using SSCQS Method

0.5

1

1.5

2

2.5

3

3.5

9.95 10.5 11.05 11.6 12.15 12.7 13.25

SNR (dB)

MO

S

Overall Image Quality with UEP Overall Image Quality without UEP

35


Depth Perception using SSCQS Method

0.5

1

1.5

2

2.5

3

3.5

9.95 10.5 11.05 11.6 12.15 12.7 13.25

SNR (dB)

MO

S

Depth Perception with UEP Depth Perception without UEP

36


Aim and Objectives





Conclusion

Future Works


37


3-D error concealment methods can utilize the information from the same video stream as well as from the corresponding video stream to recover the missing information. A Novel Frame Concealment Method for Depth Maps

Using Corresponding Colour Motion VectorsCompression efficiency

Error Concealment Scheme for Stereoscopic Video Using the Shared Motion Information Send by the Encoder

3-D Video Concealment Using Associated Shape Information

38

Frame Concealment

128

255

0

Znear

Zfar(a) (b)

0 10 20 30 40 50 60 70 80 90 100−0.05

0

0.05

0.1

0.15

0.2

Frame Number

r(A

,B)

Interview

Horizontal componentVertical component

39

Depth Frame Concealment

MVs

. . .

. . .

Depth frame n Depth frame n+1

Colour frame n Colour frame n+1

Proposed method

JSVM-FC

40

GlobalSVC Bitstream

Error Concealment Scheme for Stereoscopic Video Using the Shared Motion Information

SVC Encoder

Colour MVs

Base layer Encoder

Enhancement layer encoder

Colour

Depth

. . .

. . .

MVsMVs

#1 Depth frame #2 Depth frame

#1 Colour frame #2 Colour frame

40.2741.0537.2539.69

Proposed MV

sharing scheme

42.3441.0538.1040.13Separate MVs

Depth (dB)

Colour (dB)

Depth (dB)

Colour (dB)

Interview-PSNROrbi-PSNR

Method

41

Performance of the Proposed MethodOrbi

Colour Depth

42

Performance of the Proposed Method

Orbi Sequence @ 5% PLR with individual MVs

Orbi Sequence @ 5% PLR with shared MVs

43


Aim and Objectives





Conclusions

Future Works


44

Conclusions 3-D Video Coding

Compression of Colour and depth using scalable H.264/AVC Asymmetric coding of colour plus depth video Stereoscopic video performance over IP

Objective Quality Metrics for 3-D video Quality Error Resilience Schemes for 3-D Video

UEP scheme for 3-D video transmission over WiMAX networks Prioritization scheme for 3-D video distribution over WLAN

Error Concealment Schemes for 3-D Video Depth frame concealment using associated colour motion

information Frame concealment scheme for 3-D video using shared motion

information 3-D error concealment scheme based on associated shape

information

45

Future Works Rate Adaptive 3D Video Coding

Asymmetric coding approached can be extended to implement open/close rate adaptive schemes for 3-D video transmission.

Generic 3-D Video Quality Metric Extend this quality evaluation to cover the end-to-end

chain of 3-D video technologies from 3-D capture to display.

Associate more perceptual attributes such as presence, naturalness and eye-strain.

The Proposed Error Resilience and Concealment Schemes Can be Extended for Multi-View Video

46

List of Publications1. C.T.E.R. Hewage, S. Nasir, S. Worrall, S. Dogan and A.M. Kondoz, "Prioritized 3-D Video Transmission over IEEE

802.11e", submitted to 2008 IEEE International Symposium on Circuits and Systems, Taipei, Taiwan, May 2009.

2. C.T.E.R. Hewage, Z. Ahmad, S. Worrall, S. Dogan and A.M. Kondoz, "Unequal Error Protection for Backward Compatible 3-D Video Transmission over WiMAX", submitted to 2008 IEEE International Symposium on Circuits and Systems, Taipei, Taiwan, May 2009.

3. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "Prediction of stereoscopic video quality using objective quality models of 2-D video", Electronics Letters -- 31 July 2008 -- Volume 44, Issue 16, p. 963-965.

4. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "Quality Evaluation of Colour plus Depth Map Based Stereoscopic Video", submitted to IEEE Journal of Selected Topics in Signal Processing:, May 2008.

5. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "Frame Concealment Algorithm for Stereoscopic Video Using Motion Vector Sharing", Proceedings of IEEE International Conference on Multimedia & Expo 2008 (ICME2008), Hannover, Germany, June 2008, p. 485-488.

6. C.T.E.R. Hewage, S. Worrall, S. Dogan and A.M. Kondoz, "A Novel Frame Concealment Method for Depth Maps Using Corresponding Colour Motion Vectors", Proceedings of 2nd 3DTV conference (3DTV-CON'08-IEEE), Istanbul, Turkey, May 2008, p.149-152.

7. C.T.E.R. Hewage, S. Worrall, S. Dogan, H. Kodikara Arachchi and A.M. Kondoz, "Stereoscopic TV over IP", Proceedings of the 4th IET European Conference on Visual Media Production (CVMP'2007), London, UK, November 2007.

8. C.T.E.R. Hewage, H.A. Karim, S. Worrall, S. Dogan and A.M. Kondoz, "Comparison of Stereo Video Coding Support in MPEG-4 MAC, H.264/AVC and H.264/SVC", Proceedings of the 4th IET International Conference on Visual Information Engineering (VIE'2007), London, UK, 25-27 July 2007.

9. C.T.E.R. Hewage, H. Kodikara Arachchi, T. Masterton, A.C. Yu, H. Uzuner, S. Dogan and A.M. Kondoz, "Content Adaptation for Virtual Office Environment Using Scalable Video Coding", Proceedings of the 16th IST Mobile and Wireless Communications Summit (IST Summit'2007), Budapest, Hungary, 1-5 July 2007.

47

Thank You!..

Any Questions?

Contacts: Chaminda Hewage

[email protected]

perceptual quality driven 3-d video over...

Documents