investigating wavelet investigating wavelet based video based video...

Investigating Wavelet Investigating Wavelet Based Video Based Video

Conferencing SystemConferencing SystemTeam Members:Team Members:

oo AhtshamAhtsham Ali Ali oo Adnan Ahmed (in Newzealand for grad studies)Adnan Ahmed (in Newzealand for grad studies)oo Adil Nazir (starting MS at LUMS now)Adil Nazir (starting MS at LUMS now)oo WaseemWaseem KhanKhanoo WaseemWaseem KhanKhanoo Farah ParvaizFarah Parvaizoo Muhammad Hussnain (pursuing MS from Sweden)Muhammad Hussnain (pursuing MS from Sweden)oo Ahmed Majeed, Farooq Ali (are in USA for grad studies)Ahmed Majeed, Farooq Ali (are in USA for grad studies)

Supervisors:Supervisors:oo Dr. Shahid MasudDr. Shahid Masudoo Dr. Nadeem Dr. Nadeem A.KhanA.Khan

Lahore University of Management SciencesLahore University of Management Sciences

1

Investigating Wavelet Based Video Investigating Wavelet Based Video

ConferencingConferencing

� We are presenting a wavelet based video conferencing system.

� Openphone

� Peer to peer video conferencing system� Peer to peer video conferencing system

� Dirac

� Wavelet based video codec

2

OutlineOutline� Introduction

� Video Conferencing system

� Openphone

� Wavelet based video codec� Wavelet based video codec

� Dirac

� Embedding Process

� Improvements in Openphone

� Improvements in Dirac

� Scalability

� Summary

3

Video Conferencing Video Conferencing ––

System OverviewSystem Overview

4

Open Phone in Open Phone in Video Conferencing Video Conferencing

SystemSystem

� What is Openphone?

PWLIB� PWLIB

� SIP

� RTP

5



� Openphone


� Dirac




� Scalability

� Summary

6

Dirac: Wavelet Based Video Codec Dirac: Wavelet Based Video Codec

�� What is Dirac?What is Dirac?

�� Open Source technologyOpen Source technology

�� High Performance, simple and Modular Design Video High Performance, simple and Modular Design Video High Performance, simple and Modular Design Video High Performance, simple and Modular Design Video CodecCodec

�� Support Multiple Picture FormatsSupport Multiple Picture Formats

�� Good Subjective quality due to Psycho Visual property Good Subjective quality due to Psycho Visual property of waveletof wavelet

7

Dirac Encoder ArchitectureDirac Encoder Architecture

8



� Openphone


� Dirac




� Scalability

� Summary

9

Dirac Integration In OpenphoneDirac Integration In Openphone

�� Dirac version 0.9.1 has successfully embedded in Dirac version 0.9.1 has successfully embedded in Openphone.Openphone.

�� MethodsMethods

�� Source Code IntegrationSource Code Integration

�� Dirac Library PlugDirac Library Plug--inin

�� Support in OpenphoneSupport in Openphone

�� Registration of Dirac in OpenphoneRegistration of Dirac in Openphone

�� Dirac Bridging ClassesDirac Bridging Classes

10



� Openphone


� Dirac




� Scalability

� Summary

11

Buffer I/OBuffer I/O�� SenderSender SideSide

�� TakeTake videovideo fromfrom thethe cameracamera�� EncodesEncodes itit usingusing thethe DiracDirac encoderencoder andand directsdirects thethe outputoutput

bitbit--streamstream toto aa bufferbuffer..�� TheThe outputoutput bitbit--streamstream bufferbuffer isis thenthen handledhandled byby

OpenphoneOpenphone�� TransmitsTransmits thethe bitbit--streamstream overover networknetwork inin orderorder toto sendsend itit�� TransmitsTransmits thethe bitbit--streamstream overover networknetwork inin orderorder toto sendsend itit

toto thethe receiverreceiver sideside..�� ReceiverReceiver SideSide

�� OpenphoneOpenphone receivesreceives thethe bitbit--stream,stream,�� DecodesDecodes itit usingusing DiracDirac decoderdecoder andand�� DisplaysDisplays thethe videovideo onon thethe locallocal displaydisplay windowwindow..

12

Parameter passing from OpenphoneParameter passing from Openphone

�� WeWe havehave modifiedmodified DiracDirac encoderencoder // decoderdecoder toto getget thetheparameterparameter listlist fromfrom OpenphoneOpenphone insteadinstead ofof acquiringacquiring ititthroughthrough consoleconsole inputinput..

�� WidthWidth

�� HeightHeight�� HeightHeight

�� FrameFrame raterate

�� BitBit--raterate

�� VideoVideo formatformat

�� QualityQuality etcetc..

�� whereaswhereas DiracDirac encoder/decoderencoder/decoder usesuses thisthis listlist totoencode/decodeencode/decode asas appropriateappropriate..

13

BitBit--stream Fragmentationstream Fragmentation

� Divide the bit-stream into network supported size packets.

14

Size Up gradation (CIF)Size Up gradation (CIF)

�� We have added support for CIF size (352x288) as well We have added support for CIF size (352x288) as well as for QCIF (176x144) size.as for QCIF (176x144) size.

�� Practical issues involved were increase in encoding Practical issues involved were increase in encoding time and size of bittime and size of bit--stream.stream.time and size of bittime and size of bit--stream.stream.

�� Frame rate reduced to 2,3 fpsFrame rate reduced to 2,3 fps

15

Frame Rate OptimizationFrame Rate Optimization�� Dirac usually generates a bitDirac usually generates a bit--stream of 5 to 75 stream of 5 to 75

kilobyteskilobytes

�� modified the Openphone architecture to transmit the modified the Openphone architecture to transmit the whole bitwhole bit--stream of one frame before grabbing the stream of one frame before grabbing the next frame.next frame.next frame.next frame.

�� Increased the frame rate to 12 fps.Increased the frame rate to 12 fps.

16

INTER ModeINTER Mode�� Motion Estimation SupportMotion Estimation Support

�� BitBit--stream size reductionstream size reduction

�� Maintaining the starting and ending sequence of the Maintaining the starting and ending sequence of the frameframe

�� Maintaining the initialization and reMaintaining the initialization and re--initialization of initialization of Dirac encoder.Dirac encoder.

�� Also, some buffers are introduced for maintaining the Also, some buffers are introduced for maintaining the bitbit--stream before transmission.stream before transmission.

17

GUI SupportGUI Support�� The user can configure video conferencing options as The user can configure video conferencing options as

desireddesired

�� Dial / Listen Dial / Listen

�� Video Size Video Size

�� CIF or QCIFCIF or QCIF

�� QualityQuality

�� Encoding mode (INTER / INTRA)Encoding mode (INTER / INTRA)

18

GUI Main Window GUI Main Window

19

Crashing ProblemCrashing Problem

�� Video conferencing lasted only a couple of hoursVideo conferencing lasted only a couple of hours

�� Memory leakages and bugs have been removed from Memory leakages and bugs have been removed from the code by extensive debugging.the code by extensive debugging.the code by extensive debugging.the code by extensive debugging.

�� Tested for 50 hours.Tested for 50 hours.

20

Discrete Wavelet TransformDiscrete Wavelet Transform

�� Old DWT filterOld DWT filter

�� DD9_7DD9_7

�� 10 frames per second10 frames per second

�� Experimentations have been done over all the available Experimentations have been done over all the available �� Experimentations have been done over all the available Experimentations have been done over all the available wavelet filters wavelet filters

�� Replaced with HAAR0Replaced with HAAR0

�� Performance gain Performance gain

�� 25 %25 %

�� Frame rate increased to 15 frames per secondFrame rate increased to 15 frames per second

21

Dirac 1.0.2Dirac 1.0.2

�� Latest release of Dirac video codec version 1.0.2 has Latest release of Dirac video codec version 1.0.2 has been successfully implemented in Openphone.been successfully implemented in Openphone.

�� Frame rate is around 15 frames per secondFrame rate is around 15 frames per second

22

Client / Server ConfigurationsClient / Server Configurations

�� Quality and Size Quality and Size

�� To be set from both sidesTo be set from both sides

�� Now only Dialer has the option and whatever he Now only Dialer has the option and whatever he selects becomes the communication size for video selects becomes the communication size for video conferencing.conferencing.

23

Experiments and Results (Cont.)Experiments and Results (Cont.)� Comparison of Dirac and other video codec

Comparison of Openphone Codecs

45

50

Average PSNR (dB)

20

25

30

35

40

45

50 100 150 200 250 300 350 400 450 500

Bitrate

Average PSNR (dB)

H263-Intra

H261-Intra

Dirac-Intra

� Bit rate Vs PSNR of Openphone Codec

24

Experiments and Results (Cont.)Experiments and Results (Cont.)

� Subjective Quality Difference in H.261,H.263 and Dirac at 300 Kbps

H.261 H.263 Dirac

25



� Openphone


� Dirac




� Scalability

� Summary

26

Improvements in DiracImprovements in Dirac�� Improvement in Motion EstimationImprovement in Motion Estimation

�� 3D Recursive Search Algorithm3D Recursive Search Algorithm

�� Reduced Computational ComplexityReduced Computational Complexity

�� PerformancePerformance

�� ScalabilityScalability

�� Three Different Bit StreamsThree Different Bit Streams

�� Base Layer, Enhancement Layer1, Enhancement Base Layer, Enhancement Layer1, Enhancement Layer2Layer2

�� Three TypesThree Types

�� SNR, Spatial, TemporalSNR, Spatial, Temporal27

Improvements in Motion EstimationImprovements in Motion Estimation

Existing Motion Estimation Algorithm: Three StagesExisting Motion Estimation Algorithm: Three Stages

Motion vectors are found for each block of 1 pixel accuracy

using Hierarchical Motion using Hierarchical Motion Estimation

Vectors are refined to sub pixel accuracy

Mode Decision

28

Continued…Continued…� Process of down conversion in hierarchical motion

estimation

29

Continued…Continued…� In lowest level, two candidate vector lists are generated , one is

centered at zero motion vector and the other spatially predicted motion vector.

� In all other levels, there candidate vector lists are generated, two above and one guide vector list ,which is the best motion vector of the block at the immediate lower level.of the block at the immediate lower level.

� Sum of absolute difference is used as a cost function.

30

Continued…Continued…�� Complexity of Existing Complexity of Existing

AlgorithmAlgorithm

� Using Hierarchical Motion Estimation

Huge amount of SAD

MB in previous frame

BA

X

MB in current frame

Current MB

C

� Huge amount of SAD calculations, taking 80% of total encoding time

�� 3DRS Algorithm3DRS Algorithm

�� Spatial PredictorSpatial Predictor

�� Temporal PredictorTemporal Predictor

E

Current MB

31

Continued…Continued…�� Proposed ME Scheme Based on 3DRSProposed ME Scheme Based on 3DRS

32

Experiments and ResultsExperiments and ResultsThe Comparison of Motion Estimation (ME) Results for CIF with GOP Size = 36, B Frames = 18

SequenceAlgorithm File Size

(KB)

PSNR-Y

(dB)

Avg(SAD)

/Frame

%SAD

Reduction of

3DRS

%ME Time

Reduction of

3DRS

%Total Time

Reduction of 3DRS

Foreman

Original 1316 37.9834 122871

62% 61% 49%Foreman 62% 61% 49%3DRS 1281 37.7774 47285

Container

Original 825 35.8540 142651

59% 59% 46%3DRS 769 35.5432 58371

News

Original 791 39.8091 76566

63% 60% 43%3DRS 781 39.7951 27965

Highway

Original 7231 39.2548 107357

57% 55% 42%3DRS 6732 39.0925 46962

33

Experiments and ResultsExperiments and ResultsThe Comparison of Motion Estimation (ME) Results for CIF with GOP Size = 54, B Frames = 36

SequenceAlgorithm File Size

(KB)

PSNR-Y

(dB)

Avg(SAD)

/Frame

%SAD

Reduction of

3DRS

%ME Time

Reduction of

3DRS

%Total Time

Reduction of 3DRS

Foreman

Original 1129 37.3678 126498

62% 62% 49%Foreman 62% 62% 49%3DRS 1075 37.1229 48095

Container

Original 697 37.7331 79323

61% 59% 43%3DRS 635 37.7676 30872

News

Original 681 39.4961 7847964% 60% 44%3DRS 656 39.4300 28453

Highway

Original 6050 38.7711 11004656% 57% 43%3DRS 4692 38.5662 48299

34

ConclusionConclusion

� Implemented modified adaptive 3DRS algorithm by using temporal and spatial block motion vectors.

� Reduces the average number of SAD calculations per block by 50% to 65%.block by 50% to 65%.

� Reduces the motion estimation time from 50% to 60%.

� Reduces the total encoding time from 30% to 50%.� Average PSNR and bit stream size does not show

significant variations.

35



� Openphone


� Dirac




� Scalability

� Summary

36

ScalabilityScalability�� Three different bitThree different bit--streamsstreams

�� Base Layer (7Base Layer (7--16)16)

�� Enhancement Layer 1 (4Enhancement Layer 1 (4--6)6)


�� Three typesThree types

�� SNRSNR

�� SpatialSpatial

�� Temporal ( In progress)Temporal ( In progress)

37

Dirac Bit Stream SyntaxDirac Bit Stream Syntax

38

Dirac Bit Stream SyntaxDirac Bit Stream Syntax

39

Dirac Architecture For ScalabilityDirac Architecture For Scalability

40

Dirac Bit stream SplitterDirac Bit stream Splitter

41

Bit Stream Layers After SplittingBit Stream Layers After Splitting

� Bit Stream Layers After Splitting for Inter Frame

42

Dirac Bit stream JoinerDirac Bit stream Joiner

43

Bit Stream After joiningBit Stream After joining

� Bit Stream After Joining for Inter Frame

44

Scalability In OpenphoneScalability In Openphone�� Three different bitThree different bit--streamsstreams

�� Base Layer (7Base Layer (7--16)16)


�� Enhancement Layer 2 (1Enhancement Layer 2 (1--3 )3 )

�� Three typesThree types

�� SpatialSpatial

�� SNRSNR

�� Temporal (Not completed)Temporal (Not completed)

45

Scalability in OpenphoneScalability in Openphone•• User receive video depending on his capabilities.User receive video depending on his capabilities.

46



� Openphone


� Dirac




� Scalability

� Summary

47

Tasks CompletedTasks Completed� Understanding the code of different modules of Dirac

video codec� Documenting the algorithmic details� Improvement in encoding time of Dirac

� 3DRS implementation, Comp complexity reduction , wavelet transform improvementwavelet transform improvement

� Implementing SNR and Spatial scalability in Dirac� Integrating Dirac in Open phone� Implementing Multi-streaming in open phone� Providing Graphical User Interface support� Support up till CIF format� Improvement Dirac to real time

� Extended frame rate from 3 to 1548

Ongoing and Future WorkOngoing and Future Work

� Addressing other modules regarding performance perspective

� Implementing Wavelet coefficient parent child relationship for scalable video performance improvementimprovement

� Implementing Temporal scalability

� Support extended for HDTV.

� Multi-user support for scalability in open phone

� Association of audio stream

� Improvement of GUI according to new developments

� Chat, File sharing49

investigating wavelet investigating wavelet based video based video...

Documents