implementation and analysis of user adaptive mobile video streaming using mpeg- dash abhijith...
TRANSCRIPT
1
IMPLEMENTATION AND ANALYSIS OF USER ADAPTIVE MOBILE VIDEO STREAMING USING MPEG-
DASH
Abhijith JagannathDepartment of Electrical Engineering
University of Texas at ArlingtonAdvisor: Dr. K. R. Rao
Committee: Dr. W Alan Davis and Dr. Jonathan Bredow
2
Overview• Need for bandwidth savings• Factors affecting visual perception of mobile video• Some concepts on limits of human vision• Derivation of Maximum visible frequency• Calculation of sufficient resolution• Adaptive video streaming• MPEG-DASH Overview• User environment estimate from device sensors• Implementation of user adaptive video streaming using MPEG-
DASH• Experimental results• Conclusions
3
Increasing need for bandwidth savings
• Mobile Internet use is expanding dramatically• Video traffic is growing exponentially • Challenges:
– Mobile users expect high quality video experience– Network operators need to offer quality experience affordably
4
Ambient Light
Factors affecting visual perception of mobile videos
• Display Characteristics– Brightness– Contrast– Physical size– Pixel depth
• Viewing angle• Distance• Reflected ambient light
5
Visual Acuity
– Snellen’s Chart– Person with normal eye sight can see
details of 20/20 letter from 20 feet.
6 meters / 20 feet viewing distance
6/6 (20/20) – row letters are designed such that when viewed from 6 meters their smallest spatial details (strokes, gaps) constitute 1 minute of arc (1/60 of a degree of viewing angle)
6
Spatial Frequency
• cycles per degree
𝑢 = 1𝛽, 𝛽 = 𝜋360arctan൬𝑛2𝑑𝜌൰
n: # of pixels, p : pixel density
8
Contrast
• Ratio between brightest (typically white) to darkest (typically black) colors of a display.
• Michelson contrast
• Contrast sensitivity
maxL
minL
9
Effects of Ambient light
10
𝐿𝑚𝑖𝑛𝐷 ≤ 𝐿𝑚𝑖𝑛≤ 𝐿𝑚𝑎𝑥≤ 𝐿𝑚𝑎𝑥
𝐷
• Reflected Luminance :
• Effective Luminance of display
• Ambient contrast ratio
• Effective display contrast
Ambient light (lux)
Disp
lay co
ntras
t rati
o (
CRA
)
Effects of Ambient light
𝑠𝑚𝑖𝑛=1
𝐶𝐷
=CRA+1
CRA−1
Minimum contrast sensitivity
11
Contrast Sensitivity Function (CSF)• CSF is measured by projecting small (2-12o) Gabor patches on screen
• When CSF is measured, the contrast of a patch is progressively reduced till it becomes barely visible (50% of viewers can still see it while the rest of viewers can’t). Such threshold points form CSF.
Spatial Frequency (cpd)
Con
tras
t Sen
sitiv
ity
Visible
Invisible
Contrast sensitivity
12
CSF Models
• Many models proposed in the literature• Latest models account for variety of factors:
– Object (screen) luminance– Field size– Oblique effect– Background luminance– etc.
13
Barten-04 CSF Model [20]
• Main formula :
• Low pass branch:
14
Finding maximum visible frequency• Analytic inverse of :
• Lambert W function
• At
• is the maximum visible frequency
15
Sufficient Resolution • Minimum display density:
• Sufficient Resolution:
𝜌𝑚𝑖𝑛 ≈ 360𝜋 u𝑚𝑎𝑥𝑑
𝑊 𝑠𝑢𝑓𝑓 =𝑀 𝜌𝑚𝑖𝑛 ,𝐻 𝑠𝑢𝑓𝑓 =𝑁 𝜌𝑚𝑖𝑛
𝑊 𝑠𝑢𝑓𝑓 =𝑊 𝑑𝑖𝑠𝑝
𝜌𝑚𝑖𝑛
𝜌,𝐻 𝑠𝑢𝑓𝑓=𝐻𝑑𝑖𝑠𝑝
𝜌𝑚𝑖𝑛
𝜌
𝑀
𝑁
( pixels)
( pixels)
𝑢 = 1𝛽, 𝛽 = 𝜋360arctan൬𝑛2𝑑𝜌൰
16
Adaptive Video Streaming
• Server• Can be standard web
server• Media segment can be
prepared in-line or off-line
• Client • Sends series of HTTP GET
segment requests and receives segments
• Performs rate adaptation before sending a new GET segment request
18
Dynamic Adaptive Steaming over HTTP (DASH)
• Dash is NOT– System, protocol, presentation, codec, interactivity
• What is DASH – Enabler which provides formats to enable efficient and high-quality
delivery of streaming services over the Internet– Component of end-to-end service– Enabler to reuse existing technologies (containers, DRM (Digital Rights
Management), codecs)– Enabler for deployment on top of HTTP-CDNs– Enabler for very high user experience (low start-up, no re-buffering)– Provides simple inter-operability points (profiles)
MPEG-DASH: A standard for adaptive streaming
19
MPEG-DASH Overview
20
Media Presentation Description (MPD)
21
MPD example
22
Client centric
• Client has best view of network conditions• No session state in network• Faster innovation and experimentation• But, relies on client for operational metrics
– Only client knows what really happens
23
DASH JavaScript Player
24
DASH-JS (JavaScript Implementation)
25
Implementing User Adaptation
• Built-in Sensors used for viewing setup estimation– Device front camera : Distance measurement– Accelerometer: State detection and distance estimation.– Proximity sensor: Ambient light
• Display information provided by manufactures (Software APIs)– Backlight brightness– Device pixel density– Actual display resolution
26
Distance calculation using front camera
𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒=𝑊 ∗h𝑓𝑤
2∗𝑤∗ tan (𝜃 /2)
27
Device state estimation using accelerometer
28
Effective Contrast measurement
• Reflection coefficient given by manufacturer. • Luminance of screen with white background and maximum brightness ()• Luminance of screen with white background and minimum brightness () • Luminance of screen with black background and maximum brightness ()• Luminance of screen with black background and minimum brightness () • Brightness saturation level () at which luminance with white background reaches maximum.
• Best fit gamma () calculated from power law model.
29
Effective contrast: calculation• Reflected luminance
• Correction factor
• Bright and dark intensities
• Max and min luminance
• Effective contrast
𝐿𝑚𝑎𝑥𝐷 =𝐿 h𝑤 𝑖𝑡𝑒+𝐿𝑟𝑒𝑓 ,𝐿𝑚𝑖𝑛
𝐷 =𝐿𝑏𝑙𝑎𝑐𝑘+𝐿𝑟𝑒𝑓
𝐶𝐷=𝐿𝑚𝑎𝑥
𝐷 −𝐿𝑚𝑖𝑛𝐷
𝐿𝑚𝑎𝑥𝐷 +𝐿𝑚𝑖𝑛
𝐷
𝑄 (𝐵)=( 𝐵𝐵𝑠𝑎𝑡
)𝛾
𝐿𝑟𝑒𝑓=( 𝐼𝜋 )𝐾
30
Implementation of User adaptive DASH
31
Modified JavaScript Player
32
Configuration• Device used : Microsoft surface Pro
• Device state vs distance usedDevice State Corresponding distance
In Hand 12”
On Lap 22”
On Stand 30”
On Table facing up 60”
33
Representations of media used
Representations available Resolutions (pixels)
Corresponding bitrates (bits /sec)
1 1280 x 720 30000002 1024 x 576 20000003 704 x 396 10000004 480 x 270 6000005 320 x 180 349952
Representations available Resolutions (pixels)
Corresponding bitrates (bits /sec)
1 1920 X 1080 60000002 1568 X 880 44410003 1280 X 720 32870004 1056 X 592 24330005 848 X 480 18010006 688 X 384 13330007 576 X 320 9860008 448 X 256 7300009 368 X 208 540000
10 320 X 176 400000
Representations available Resolutions (pixels)
Corresponding bitrates (bits /sec)
1 1920 X 1080 7706632 1280 X 720 5147933 640 X 360 1948344 320 X 180 50842
Representations available Resolutions (pixels)
Corresponding bitrates (bits /sec)
1 1920 X 1080 59334862 1280x720 33604413 960 X 540 22223524 640 X 360 9853215 320 X 180 391544
A B
C D
35
Results• Bandwidth savings for ‘A’ when all the
resolutions are supported by network
In hand On Lap On Stand On Table facing up1280 x 720 1024 x 576 704 x 396 480 x 270
0
10
20
30
40
50
60
70
80
90
100
Sequence A
Device state and selected resolution
Band
wid
th S
avin
gs i
n %
Available resolution: 1280 x 720
36
Results• Bandwidth savings for ‘B’ when all the resolutions are
supported by network
In Hand On Lap On Stand On Table facing up1568 X 880 1280 X 720 1056 X 592 848 X 480
0
10
20
30
40
50
60
70
80
90
100
Sequence B
Device state and selected resolution
Band
wid
th sa
ving
s in
%
Available Resolution: 1920 X 1080
37
Results• Bandwidth savings for ‘C’ when all the resolutions are
supported by network
In Hand On Lap On Stand On Table facing up1920 X 1080 1280 X 720 1280 X 720 640 X 360
0
10
20
30
40
50
60
70
80
90
100
Sequence C
Device state and selected resolution
Band
wid
th sa
ving
s in
%
Available Resolution: 1920X1080
38
Results• Bandwidth savings for ‘D’ when all the resolutions are
supported by network
In Hand On Lap On Stand On Table facing up1920 X 1080 1920 X 1080 960 X 540 640 X 360
0
10
20
30
40
50
60
70
80
90
100
Sequence D
Device state and selected resolution
Band
wid
th sa
ving
s in
%
Available Resolution: 1920X1080
39
Summary• Limitations of human visual system are reviewed and
sufficient resolution of a display to discern information is derived
• MPEG-DASH offered a good platform to implement the concept and analyze the user adaptation technology.
• Using several sensors present in the device, the viewing environment is estimated. And hence the resolution sufficient to display information is calculated.
• Reference DASH player was modified to incorporate the sufficient resolution to select from the bit streams available in MPD.
• Considerable bandwidth savings (10-40%) are observed with this implementation.
40
Conclusions
• By incorporating the characteristics of viewing environment and by understanding the limits of HVS, the streaming of videos to mobile phones can be adaptive to viewing preferences of the user.
• This can be done through MPEG-DASH standard which enables design of intelligent streaming systems adapting not only to bandwidth but also to factors affecting user ability to see visual information.
• It is shown that such adaptation can result in reduced bandwidth usage, increased battery life, and improved quality of user experience
41
Future work
• HVS is a vast area of research, there can be more limits to human other than ones that are described in this thesis. Exploring more in this area may add additional benefits.
• Further research in the direction of detecting user attention while playing the multimedia content may help in improving not only the delivery of media but also the quality of the content played.
42
References[1] D. Wu, Y. T. Hou, W. Zhu, Y.-Q. Zhang and J. M. Peha, "Streaming video over the Internet: approaches and directions," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, no. 3, pp. 282-300, Mar 2001. [2] G. J. Conklin, G. S. Greenbaum, K. O. Lillevold, A. F. Lippman and Y. A. Reznik, "Video coding for streaming media delivery on the Internet," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, no. 3, pp. 269-281, Mar 2001. [3] I. Sodagar, "The MPEG-DASH Standard for Multimedia Streaming Over the internet," IEEE MultiMedia, vol. 18, no. 4, pp. 62-67, Apr 2011. [4] ISO/IEC 23009-1, "Dynamic adaptive streaming over HTTP (DASH) - Part 1: Media presentation description and segment formats," Information Technology, 5 Jan 2012. [5] A. Talukdar, M. Cudak and A. Ghosh, "Streaming Video Capacities of LTE Air-Interface," IEEE International Conference on Communications (ICC), pp. 1-5, May 2010. [6] J. Bergquist, "Resolution and contrast requirements on mobile displays for different applications in varying luminous environments," 2nd Int Symp Nanovision Sceince, pp. 143-145, 2005. [7] J. Xue and C. W. Chen, "A study on perception of mobile video with surrounding contextual influences," Fourth International Workshop on Quality of Multimedia Experience, vol. 5, no. 7, pp. 248-253, Jul 2012. [8] Y. A. Reznik, "User-adaptive mobile video streaming using MPEG-DASH," Proc. SPIE, Applications of Digital Image Processing XXXVI, vol. 8856, Sep 2013. [9] Wikipedia, "Snellen's Chart," [Online]. Available: http://en.wikipedia.org/wiki/Snellen_chart.[10] H. Snellen, "Probebuchstaben zur Bestimmung der Sehschärfe (Sample letters to determine the visual acuity)", Berlin: Hermann Peters, 1873.
43
References[11] K. Michael and L. Charles, "Visual Acuity," Web Vision, 5 June 2007. [Online]. Available: http://webvision.med.utah.edu/book/part-viii-gabac-receptors/visual-acuity/. [Accessed 29 June 2014].[12] D. A. Owens and K. Wolf-Kelly, "Near work, visual fatigue, and variations of oculomotor tonus," Investigative ophthalmology & visual science, vol. 28, no. 4, pp. 743-749, 1987. [13] J.Anshel, "Visual ergonomics in the workplace", CRC Press, 2002. [14] Y. Bababekova, M. Rosenfield, J. E. Hue and R. R. Huang, "Font Size and Viewing Distance of Handheld Smart Phones," Optometry and Vision Science, vol. 88, no. 7, pp. 795-797, Jul 2011. [15] G. E. Legge, G. S. Rubin and A. Luebker, "Psychophysics of reading. V. The role of contrast in normal vision," Vision Research, vol. 27, no. 7, pp. 1165-1177, 1987. [16] Recommendation ITU-R BT.500-13, "Methodology for the subjective assessment of the quality of television pictures," International Telecommnication Union, 2012. [17] R. Vanam and Y. Reznik, "Improving the Efficiency of Video Coding by using Perceptual Preprocessing Filter," IEEE Data Compression Conference, 2013. [18] J. A. Movshon and L. Kiorpes, "Analysis of the development of spatial contrast sensitivity in monkey and human infants.," Journal of the Optical Society of America A, vol. 5, no. 12, pp. 2166-2172, 1988. [19] P. G. J. Barten, "Evaluation of subjective image quality with the square-root integral method," Journal of the Optical Society of America A, vol. 7, no. 10, pp. 2024-2031, Oct 1990. [20] P. G. J. Barten, "Formula for the contrast sensitivity of the human eye," Proc. SPIE , Image Quality and System Performance, vol. 5294, pp. 231-238, 2003.
44
References• [21] S. J. Daly, "Visible differences predictor: an algorithm for the assessment of image fidelity,"
Proc of SPIE in Human Vision, Visual Processing, and Digital Display III, vol. 1666, pp. 2-15, 1992. • [22] Y. A. Reznik and R. Vanam, "Improving coding and delivery of video by exploiting the oblique
effect," IEEE Global Conference on Signal and Information Processing, Dec 2013. • [23] L. J. Kerofsky, R. Vanam and Y. A. Reznik, "Improved Adaptive Video Delivery System Using a
Preceptual Pre-processing Filter," IEEE Global Conference on Signal and Information Processing, 2014. • [24] R. M. Corless, G. H. Gonnet, D. E. G. Hare, D. J. Jeffrey and D. E. Knuth, "On the LambertW
function," Advances in Computational mathematics, vol. 5, no. 1, pp. 329-359, 1996. • [25] A. Saamer, A. C. Begen and C. Dovrolis, "An Experimental Evaluation of Rate-Adaptation
Algorithms in Adaptive Streaming over HTTP," Proceedings of the second annual ACM conference on Multimedia systems, pp. 157-168, 2011.
• [26] "Overview of MPEG-DASH," DASH Industry Foundation, [Online]. Available: http://dashif.org/mpeg-dash/.
• [27] DASH-IF, "DASH Industry Forum," [Online]. Available: http://dashif.org/.• [28] DASH-IF, "Reference Client 1.0.0," DASH Industry Forum, [Online]. Available:
http://dashif.org/reference/players/javascript/1.0.0/index.html.• [29] B. Rainer, S. Lederer, C. Muller and C. Timmerer, "A seamless Web integration of adaptive
HTTP streaming," IEEE Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp. 1519-1523, Aug 2012.
• [30] M. Katsikitis, “The Human Face: Measurement and Meaning”, Springler, 2003.
45
References• [31] J. Young, M. Trudeau, D. Odell, K. Marinelli and J. Dennerlein, "Touch-screen tablet user configurations
and case-supported tilt affect head and neck flexion angles," Work: A Journal of Prevention, Assessment and Rehabilitation, vol. 41, no. 1, pp. 81-91, 2012.
• [32] Microsoft, "Surface pro," [Online]. Available: http://www.microsoft.com/surface/en-us/products/surface-pro-2.
• [33] T. Montgomery, "Anatomy, Physiology & Pathology of the Human Eye," [Online]. Available: http://www.tedmontgomery.com/the_eye/acuity.html.
• [34] J. Chipchase, C. Yanqing and Y. Jung, "Personal Television: A Qualitative Study of Mobile TV Users", Springer, 2007.
• [35] K. R. Rao and H. Wu, "Digital Video Image Quality and Perceptual Coding", CRC Press, 2005. • [36] S. Dernbach, B. Das, N. C. Krishnan, B. L. Thomas and D. J. Cook, "Simple and Complex Activity
Recognition through Smart Phones," 8th International Conference on Intelligent Environments (IE), vol. 26, no. 29, pp. 214-221, 2012.
• [37] D. Lamming, "Contrast Sensitivity," in Vision and visual dysfunction, London, Macmillan Press, 1991. • [38] J. Mannos and D. J. Sakrison, "The effects of a visual fidelity criterion of the encoding of images," IEEE
Transactions on Information Theory, vol. 20, no. 4, pp. 525-536, Jul 1974. • [39] V. Cerf and R. E. Kahn, "A Protocol for Packet Network Intercommunication," IEEE Transactions on
Communications, vol. 22, no. 5, pp. 637-648, May 1974. • [40] Wikipedia, "Adaptive bitrate streaming," [Online]. Available:
http://en.wikipedia.org/wiki/Adaptive_bitrate_streaming.• [41] K. R. Boff and L. E. Janet, "Engineering data compendium. Human perception and performance. User's
guide," 1988.
46
Thank you