user modeling 13/9 – 2004 inf serv – media storage and distribution systems:

User ModelingUser Modeling

13/9 – 2004

INF SERV – Media Storage and Distribution Systems:

2004 Carsten Griwodz & Pål Halvorsen

INF5070 – media servers and distribution systems

Why user modeling?

Multimedia approach If you can’t make it, fake it

Translation Present real-life quality If not possible, save resources where it is not

recognizable

Requirement Know content and environment Understand limitations to user perception If these limitations must be violated, know least

disturbing saving options



User Modelling

What? Formalized understanding of

users’ awareness user behaviour

Why? Achieve the best price/performance ratio Understand actual resource needs

achieve higher compression using lossy compression potential of trading resources against each other potential of resource sharing relax relation between media



Applications of User Modelling Encoding Formats

Exploit limited awareness of users JPEG/MPEG video and image compression MP3 audio compression

Based on medical and psychological models Quality Adaptation

Adapt to changing resource availability no models - need experiments

Synchronity Exploit limited awareness of users

no models - need experiments

Access Patterns When will users access a content? Which content will users access? How will they interact with the content?

no models, insufficient experiments - need information from related sources

User Perception ofQuality Changes



Quality Changes

Quality of a single stream Issue in Video-on-Demand, Music-on Demand, ... Not quality of an entire multimedia application

Quality Changes Usually due to changes in resource availability

overloaded server congested network overloaded client



Kinds of Quality Changes Long-term change in

resource availability Random Planned

Short-term change in resource availability Random Planned




resource availability Random

no back channel no content adaptivity continuous severe

disruption Planned





resource availability Random

no back channel no content adaptivity continuous severe

disruption Planned

change to another encoding format

change to another quality level

requires mainly codec work






Short-term change in resource availability Random

packet loss frame drop alleviated by protocols and

codecs Planned






packet loss frame drop alleviated by protocols and

codecs Planned

scaling of data streams appropriate choices require

user model



Kinds of Quality Changes Long-term change in resource availability

Random no back channel no content adaptivity continuous severe disruption

Planned change to another encoding format change to another quality level requires mainly codec work


packet loss frame drop alleviated by protocols and codecs

Planned scaling of data streams appropriate choices require user model



Planned quality changes Audio

Lots of research in scalable audio No specific results for distribution systems Rule-of-thumb

Always degrade video before audio

Video Long-term changes Short-term changes



Planned quality changes Audio Video

Long-term changes Use separately encoded streams Switch between formats Non-scalable formats compress better than scalable ones (Source:

Yuriy Reznik, RealNetworks) Short-term changes

Switching between formats Needs no user modeling Is an architecture issue




Long-term changes Short-term changes

Use scalable encoding Reduce short-term fluctuation by prefetching and buffering

Two kinds of scalable encoding schemes Non-hierarchical

encodings are more error-resiliento fractal single image encoding

Hierarchial encodings have better compression ratios

Scalable encoding Support for prefetching and buffering is an architecture issue Choice of prefetched and buffered data is not




Long-term changes Short-term changes

Use scalable encoding Reduce short-term fluctuation by prefetching and buffering

Short-term fluctuations Characterized by

frequent quality changes small prefetching and buffering overhead

Supposed to be very disruptive

See for yourself: subjective assessment



Subjective Assessment A test performed by the Multimedia Communications

Group at TU Darmstadt Goal

Predict the most appropriate way to change quality Approach

Create artificial drop in layered video sequences Show pairs of video sequences to testers Ask which sequence is more acceptable

Compare two means of prediction Peak signal-to-noise ratio (higher is better)

compares degraded and original sequences per-frame ignores order

Spectrum of layer changes (lower is better) takes number of layer changes into account ignores content and order



Subjective Assessment




Used SPEG (OGI) as layer encoded video format

frames

frames

frames

frames

laye

rs

laye

rsla

yers

laye

rs

amplitude of layer variation

frequency of layer variation




What is better?

frames

frames

frames

frames

laye

rs

laye

rsla

yers

laye

rs

First gap first or lowest gap first?

Early or late high quality?



Subjective Assessment How does the spectrum correspond with the results of the

subjective assessment? Comparison with the peak signal-to-noise ratio

#Farm 1 Farm 2 M&C1

ts 1 ts 2 ts 1 ts 2 ts 1 ts 2

1 Subjective assessment 0.35 0.55 0.73

2 PSNR (higher is better)62.8

649.47

61.46

73.2863.1

552.3

8

3Spectrum (lower is

better)2 2 6.86 4 2 1

According to the results of the subjective assessment the spectrum is a more suitable measure than the PSNR

#M&C3 M&C4 T-Tennis3

ts 1 ts 2 ts 1 ts 2 ts 1 ts 2

1 Subjective assessment 1.18 1.02 2.18

2 PSNR (higher is better)48.0

125.08

49.40

26.9566.0

263.2

8

3Spectrum (lower is

better)2 0 2 0 0.5 0.5

Metric

MetricClip

Clip




Conclusions Subjective assessment of variations in layer encoded

videos Comparison of spectrum measure vs. PSNR measure

Observing spectrum changes is easier to implement Spectrum changes indicate user perception better than PSNR Spectrum changes do not capture all situations

Missing Subjective assessment of longer sequences Better heuristics

"thickness" of layers order to quality changes target layer of changes

User Model for Synchronity



Synchronization Content Relation

se.g.: several views of the same data Spatial Relations

Layout Temporal Relations

Intra-object Synchronization Intra-object synchronization defines the time relation between

various presentation units of one time-dependent media object Inter-object Synchronization

Inter-object synchronization defines the synchronization between media objects

Relevance Hardly relevant in current NVoD systems Somewhat relevant in conferencing systems Relevant in upcoming multi-object formats: MPEG-4, Quicktime



Inter-object Synchronization

Lip synchronization demands for a tight coupling of audio and video

streams with a limited skew between the two media streams

Slide show with audio comment

Main problem of the user model permissible skew



Inter-object Synchronization A lip synchronized audio video sequence (Audio1 and Video) is

followed by a replay of a recorded user interaction (RI), a slide sequence (P1 - P3) and an animation (Animation) which is partially

commented using an audio sequence (Audio2). Starting the animation presentation, a multiple choice question is presented to

the user (Interaction). If the user has made a selection, a final picture (P4) is shown

Main problem of the user model permissible latency

analysing object sequence allow prefetching user interaction complicates prefetching



Synchronization Requirements – Fundamentals

100% accuracy is not required, i.e., skew is allowed

Skew depends on Media Applications

Difference between Detection of skew Annoyance of skew

Explicit knowledge on skew Alleviates implementation Allows for portability



Experimental Set-Up Experiments at IBM ENC Heidelberg to quantify synchronization

requirements for Audio/video synchronization Audio/pointer synchronization

Selection of material Duration

30s in experiments 5s would have been sufficient

Reuse of same material for all tests Introduction of artificial skew

By media composition with professional video equipment With frame based granularity

Experiments Large set of test candidates

Professional: cutter at TV studios Casual: every day “user”

Awareness of the synchronization issues Set of tests with different skews lasted 45 min



Lip Synchronization: Major Influencing Factors

Video Content

Continuous (talking head) vs. discrete events (hammer and nails) Background (no distraction)

Resolution and quality View mode (head view, shoulder view, body view)

Audio Content Background noise or music Language and articulation



Lip Synchronization: Level of Detection

Areas In sync QoS: +/- 80 ms Transient Out of sync



Lip Synch.: Level of Accuracy/Annoyance

Some observations Asymmetry Additional tests with long movie

+/- 80 ms: no distraction -240 ms, +160 ms: disturbing



Pointer Synchronization Fundamental CSCW shared workspace issue

Analysis of CSCW scenarios Discrete pointer movement (e.g. “technical sketch”) Continuous pointer movements (e.g. “route on map”)

Most challenging probes Short audio Fast pointer movement



Pointer Synchronization: Level of Detection

Observations Difficult to detect “out of sync”

i.e., other magnitude than lip sync Asymmetry

According to every day experience



Pointer Synchronization: Level of Annoyance

Areas In sync: QoS -500 ms, +750 ms Transient Out of sync



Quality of Service of Two Related Media Objects

Expressed by a quality of service value for the skew Acceptable skew within the involved data streams Affordable synchronization boundaries

Production level synchronization Data should be captured and recorded with no skew

at all To be used if synchronized data will be further

processed Presentation level synchronization

Reasonable synchronization at the user interface To be used if synchronized data will not be further

processed




Media Mode, application QoS

Video Animation Correlated +/- 120 ms

Audio Lip synchronization +/- 80 ms

Images Overlay +/- 240 ms

No overlay +/- 500 ms

Text Overlay +/- 240 ms

No overlay +/- 500 ms




Media Mode, application QoSAudio Animation Event colleration +/- 80 ms

Audio Tightly coupled (stereo) +/- 11 μs

Loosely coupled (dialog mode with various participants)

+/- 120 ms

Loosely coupled (background music)

+/- 500 ms

Image Tightly coupled (music with notes)

+/- 5 ms

Loosely coupled (slide show)

+/- 500 ms

Text Text annotation +/- 240 ms

Pointer Audio related to shown item

-500 - +750 ms

User Model for Access Patterns



Modelling

User behaviour The basis for simulation and emulation

In turn allows performance tests Separation into

Frequency of using the VoD system Selection of a movie

User Interaction Models exist

But are not verified

Selection of a movie Dominated by the access probability Should be simulated by realistic access patterns



Focus on Video-on-Demand

Video-on-demand systems Objects are generally consumed from start to end Repeated consumption is rare Objects are read-only Hierarchical distribution system is the rule

Caching approach Simple approach first Various existing algorithms

Simulation approach No real-world systems exist Similar real-world situations can be adopted



Using Existing Models

Use of existing access models ? Some access models exist Most are used to investigate single server or cluster

behavior Real-world data is necessary to verify existing models

Optimistic model Cache hit probabilities are over-estimated Caches are under-dimensioned Network traffic is higher than expected

Pessimistic model Cache hit probabilities are under-estimated Cache servers are too large or not used at all Networks are overly large



Existing Data Sources for Video-on-Demand

Movie magazines Data about average user behaviour Represents large user populations Small number of observation points (weekly)

Movie rental shops Actual rental operations Serves only a small user population Initial peaks may be clipped

Cinemas Actual viewing operations Serves only a small user population Few number of titles Short observation periods



Model for Large User Populations Zipf Distribution

Verified for VoD by A. Chervenak N - overall number of movies ξ – skew factor i - movie i in a list ordered by descreasing popularities z(i) - hit probability

Many application contexts all kinds of product popularity investigations http://linkage.rockefeller.edu/wli/zipf/ collects applications of

Zipf’s law natural languages, monkey-typing texts, web access statistics,

Internet traffic, bibliometrics, informetrics, scientometrics, library science, finance, business, ecological systems, ...

N

j

jCi

Ciz

1

/1/1,)(



Verification: Movie Magazine Movie magazine

Characteristics of observations on large user populations Smoothness Predictability of trends Sharp increase and slower decrease in popularities

weeks5 10 15 20 250

Highlander 3

med

ia c

ont

rol i

nde

x

0

20

40

60

80

100

weeks5 10 15 20 250

Highlander 3

top

100

ra

nkin

g

0

20000

40000

60000

80000



Comparison with the Zipf Distribution

Well-known and accepted model Easily computable Compatible with the 90:10 rule-of-thumb

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 20 40 60 80 100

ren

tal

pro

bab

ility

movie index

probability curves for 250 movie titles

z(i)4/3/96

4/6/96



Verification: Small and Large User Populations



Verification: Small and Large User Populations

Similarities Small populations follow the general trends Computing averages makes the trends better visible Time-scale of popularity changes is identical No decrease to a zero average popularity

Differences Large differences in total numbers Large day-to-day fluctuations in the small

populations

Typical assumptions 90:10 rule Zipf distribution models real hit probability



Problems of Zipf Does not work in distribution hierarchies

Access to independent caches beyond first-level are not described

Not easily extended to model day-to-day changes Is timeless Describes a snapshot situation

Optimistic for the popularity of most popular titles Chris Hillman, bionet.info-theory, 1995

Any power law distribution for the frequency with which various combinations of ‘‘letters’’ appear in a sequence

is due simply to a very general statistical phenomenon,and certainly does not indicate some deep underlying process

or language. Rather, it says you probably aren’t looking at your problem the

right way!



Approaches to Long-term Development

Model variations for long-term studies Static approach

No long-term changes Movie are assumed to be distributed in off-peak hours

CD sales model Smooth curve with a single peak Models the increase and decrease in popularity

Shifted Zipf distribution Zipf distribution models the daily distribution Shift simulates daily shift of popularities

Permutated Zipf distribution Zipf distribution models the daily distribution Permutation simulates daily shift of popularities



Verification: Zipf Variations Rotation model for day-to-day relevance changes

0

20

40

60

80

100

0 50 100 150 200 250po

pu

lari

ty i

nd

ex

ch

an

ge

age in days

relevance change of a real movie

0

20

40

60

80

100

0 50 100 150 200 250po

pu

lari

ty in

de

x ch

ang

e

age in days


0

20

40

60

80

100

0 50 100 150 200 250po

pu

lari

ty i

nd

ex

ch

an

ge

age in days

relevance change



Verification: Zipf Variations Permutation model for day-to-day relevance changes

0

20

40

60

80

100

0 50 100 150 200 250po

pu

lari

ty i

nd

ex

ch

an

ge

age in days


0

20

40

60

80

100

0 50 100 150 200 250po

pu

lari

ty in

de

x ch

ang

e

age in days


0

20

40

60

80

100

0 50 100 150 200 250po

pu

lari

ty i

nd

ex

ch

an

ge

age in days

relevance change



Modelling: Requirements Model should represent movie life cycles

To reflect the aging of titles To observe movement of movies through a hierarchy of servers To make observations with respect to a single movie To support the idea of pre-distribution

Model should work for large and small user populations To allow variations in client numbers To prevent from built-in smoothing effects

Model can not be trace-driven The number of movies is too small The observation time is too short The user population size is not variable One title can not be re-used without similarity effects



New Model: Movie Life Cycle

Characteristics Quick popularity increase Various top popularities Various speeds in popularity decrease Various residual popularity



New Model: User Population Size

Smoothing effect of larger user populations Day-to-day relevance changes Probability distribution of all movies by „new releases“

0

0.5

1

1.5

2

0 50 100 150 200 250

mo

vie

hit

s

days

50 draws per day

0123456789

0 50 100 150 200 250

mo

vie

hit

s

days

500 draws per day

0

10

20

30

40

50

60

70

0 50 100 150 200 250

mo

vie

hit

s

days

5000 draws per day

0

100

200

300

400

500

600

700

0 50 100 150 200 250m

ovi

e h

its

days

50000 draws per day



Problems with Data Sources Lack of additional real-world data

No verification data for medium-sized populations available

Missing details Genres

Popularity rise and decline depends on genres Single users´ behaviour can be predicted

Single day probability variations Children´s choices at daytime, adults´ choices at night

Regional popularity differences Ethnic groups Regional information

Comebacks Sequels inspire comebacks

Detail overload Simplifications are required for large simulations



Video Access Modeling Simple Zipf models are not suited for simulation

of server hierarchies Trace-driven simulation can not be used Our model is sufficient for general investigation

on caching Long-term movie life cycles can be modeled nicely Optimistic assumptions due to smoothness are

removed Variations in movie behavior are supported Day-to-day popularity changes are realistic

It is not sufficient yet for advanced caching mechanisms Single-day variations are missing Genres are missing



Summary

User modeling helps achieving a good price/performance ratio for multimedia systems

User modeling allows cheating

Examples seen: Modeling quality assessment of layered video Modeling audio/video synchronization Modeling video access probability



References Ann Chervenak: Tertiary Storage: An Evaluation of New Applications, PhD thesis, University of

California, Berkeley, 1994 Carsten Griwodz, Michael Bär, Lars Wolf: Long-Movie Popularity Models in Video-on-Demand

Systems, ACM Multimedia, Seattle, WA, USA, Nov. 1997 Charles Krasic, Jonathan Walpole: Priority-Progress Streaming for Quality-Adaptive Multimedia,

ACM Multimedia Doctoral Symposium, Ottawa, Canada, Oct. 2001 Ralf Steinmetz, Klara Nahrstedt: Multimedia Fundamentals, Volume I: Media Coding and Content

Processing (2nd Edition), Prentice Hall, 2002, ISBN 0130313998 Michael Zink, Oliver Künzel, Jens Schmitt, Ralf Steinmetz: Subjective Impression of Variations in

Layer-Encoded Videos, IWQoS, Monterey, CA, USA, Jun. 2003 Michael Zink, Jens Schmitt, and Carsten Griwodz. Layer-Encoded Video Streaming: A Proxy's

Perspective. In IEEE Communications Magazine, Vol. 42, No. 8, August 2004

user modeling 13/9 – 2004 inf serv – media storage and distribution systems:

Documents

resource availabilityrandomno

otherpotential of resource

resource availabilityno

codec workshortterm

quality levelrequires

inf serv media storage

user modeling139

user perceptionif