W H I T E P AP E R

S t r e a m i n g V i d e o Q u a l i t y a n d U s e r E n g a g e m e n t

Sponsored by: Akamai

Karsten Weide

July 2011

I D C O P I N I O N

In an industry first, IDC conducted a statistical analysis of the server log files of six

major 2010 sports events that were streamed live to consumers in both North

America and Western Europe with a total of more than 2 million users.

The analysis found that both user engagement (measured as session length) and,

consequently, unique user numbers were influenced by video quality. Several factors

were shown to have an impact:

Higher bit rates do increase user engagement. For each event, after a certain bit

rate threshold, a further increase of bit rates had no additional positive effect on

user engagement anymore.

An important factor negatively impacting user engagement was the number of

rebuffering events per hour.

Other, less influential negative factors were the share of time the video player

spent rebuffering during users' sessions and the number of dropped frames per

hour.

Our research suggests that measuring and monitoring key performance indicators (KPIs)

for video quality is of critical importance for publishers because they affect user

engagement and audience reach and therefore publishers' revenue and competitiveness.

M E T H O D O L O G Y

Akamai tasked IDC with a research project to explore the impact of different aspects

of online streaming video quality on user engagement. To that end, Akamai provided

IDC with the server log files of six major 2010 sports events that were streamed live

to consumers via Akamai's HD Network employing HTTP streaming, using adaptive

bit rate technology as outlined in Table 1.

For all of the following analyses, keep in mind that the bit rates offered by publishers

were vastly different between events. The FIFA World Cup soccer events in particular

were offered at comparatively low bit rates because of the massive crowds expected

to watch.

Glo

bal H

eadquart

ers

: 5 S

peen S

treet

Fra

min

gham

, M

A 0

1701 U

SA

P

.508.8

72.8

200 F

.508.9

35.4

015

w

ww

.idc.c

om

2 #229083 ©2011 IDC

T A B L E 1

E v e n t O v e r v i ew

Event Region Number of Users Bit Rates Served

Soccer A: FIFA World Cup 2010 North America 235,052 400, 750, 1000, 1300, 1800 kbps

Soccer B: FIFA World Cup 2010 Western Europe 76,843 700, 1300, 2200, 3000 kbps

Soccer C: FIFA World Cup 2010 Western Europe 125,244 400, 800, 1200, 1600 kbps

Sports Event A: Major 2010 sports event North America 1,376,727 564, 1064, 1564, 2200 kbps

Sports Event B: Major 2010 sports event North America 25,644 564, 1064, 1564, 2200 kbps

Sports Event C: Major 2010 sports event North America 469,876 Four bit rates under 2000 kbps

Source: IDC, 2011

The log files were cleaned up before statistical analysis as follows:

Sessions were consolidated by user ID. Each log file entry originally represented

one viewing session. Where there were two or more separate sessions for the

same user ID, these sessions were consolidated so that log entries represented the

complete viewing experiences for each user ID for each event.

Logs were cleaned up. We removed any entry where it was clear from the data

that it was either impossible for the respective user to have seen the video

(average playback bit rate was zero, number of average frames per second

[FPS] was zero) or where more than 20% of total session time was spent

rebuffering (with the picture frozen), making it unlikely for the user to have

endured that bad of a viewing experience. We also ignored cases where the total

aggregated session time was less than one minute, assuming that shorter

sessions could not be counted as "viewing" a live video.

We also assumed that each user ID related to one person, even though several

persons or different persons at different times may have watched the video.

Furthermore, we assumed that users had spent the entire total aggregated session

time watching the video. In practice, users might have walked away from their PC or

could have had the video run in the background. For both assumptions, there was no

way for us to determine from the log files whether they held true.

IDC then conducted a statistical analysis of the remaining cases using the statistical

software package SPSS. The approach was to correlate user engagement (measured

as the total aggregated session time per user ID per event [short: session length or

session duration]) with certain measurements of streaming video quality for that

user's session during that event (see the Correlation section for details on the

statistical method):

Average playback bit rate: The average bit rate at which the video was rendered

on the user's screen as reported to the server by the user's video player

©2011 IDC #229083 3

Rebuffering events per hour: The number of times the buffer ran out of data

and had to be replenished, possibly with the picture frozen if the rebuffering

event was long enough to be noticeable by the user

Percent of time spent rebuffering: The share of the total aggregated session

time that was spent rebuffering

Dropped frames per hour: The number of frames that the user's video player

did not show

(The project did not analyze the impact of video start-up times on user engagement

because the log files did not include that information.)

The hypotheses were that:

Where positive KPIs such as playback bit rate or average FPS were higher (i.e.,

video quality was better), user engagement would also be higher (i.e., session

times would be longer). The expected correlation coefficient would be > 0.

Where positive KPIs were lower (i.e., video quality was worse), we expected user

engagement to also be lower (i.e., session times would be shorter). The

expected correlation coefficient would be < 0.

Conversely, where negative KPIs were higher (i.e., video quality was worse), we

expected user engagement to also be lower (i.e., session times would be

shorter). The expected correlation coefficient would be < 0.

Where negative KPIs were lower (i.e., video quality was better), user

engagement would also be higher (i.e., session times would be shorter). The

expected correlation coefficient would be > 0.

All correlation coefficients reported in this document were significant at the 0.01 level.

This means that mathematically, there is only a 1% likelihood that the reported

correlation occurred by chance.

C o r r e l a t i o n

Correlation is a statistical method that analyzes the relationship between two sets of

data and expresses the closeness of their relation in a "correlation coefficient," a

single number between 1 and -1.

For instance, we compared the average bit rates at which thousands of users

watched a video and the total time they spent watching the video.

If higher bit rates in each case translate into longer viewing times in a certain

proportion, the correlation coefficient would be 1.

If there was no relation at all between bit rates and viewing times, the coefficient

would be 0.

If higher bit rates in each case translate into shorter viewing times in a certain

proportion, the correlation coefficient would be -1.

Values between 0 and 1 and 0 and -1 would express varying degrees of relationship.

4 #229083 ©2011 IDC

Correlation does not necessarily indicate causation (i.e., two sets of data might be

shown to relate to each other statistically even though there is no relation between

the two data sets in the real world).

I N T H I S W H I T E P AP E R

This IDC white paper explores the impact of different aspects of online streaming

video quality on user engagement based on the statistical analysis of server log files

of six major 2010 sports events that were streamed live to consumers.

S I T U AT I O N O V E R V I E W

S t a t i s t i c a l A n a l y s i s o f t h e I m p a c t o f

S t r e a m i n g V i d e o Q u a l i t y o n U s e r E n g a g e m e n t

IDC's statistical analysis of the server log files of six major sports events that were

streamed live to users found that user engagement was influenced by video quality.

We found the following two factors had a positive impact on session durations (i.e.,

they tended to improve user engagement):

Higher playback bit rates (which of course are based on higher transferred bit

rates) had the greatest impact in terms of extending session lengths, but only up

to a certain optimal bit rate threshold. If bit rates were further increased beyond

that threshold, session durations were not further increased, or they were not

increased as much.

Higher frame rates (frames per second) also had a positive impact on session

lengths, but to a lesser extent than higher bit rates.

The following factors had a negative impact on session lengths — that is, they tended

to worsen user engagement (in sequence of their level of impact):

The number of rebuffering events

The share of the session time spent rebuffering

The number of dropped frames

Of the preceding factors negatively impacting user engagement, one of the most

important was the number of rebuffering events. The share of viewing time spent

rebuffering and the number of dropped frames/s had less of an impact.

Our research suggests that measuring and monitoring KPIs for video quality is of

critical importance for publishers because they affect user engagement and audience

reach and therefore publisher revenue and competitiveness.

©2011 IDC #229083 5

Playback Bit Rate

Online video publishers all adopt high-quality or high-definition video for competitive

and branding purposes, also based on the experience in cable TV, where higher

resolutions translated into greater user engagement.

There has been a lot of discussion in the industry about whether increasing the bit

rate available to the user and thereby improving video resolution has a positive

impact on user engagement. Therefore, we began our analysis by correlating users'

average playback bit rates and session lengths. We expected a positive correlation

(i.e., that higher bit rates come with longer sessions).

Our statistical analysis showed that users did watch the video streams for a longer

time if they watched the event at higher playback bit rates (i.e., at higher video

resolutions) — but only up to a certain bit rate. That is, for each event, if we analyzed

only the cases up to that event's optimal bit rate threshold, correlation between

playback bit rates and session length was positive, which means that higher bit rates

tended to go with longer sessions (see Figure 1). The impact was slight, but

statistically significant.

F I G U R E 1

C o r r e l a t i o n B e t w e e n S e s s i o n T i m e a n d P l a y ba c k B i t R a t e s f o r

C a s e s u p t o t h e O p t i m a l B i t R a t e T h r e sh o l d f o r E a c h E v e n t

Note: For the optimal bit rate threshold (i.e., the playback bit rates up to which cases were

analyzed for each of the above events), see Table 2.

Source: IDC, 2011

-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4

Sports Event C

Sports Event B

Sports Event A

Soccer C

Soccer B

Soccer A

(Correlation coef f icient)

6 #229083 ©2011 IDC

After that threshold, there was no additional positive effect on session times, or

the effect decreased. For the six events analyzed, the threshold was at different

levels (see Table 2). For publishers, this means that it is necessary to carefully

measure and monitor the impact of bit rate on session lengths to establish the optimal

bit rate range.

T A B L E 2

M a x i m u m P l a y b a c k B i t R a t e L e v e l S h o w i n g P o s i t i v e I m p a c t o n S e s s i o n L en g t h

Maximum Bit Rate (kbps) up to Which

Higher Bit Rates Further Improved

Impact on Session Lengths Correlation Coefficient

Soccer A 1000 0.120

Soccer B 2500 0.148

Soccer C 1000 0.031

Sports Event A 2000 0.021

Sports Event B 3500 0.048

Sports Event C 1500 0.023

Source: IDC, 2011

It is difficult to arrive at a formula that would express how much user engagement

(i.e., session lengths) increases as bit rate increases given the many factors that have

an impact on video performance (see next paragraph). But based on the kind of

performance increases we have seen in the data, we would expect to see, as a rule

of thumb, an increase of 10% in session lengths per 500 kbps increase in average

playback bit rate.

Keep in mind that the bit rates offered by publishers were vastly different between

events. This may be one reason why the cutoff is at different levels for different

events. We also theorize that there may be other effects at work as well. For instance,

those users who watch video at the highest bit rates also must have the infrastructure

(e.g., a high broadband access speed) in place to be able to watch at these rates.

Those users are also more likely to have incomes and busier lives, which could

explain why they are more likely to watch for shorter periods of time. More research is

needed, taking into account cultural, social, and situational factors.

Rebuffering

Rebuffering events are among the most frustrating experiences when watching a live

video stream. We analyzed the impact of rebuffering events per hour. These are

incidents where the buffer of the user's video player runs out of data and must be

©2011 IDC #229083 7

replenished by the server while the picture freezes. Rebuffering is caused either by a

connection slowdown or by bad heuristics (i.e., when the player waits too long to

switch to a lower bit rate).

We expected a negative correlation between the number of rebuffering events and

session durations (i.e., for more rebuffering events to go with shorter sessions)

because with more interruptions of the video stream, users would become more

frustrated and more likely to stop watching it.

That is precisely what we found. The number of rebuffering events turned out to be

one of the worst factors impacting user engagement (see Figure 2).

F I G U R E 2

C o r r e l a t i o n B e t w e e n S e s s i o n T i m e a n d R e bu f f e r i n g E v e n t s p e r

H o u r

Source: IDC, 2011

Then we looked at the impact of the total share of the session time that users had to

spend waiting for the rebuffering to complete and the video to resume (aggregating

the waiting time incurred by all rebuffering events) on session lengths.

Again, we expected a negative correlation (i.e., for higher rebuffering time shares to

go with shorter session durations). And again, we found that to be the case (see

Figure 3). Rebuffering duration was the second most important factor negatively

impacting user engagement. This also means the negative impact of rebuffering time

was smaller than that of the number of rebuffering events. Apparently, viewers find

that disruption as such is worse than waiting for it to end.

-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4

Sports Event C

Sports Event B

Sports Event A

Soccer C

Soccer B

Soccer A

(Correlation coef f icient)

8 #229083 ©2011 IDC

F I G U R E 3

C o r r e l a t i o n B e t w e e n S e s s i o n T i m e a n d P e r c en t o f T i m e S p en t

R e bu f f e r i n g

Source: IDC, 2011

Video Frames

The number of frames per second (FPS) or frame rate expresses the number of

consecutive images shown in a video transmission per second that create the illusion

of motion. Higher FPS numbers translate into better video quality because the video

plays more smoothly; lower FPS numbers conversely result in worse video streams.

Dropped frames are images that are not displayed by the user's video player, either

because local resources (CPU, graphics adapter, memory, etc.) are not sufficient or

because there is a disruption in the video transmission. From the user's perspective,

dropped frames translate into choppy video.

For the number of dropped frames per hour, we expected a negative correlation with

session durations (i.e., for more dropped frames to coincide with lower user

engagement) because dropped frames disrupt the viewing experience. This is what

we found in the numbers, too. Dropped frames per hour were the third most important

factor negatively impacting session lengths. However, the impact was fairly minimal

(see Figure 4).

-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4

Sports Event C

Sports Event B

Sports Event A

Soccer C

Soccer B

Soccer A

(Correlation coef f icient)

©2011 IDC #229083 9

F I G U R E 4

C o r r e l a t i o n B e t w e e n S e s s i o n T i m e a n d A v e r ag e N u m b e r o f

D r o p p e d F r am e s p e r H o u r

Source: IDC, 2011

F U T U R E O U T L O O K

This research has established that there is an impact of streaming video quality on

user engagement (i.e., session durations) and, therefore, on audience reach. As

consumers embrace online video distribution as a viable alternative to cable and

broadcast TV, their expectations of video quality will continue to increase. This will be

even more so as online video makes its way into consumers' living rooms, where it

ends up on high-definition television sets and will have to compete with the quality

that cable routinely provides. Here, a reliable online transmission at a resolution of

720p is only the beginning.

Publishers will need to embrace measuring and monitoring video quality such as

buffering, drop-off, and bit rate consumption on an ongoing, routine basis to tune the

experience and avoid dips in video quality and the resulting drop in user engagement

in order to protect their financial performance and competitiveness.

Given the wide range of bit rates offered in the six events analyzed, and the different

infrastructures given for them, it is difficult to arrive at a universal bit rate benchmark.

Two of the three soccer events in particular offered comparatively low bit rates

because of the expected huge numbers of viewers. If one looked only at the United

States, recommended bit rates would have to be set quite a bit higher.

Based on the given events, for the bit rate provided by publishers, IDC suggests

maintaining a level of at least 1200 kbps to attract the kind of audience numbers we

saw in the events analyzed. If one wanted to increase audience reach and user

engagement beyond that level, it would be prudent to increase the average bit rate to

1500 kbps.

-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4

Sports Event C

Sports Event B

Sports Event A

Soccer C

Soccer B

Soccer A

(Correlation coef f icient)

10 #229083 ©2011 IDC

The single most negative impact on engagement was the number of rebuffering

events. We believe the best practice is not about keeping rebuffering events to a

certain exact number; rather, it is about ensuring that the largest share of your

audience experiences no buffering at all.

Of course, only part of the occurrence of rebuffering events can be controlled by

publishers. Again, an optimized distribution technology and measuring and monitoring

rebuffering events are key to an optimized user engagement.

The second most important negative impact on viewer engagement is the amount of

time spent rebuffering. IDC recommends, as a rule of thumb, maintaining a level of

video quality at which users experience rebuffering for a maximum of 1% of the time.

This lines up with the experiences that publishers have had in practice. In an

interview with IDC, Glenn Goldstein, MTV's VP, Video Technology Strategy, said,

"Once rebuffering time hits 1% of the playback time, we know we're in trouble."

More research is needed regarding the impact of start-up times on user engagement

(which was not explored in this research) and the influence of demographic,

psychographic, and situational factors on video consumption.

C o p y r i g h t N o t i c e

External Publication of IDC Information and Data — Any IDC information that is to be

used in advertising, press releases, or promotional materials requires prior written

approval from the appropriate IDC Vice President or Country Manager. A draft of the

proposed document should accompany any such request. IDC reserves the right to

deny approval of external usage for any reason.

Copyright 2011 IDC. Reproduction without written permission is completely forbidden.