pre-fetch based content distribution system for software update: design and evaluation

July 2014, 21(Suppl. 1): 12–19 www.sciencedirect.com/science/journal/10058885 http://jcupt.xsw.bupt.cn

The Journal of China Universities of Posts and Telecommunications

Pre-fetch based content distribution system for software update: design and evaluation

ZHANG Yi ( ), GUO Yuchun, CHEN Yishuai

School of Electronics and Information Engineering, Beijing Jiaotong University, Beijing 100044, China

Abstract

Today, with rapid growth of apps (applications) for computers and mobile terminals, software update happens frequently. As a result, the flash crowd appearing in the early stage of a software patch release cannot be handled smoothly with abruptly arriving requests, thus considerable server bandwidth is needed. Unfortunately, it is still unknown how much bandwidth a content provider should provision to handle this case. In this paper, through measurements in one of the largest game patch distribution systems in China, we propose a simple but accurate user demand prediction model, since our finding that a highly predictable pattern of user requests exists in such a system. We further present the design of a pre-fetch based software update system. In this system, users who are idle and capable in hours before the patch release time could fetch the patches in advance and then serve other users requiring the patches. The design of this system mainly includes demand prediction and pre-fetch scale computation. After the deployment of this system, the contribution rate of users increases about 20% in the release day and is more stable than that before. Moreover, the peak of server bandwidth decreases by about 30%, which significantly saves bandwidth cost for providers.

Keywords software-patch-distribution, demand prediction, pre-fetch

1 Introduction

Software update is critical for ensuring software system’s lifetime and directly affects the software providers’ revenue. In recent years, online games have been becoming the most popular social network applications. According to a study result about the online game market (http:// www.91danji.com/news/2465.html), the number of online game player surges crazily, especially in China, about 110 million players in 2010, i.e., 37 percent growth compared with the number in 2009. For a successful online game, to attract players, the operators must deliver game patches periodically to add new game contents, such as new maps and new weapons.

It is, however, a challenge to provide high quality software update with reasonable cost, especially for online game systems, there are too many players waiting for Received date: 12-06-2014 Corresponding author: ZHANG Yi, E-mail: [email protected] DOI: 10.1016/S1005-8885(14)60518-5

downloading. According to the measurement study, high concurrent requests, i.e., flash crowd, occur at the beginning of the patch release. Flash crowd results in network congestion and affects user experience. Moreover, operators are suffering from too much bandwidth lease fee. A direct method to solve the problem is buying more servers and bandwidth, which increases cost of operators. In China, the bandwidth lease fee is measured by the peak value, about 60 000 yuan per GB for a year, and is the main cost of operators. Thus, this solution is not economical.

Considering the success of peer-to-peer (P2P) content distribution network [1–3] (e.g., BitTorrent (BT)), P2P technology has been widely used for the delivery of software update. However, at the beginning time of a release, a large number of users request the patches but few users own the patches. The phenomenon is more obvious in online game system, because players of old version client cannot play the game, all the players must

Supplement 1 ZHANG Yi, et al. / Pre-fetch based content distribution system for software update: design and evaluation 13

download the new content completely, so when the patch release begins, huge number of players are waiting for downloading.

In this paper, we present our analysis on bandwidth prediction and our design and evaluation of such a pre-fetch based content distribution system for online game update, based on a large-scale trace obtained from the game patch distribution system for Dungeon and fighter (DNF), which is one of the most popular online games in China. The operator must prepare enough bandwidth for the game release, even with the help of peers, because more than 2 million players stay online at the peak. We first conduct a comprehensive analysis of user requests arriving pattern to obtain the characteristics of user requests of game patch release. Measurement results reveal that the demand of game patches (which is decided by users’ arriving pattern) is highly periodic and can be easily predicted. Then, inspired by the measurement results, we propose a pre-fetch-based P2P scheme to distribute software update, i.e., the system first predicts the user demand based on the historical user demand, next, some reasonable and suitable super nodes could download game patches in advance. Thus, once the process of patch release starts, these nodes can share their patches to help servers.

We focus on two key problems for designing such a system:

1) How many peers will pre-fetch the software. 2) When to start the pre-fetch scheme. Systematic

answers of these questions need comprehensive understanding of user arriving and leaving pattern and peer capacity (e.g., uploading bandwidth), which are difficult to clearly obtain in the highly volatile Internet environment. In this paper, we adopt an experiment-based method to obtain a realistic pre-fetch scheme. Specifically, we implement a pre-fetch scheme in our real world large-scale game patch distribution systems and deploy it in the wild, and then verify the effectiveness of the system.

Evaluation results show that the scheme can not only utilize the uploading capacity of peers to a considerable level and significantly save the bandwidth consumption (the peak hour traffic is decreased more than 10%), but also can obtain stable and timely patch distribution. Besides, we also establish a simple mathematical model to analyze the relationship between the pre-fetch performance and the number of pre-fetch peers. Moreover, our approach does not modify the protocol or architecture of the

network. The contributions of this paper are two folds: 1) We propose a model for predicting user demand in

software system. In detail, we find user requests arriving pattern has great effects to the bandwidth those operators provide to players, moreover, the file size of the patch is a factor determines the scale of the release, finally we present an accurate prediction method. Our method could be used to compute the bandwidth needed to be prepared in advance for content distribution.

2) Based on our model, we study the number of peers needed to pre-fetch the patches and design the pre-fetch scheme. After the deployment in practical system, we find the contribution rate of peers becomes more stable and the peak of the bandwidth decreases by 20%.

2 Related work

There has been some work concentrated on how to improve P2P efficiency, such as replicas replication method [4–5], they proposed effective replicas replication mechanisms, which are also suitable in software distribution systems. And pre-fetch mechanism [6] used in streaming applications could improve the seeking performance. But for such a software distribution system, pre-fetch patches during the release time cannot handle flash crowd in the early time.

The issue of bandwidth prediction has attracted some researchers in VoD (video of demand) systems. The main approaches are auto-regressive and moving average (ARMA) and auto-regressive integrated moving average model (ARIMA) models [7]. However, these models cannot work well in game release systems, for they are based on relatively stable user arrival rate. In our scene, the number of players’ requests fluctuate fiercely especially in the early stage of the distribution, that is to say, it is hard to predict the population with ARMA or ARIMA. For comparison, we systematically analyze the relationship among file size, downloading counts as well as server bandwidth and present a useful model.

Handing flash crowd is also a popular topic in network researches [8]. Existing studies propose to implement flash crowd mitigation via adaptive application-level admission control. However, it is impossible to accomplish application-level admission control for software service provider, otherwise, throwing the requests into a queue results in update delay, which cannot be acceptable for

14 The Journal of China Universities of Posts and Telecommunications 2014

players. P2P caching schemes [9] has been widely used to

address flash crowd problem, but facing the large scale patch release, in early term, few peers can hardly supply considerable assistance due to hundreds of thousands of simultaneous requests. For comparison, our system focuses on the early stage and works well during the release.

The push strategy [10] has been proposed for many years, they focus on data placement strategy, which is not our objective. Some work [11–13] adopt network coding technology to improve the quality in VoD system which can be used in our system in future. Zheng et al., proposed a method to push patches to peers [14], but they use idle peers in other swarms. For comparison, the users can only pre-fetch the patches they need in our system.

3 Background

In this section, we first present background knowledge about online game patch release system, then state the special quality of service requirement of the system.

There are some differences among traditional BT networks or peer-to-server and peer (P2SP) networks and the game-patch-release system we studied. Game client was installed completely before, inside it, a downloading core is embed in, those cores and servers make up the traditional P2SP overlay network. Before a release, thousands of patches are going to be downloaded by players, these files are necessary for each player. Operators prepare patches in advance, when the release time comes, all the players cannot play the game, until they download the whole update completely. The game patches often released at 8 o’clock. Due to the temptation of the game, players like students or adults spend as much time as possible in the game, so when the game needs to be updated, they will be more sensitive to the downloading time, because they cannot wait for playing the game. In other words, in such a scene, players need more guarantee on the downloading speed, so unlike traditional BT network, service provider has to buy more bandwidth.

The periodic change of the system working mode is another difference. On one hand, release mode (RM) often happens in game-patch-release days. On the other hand, after resources become unpopular after the day their released, the distribution works in normal mode (NM). We present our measurement results in the next section.

The data we obtained is from the DNF operator.

Collected by a core of downloading software, the data logs system performance and users’ downloading behavior. The fields of data include downloading timestamp, downloading time, downloading speed, HTTP speed, P2P speed and file information. There are about six hundred million records from April 1st to April 19th in 2012.

4 Problem statement

In this section, we present the measurement result of a system to demonstrate the performance problems in the system for game patch distribution.

To learn the characteristics of bandwidth demand of game-patch-release-system, we extract data from April 1st to April 19th in 2012, which shows great differences of bandwidth consumption modes between RM and NM.

As a characteristic of game-patch-release-system, about one update event happens in a week. During the early 19 days in April, the release days are the 6th and 13th day (RM), as is shown in Fig. 1, and the system works in NM in other days. We can notice the huge gap between these two modes, the peak of total bandwidth is about 40 Gbps in NM, while about 210 Gbps in RM, and the peak of HTTP bandwidth is about 120 Gbps which lasts about 2 hours in the release days, operators have to prepare enough bandwidth for the 2 hours. In China, the bandwidth is hired by month, thus the major cost is the peak of bandwidth in some hours in a month.

Fig. 1 The bandwidth situation of the April 19th.

Therefore, our attention is focused on RM, we choose the 6th (RM) and the 8th day (NM) so as to find the reason why in RM the HTTP bandwidth is so high. Generally, in a P2P system, the P2P rate is a key index to describe the efficiency of the P2P mechanism, so we introduce P2P rate to find the reason. The P2P rate is computed by the equation as follows:


( )( ) ( )

P2P

HTTP P2P

( )B t

tB t B t

ρ =+

(1)

where ( )tρ means the P2P rate, and B means

bandwidth. As what is shown in Fig. 2, P2P rate of the 8th day remains at 52% whereas about 36% in 6th. Fig. 2 illustrates that the P2P rate is too low to support downloading at the beginning of release, especially at 12 o’clock, P2P rate is about 28%, and the HTTP bandwidth is very high. In a mature system, the P2P rate often has an upper boundary about 55% concluded by our experience.

(a) HTTP bandwidth comparison between Arp. 6th and Apr. 8th

(b) P2P rate comparison between Apr. 6th and Apr. 8th

Fig. 2 The HTTP bandwidth and P2P rate of 6th day and 8th day in April

From Fig. 1 we can see that in RM days after 8 o’clock, about 20 thousand times of downloading behaviors occurred at that time, the P2P rate is very low, this is because few replicas exist in the network and the system cannot afford thousands of concurrent requests.

Based on the shortcomings above in current system, we design a pre-fetch-based system to reduce bandwidth in RM days.

5 Pre-fetch-based system design

This section presents our design of the pre-fetch-based system. We first introduce the system architecture. Then, we focus on the design of two key system modules, i.e., demand prediction and the detailed pre-fetch algorithms.

5.1 Architecture

P2P overlay network is good at content distribution, however, its performance is not stable enough for peers’ churn or files’ scarcity, especially in the early term of a release.

Therefore, we design a system called as pre-fetch

system. The system shown in Fig. 3 is based on traditional P2P overlay network, and the key technology is called pre-fetch, which means clients could get game patches in advance, so they will complete the update soon, and enjoy the software.

Fig. 3 The architecture of pre-fetch system

The pre-fetch system module introduced in the left part of Fig. 3 is the main difference compared to traditional P2P network. Unlike normal downloading process, the day before a game-patch-release, when peers start the game, the clients start to register itself to track servers and pre-fetch-strategy-server to get pre-fetch job, then the pre-fetch-strategy-server judges that whether this peer is suitable to be a pre-fetch peer with the peer information history. Once a peer becomes a pre-fetch peer, it start to get URL from strategy server and download the patches. Thus, when the game-patch release, these nodes can help the server. But how many peers should become pre-fetch peer, and how to judge a peer is suitable to be a pre-fetch peer are key questions, in the following part, we propose a method of predicting bandwidth to answer the first question.

5.2 Bandwidth model

In this subsection, we formulate the problem of uploading bandwidth provision in the pre-fetch system.

To compensate the gap between the two modes, we introduce pre-fetch bandwidth to assist servers. Thus the bandwidth supply and demand function becomes

( ) ( ) ( ) ( )d h pf prB t B t B t B t= + + (2) where dB means the demand from game players, hB denotes HTTP bandwidth from servers, prB is the

bandwidth from other peers who formed after the release time. The only thing different from conventional system is


the pre-fetch bandwidth from pre-fetch peers abbreviated as pfB . Like other P2P systems, the index we evaluate the

system is P2P rate, the equation B shown formulated as follows

( ) ( ) ( )( )

pr pf

d

B t B tt

B tρ

+= (3)

From Eq. (3) we can infer that, if the demand is a constant value, P2P rate increases with pfB .

5.3 Demand prediction

This subsection presents our measurement result of the game patches demand. Based on the measurement results, we propose our demand prediction algorithms.

To predict the HTTP bandwidth with the phenomenon of flash crowd is very hard in normal P2P system, for users’ high dynamics and heterogeneity. However, in a game-patch-release system, due to the stability of player scale in a mature game, some properties can be excavated out.

Downloading counts is one of the most important factors which influence the bandwidth. In Fig. 1, the downloading counts line is very similar with the HTTP bandwidth line, the Pearson correlation coefficient is 0.987 1, which means the same trend in the system. Fig. 4 plots the downloading ratio of different patches which is computed as follows

cdc

DR

μσ−

= (4)

where dcR denotes the normalized ratio of downloading counts. We plot the average normalized downloading counts of all the patches during releases, and take six typical patches for example. Patch1 and patch 2 are selected from the same release, patch 3 and patch 4, patch 5 and patch 6 from other two releases. And the patches of odd order are big files, patches of even order are small files. Every pair is from the same release, the two files must be downloaded finally. The average RMSE between the six patches and average lines is 0.001 73. The normalized ratio of downloading counts are almost consistent with each other, moreover, the same situation occurs among different pairs, the average Pearson correlation coefficient is 0.996 5. We can infer that there must be a downloading pattern among different patches. Fig. 5 plots the downloading counts of twenty patches, we can see that the downloading counts lines are very

concentrated, and the Pearson correlation coefficient is 0.927 3. The first peak arrives at about ten o’clock, because the players are mainly college students, white-collars, freelancers, people unemployed by questionnaire, they always have considerable time, so the downloading peak often arrives at the unwanted time, such as 11 a.m., 2 p.m. and 6 p.m.

Fig. 4 The normalized ratio of downloading counts in different releases

Fig. 5 The downloading counts of different patches in Apr.6th

Inferred from the analysis, the average curve in Fig. 4 could be considered as a template of downloading counts.

According to the result above, the relation can be expressed as

( ) ( )hB t C t∝ (5)

where ( )C t denotes the downloading counts at time t .

Due to the downloading counts template, the trend of downloading counts during the release day can be predicted by that of counts in a release before, thus the equation becomes

( ) ( )h N 0B t C t t∝ − (6) where 0t is the interval between two releases, and NC is the normalized ratio of downloading counts. We could use


the history information to predict the bandwidth, inferred from the Fig. 1, the distribution of normalized downloading counts of different patches are nearly the same except the latitude. Although the downloading counts of Apr.6 and Apr.13 are similar, the latitude of downloading bandwidth is different. This is to say, downloading counts is not the only factor. Before the release, the thing we can know is the information about files, we need to find the relation between file size and bandwidth.

Fig. 6(a) describes the file size CDF in a release. We find that the biggest ten files will cost about 55% of the bandwidth. Through the phenomenon, we can conclude that big files will cost more bandwidth.

(a) CDF of file size of Apr. 6th

(b) Bandwidth consumption of top N sized files in Apr. 6th

Fig. 6 The CDF of file size in Apr. 6th and the bandwidth consumption of top N sized files

The relationship can be expressed as ( )h fB t S∝ (7)

in which the alphabet f denotes files to be delivered, and the file size is abbreviated as fS , inspired by the Fig. 4, the equation can be expressed as:

( ) ( )h f N 0B t S C t t∝ − (8)

The simple equation describes key elements about the bandwidth in a game-patch-release by measurement, with the equation, we can predict the demand bandwidth trend in the future.

From Fig. 1, the bandwidth of Apr. 6th and that of Apr. 13th are almost the same, we can assume that the periodicity is about a week, thus, we have

( ) ( )N 0C t t C t T− ∝ − (9)

where T is the multiple of 7 days, let α be the result of their product, Eq. (9) becomes

( ) ( )h fB t S C t Tα= − (10)

To verify the correction of the Eq. (10), we predict the

bandwidth of April 13th day with the data of April 6th. The total file size in 6th is 1.291 GB while 453 MB in April 13th, from the equation, we can find that the bandwidth in 6th is 2.7 times as that in 13th. Fig. 7 plots the prediction result and the practical data of April 13th.

Fig. 7 The HTTP bandwidth of practical system and the prediction result in Apr. 13th

From Fig. 7, we can find that the bandwidth we predict is very similar to that in practical system, the coefficient is 0.983 4, RMSE is 3.9, which means a successful prediction.

5.4 Pre-fetch-strategy design

This section we propose some strategies in detail for the system design, including two problems:

1) How many pre-fetch peers the system needs. 2) The condition of normal peer becomes pre-fetch peer. (1) Calculate the number of pre-fetch peers needed According to the bandwidth prediction introduced in the

third section, we can predict the bandwidth in a release. The flash crowd often arrives at about 12 and 18 o’clock, because the bandwidth latitude is very big, and the average curve gradient near 12 o’clock is 0.89, while about 0.83 near 18 o’clock. So we select 12 o’clock as the bandwidth peak time. The bandwidth gap could be obtained by prediction:

gap peak troughB B B′ ′= − (11)

The number of pre-fetch peers can be computed by gapB

Nv

= (12)

where peakB′ means the peak HTTP bandwidth that day,

and troughB′ is the trough bandwidth after 8 o’clock,

usually happens at 14 o’clock, and v is the average upload speed of pre-fetch peers.


(2) Conditions of pre-fetch peers In such a system, hundreds of thousands of peers stay

online at the same time, some players have perfect input/output (I/O) devices. To assist servers, those computers have priorities to become pre-fetch peers. The conditions we use are mainly the average online time and the ability of network I/O, In order to guarantee the experience of players, we also take CPU and memory utilization into consideration.

(3) The start time of pre-fetch scheme From Fig. 4, the idle time of the system is concentrated

at zero o’clock, thousands of players are still online after zero o’clock, so the time pre-fetch-peers get pre-fetch jobs can be selected at 0 o’clock.

6 Performance evaluation

This section presents the performance result of the real system. The scheme is effective from the result of simulation in testing environment. Finally, we deploy our method in the real system for the new release in April 20th.

According to the history information, the file size is 417.31 MB, from Eq. (10), the is peakB about 35 Gbps and the troughB is about 13 Gbps. To guarantee users’

experience, the upload speed is limit to 40 KBps at release day, finally we select 72 000 peers, the pre-fetch files are the biggest top five files. The release began at 0 o’clock, and the maximum bandwidth is about 45 Gbps.

Fig. 8 plots the performance and measurement results we get from the real system with pre-fetch. Fig. 8(a) plots the total bandwidth, the HTTP bandwidth and the bandwidth contributed by pre-fetch peers in April 20th. Fig. 8(b) shows the bandwidth proportion (HTTP bandwidth divide by total bandwidth) of the two days. Fig. 8(c) and Fig. 8(d) describe the HTTP speed and average speed of the two days respectively.

1) Comparing the curves in Fig. 8(a), the pre-fetch traffic takes up about 30% of the total bandwidth, that is to say the HTTP bandwidth decreases by more than 30%. Moreover, the trend of pre-fetch bandwidth is the same as that of HTTP bandwidth, we can infer that the pre-fetch bandwidth can be self-adaptive with the HTTP bandwidth.

2) Comparing the bandwidth proportion of Apr. 6th and Apr. 20th in Fig. 8(b), the P2P rate becomes more stable with pre-fetch technology. The mean of the proportion in 6th day is 0.63, and the variance is 0.205, while in 20th, the mean is 0.437, and the variance is 0.037 2.

(a) Variation of bandwidth in Apr. 20th

(b) Bandwidth proportion comparison between Arp. 6th and Apr. 20th

(c) HTTP speed comparison between Arp. 6th and Apr. 20th

(d) Average speed comparison between Apr. 6th and Apr. 20th

Fig. 8 The system performance of Apr. 20th


3) Comparing the HTTP speed and average speed of Apr.6th and Apr.20th, the average HTTP downloading speed is lower and more stable whereas the average total downloading speed is the same as before from Fig. 8(c) and Fig. 8(d). In April 6th, when the flash crowd occurs, the HTTP speed decreases fiercely, for the expectation is 127 KBps, and the variance is 5.203. However, in 20th, with the help of pre-fetch peer, the HTTP speed is not influenced by the flash crowd, the expectation is 95.3 KBps, and the variance is 0.426.

7 Conclusions

In this work, we build a simple method to help software operators save money from the heavy traffic cost. From our measurement, some particular characteristics have been found out, i.e., the user requests pattern determines the HTTP bandwidth pattern while the file size determines the scale of a release. We design a system based on pre-fetch. After the deployment in real system, the results prove the effectiveness of our system.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 61271199, 61301082 and the Fundamental Research Funds for the Central Universities under Grant W14JB00500.

References

1. Zhang Y, Lei L, Chen C. Characterizing peer-to-peer traffic across Internet Grid and Cooperative Computing. Springer Berlin Heidelberg, 2004:

388−395 2. Gai A T, Mathieu F, Reynier J, et al. Stratification in P2P

networks-application to bittorrent. ArXiv preprint cs/0612130, 2006 3. Gummadi K P, Dunn R J, Saroiu S, et al. Measurement, modeling, and

analysis of a peer-to-peer file-sharing workload. ACM SIGOPS Operating Systems Review. ACM, 2003, 37(5): 314−329

4. On G, Schmitt J, Steinmetz R. The effectiveness of realistic replication strategies on quality of availability for peer-to-peer systems. Peer-to-Peer Computing, 2003 (P2P 2003). Proceedings of the 3rd Third International Conference on IEEE, 2003: 57−64

5. Zhou Y, Fu T Z J, Chiu D M. Statistical modeling and analysis of p2p replication to support vod service. INFOCOM, 2011 Proceedings IEEE. IEEE, 2011: 945−953

6. Zheng C, Shen G, Li S. Distributed prefetching scheme for random seek support in peer-to-peer streaming applications. Proceedings of the ACM Workshop on Advances in Peer-to-Peer Multimedia Streaming. ACM, 2005: 29−38

7. Niu D, Liu Z, Li B, et al. Demand forecast and performance prediction in peer-assisted on-demand streaming systems. INFOCOM, 2011 Proceedings IEEE. IEEE, 2011: 421−425

8. Zhou W, Wang D. A dynamic-resource-allocation based flash crowd mitigation algorithm for video-on-demand network. Computer Science and Information Technology (ICCSIT), 2010 3rd IEEE International Conference on IEEE, 2010, 1: 388−392

9. Huang Y, Fu T Z J, Chiu D M, et al. Challenges, design and analysis of a large-scale p2p-vod system[C]. ACM SIGCOMM Computer Communication Review. ACM, 2008, 38(4): 375−388

10. Suh K, Diot C, Kurose J, et al. Push-to-peer video-on-demand system: Design and evaluation. Selected Areas in Communications, IEEE Journal on, 2007, 25(9): 1706−1716

11. Ren D, Wong W, Chan S H G. Toward continuous push-based P2P live streaming. Global Communications Conference (GLOBECOM), 2012 IEEE. IEEE, 2012: 1969−1974

12. Cassagnes C, Magoni D, Chang H, et al. On the scalability of P2P-based push-driven live streaming systems. Communications (ICC), 2010 IEEE International Conference 2010: 1−6

13. Wang M, Li B. R2: Random push with random network coding in live peer-to-peer streaming. Selected Areas in Communications, IEEE Journal, 2007, 25(9): 1655−1666

14. Zheng Y, Huang D, Chen C J, et al. Adaptive resource scheduling mechanism of P2P file sharing system. Accepted by Journal of Frontiers of Computer Science in China, 2012

pre-fetch based content distribution system for software update: design and evaluation

Documents