Digital Library Engine: Adapting Digital Library for Cloud Computing

Weiming Lu, Liangju Zheng, Jian Shao, Baogang Wei, Yueting Zhuang College of Computer Science and Technology

Zhejiang University Hangzhou, China

{luwm, lj_zheng, jshao, wbg, yzhuang}@zju.edu.cn

Abstract— With the rapid growth of digital libraries, more data and smart services are involved. People come to recognize the importance of digital libraries and the convenience they might bring to the society. However, the cost of owning a digital library is quite high, and many institutions do not have the ability to run and maintain a digital library by themselves,especially for massive data and complex services which require lots of storage and computing resources. In this paper, we proposed the Digital Library Engine, which aims to provide a new Platform as a Service for fast developing and deploying digital libraries in cloud. To the best of our knowledge, it is the first work to create a PaaS system for digital libraries. With the help of Digital Library Engine, institutions only need to develop some service bundles, which can be deployed in the engine, and then their own digital libraries could be running well with features of scalability, reliability, security, extensibility, availability and manageability. The practice in CADAL and the experiments demonstrate the feasibility and efficiency of our engine.

Keywords-Digital Library Engine; PaaS; resource management; QoS; multi-tenancy

I. INTRODUCTION

With the rapid development of digital libraries, more resource such as books, images, audio and video are digitized, preserved and shared on the Web. Besides, digital libraries are transiting to smart libraries. That is to say, various smart services such as personalized service, recommendation service, knowledge-based service and translation service are being developed and provided in digital libraries. For instance, CADAL Digital Library (China Academic Digital Associate Library)1 has developed several smart services for its 2.5 millions digital books and other multimedia resources, including multi-faceted book search [1], personalized service [2], recommendation service [3], Chinese Literature Chronicle service [4], Chinese Traditional Medicine service [5] and Chinese Calligraphy service [6].

The digital libraries become essential not only in preserving the cultural heritage but also in knowledge sharing for the society. More people come to recognize the importance of digital libraries and the convenience they might bring to the society. Many organizations such as schools, universities, colleges, institutions and companies want to have their own digital libraries to digitize and

1 www.cadal.zju.edu.cn

organize the resource in the organization, provide services to their members or the public. However, the cost of ownershipis quite high, and many organizations do not have the ability to run and maintain the digital libraries by themselves. For example, IT infrastructure and software should be purchased, maintained and updated. Moreover, some dedicated IT personnel and a server room are also required.

In addition, the traditional services such as browsing and reading services in digital libraries are not satisfying,especially when the volume of digital resources are managed in digital libraries grows with an unprecedented rate in the coming years. For example, more intelligent services are needed to discover values in these large volumes of data, and more personalized services are needed to meet different people’s needs. However, these complex services require lots of computing resources, which is hard for institutions to provide them.

Cloud computing can decouple the responsibility of running the infrastructure from the digital library that use it. With this decoupling, users need only to focus on the services of digital libraries, while do not need to consider the reliability, scalability, usability and security of the infrastructure. In addition, cloud computing can also obtain more economic benefits with infrastructure and administration sharing, and the dynamic resource scaling ability. Moreover, cloud computing can provide elastic computing resources for complex services in digital libraries.

The current cloud computing architecture can be classified into three categories: SaaS (Software as a Service), PaaS (Platform as a Service) and IaaS (Infrastructure as a Service). SaaS is a way of offering complete applications over the Internet as a service, such as Google Docs, Gmail and Salesforce.com [7]. PaaS provides a platform for developing and deploying custom applications on it, such as Google App Engine [8] and Microsoft Windows Azure [9].IaaS can provide storage resources and elastic computing resources on demand, such as Amazon’s Simple Storage Service (S3) [10], Elastic Compute Cloud (EC2) [11], and some open source implementations such as Eucalyptus [12]and OpenStack [13].

In this paper, we proposed the Digital Library Engine (DL Engine), which aims to provide a new Platform as a Service for fast developing and deploying digital libraries in cloud. With the help of Digital Library Engine, organizations only need to develop some service bundles, which can be deployed in the DL Engine, and then their own digital libraries could be running well with features of scalability,

2013 IEEE Sixth International Conference on Cloud Computing

978-0-7695-5028-2/13 $26.00 © 2013 IEEE

DOI 10.1109/CLOUD.2013.58

934

reliability, security, extensibility, availability and manageability. In addition, the Quality of Service (QoS) could be guaranteed in digital libraries on the platform through dynamic resource scaling. Moreover, if people build their digital libraries under the Digital Library Engine, it would be quite convenient for them to share the digital resources and services each other, which would connect information isolated islands together.

The paper is organized as follows. Section 2 gives some related works about PaaS. Then, we will give the architecture overview of the Digital Library Engine in section 3 and describe some implementation details in section 4. We evaluate the performance of the engine in section 5. Finally, we conclude in section 6.

II. RELATED WORKS

Platform as a Service is one of the key services in cloud computing, and it represents the glue that integrates custom and commercial apps and corporate IaaS and SaaS, without requiring cumbersome programming.

Recently, many PaaS systems have been developed. Google App Engine, Microsoft Windows Azure, IBM’sSmartCloud Application Services [14] and Oracle Cloud Platform [15] are the representative set of such systems. Google App Engine supports a scalable, reliable and secure runtime environment, which enables users to develop and deploy web applications more conveniently. Windows Azure is an open and flexible cloud platform that enables users to quickly build, deploy and manage applications across a global network of Microsoft-managed datacenters. IBM SmartCloud Application Services offer a comprehensive array of tools for developing, deploying, managing, and integrating corporate applications in the cloud. Oracle also provides a shared and elastically scalable platform for consolidation of existing applications and new application development and deployment. However, all these platformstry to provide a general solution for developing and deploying applications in the cloud, but they are usually heavily restricted [16]. Moreover, they don’t provide supports for creating a digital library.

Nowadays, several specific computing platforms and storage platforms begin to emerge on the cloud. Yang et al. [17] designed a platform for partitioning and execution of data stream applications in mobile cloud computing, which can move some computation to the cloud to compensate the lack of significant computation capability and energy power of mobile devices. Satzger et al. [18] also proposed an elastic stream computing platform in the cloud, which mainly to deal with the distributed online processing of event streams. In addition, Database as a Service [19, 20, 21] also has gain significant momentum, which can provide DB Instance for users in the cloud.

Digital libraries are also trying to use cloud computing to reduce the maintenance needs and encourage efficient resource use. For example, CiteSeerx [22,23], which is an application instance of SeerSuite [24], tried to migrate some services and components to the cloud. But it does not deploy SeerSuite in the cloud as a PaaS to create and manage multiple digital library instances. LDPaaS (Long-Term

Digital Preservation as a Service) [25] was proposed to address the problem of archiving and preserving digital records in digital libraries. But it only supports the services of preservation and doesn’t support more customized services for users.

There are not only massive data in digital libraries, but also various types of services, so the features of extensibility,customization and dynamic are very important for digital libraries. OpenDLib [26] and D-Net [27] tried to build an expandable digital library based on service oriented architecture. But they emphasized on service reusability and interoperability between different institutions, and didn’tbuild and maintain digital libraries for institutions. We also proposed a scalable, adaptive, self-management and fault-tolerant architecture for digital library [28], which tries to manage various services uniformly in digital library and make them scalable, adaptive, self-management and fault-tolerant. However, cloud computing was not exploited in these approaches.

III. ARCHITECTURE OVERVIEW

In this section, we will describe the diagram of Digital Library as a Service at first, and then give the architecture of Digital Library Engine.

Figure 1 depicts the diagram of the Digital Library as a Service, which consists of four layers including a physical cluster, virtual infrastructure, Digital Library Engine and several digital libraries.

Virtual Infrastructure(OpenStack )

VMOS

DL App VMOS

DL App

VMOS

DL App

VMOS

DL AppVMOS

DL App

VMOS

DL App

VMOS

DL App

DL Instance DL Instance

DL InstanceDL Instance

DL Instance

DL Instance

DL Instance

Digital Library Engine

Physical ServersStorageNetworks

Figure 1. The Architecture of Digital Library as a Service

Digital Library Engine is deployed over the physical cluster, and can allocate individual VMs (virtual machine) with a desired configuration of CPU, memory and disk space to set up a digital library for users through IaaS system (we use OpenStack in our system now, and plan on adding additional IaaS systems, such as Amazon EC2). The produced digital library is called DL Instance (Digital Library Instance), which is running on a virtual cluster. That is to say, when a user wants to set up a digital library, he only needs to send a simple XML description of the desired

935

virtual cluster to the Digital Library Engine. Then our Digital Library Engine will provision a collection of VMs at first, and load a VM image for each VM to generate a DL Instance with basic services to the user. If users want to enrich services in the digital library, he can develop some services in the form of SOA (service oriented architecture) bundle and then deploy them into the digital library.

When a DL Instance runs in a cluster of VMs, it can take advantage of the capabilities provided by the virtualization infrastructure such as live migration, elastic scaling and better sharing of physical resources.

Digital Library Engine can manage several DL Instances efficiently, and the architecture is shown in Figure 2.

DLManagement

ResourceManagement

UserManagement

Infrastructure as a Service(OpenStack)

ClusterManagement

BundleManagement

QoSManagement Load Balance

FaultTolerance

MessageQueue

Digital Library Framework

IaaS API

ServiceManagement

ServiceScheduler

SystemMonitoring

RESTfulA

PI

Digital Library Engine

CLI

Admin

Adminconsole

Developer

D-Ocean

Figure 2. The Architecture of Digital Library Engine

Digital Library Engine consists of several key components including Digital Library Framework, DL Management, Resource Management, User Management, RESTful API and IaaS API.

Digital Library Framework is the most important component in the architecture, which abstracts the digital library and is just like a digital library template. Therefore, Digital Library Engine can produce DL Instances according to the framework. The framework is based on OSGi (Open Service Gateway initiative) framework [29], which can deploy and manage bundles dynamically, where services in digital library are implemented as OSGi bundles. Since OGSi framework has to run applications in a single Java Virtual Machine, which means all these services should be in a single server. That is to say, it is not suitable for various and high-concurrent access and it also has the single point of failure (SPOF). So we extend the OSGi framework to adistributed environment by deploying services in multiple VMs as replicas for scalability and fault-tolerance, which enables the DL Instance to be distributed in multiple VMs. The distributed OSGi framework is based on a master-slave architecture, where the OSGi framework is deployed on each slave, and they can cooperate with each other in the cloud toprovide a coherent logical function.

In addition to bundle management and service management, Digital Library Framework also implements the functions of cluster management, QoS management, load balance, fault tolerance, service schedule and system monitoring, and we will expatiate on them in section 4.

Digital Library Instance can use D-Ocean to manage the digital books and multimedia resource. Since D-Ocean is not the emphasis in this paper, we describe it simply as follows. D-Ocean is a data management system we developed to store, analyze, index and search various types of unstructured data, which is built on a cloud computing platform, Hadoop[30].In D-Ocean, data modeling can represent different types of data into a unified digital object through an object-oriented model. Then it provides a unified interface to store different types of data in a suitable store, such as MySQL, HBase, HDFS, XMLDB and GraphDB, which are all managed by D-Ocean. Finally, these data can be analyzed and indexed through system-provided or user-provided plug-ins, which can run in a MapReduce way for batch processing. System-provided plug-ins include data analysis, data mining and data indexing toolkits, such as document metadata extraction, image feature extraction, video caption detection and recognition, clustering, classification, full text indexing and high dimensional indexing.

Resource Management focuses on VM management in the cluster by invoking IaaS API. It can launch a VM cluster for new DL Instance deployment, when a user wants to build a new digital library by Digital Library Engine. It can also create a new VM or delete a VM for an existing DL Instance to guarantee load balance and fault tolerance. Resource Management also collects the status information from physical servers for VM schedule and load balance.

User Management is used to manage users of Digital Library Engine, especially for Digital Library Engine should be multi-tenant. User’s unique name can be used to distinguish the DL Instance and data in D-Ocean.

DL Management is mainly used for managing Digital Library Instances in the cloud. It maintains all metadata about each DL Instance, including the location of the DL Instance, VMs of the DL Instance, the mapping from users to DL Instances. With this metadata, DL Engine can transmit user’s query to the right DL Instance, and control cloud-wide activities such as life-cycle management for DL Instance.

Administrator can access the Digital Library Engine through well-defined interfaces to manage the engine, such as start, stop, or upgrade the engine. Users can build a digital library on the scalable cloud platform through RESTful APIs, which are provided by DL Engine. When the library is built, users can access the DL Instance via HTTP(S) for DL management such as bundle installation, bundle start, bundle stop and bundle un-installation.

IV. IMPLEMENTATION In this section, we only describe the implementations for

some important functions of Digital Library Engine in detail,including Digital Library Framework, resource management, QoS management and multi-tenancy support.

A. Digital Library Framework Digital Library Framework is the most important

component in our Digital Library Engine, and it is implemented based on the distributed OSGi framework,which consists of a single Master, a SecondMaster and multiple ServiceManagers and provides several RESTful

936

D-Ocean

Zookeeper

Web Interface or RESTful API

bundle

data

AdminconsoleCLI

Master

ClusterManage

ServiceSchedule

ServiceManage

MessageQueue Cli

SystemMonitoring

ServiceManager

BundleManagement

OSGi Framework(Equinox)

ASB

BSB

ServiceManager

Bundle Management

Start Unregister

Register StopD-Ocean Bundle (BSB)

D-Ocean Client (API)

Data Access Data Search

Data Modeling PluginManagementData Analysis

IndexManagement

Application Service Bundle (ASB)

BookService

ImageService

VideoService

CalligraphyService

LiteratureChronicleService

BSBcache

log

session

monitoring

OSGi Framework(Equinox)

Load Resolve

bundle

SecondMaster

MessageQueue Cli

ServiceManager

BundleManagement

OSGi Framework(Equinox)

ASB

BSB

MessageQueue Cli

MessageQueueClient

Virtual Machine

Virtual MachineVirtual Machine

Digital Library Instance

Virtual Machine

QoSManagement

Message Queue

Figure 3. Digital Library Framework

APIs for administration and digital library portals, as shown in Figure 3.

Master, SecondMaster and ServiceManagers are all deployed on VMs, which forms a distributed digital library system.

The Master manages the ServiceManagers through Zookeeper [31], which is a high-performance coordination service for distributed applications. When one ServiceManager is down, the Master would be notified through the Zookeeper. Then the Master could re-schedule services through VM migration or bundles re-dispatch.

The Master maintains all metadata about bundles and services, including the mapping from service to ServiceManager, the mapping from SessionID to ServiceManager and the locations of bundles. The Masteralso collects the status information from ServiceManager,which is used for service schedule and load balance. In addition, the Master controls system-wide activities such as services migration between ServiceManagers, life-cycle management for bundles, and fault tolerance for bundles, services and ServiceManagers.

The SecondMaster is the standby for the Master. When the Master fails, the SecondMaster could take over the Master, so the system could still respond to user’s requests even when the Master fails.

The ServiceManager is implemented based on an OSGi framework (Equinox is used in our implementation), and it is responsible for the bundles management. Bundles in ServiceManager are divided into two categories: BSB (Basic Service Bundle) and ASB (Application Service Bundle). BSB is deployed in every ServiceManager, including D-Ocean bundle, monitoring bundle, cache bundle, log bundle and session bundle, while ASB is only deployed to some ServiceManagers according to load and bundle dependency. In Digital Library Engine, BSBs would be packed into a VM image with ServiceManager, and ABSs are developed and installed by users. When developing ABSs, users can utilize some underlying services provided by DL Engine, such as D-Ocean service for data modeling, storage, analysis, index and search, cache service, etc. In CADAL, we implemented several ASBs including book service bundle, image service bundle, video service bundle, calligraphy service bundle, recommendation service bundle, Chinese traditional medicine service bundle and Chinese literature chronicle service bundle.

In addition, Message Queues are used in a DL Instance for QoS management, and Master and ServiceManagers all have message queue client to operate the message queues.

937

Bundle management, cluster management, and fault tolerance are the main functions in a DL Instance. We explain them as follows.

1) Bundle management Bundle management is the core function in a DL Instance,

and it is the basis for managing various services, load balance and fault tolerance. Bundle management involves the life-cycle of bundles, including bundle loading from D-Ocean, bundle installation, bundle resolution, bundle start, bundle stop and bundle un-installation.

Each bundle is described by specific bundle definition,which consists of bundle name, services provided by the bundle, data model involved in the bundle, and the deployment information such as hardware resources demand, the number of service to be deployed.

Figure 4 shows the framework for bundle management in a DL Instance.

Http Server

Client

MasterCmds

Monitoring data

Zookeeper

Data Collector

HardwareStatus

BundleStatus

ServiceStatus

Monitoring bundle

Netty Server

OSGi Container

ServiceManager

Data Collector

HardwareStatus

BundleStatus

ServiceStatus

Monitoring bundle

Netty Server

OSGi Container

ServiceManager

Data Collector

HardwareStatus

BundleStatus

ServiceStatus

Monitoring bundle

Netty Server

OSGi Container

ServiceManager

Bundle and its definition

Figure 4. Bundle Management Framework

When a bundle is developed, it can be uploaded to Master with its definition for installation. The Master could parse the definition, and then store the bundle in D-Ocean. Then the Master would notify several ServiceManagers to install the bundle according to the deployment information and the load information of ServiceManagers. The ServiceManagers would obtain the bundle from D-Ocean, and then install it after bundle dependency resolution. Finally, the ServiceManager will notify the Master for its new installation.

2) Cluster management Zookeeper is a simple way to manage the

ServiceManagers in a DL Instance. The Master and each ServiceManager are registered as a Zookeeper node, and the Master is listening to the nodes of ServiceManagers.

When a ServiceManager fails, the Master would be notified. Then the Master could ask the Resource Manager for a new VM to replace the failed ServiceManager. After a new VM is allocated, which would be deployed with a basic ServiceManager, it will add a node to Zookeeper, so the Master would know the new ServiceManager, and then connect to it for bundle installation and service schedule.

3) Fault tolerance The failure of a ServiceManager could be handled by

Cluster management as mentioned above. While for the Master, a SecondMaster is used as a

standby through Zookeeper. The SecondMaster is also listening to the Zookeeper, and when some information such as the mapping from service to ServiceManager, the mapping from SessionID to ServiceManager and the locations of bundles changes, the SecondMaster would change its information accordingly. With the help of VM replication [32], the state of the active and standby could be consistent, so when the Master fails, the SecondMaster can take over the failed Master.

B. Resource Management The Resource Management of Digital Library Engine has

the similar idea with Apache Hadoop NextGen MapReduce (YARN) [33], which splits the two major functions of Hadoop’s Job Tracker: Resource management and Job scheduling/monitoring into separate daemons. We also have a global Resource Manager and per-DL Master, as shown in Figure 5.

Physical Server

Node Manager

Resource Manager

Physical Server

Master ServiceManager

Physical Server

Node Manager

ServiceManager

ServiceManager

Physical Server

Node Manager

ServiceManager

ServiceManager

Master

RESTful API

DL Catalog

User Manager

VM AllocationVM Request

User requestForwardNode status

Figure 5. Resource Management in Digital Library Engine

The global Resource Manager cooperates with NodeManager in each physical server manages and arbitrates all VMs among DL Instances. It collects the resource usage (cpu, memory and disk) of every physical server. With this information, Resource Manager can create VMs in appropriate physical servers and move VMs between physical servers for load balance.

Node Manager can manage its local VMs in the physical server, and it is responsible to create, move or delete a VM when receiving a command from Resource Manager. It also collects the resource usage of each VM and the physical server, and reports them to the Resource Manager.

When a user wants to set up a digital library, he sends a desired configuration of CPU, memory and disk space to the Resource Manager. The Resource Manager will ask some suitable Node Manager to allocate VMs according to the resource usage of each physical server. When the collection

938

of VMs is prepared, the Digital Library Instance would be deployed on it.

The Master of each DL Instance is monitoring each ServiceManager in the instance. When it finds ServiceManagers are overburdened such as CPU utilization is greater than 80%, and has to add more VMs to guarantee the QoS, it would resort to Resource Manager for more VMs. Then Resource Manager will ask one Node Manager with a light load to create a new VM according to the load of eachphysical sever, and notify the Master. The Master will add the new VM to its cluster and then deploy some bundles for high performance.

For digital library services, the requests are transited to DL Manager at first, and then the DL Manager would forward the requests to the Master of the right digital library according to user’s identifier which can be parsed from the requests. The Master would then forward the requests to the right service.

C. Multi-Tenancy Support Since multiple DL Instances could be managed by DL

Engine, multi-tenancy should be supported to isolate them among one another. Besides isolation, multi-tenancy can also improve the utilization of servers.

Multi-tenancy is regarded as one of the essential attributes of cloud computing, and there are many different models of multi-tenancy. For example, multi-tenancy for DBaaS can be implemented through shared hardware, shared OS, shared database and shard table [34]. Different models are suitable for different cloud paradigms.

In Digital Library Engine, service and data are managed quite different, and we used different models of multi-tenancy for them, as show in Figure 6.

OS

DL App

OS

DL App

OS

DL App

OS

DL App

OS

DL App

Hardware Hardware Hardware

OS

DL App

OS

DL App

D-OceanData Management

Service Management

Figure 6. Multi-Tenancy support for Digital Library Engine

In each digital library, service bundles are developed by users themselves, so every tenancy should has its own independent digital library instance. That is to say, for each DL Instance, a collection of VMs are created. Although VM-based isolation would reduce the resource utilization, we improved resource utilization in our Digital Library Engine through a global Resource Manager cooperating with several Node Managers.

D-Ocean can represent different types of data into a unified digital object through an object-oriented model, and it supports namespace for isolation. So only one D-Ocean instance is running for DL Engine to improve the resource

utilization, and tenancy information can be encoded into the namespace, which makes each user only see the data under his namespace.

D. QoS Management In DL Engine, message queues are used with the global

Resource Manager to guarantee the QoS by allocating enough resources to a running service.

When deploying a service in a DL Instance, the Mastercould ask several ServiceManagers to deploy the corresponding service bundle on them, and then create a message queue (ActiveMQ was used in our implementation) for request processing, which would be listened by the ServiceManagers. When a request comes, it could be pushed into the queue by the Master, and then the ServiceManagerscould obtain the request from the queue. This listening mechanism can ensure that all ServiceManagers would not idle unless there is no request in queue.

The Master also monitors the message queue, in addition to the status of each physical server. When the length of message queue is larger than a threshold, which means some requests could not be processed in time, so more service bundles should be deployed in the cluster. However, if all ServiceManagers have a heavy load, a new ServiceManagershould be added to the cluster to deploy more busy bundles. Therefore, the Master could allocate a new VM for the new ServiceManager. Conversely, if some ServiceManagers idle, the VMs could be reclaimed.

If a ServiceManager fails, the Master could be notified and then tries to restart the VM or asks the ResourceManager to add a new VM to guarantee the QoS.

V. EXPERIMENTS

We have used Digital Library Engine in our CADAL Digital Library, where Chinese Traditional Medicine service, Chinese Literature Chronicle service and Chinese Calligraphy service are implemented as DL Instances for high performance. That is to say, CADAL Digital Library is composed of several DL Instances.

In this section, we performed experiments on a cluster of commodity machines with 4 Intel Xeon CPU 3.10GHz, 8Gmemory and 2T SATA disks, and all machines are interconnected with 1Gbps Ethernet network. Each machine was installed with OpenStack.

The first experiment tests the performance of building a digital library on the Digital Library Engine with different configurations. As discussed above, when a user wants to build a digital library, the Digital Library Engine should create a cluster of VMs at first and then start the digital library, which includes OSGi framework startup, bundle loading, bundle installation, bundle resolution and bundle start. So the duration of creating and running a digital library consists the time of creating a cluster of VMs and the time of starting a digital library.

Firstly, we computed the startup time of a VM with some service bundles. We selected some bundles from CADAL for the experiment, such as cache bundle, D-Ocean bundle, StoreAccess bundle and BookService bundle, where cache bundle is implemented for data cache, D-Ocean bundle

939

encapsulates the D-Ocean client for data modeling, storage, analysis, indexing and search, StoreAccess bundle depends on D-Ocean bundle and cache bundle for CRUD (create, read, update and delete) operations in high performance, while BookService bundle depends on StoreAccess bundle. If a bundle wants to be started, all its dependents should be started before.

Figure 7 shows the startup time of a VM with different bundles. From the figure, we can see that a pure VM would start in 13.32 seconds, and then a VM with a cache bundle or a D-Ocean bundle would start in 15.802 seconds and 15.936 seconds respectively. The startup time of a VM with a StoreAccess bundle or a BookService bundle would be much longer. This is mainly because of the bundle dependency. More dependency could lead to more startup time.

0 5 10 15 20 25 30

pure VM

VM + Cache

VM + D-Ocean

VM + StoreAccess

VM + BookService

startup time(s)

Figure 7. The startup time of a VM with service bundles

Figure 8 shows the time of expanding an existing DL instance by integrating a new cluster of VMs with the basic service bundles for more capacity.

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8

Star

t tim

e(s)

The Number of VMs

VM DL Total

Figure 8. Time of expanding a DL Instance with different number of VMs

From the figure, we can see that although the total time of expanding a digital library increases greatly as the number of VMs increases, the starting of the services only consumesa little time on a large cluster of VMs, since OSGi framework startup, bundle loading, bundle installation, bundle resolution and bundle start can be executed in parallel for each VM. If VMs are created in a pool in advance, digital libraries could be started and expanded more quickly.

Figure 9 shows the number of VMs changes with the increment of requests because of resource management. In the experiment, we initialized the digital library with only 1

VM. We increased 1000 more requests per 100 seconds until to 4000 requests per 100 seconds, and then decreased the number of requests. From the figure, we found that the number of VMs increases with the increment of requests and then decreases when the request reduces. This indicates the resource management of the Digital Library Engine can work well.

0

0.5

1

1.5

2

2.5

3

3.5

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 100 200 300 400 500 600 700 800 900

Num

ber o

f VM

s

Num

ber o

f Req

uest

s

Time (seconds)

Requests

VMs

Figure 9. The number of VMs changes with the requests

Moreover, we also measured the average response time for each request, and the result is shown in Figure 10.

0

0.005

0.01

0.015

0.02

0.025

500 1000 1500 2000 2500 3000 3500 4000 4500

Ave

rage

resp

onse

tim

e (s)

The Number of Requests

Add a VM

Add a VM

#VM = 1

Figure 10. Response time with different number of requests and VMs

At the beginning, only 1 VM was needed for the digital library when the concurrent requests were below 500 requests per 100 seconds. With the increment of the requests, more VMs were needed to ensure the response time was below 0.02 second.

Figure 11 shows the changes of throughout when a service fails in a digital library with 3 ServiceManagers.

0

500

1000

1500

2000

2500

3000

3500

4000

4500

180 240 300 360 420 480 540

Thro

ughp

ut(tp

m)

Elapsed Time(s)

restart servicerestart vmrecreate vm

432.838

430

438.496

448.25

Figure 11. Throughput when a service bundle fails

940

In a digital library, a service bundle is always deployed in several ServiceManagers for high performance. When a service fails, the Master would be notified, and then to restart the service for QoS guarantee. Three cases could be happened, when a service fails, which includes the service bundle crashes, VM breakdown and VM crashes. In each case, the throughput would decrease at first, and then recover quickly. In Figure 11, a service failed at 420s, and then the Master detected the failure at 430s for recovery. The fastest recovery is restarting the service, which needs only 2.838 seconds, and restarting a VM needs 8.496 seconds. The slowest is recreating a VM and then restarting the VM, which needs 18.25 seconds.

VI. CONCLUSION

In this paper, we proposed a new Platform as a Service for fast developing and deploying digital libraries, called Digital Library Engine. The experiments show that our platform can create and deploy a digital library more convenient, and the performance is quite well.

In future, we would like to deploy the Digital Library Engine in a public cloud, such as EC2, and we would also like to implement more efficient approaches for QoS management, resource management and multi-tenancy support, especially to utilize the capabilities provided by the virtualization infrastructure.

ACKNOWLEDGMENT

This work is supported by CADAL Project, HEGAOJI Project under grant No.2010ZX01042-002-003-001, and the National Natural Science Foundation of China No.61103099.

REFERENCES

[1] Deng Yi; Yin Zhang; Haihan Yu; Yanfei Yin; Jing Pan; Baogang Wei. Improving Multi-Faceted Book Search by Incorporating Sparse Latent Semantic Analysis of Click-Through Logs. JCDL 2012.

[2] Yin Zhang, Xiaojun Wang, Haihan Yu, Ruifeng Li, Baogang Wei, Jing Pan: When personalization meets socialization: an iCADAL approach. JCDL 2011.

[3] Chenxing Yang, Baogang Wei, Jiangqin Wu, Yin Zhang, Liang Zhang: CARES: a ranking oriented CADAL recommender system. JCDL 2009.

[4] Zhenchao Ye, Ling Zhuang, Jiangqin Wu, Chenyang Du, Baogang Wei, Yin Zhang: In depth utilization of Chinese ancient maps: a hybrid approach to digitizing map resources in CADAL. JCDL 2010.

[5] Wenhao Zhu, Baogang Wei, Jiangqin Wu, Shaomin Shi, Yan Yang: Knowledge extraction and integration for semi-structural information in digital libraries. JCDL 2009.

[6] Pengcheng Gao, Jiangqin Wu, Yang Xia, and Yuan Lin. CADAL digital calligraphy system. JCDL 2012.

[7] Salesforce Customer Relationships Management (CRM) System. http://www. salesforce.com/

[8] Google AppEngine. code.google.com/appengine. [9] Windows Azure Platform. http://www.microsoft.com/windowsazure/. [10] Amazon Simple Storage Service, http://aws.amazon.com/s3/. [11] Amazon Elastic Compute Cloud, http://aws.amazon.com/ec2/.

[12] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youse, and D. Zagorodnov. The Eucalyptus Open-Source Cloud-Computing System. CCGRID 2009.

[13] OpenStack: The open source, open standards cloud. http://openstack.org.

[14] IBM SmartCloud, http://www.ibm.com/cloud-computing/us/en/ [15] Oracle Cloud Platform,

http://www.oracle.com/us/solutions/cloud/platform/index.html [16] Kächele, Steffen, Jörg Domaschka, and Franz J. Hauck. COSCA: an

easy-to-use component-based PaaS cloud system for common applications. In Proceedings of the First International Workshop on Cloud Computing Platforms, 2011.

[17] Yang, Lei, Jiannong Cao, Shaojie Tang, Tao Li, and Alvin TS Chan. A Framework for Partitioning and Execcution of Data Stream Applications in Mobile Cloud Computing, IEEE Cloud 2012

[18] Satzger, Benjamin, Waldemar Hummer, Philipp Leitner, and Schahram Dustdar. Esc: Towards an elastic stream computing platform for the cloud. IEEE Cloud 2011.

[19] C. Curino, E. P. C. Jones, R. A. Popa, N. Malviya, E. Wu, S. Madden, H. Balakrishnan, and N. Zeldovich. Relational cloud: A database-as-a-service for the cloud. In Proceedings of the 5th Biennial Conference on Innovative Data Systems Research, 2011.

[20] Yu Tao, Jie Qiu, Berthold Reinwald, Lei Zhi, Qirong Wang, and Ning Wang. Intelligent Database Placement in Cloud Environment. ICWS 2012.

[21] Bernstein, Philip A., Istvan Cseri, Nishant Dani, Nigel Ellis, Ajay Kalhan, Gopal Kakivaya, David B. Lomet, Ramesh Manne, Lev Novik, and Tomas Talius. Adapting microsoft SQL server for cloud computing. ICDE 2011.

[22] Teregowda, Pradeep B., Bhuvan Urgaonkar, and C. Lee Giles. CiteSeerx: a Cloud Perspective. Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, 2010.

[23] Teregowda, Pradeep, Bhuvan Urgaonkar, and C. Lee Giles. Cloud computing: A digital libraries perspective. IEEE Cloud 2010.

[24] SeerSuite Source, http://sourceforge.net/projects/citeseerx. [25] Nguyen, Quyen L., and Alla Lake. Content Server System

Architecture for Providing Differentiated Levels of Service in a Digital Preservation Cloud. IEEE Cloud 2011

[26] Donatella Castelli, Pasquale Pagano, OpenDLib: A Digital Library Service System, ECDL 2002:292-308.

[27] D-Net. http://www.d-net.research-infrastructures.eu/. [28] Zheng, Liangju, Weiming Lu, Meng Zhang, Baofeng Zhang, and

Baogang Wei. A Scalable, Adaptive, Self-management and Fault tolerant Architecture for Digital Library, ICADL 2012

[29] OSGi Alliance. OSGi Service Platform: Core Specification, Release 4, Version 4.1. Technical report, 2007.

[30] The Apache Hadoop Project. http://hadoop.apache.org. [31] Hunt, P.; Konar, M.; Junqueira, F.P.; Reed, B.: ZooKeeper: Wait-free

coordination for Internet scale systems. USENIX Annual Technology Conference, 2010.

[32] Minhas, U. F., Rajagopalan, S., Cully, B., Aboulnaga, A., Salem, K., & Warfield, A. Remusdb: Transparent high availability for database systems. In Proc. of VLDB. 2011.

[33] Apache Hadoop NextGen MapReduce (YARN).http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html

[34] Agrawal, Divyakant, Sudipto Das, and Amr El Abbadi. Big Data and Cloud Computing: Current State and Future Opportunities, EDBT 2011

941