Multimed Tools ApplDOI 10.1007/s11042-015-3121-4

vConnect: perceive and interact with real worldfrom CAVE

Yifeng He1 · Ziyang Zhang1 · Xiaoming Nan1 ·Ning Zhang1 · Fei Guo1 · Edward Rosales1 ·Ling Guan1

Received: 26 March 2015 / Revised: 27 September 2015 / Accepted: 26 November 2015© Springer Science+Business Media New York 2015

Abstract The Cave Automatic Virtual Environment (CAVE) is a fully immersive VirtualReality (VR) system. CAVE systems have been widely used in many applications, such asarchitectural and industrial design, medical training and surgery plan, museums and edu-cation. However, one limitation for most of the current CAVE systems is that they areseparated from the real world. The user in the CAVE is not able to sense the real worldaround him or her. In this paper, we propose a vConnect architecture, which aims to estab-lish real-time bidirectional information exchange between the virtual world and the realworld by utilizing the advanced technologies in cloud computing, mobile communications,wireless sensor networks, and computer vision. Specifically, we address three technicalchallenges in the proposed vConnect architecture. First, we propose an optimal allocationscheme for the wireless sensor networks to ensure that the data streams captured by the sen-sors can be delivered to the cloud servers in a reliable and prompt way. Second, we optimizethe allocation of the cloud resources to ensure that the data streams sent from the clients canbe processed promptly by the cloud servers at a minimal resource cost. Third, we proposeto use marker-based finger interactions such that the user in the CAVE can manipulate theinformation in a natural and intuitive way. Fourth, we implemented a vHealth prototype, aCAVE-based real-time health monitoring system, to validate the proposed vConnect archi-tecture. We demonstrated in the vHealth prototype that the user in the CAVE can visualizeand manipulate the real-time physiological data of the patient who is being monitored, andinteract with the patient.

Keywords Cave Automatic Virtual Environment (CAVE) · Virtual reality ·Human-Computer Interaction (HCI) · Cloud computing · Wireless sensor networks ·Optimal resource allocation · Quality of Service (QoS)

� Yifeng Heyhe@ee.ryerson.ca

1 Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada

Multimed Tools Appl

1 Introduction

Virtual Reality (VR) refers to computer-generated Three-Dimensional (3D) environmentsthat allow the user to perceive the virtual objects and interact with them. The Cave Auto-matic Virtual Environment (CAVE) [7] is a fully immersive VR system. The framework ofCAVE consists of three main systems: 1) the projection system, 2) the tracking system, and3) the computer system. In the projection system, multiple stereoscopic projections are usedto project a 3D environment into a room-sized cube [7], which consists of three walls anda floor. The tracking system is used to track the real-time position and orientation of theeyes of the user inside the CAVE. The computer system accepts the inputs of the real-timetracking data from the tracking system, and then controls the outputs of the images from theprojectors. The user can perceive the true 3D environment through a pair of stereoscopicshutter glasses, which alternately block the left or right eye such that each eye only seesthe corresponding images. The user inside the CAVE can see objects apparently floatingin the air, and can walk around them, getting a proper view of what they would look likein reality. Such fully immersive experience provided by the CAVE is not achievable withdesktop-based computers.

CAVE systems have been widely used in many applications, for example, 3D simulationand training, architectural and industrial design, medical training and surgery plan, muse-ums and education, oil and gas exploration, scientific research, and topographical analysis.However, one limitation for the current CAVE systems is that they are typically separatedfrom the real world. First, the CAVE is a virtual environment, and everything shown in theCAVE is not real. The user in the CAVE cannot sense the real world around him or her. Forexample, an Avatar can be used to represent a person in the real world, and be shown inthe CAVE. However, the Avatar is just a virtual object, and it does not have the real-timecharacteristics, such as body temperature, heart rate, that the person in the real world has.Therefore, the user in the CAVE cannot access the information of the real person. Second,the user in the CAVE cannot interact with the real world. Any instructions generated in theCAVE cannot directly be applied to the real world.

In this paper, we propose a vConnect architecture, which aims to establish real-timebidirectional information exchange between the virtual world and the real world byutilizing the advanced technologies in cloud computing, mobile communications, wire-less sensor networks, and computer vision. The motivation of the proposed vConnectarchitecture is to overcome the limitation for the current CAVE systems in separationbetween the virtual world and the real world. The proposed vConnect architecture, whichinterconnects the virtual world and the real world, can be applied in a wide range ofapplications, such as remote health monitoring, real-time traffic monitoring, and smartcity.

The proposed vConnect architecture is shown in Fig. 1. It consists of two major com-ponents: 1) cloud server and 2) clients. Cloud server is the coordinator which aggregatesthe information from various sources, processes the information, and then transmit it to theclients. Clients can be classified into bidirectional clients and and unidirectional clients. Thebidirectional clients, such as remote desktop users and mobile users, can receive and sendinformation. The unidirectional clients can only receive information, such as actuators, orsend information, such as sensors. Clients are heterogenous in terms of power, processingand display capacity. For example, the remote desktop user has a much higher processingcapacity than a sensor. The CAVE is a special bidirectional client, which represents thevirtual world. The other clients can capture, send, or receive the information of the realworld.

Multimed Tools Appl

Real World

Virtual object/scene

CAVE

Finger interactions Wireless sensor network

Base station

Actuator

Remote mobile user

Cloud server

Virtual World

Remote desktop user

Sensor

Fig. 1 The proposed vConnect architecture with bidirectional information exchange between the virtualworld and the real world

In the proposed vConnect architecture, the CAVE receives the real-time information,including the information captured by the sensors and the information provided by theremote users, from the real world. The received real-world information is then visual-ized in the virtual world, such that the user in the CAVE can perceive it. The user inthe CAVE can also manipulate the information, and then make corresponding actions,which can be transmitted back to the real world for executions. For example, the user inCAVE can perform operations on the remote actuators, or place a video call with a remoteuser.

The problem to be tackled in this paper is how to achieve a high quality of user experiencefor the user in the CAVE. The quality of user experience is mainly dependent on two fac-tors, the quality of the received data streams and the performance of the Human-ComputerInteraction (HCI). There is correlation between the quality of the received data streams andHCI performance in the CAVE. If the data captured from the real world is lost or delayedduring network transmissions, or the data processing at the cloud is delayed, the informa-tion from the real world will not be presented promptly and correctly in the CAVE, and theuser in the CAVE will not be able to manipulate the information correctly, thus degradingthe HCI performance.

In order to achieve a high-quality experience for the proposed vConnect architecture, weneed to address a number of technical challenges, as listed below:

1) QoS guarantee for real-time delivery of data streams in the wireless networks. Thedata streams generated at the clients, such as sensors, need to be delivered to the cloudpromptly. The information that arrives after its visualization deadline will be regardedas useless. However, it is quite challenging to ensure a reliable and prompt deliv-ery of data streams in wireless networks, especially when the channel is impaired byinterference or fading.

Multimed Tools Appl

2) QoS guarantee for real-time processing of data stream at the cloud. The cloud serversreceive the data streams from the clients, process the data, and then store the resultsinto the database for future analysis, or immediately send the results back to the CAVEor other clients for instant visualization. The data streams have QoS requirementson the service response time at the cloud. However, it is challenging to optimallyallocate the cloud resources to satisfy the QoS requirements at a minimal resourcecost.

3) Natural HCI for information manipulation in the CAVE. The information captured fromthe real world is overlaid on the virtual objects in the CAVE. Information manipula-tion in the CAVE includes the information retrieval from the database, the informationvisualization and interactive information processing in the CAVE, and the informationfeedback to the real world. The user in the CAVE needs to manipulate the information ina natural way. However, the conventional interaction tool for CAVE, the wand, requiresa learning process, such as memorizing button functions, which is neither natural norintuitive.

The major contribution of this paper is that we propose a vConnect architecture to enablethe user in the CAVE to perceive the real world and interact with it. Specifically, theproposed vConnect architecture has the following three technical novelties:

1. We optimize the resource allocation for the wireless sensor networks such that the datastreams captured by the sensors can be delivered to the cloud servers in a reliable andprompt way.

2. We optimize the allocation of the cloud resources such that the data streams fromthe clients can be processed promptly by the cloud servers at a minimal resourcecost.

3. We propose to use finger interactions such that the user in the CAVE can manipulate theinformation in a natural and intuitive way. Finger interactions are achieved by trackingthe markers worn on the hands. We define the specifications of five basic actions, whichare: 1) activating the main menu, 2) selecting a menu item, 3) moving an object, 4)rotating an object, and 5) scaling an object.

4. We implemented a health monitoring prototype, vHealth, which is an instance of theproposed vConnect framework. We demonstrated in the vHealth prototype that the userin the CAVE can visualize and manipulate the real-time physiological data of the patientwho is being monitored, and interact with the patient.

To the best of our knowledge, nobody has proposed the similar architecture which canbridge the virtual world and the real world, and nobody has considered the QoS guaranteefor wireless data transmission and cloud computing, and finger interactions in the similararchitecture.

The remainder of this paper is organized as follows: Section 2 discusses the related work.In Section 3, we address QoS problem for the real-time delivery of the data streams in thewireless sensor networks. In Section 4, we solve the resource cost minimization problemat the cloud to enable prompt processing of the data streams at a minimal resource cost. InSection 5, we describe information manipulation in the CAVE with finger interactions. InSection 6, we present the implementation of the vHealth prototype. Finally, the conclusionsare drawn in Section 7.

Multimed Tools Appl

2 Related work

2.1 QoS for wireless sensor networks

QoS is important for real-time data delivery over wireless sensor networks. Cheng et al.formulated the Geographic Opportunistic Routing (GOR) problem as a multi-objectivemulti-constraint optimization problem, and proposed an efficient QoS-aware GOR proto-col for QoS provisioning in Wireless Sensor Networks (WSNs) [6]. Shah et al. proposeda novel cross-layer framework for QoS support in Wireless Multimedia Sensor Net-works (WMSNs). The objective of the proposed framework is to maximize the capacityof the deployed network to enhance the number of video sources given that the QoSconstraint of each individual source is also preserved [26]. Zhang et al. proposed anenergy-balanced routing method based on forward-aware factor, in which the next-hopnode is selected according to the awareness of link weight and forward energy density[37].

2.2 QoS for data processing at the cloud

Response time is typically used to measure the QoS performance for real-time data pro-cessing at the cloud. The response time can be minimized by optimally scheduling theworkload to different Virtual Machines (VMs) [18]. In addition, a job can be decomposedinto multiple sub-tasks. The overall execution time for the job can be minimized by opti-mally assigning each sub-task to different VM. Nan et al. studied the task-level schedulingproblem for cloud based multimedia applications [17]. In multi-site cloud, it is challeng-ing to balance the workload among different data centers. Nan et al. minimized the averageround-trip-time in multi-site cloud by jointly optimizing the workload assignment and thelocal VM reconfiguration [16].

2.3 Human computer interactions in CAVEs

In terms of what is the most natural interaction with the CAVE, various tools have beenproposed and tested. Traditionally, a so called Flystick™, a wand type remote control is usedwith various buttons. Individual button and a combo of buttons corresponding to certainactions such as menu selection, directional up/down, etc. Abramyan et al. used two types ofwands (Nintendo Wii controller and Nunchuk joystick) in angle viewing and manipulationcontrol [1]. Wand was also used in a virtual table tennis game as the hand tracker to mimicthe racket in Li et al.’s work [12]. Koike and Makino proposed a 3D solid modeling systemusing wand to draw sketches on the screen, and 3D model was then converted from thebasic sketch [11]. A pinch glove, worn on the user’s real hand, was used to manipulate thevirtual object in the CAVE [30]. A CAVE-based virtual design environment using fingerinteractions was studied in our previous work [19].

Various approaches were proposed in transferring command from 2-D touch screens tothe 3-D CAVE™. Kim et al. clicked button and drew arrows on the iPhone/iPod for touchscreen to command the CAVE in menu selection and navigation (walking and view-anglerotation in virtual space) [10]. In data mining, Prachyabrued et al. developed an interfacebased on iPod touch technology, in order to achieve complicated and cluttered 3-D data

Multimed Tools Appl

visualization and manipulation in the CAVE environment [22]. In this interface, occludeddata in 3-D is simplified to the 2-D overview and presented in iPod, while the selectionand navigation commands executed at the iPod are transferred back to the CAVE system.Advanced 3-D data manipulation is also introduced using 2-D device to control 3-D volu-metric data in exploration and annotation. Song et al. proposed a set of interaction commandbased on iPod touch, including volume data slicing, drawing, and annotation [28].

Although 2-D based interaction utilized recently developed touch screen technology andprovided an easy access interface, it still hasn’t reached to the full capacity that immersivevirtual reality provides. Intuitive and natural manipulation in the CAVE system with virtualobjects should be as similar as in the real world with tangible objects. This means thatusers are able to command the CAVE via their gesture and finger interactions. Initial workshave been proposed in tracking hand and limbs in commanding the CAVE. Kapri et al.used marker-based hands and head tracking method for steering-by-pointing in directionalcontrol [32]. Virtual ball juggling using hand motion tracking technique was also reportedin literature [15].

2.4 Collaborations with CAVEs

CAVE is also used in collaborative and distributed environments connected by networks.Schaeffer et al. proposed a network interaction in simulating multi-participant physicalactivities in dancing [25]. In this application, two dancers from two locations workedtogether in a way that the virtual Avatar of one dancer was created by motion capture. Themotion was transmitted from one stage to the other one to form a collaborating dance. Asimilar example was also created, such that a dancer remotely controls the Avatar with realdance movement to interact with the audience, with a video camera setup to assist her/him inobserving the audience response accordingly [35]. Collaborative work on volumetric med-ical data is also achieved using CAVE system [3]. In this work, a local CAVE user and aremote user, whose Avatar appears in CAVE, were able to collaboratively work on a volu-metric model. Roberts et al. also proposed a virtual collaboration that the joint effort task isvirtually displayed in the CAVE, while three remotely located partners have to cooperate inthis collaborative virtual environment [24].

3 QoS guarantee for real-time delivery of data streams in wireless sensornetworks

As illustrated in Fig. 1, the sensors are unidirectional clients, which capture the informationand send it to the cloud. The coverage area of a base station is defined as a cell area.The sensors in the same cell area send the data to the common base station via one-hopcommunications. In this section, we optimize the resource allocation for the sensors in acell area to provide QoS guarantee, in terms of Packet Loss Rate (PLR) and delay, to thedelivery of data streams.

3.1 System models

3.1.1 Queueing model

Since the arrivals of data traffic are dynamic, we study the QoS problem in a discrete-timemanner. The set of the sensors in the cell area is denoted by a set M. Each sensor senses

Multimed Tools Appl

the environment and generates the data packets. We assume that the arrivals of the packetsat sensor m, where ∀m ∈ M, follow a Poisson process with a rate λ

(t)m during time slot t .

We model each sensor as an M/M/1 queueing system [9]. The transmission rate at sensor

m at time slot t is denoted by R(t)m . The service rate u

(t)m for the packets at sensor m at time

slot t is given by u(t)m = R

(t)m /Lm where Lm is the average packet length at sensor m. In

order to maintain a stable M/M/1 queueing system, it is required that λ(t)m ≤ u

(t)m ,∀m ∈ M

[9].In the queueing theory, the queueing delay of a packet is defined as the duration from

the time when the packet arrives in the queue to the time when the packet is sent out of thequeue. In the M/M/1 queueing model, the average queueing delay of the packets at sensor

m at time slot t is given by T(t)m = 1/u

(t)m

1−λ(t)m /u

(t)m

, ∀m ∈ M [9].

In the M/M/1 queueing model, the tail probability is defined as the probability thatthe number of packets in the system is larger than a threshold ρm [9]. Let the thresh-old ρm represent the length of the queue at sensor m. The tail probability represents thepacket drop probability due to queue overflow. The tail probability for the packets at sen-

sor m at time slot t is given by P cm = Pr(Nm > ρm) = (λ

(t)m /u

(t)m )(ρm+1),∀m ∈ M

[9], where Nm is a random variable representing the number of the packets at sensorm.

3.1.2 CDMA model

We assume that Code-Division Multiple Access (CDMA) is used to address the simul-taneous access issue for the sensors in the same cell area. In the CDMA model, thespread-spectrum bandwidth is denoted by W , the power spectrum density of the AdditiveWhite Gaussian Noise (AWGN) is denoted by N0. The channel gain from sensor m tothe base station at time slot t is denoted by h

(t)m . The received Bit-Energy-to-Interference-

Density Ratio (BEIDR) at the base station from sensor m at time slot t is denoted by

y(t)m , which is given by y

(t)m =

(W

R(t)m

)(h

(t)m P

(t)m

δ�j∈M,j �=mh(t)j P

(t)j +N0W

), ∀m ∈ M, where δ is the

orthogonality factor representing Multiple Access Interference (MAI) from the imperfectorthogonal spreading codes [27].

We assume Binary Phase Shift Keying (BPSK) modulation is used in the CDMA system.The Bit Error Rate (BER) of the data transmitted from sensor m to the base station is given

by em = Q

(√2y

(t)m

), where Q(x) is a Q-function [36]. If a packet is received in error,

it will be dropped at the base station. We assume the bit errors occur independently in apacket. Therefore, the PLR due to transmission errors of the packets from sensor m is then

given by P em = 1 − (1 − em)Lm = 1 −

(1 − Q

(√2y

(t)m

))Lm

, where Lm is the average

packet length in bits at sensor m.

3.2 Resource optimization problem

We optimize the resource allocation in the cell area to provide QoS guarantee to the deliveryof the data streams. The resource optimization problem can be stated as: to minimize thesum of the transmission powers of all the sensors by optimizing the transmission powerand the transmission rate at each sensor, subject to the power constraints, the requirements

Multimed Tools Appl

of the congestion PLR, the transmission BER, and the delays of the packets, respectively.Mathematically, the problem is formulated as follows.

minimize{R

(t)m ,P

(t)m

}∑

m∈MP (t)

m

subject to Q

(√2y

(t)m

)≤ eth,∀m ∈ M,

y(t)m =

(W

R(t)m

)(h

(t)m P

(t)m

δ�j∈M,j �=mh(t)j P

(t)j + N0W

),∀m ∈ M,

u(t)m = R(t)

m /Lm,∀m ∈ M,

1/u(t)m

1 − λ(t)m /u

(t)m

≤ Tth, ∀m ∈ M,

(λ(t)

m /u(t)m

)(ρm+1) ≤ Pth,∀m ∈ M,

λ(t)m ≤ u(t)

m ,∀m ∈ M,

0 ≤ P (t)m ≤ Pmax,∀m ∈ M,

R(t)m > 0, ∀m ∈ M, (1)

where P(t)m is the transmission power at sensor m at time slot t , R

(t)m is the transmission rate

at sensor m at time slot t , eth is the threshold of BER, u(t)m is the service rate for the packets

at sensor m at time slot t , Tth is the threshold of the queueing delay, Pth is the threshold ofcongestion PLR, and Pmax is the maximum transmission power at the sensor.

Q function is a monotonically decreasing function. Therefore, we can convert the con-

straints, Q

(√2y

(t)m

)≤ eth and y

(t)m =

(W

R(t)m

)(h

(t)m P

(t)m

δ�j∈M,j �=mh(t)j P

(t)j +N0W

)in the optimization

problem (1), to the following equivalent form:(W

R(t)m

)(h

(t)m P

(t)m

δ�j∈M,j �=mh(t)j P

(t)j + N0W

)≥ (Q−1(eth))

2

2, (2)

where Q−1(x) is the inverse Q-function [36]. Let γth = (Q−1(eth))2

2 , representing the thresh-

old of the received BEIDR. Equation (2) is equivalent to δγthR(t)m

h(t)m P

(t)m W

�j∈M,j �=mh(t)j P

(t)j +

γthN0

h(t)m

(R

(t)m

P(t)m

)≤ 1.

Therefore, the optimization problem (1) is then converted to the following equivalentform:

minimize{R

(t)m ,P

(t)m

}∑

m∈MP (t)

m

subject toδγthR

(t)m

h(t)m P

(t)m W

�j∈M,j �=mh(t)j P

(t)j + γthN0

h(t)m

(R

(t)m

P(t)m

)≤ 1,

∀m ∈ M,

0 ≤ P (t)m ≤ Pmax,∀m ∈ M,

R(t)m ≥ RLB

m ,∀m ∈ M, (3)

Multimed Tools Appl

where RLBm = max

{Lm+Tthλ

(t)m Lm

Tth,

λ(t)m Lm

P1/(ρm+1)th

, λ(t)m Lm

}.

The optimization problem (3) is a geometric programming that can be transformed intoa convex optimization problem, which can then be solved efficiently using the primal-dualinterior-point method [5].

3.3 Simulation results on QoS guarantee in wireless sensor networks

In the simulations, the number of the sensors in the cell area is set to 10. The distancefrom the sensor to the base station is uniformly distributed between 10 and 100 m. Thechannel gain from sensor m to the base station is given by hm = 100/d4

m, where dm is thedistance from sensor m to the base station. The mean arrival rate of the packets at a sensoris uniformly distributed between 50 and 80 packets/second. In the CDMA model, we setW = 400KHz, δ = 0.1, and N0 = 10−13W/Hz. In the setting of QoS thresholds, we setthe threshold of transmission BER to 10−5, the threshold of the congestion PLR to 0.01,and the threshold of the queueing delay to 0.02 s.

In Fig. 2, we compare the PLRs and the queueing delays, respectively, between the twoschemes: 1) the proposed allocation scheme, which is the optimal solution to the optimiza-tion problem (3), and 2) the equal allocation scheme, in which the transmission powers andthe transmission rates are equally allocated among the sensors. In order for a fair compari-son, the sum of the transmission powers and the sum of the transmission rates of all sensorsare equal in the two schemes. In the proposed scheme, the PLRs at all sensors are less than0.01, and the queueing delays at all sensors are no greater than 0.02 s. The equal allocationscheme does not allocate the resources appropriately, thus leading to the large PLRs at thefar-way sensors.

4 QoS guarantee for real-time data processing at the cloud

After receiving data from clients, cloud servers will process clients’ data and store resultsinto the database server for future analysis. The medical data processing in cloud has strict

1 2 3 4 5 6 7 8 9 100

0.2

0.4

0.6

0.8

1

Pac

ket l

oss

rate

Sensor no.

Equal

Proposed

(a)

1 2 3 4 5 6 7 8 9 100

0.005

0.01

0.015

0.02

0.025

0.03

Que

uein

g de

lay

[s]

Sensor no.

Equal

Proposed

(b)

Fig. 2 Comparison between the equal allocation scheme and the proposed allocation scheme: a packet lossrate, and b queueing delay

Multimed Tools Appl

requirements on service response time, which is defined as the duration from the time whenthe data arrives at the data center to the time when the result is completely stored intodatabase. In this section, we optimize the cloud resources to minimize the resource costwhile guaranteeing the service response time requirements for data processing.

4.1 System models

Most of clouds are built in the form of data centers. In our study, the data center con-sists of a master server, a number of computing servers, and a database server. All theseservers are VM instances generated from physical computation resources. The master serverserves as a scheduler, receiving all data streams and then distributing them to the computingservers. The computing servers process the data with the allocated computation resources.The results out of the computing servers will be stored in the database server. Due to thetime-varying workload, the resources in cloud have to be dynamically adjusted. Therefore,we divide the time domain into time slots. The cloud resources will be dynamically allocatedin every time slot t .

The allocated cloud resources include the resources at the master server, computingservers, and the database server. At the master server, the resource is represented by thescheduling rate S in terms of the number of data packages scheduled per second. Supposethat there are N computing servers. The allocated resource at a computing server is repre-sented by the processing rate C in terms of the number of instructions executed per second.For the database server, the Input/Output (I/O) rate is a common performance measurement.Thus, the allocated resource at the database server is represented by the I/O rate B in termsof the I/O operations per second.

We propose a queueing model to study the optimal resource allocation at the cloud. Theproposed queueing model for cloud data center is shown in Fig. 3. The model consistsof three concatenated queueing systems, which are the scheduling queue, the computationqueue, and the database queue. The master server maintains the scheduling queue. Since

Fig. 3 The proposed queueing model for cloud data center

Multimed Tools Appl

the packages are sent from different clients, the arrivals of the packages can be modeled asa Poisson Process [9]. The average arrival rate at time slot t is denoted by λ(t). The datapackages are scheduled to the computing servers at the rate S(t). Each computing serverhas a corresponding computation queue. After processing, the results are inserted into thedatabase at the rate B(t) by the database server. The service availability is guaranteed by theService Level Agreement (SLA) and no package is dropped during the process. Therefore,the number of results inserted into database is equal to the number of received packages.

The total resource cost �(t)tot at time slot t can be modeled as

�(t)tot =

(αS(t) + β

N∑i=1

C(t)i + γB(t)

), (4)

where is the time slot length, S(t) is the allocated scheduling resource, C(t)i is the allocated

computation resource at the computing server i, B(t) is the allocated database resource, α,β, and γ are the cost rates for scheduling, computation, and database access, respectively.The linear cost model in (4) has been justified by the numerical analysis in [14].

4.2 Resource cost minimization problem

When data streams arrive at cloud data center, all packages enter into the scheduling queuefirst. The scheduling queue can be modeled as an M/M/1 queueing system with a meanservice rate S(t) at the master server. In order to maintain a stable queue, λ(t) < S(t) is

required. The response time of the scheduling queue is given by T(t)sch = 1/S(t)

1−λ(t)/S(t) [9].Since every computing server has the same service procedure, a weighted scheduling

scheme is employed. A package is assigned to computing server i with a probability pi ,and thus

∑Ni=1 pi = 1. According to the decomposition property of Poisson distribution,

the average arrival rate at computing server i is piλ(t). Suppose that the average task size

is denoted by F in terms of the number of instructions. The execution time can be approxi-mated as the exponential distribution with an average of F/C

(t)i [4]. Therefore, the service

at computing server i is modeled as an M/M/1 queueing system. To maintain the queue sta-bility, the constraint piλ

(t) < C(t)i /F should be satisfied. The response time at computing

server i is given by T(i)(t)com = F/C

(t)i

1−piλ(t)F/C

(t)i

. Thus, the mean response time in computation

is formulated by T(t)com = ∑N

i=1 piT(i)(t)com = ∑N

i=1piF/C

(t)i

1−piλ(t)F/C

(t)i

.

Since no package is dropped during the service, the arrivals of the results at the databasequeue is equal to λ(t). The I/O operation time for the database server is a random variablewhich can be modeled as the exponential distribution with an average of D/B(t), whereD is the number of I/O operations. To maintain a stable database queue, λ(t) < B(t)/D is

required. The response time in database queue is given by T(t)db = D/B(t)

1−λ(t)D/B(t) .The total service response time in the cloud data center is then given by

T(t)tot = T

(t)sch + T (t)

com + T(t)db

= 1/S(t)

1 − λ(t)/S(t)+

N∑i=1

piF/C(t)i

1 − piλ(t)F/C(t)i

+ D/B(t)

1 − λ(t)D/B(t). (5)

We optimize the resources at the cloud to provide satisfactory services at the minimalresource cost. The resource cost minimization problem can be stated as: to minimize the

Multimed Tools Appl

total resource cost at the cloud by jointly optimizing the allocated scheduling resource,the computation resource, and the database resource, subject to the queueing stability con-straint in each queueing system and the service response time constraint. Mathematically,the problem can be formulated as follows:

minimize{S(t),C

(t)1 ,··· ,C(t)

N ,B(t)} �

(t)tot

subject to λ(t) < S(t),

piλ(t) < C

(t)i /F, ∀i = 1, . . . , N,

λ(t) < B(t)/D,

T(t)tot ≤ τ, (6)

where �(t)tot is given by (4), T

(t)tot is given by (5), and τ is the upper bound of the service

response time.We employ the Lagrange multiplier method [5] to solve the optimization problem (6),

and get the optimal analytical solution as follows:

S(t)∗ =√

α + √βF

∑Ni=1

√pi + √

γD√ατ

+ λ(t),

C(t)∗i =

(√α + √

βF∑N

i=1√

pi + √γD

)√piF√

βτ+ piλ

(t)F,

∀i = 1, . . . , N,

B(t)∗ =(√

α + √βF

∑Ni=1

√pi + √

γD)√

D

√γ τ

+ λ(t)D. (7)

4.3 Simulation results on QoS guarantee at the cloud

In the simulation, we set parameters based on Windows Azure [14], which provides on-demand computation, storage, and networking resources as utilities through Microsoft datacenters. We compare the performance between the proposed optimal allocation scheme, inwhich cloud resources are allocated optimally by solving the resource cost minimizationproblem (6), and the equal allocation scheme, in which the resource cost for scheduling,computation, and database are allocated equally. The comparison of resource cost is shownin Fig. 4, from which we can see that the proposed optimal allocation scheme achieves amuch lower resource cost compared to the equal allocation scheme. In Fig. 4, the maximaldifference of resource cost between the two schemes is US$ 2583.5.

5 Information manipulation in the CAVE via finger interactions

We propose to use finger interactions for information manipulation in the CAVE. In thissection, we first define our own specifications on the finger interactions in the CAVE, thenjustify why finger interactions provide a natural way of information manipulation in theCAVE.

Multimed Tools Appl

0.6 0.8 1 1.2 1.4 1.6

x 104

1000

2000

3000

4000

5000

6000

Arrival rate λ (packages/s)

Res

ourc

e co

st (

$)

Proposed optimal allocation schemeEqual allocation scheme

Fig. 4 Comparison of resource cost between the proposed optimal resource allocation scheme and the equalallocation scheme

5.1 Specifications of finger interactions in the CAVE

Finger interactions in the CAVE are achieved by the markers worn on both hands of theuser. Each marker in the CAVE is tracked in real time by the tracking system. The trackingsystem that we used in this work is the DTrack, which is an infrared (IR) optical trackingsystem provided by Advanced Realtime Tracking (ART).1 The tracking system can measurethe positions of up to 20 targets in a previously defined measurement volume and deliver theresults with a small time delay of 20 to 40 ms for real time tracking. The target to be trackedis equipped with light reflection markers. In addition, the markers are arranged at a knowngeometry to measure the orientation of the target. Tracking cameras scan a certain volumeand detect the light reflect from the markers. The captured images are processed to calculatethe potential marker position with a high accuracy. According to ART’s specification, amean accuracy of 0.04 pixels can be achieved in ART tracking systems. At any time, we canget the 6 Degrees-Of-Freedom (DOF) tracking data in the format of (x, y, z, η, θ, φ), forany marker. The coordinates (x, y, z) represent the position of the marker in the 3D space,and the Euler angles (η, θ, φ) represent the rotation of the marker around its local coordinatesystem. We use the left marker to represent the left index finger and the right marker torepresent the right index finger. That is why we call such marker-based interactions fingerinteractions. The trigger of an action is determined by the positions of the markers and theposition of the virtual object to be manipulated. For example, a menu item is selected whenthe distance between the marker on the right hand and the center of the menu item is lessthan a threshold.

We define the specifications of five basic actions, which are: 1) activating the main menu,2) selecting a menu item, 3) moving an object, 4) rotating an object, and 5) scaling an object.

1http://www.ar-tracking.com/

Multimed Tools Appl

The activation of main menu is triggered by the pull-down action performed by the righthand. The selection of a menu item is trigger by touching the menu item with the right indexfinger.

The moving action is determined by two factors: moving direction and moving distance.Let pm

t denote the position of the midpoint of the segment between the two markers at timet , pm

t−τ denote the position at time (t − τ). The moving direction of the object is from pointpm

t−τ to point pmt . The moving distance dm−m

t is the distance between point pmt−τ and point

pmt . The moving manipulation is shown in Fig. 5a. At the current time t where t is the

beginning time of a time slot, the object is moved by dm−mt along the direction from point

pmt−τ to point pm

t .

(b)

(a)

(c)

Fig. 5 Object manipulations with the proposed finger interactions: a moving an object, b rotating an object,and c scaling an object

Multimed Tools Appl

The rotating action is determined by four factors: rotation plane, rotation axis, rotationdirection, and rotation angle. Let Lt denote the line passing through the two markers at timet , and Lt−τ denote the line passing through the two markers at time (t −τ). The intersectionpoint between lines Lt and Lt−τ is denoted by pi

t . The rotation plane is the plane containinglines Lt and Lt−τ . The rotation axis is the line perpendicular to the rotation plane andthrough point pi

t . The rotation direction is the same to the rotation direction of the twomarkers. The rotation angle is the angle through which line Lt−τ is rotated to coincide withline Lt around the rotation axis along the rotation direction. At the current time t where t isthe beginning time of a time slot, the object is rotated by the rotation angle αt around therotation axis along the rotation direction, as shown in Fig. 5b.

The scaling action that we use is uniform scaling, which is a linear transformation thatenlarges or shrinks the object by a scale factor that is the same in all directions. At thecurrent time t where t is the beginning time of a time slot, the scale factor is given byst = dM

t /dMt−τ , where dM

t is the distance between the two markers at time t , and dMt−τ is

the distance between the two markers at time (t − τ). The object is enlarged when st > 1,shrunk when 0 < st < 1, or kept the same when st = 1. When st �= 1, the scaling action isaccompanied with a moving action from the midpoint pm

t−τ to the midpoint pmt , as shown

in Fig. 5c.

5.2 Information manipulation in the CAVE

5.2.1 CAVE vs. desktop-based 3D visualization

Compared with the desktop-based 3D visualization, such as Maya™, CATIA™, Unityetc™, CAVE has several advantages in visualizing 3D models. First, the nature of the vol-umetric data requires a big stage for visualization. For instance, in the automobile design,a pervasive conventional approach is to have a real 1:1 model for examination before final-izing the design. The reason is that viewing a 3D model on desktop-based software is notenough in presenting both overall and detail information at the same time. In the CAVE,this is achievable since the 3D glasses the user wears have a tracking marker so that withphysical movement, the 3D projected images will change accordingly. This simulates thenatural user interaction with the real model, while desktop-based counterpart is not capa-ble in providing such a platform for viewing the scalable data. Second, CAVE provides animmersive experience, which ultimately presents user a real feeling and better judgementin decision making. This immersive feeling is not achievable with desktop-based comput-ers. Third, CAVE is capable in providing several layers of information, this is especiallyimportant for data mining [13, 33, 34]and medical data visualization [3].

5.2.2 The reason of finger interactions in the CAVE

The dominant remote controller (see the comprehensive review in Section 2.3) approachrequires a learning process. All the buttons and their functions need to be familiarized withand memorized after a period of usage. On the contrary, finger-based interaction is intuitiveand natural, such that a user can manipulate a virtual object just as a real but weightlessone. Another advantage to have finger interactions is due to the 3D space the CAVE sys-tem provided, compared with a 2D screen on the desktop computer which only allows 2Dmovement. Thus, 3D position and angle data can be captured by the tracking system whichcan support the 3D gesture tracking and recognition effectively. Last but not the least, peo-ple get used to manipulating objects with their hands or fingers. Object manipulations with

Multimed Tools Appl

finger interactions in the CAVE resemble those in real life. With finger interactions, the userin the CAVE can feel more engaged with the real world. This proves to be useful in seriousgames and patient rehabilitation [8].

We compared the proposed work with the related work for 2D display environment [20,29, 31]. The proposed work is different in the following senses: 1) The proposed inter-actions are specifically for the immersive CAVE system, but the work in [20, 29, 31]is proposed for 2D display environment. In the 2D environment, the user has to alwayslook at the 2D screen. But in the CAVE system, the user can move around freely. As aresult, we need to consider the user’s orientation in our work. For example, when acti-vating the main menu, we must make sure that the main menu appears in front of theuser no matter which direction the user is facing to. The object manipulations also haveto be conducted in front of the user. Otherwise, the user will lose the immersive experi-ence. Such issues are not considered in a 2D environment. 2) The proposed interactionsallow users to freely manipulate a 3D object from any direction, even if the user is mov-ing in the CAVE. But the work in [N4, N5, N6] only provides the pan-and-zoom functionand the movement function on the display plane. Therefore, if the user wants to movean object to an accurate position along the direction perpendicular to the display plane,the interactions in [20, 29, 31] will be much less convenient compared to the proposedinteractions.

5.2.3 Beyond a tool of visualization display

Conventionally, the primary role for the CAVE is its visualization and immersive experi-ence. The change and modification of the associated information with the VR scene has tobe done in the desktop-based 3D modeling softwares running at the back stage. In this work,we focus on presenting, editing, and storing information inside CAVE via a natural userinterface. The proposed system enables users to select menus, choose items, and add/removeinformation from the database directly with their finger interactions. In addition, the pro-posed VR interface is also able to feedback the user the edited information in CAVE, so thatusers are able to interact with the back-end database and cloud directly, without extra workdone at the computer.

6 Case study: vHealth prototype

6.1 Overview of vHealth prototype

We developed a vHealth prototype to validate the proposed vConnect architecture. ThevHealth prototype is a CAVE-based real-time health monitoring system with the bidirec-tional information exchange between the virtual world and the real world. The relationshipbetween the vHealth prototype and the vConnect architecture is clarified as follows. ThevConnect architecture is a generic framework proposed in this paper. The major contributionof the framework is that it introduces the mixed reality experience in the virtual environmentthrough real-time bidirectional information exchange. The proposed vConnect architecturecan be applied in a wide range of applications, such as entertainment, tourism, education,healthcare, and others. The vHealth prototype is an application of the vConnect frameworkin the healthcare field.

Based on our literature review, the VR application in healthcare can be categorizedinto four groups: 1) communication interface, 2) medical educational training, 3) surgical

Multimed Tools Appl

simulation, and 4) therapy and rehabilitation [21, 23]. We feel that our vHealth system fallsinto category 1) in our medical application scenario. The interactive function of VR systemsuch as CAVE has been proved to be successful in various industries, including healthcare[21]. CAVE with its 3D visualization capability, provides an integrated tool to visualize mul-tiple data sources with realism of virtual experience [23]. Based on our literature review inmanufacture industry, it has been demonstrated that 3D visualization is more efficient andeffective in spotting errors, describing models for users in the industrial design, comparingwith its 2D counterpart [2].

The vHealth prototype is illustrated in Fig. 6. The patient who is monitored is requiredto wear a set of body sensors, which collect the physiological parameters including elec-trocardiogram (ECG), heart rate, respiration, body temperature, posture, and acceleration.The physiological data of the patient is transmitted to a client computer, which forwardsthe data to a database server over the Internet. The CAVE computers, which are the com-puters used to drive the projectors, can retrieve the physiological data of the patient, andthen visualize the data in the CAVE. The healthcare professional in the CAVE can inter-actively manipulate the information via fingers. The patient information, such as phonenumber, can be updated in the CAVE, and then stored back to the database. Also, thehealthcare professional can send instant messages to the patient’s smartphone over theInternet.

The data flows in the vHealth prototype can be divided into three sequential steps: 1)collection of real-time physiological data from the real world, 2) visualization and manip-ulation of physiological data in the CAVE via Finger Interactions, and 3) notificationsto the real world. The three steps will be elaborated in the following three subsections,respectively.

Smartphone

Database server

CAVE computers

The doctor in the CAVE

Client computerThe patient wearing body sensors

Fig. 6 Illustration of the vHealth prototype

Multimed Tools Appl

6.2 Collection of real-time physiological data from the real world

The patient wears a set of body sensors, which capture the real-time physiological data,such as ECG, heart rate, and body temperature. The physiological data is first transmittedto a client computer via wireless channels. The client computer receives the physiologicaldata, and then forwards the data to the database server over the Internet. The database serveris used to store the real-time information from the real world. We set up two tables for eachpatient. One is the health table which records the real-time physiological data. The otherone is the personal information table which contains the patient’s personal information suchas weight, height, date of birth, contact number, and the name of the physician who takescare of the patient.

6.3 Visualization and manipulation of physiological data in the CAVE via fingerinteractions

In the vHealth prototype, the user in the CAVE can perform a series of informationmanipulations, including information retrieval from the database, information visualizationand processing in the CAVE, and information feedback to the patients. All informationmanipulations are done with finger interactions. In this section, we will first present theperformance evaluation on finger interactions, and then describe the visualization andmanipulation of physiological data in the CAVE.

6.3.1 Performance evaluation on finger interactions

We conducted user tests to evaluate the performance of finger interactions. We recruited agroup of students to perform extensive user experience tests. Fifteen students (2 females and13 males) in Ryerson University were invited to participate in the user tests. The averageage was 28 years. Eight participants were in the expert group, who had been using the wandand the finger interactions for more than 6 months. The other participants were in the novicegroup, who had used wand occasionally and had not used any finger interactions before thetest. The novice group was trained on the usage of the wand and finger interactions for 10min before the test.

We conducted two user tests: menu navigation and object manipulation. In the menu nav-igation task, users need to select a target menu item by navigating a 6-level menu. With theproposed interactions, users can activate the main menu with a pull-down action and selectone item or trigger the next level menu by touching the corresponding menu item. Withthe wand interactions, the menu navigation is performed by pressing buttons. In the objectmanipulation task, users were asked to move, rotate, and scale a desktop computer from theinitial state to the target state. The initial state represents the position, the orientation, andthe size of the object before any manipulation, while the target state represents the expectedstate after object manipulations.

We first compare the time spent on the menu navigation task between the wand interac-tions and the proposed finger interactions. Figure 7 shows the mean and standard deviationof the time to complete the menu navigation task. The expert group completes the 6-levelmenu navigation using the proposed interactions within 8 s, reducing the time by 22.5 %compared to the wand interactions. The average time for selecting a menu item is 1.33 swith the hand by the expert, which is lower than that (1.72 s) with the wand. In the novicegroup, the proposed finger interactions reduce the time by 27.2 % compared to the wand

Multimed Tools Appl

Expert Novice0

5

10

15

20

Tim

e (s

)

Finger (proposed)Wand

Fig. 7 Comparison of the mean and standard deviation of time to complete the menu navigation task

interactions. The result in Fig. 7 demonstrates that the proposed finger interactions canselect menu faster than the wand interactions.

We next compare the mean and the standard deviation of the time for the object manip-ulation task between the wand interactions and finger interactions. As shown in Fig. 8, theproposed finger interactions reduce the completion time by 24.3 % for the expert group, and19.6 % for the novice group, compared to the wand interactions. The result in Fig. 8 demon-strates that the proposed finger interactions can manipulate the object faster than the wandinteractions. The standard deviation in each case is low, indicating that the total time tendsto be very close to the mean. The time taken by the novice group with the finger interactions

Expert Novice0

1

2

3

4

5

6

7

Tim

e (s

)

Finger (proposed)Wand

Fig. 8 Comparison of the mean and standard deviation of time to complete the object manipulation task

Multimed Tools Appl

is close to the expert, indicating that the novice users can perform the finger interactionsquite well after a short time of training.

6.3.2 Visualization and manipulation of physiological data in the CAVE

A healthcare professional in the CAVE is responsible for monitoring the health status ofpatients. This vHealth prototype is similar to the real-time traffic monitoring center, whereoperators monitor the traffic status in a centralized control room. In vHealth, a health pro-fessional operates this monitoring system and spots any abnormal activities of patients whohave the wearable sensors. We use a 3D Avatar to represent each patient in the CAVE. Withsuch 3D Avatar, we can associate the Magnetic Resonance Imaging (MRI) data with thecorresponding organ in the Avatar, such that we can walk into the organ to view it fromdifferent angle, which is not available on 2D desktop system. The high cost of such CAVE-based monitoring center can be justified if it is responsible for monitoring a lot of patients(for example, the patients in a large city).

A healthcare professional who is stationed in the CAVE, can retrieve the informationfrom the database server, and then visualize and manipulate it in the CAVE. Figure 9presents an overall schematic of the information visualization and manipulation for a spe-cific patient. It has four modules. Part A is a local Toronto map view, which shows thelocations of active users who wear Biopac, a bio-sensor hardware collecting real-time heart

Fig. 9 An overall schematic of the information visualization and manipulation for a specific patient

Multimed Tools Appl

related physiological data, heart rate, body temperature and ECG data. This map is demon-strated in Fig. 10. At normal monitoring level, every user with normal bio-status record arerepresented as a green dot. When a user has an abnormal bio status, such as heart rate orbody temperature irregularity, his representation dot turns to red. Then an alert monitoringlevel is triggered and a healthcare professional in the vHeath CAVE is notified.

Part B is a CAVE-based monitoring system. The healthcare professional who is oncenotified for an alert monitoring level, is able to visualize and process the red dot patient’sinformation in a detail level. He can examine heart rate, body temperature, and ECG datain real-time. The healthcare professional can also access and process patient’s personalinformation and other physiological record to ensure an effective and holistic medical strat-egy and treatment for this patient. For example, when a patient encounters a cardiac arrestwithout rapid emergency rescue, his brain is likely to get damaged because of a lack ofblood and oxygen delivery. Moreover, it is utterly important for a healthcare professional tohave a complete knowledge of this patient’s medical record before making a critical rescueplan. Among those physiological data, brain is probably the most important organ. This isbecause patients who are under the vHealth monitor are probably already have cardiac arresthistory and likely brain injury at the same time. The vHeath system offers a visualizationplatform to examine brain MRI data, by querying the database. This is because, currently,the MRI is the most efficient tool for examining the complicated brain anatomy. Scanningdeck of MRI images is obviously slow and not quick enough to make a life-threateningdecision. However, CAVE equipment provides an inside-out capability to quickly viewthe patient’s 3D MRI model and catch potential time-bombing medical condition duringthe emergency rescue. Both the visualization and tracking capacity in CAVE enables thehealthcare professional a very quick examination and accurate rescue decision.

Part C is the database connected with the CAVE system. Input queries from the CAVEuser are processed, and fetched data is passed from the database to the CAVE for presenta-tion. Part D is a real-time outbound communication module for the healthcare professionalto send alert phone calls, messages, and emails. Currently, we have the email communica-tion set up. The healthcare professional is able to send email alerts to the patient. He is alsoable to send the abnormal ECG graph to cardiologist, after an image cropping action donein the CAVE.

Figure 11 shows an Avatar of the patient. After the healthcare professional in CAVEchoosing the red dot from the monitoring map, an Avatar representing the patient, as shownin Fig. 11, appears in the front of the healthcare professional. The personal information,such as the name and the age, and the real-time physiological data, such as the heart rate

Fig. 10 A map monitoring thecurrent locations of the patients

Multimed Tools Appl

Fig. 11 An Avatar with real-time monitoring status as well as a menu selection

and the body temperature, of the patient are shown beside the Avatar. The system alsoprovides a menu selection to visualize and manipulate the information such as personalcontact, physiological record, and ECG data.

The real-time ECG signals of the patient can be displayed in a graph, as shown in Fig. 12.The scenario of this scene is that after examining the basic biological statistics, the health-care professional decides to carefully examine the ECG streaming data of this patient. Afterrecognizing the abnormal ECG segment, the healthcare professional is able to crop that seg-ment using finger interaction, and saves as an image file in the database for record. The

Fig. 12 An operation to crop the abnormal ECG segment and send it outbound via email

Multimed Tools Appl

image and the ECG measurement data can be sent to a cardiologist via the outbound emailcommunications for a further diagnosis.

While making a rescue decision for saving the patient’s life. the healthcare professionalalso needs to examine the physiological history of the patient. He can choose the medicalrecord menu. The CAVE then will bring the healthcare professional to visualize the previ-ously stored physiological record. The physiological record, which is stored at the cloud,can be delivered to the CAVE for visualization upon request. In our scenario, a 3D MRIDigital Imaging and Communications in Medicine(DICOM) image of the patient’s brain isused for visualization. Figure 13 depicts this scene. It can be observed that, with the assis-tance of the CAVE, the healthcare professional is able to freely examine the 3D brain modelwithout tedious remote control (as if on the desktop-computer). The healthcare professionalis also able to zoom in/out, rotate, move the 3D model using remote-control-free hand ges-tures, and even walk around inside a brain to do a careful examination. Such an approachcombines the advantage of the conventional CAVE visualization and the information pro-cessing ability provided by the proposed vHealth system. This can save a lot of time andhelp the healthcare professional to make an accurate and fast rescue plan.

6.4 Notifications to the real world

In order to ensure that fully immersive interaction between the patient and the healthcareprofessional is achievable in the vHealth system, communication between the healthcareprofessional and the patient must be achieved. In the vHealth prototype, notifications to thereal world are done through email. The healthcare professional inside the CAVE is capableof sending the patient an email with text and an image attachment. Thus, without the useof a keyboard or any additional hardware, the healthcare professional inside the CAVE iscapable of notifying the patient of any updates or emergencies.

Email transfer was implemented by creating a Simple Mail Transfer Protocol (SMTP)instance linked to a public SMTP server. Once a link is made, authentication is done to

Fig. 13 A 3D brain visualization in the CAVE

Multimed Tools Appl

Fig. 14 The text message andthe ECG graph received on asmartphone

ensure only authorized users have access to the server. In the process of sending an emailwith an image, the location of the image must be specified, along with the file name andextension. This automatically adds the image to the email while also allowing the healthcareprofessional to incorporate text into the email. Figure 14 shows the text message and theECG graph received on a smartphone. The email communications enable the user in thecave to interact with the real world.

Though emailing the patient is effective, a more efficient method for sending notifica-tions would be to text or directly call the patient. Through simple on-screen buttons, thehealthcare professional has the ability to call the patient or to send a text message to thepatient using pre-defined messages. This procedure allows the healthcare professional todirectly talk to the patient through a headset. While emailing has been completed, callingand texting will be integrated into the vHealth prototype in the future.

7 Conclusions

In this paper, we proposed a vConnect architecture to establish the real-time bidirectionalinformation exchange between the virtual world and the real world. With such informationexchange, the user in the CAVE can perceive the real world and interact with it. Specifi-cally, we addressed three technical problems in the proposed vConnect architecture. First,we optimized the resources in the wireless sensor networks such that the data streams cap-tured by the sensors can be delivered to the cloud in a prompt and reliable way. Second,

Multimed Tools Appl

we optimized the cloud resources such that the data streams can be processed promptly ata minimal resource cost. Third, we proposed finger interactions such that the user in theCAVE can manipulate the information in a natural way. Furthermore, we implemented avHealth prototype, a real-time health monitoring system, to validate the proposed vCon-nect architecture. The vHealth prototype demonstrated that the user in the CAVE canperceive the real-time information from the real world and provide feedbacks to the realworld.

Acknowledgments This work is partially supported by the Discovery Grant of Natural Science and Engi-neering Council of Canada (No. 238813/2010) and the Key International Collaboration Grant of NationalNatural Science Foundation of China (No. 61210005).

References

1. Abramyan L, Powell M, Norris J (2012) Stage: Controlling space robots from a cave on earth. In:Proceedings of IEEE aerospace conference, pp 1–6

2. Akpan IJ, Brooks RJ (2014) Experimental evaluation of user performance on two-dimensional and three-dimensional perspective displays in discrete-event simulation. Decision Support Systems

3. Al-Khalifah AH, Woff R, Alexandrov VN, Roberts DJ (2005) Case study: interacting with volumetricmedical datasets in networked cave environments. In: Proceedings of IS&T/SPIE electronic imaging,pp 350–359

4. Ardagna D, Casolari S, Panicucci B (2011) Flexible distributed capacity allocation and load redirectalgorithms for cloud systems. In: Proceedings of IEEE international conference on cloud computing(CLOUD), pp 163–170

5. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge6. Cheng L, Niu J, Cao J, Das S, Gu Y (2014) Qos aware geographic opportunistic routing in wireless

sensor networks. IEEE Trans Parallel Distrib Syst 25(7):1864–18757. Creagh H (2003) Cave automatic virtual environment. In: Proceedings of electrical insulation conference

and electrical manufacturing & coil winding technology conference, pp 499–5048. Gieser SN, Becker E, Makedon F (2013) Using cave in physical rehabilitation exercises for rheumatoid

arthritis. In: Proceedings of ACM International conference on PErvasive technologies related to assistiveenvironments

9. Gross D (2008) Fundamentals of queueing theory. Wiley-India10. Kim J-S, Gracanin D, Matkovic K, Quek F (2010) The effects of finger-walking in place (fwip) for spatial

knowledge acquisition in virtual environments. In: Proceedings of springer international symposium onsmart graphics, pp 56–67

11. Koike M, Makino M (2009) Crayon a 3d solid modeling system on the cave. In: Proceedings of IEEEinternational conference on image and graphics, pp 634–639

12. Li Y, Shark L-K, Hobbs SJ, Ingham J (2010) Real-time immersive table tennis game for two playerswith motion tracking. In: Proceedings of IEEE international conference on information visualisation,pp 500–505

13. Lin AY-M, Novo A, Weber PP, Morelli G, Goodman D, Schulze JP (2011) A virtual excavation: com-bining 3d immersive virtual reality and geophysical surveying. In: Proceedings of springer internationalsymposium on advances in visual computing, pp 229–238

14. Microsoft windows azure: [online] http://www.microsoft.com/windowsazure/15. Mizutori M, Hirota K, Ikei Y (2012) Skillful manipulation of virtual objects: implementation of jug-

gling in a virtual environment. In: Proceedings of IEEE international conference on virtual systems andmultimedia, pp 79–86

16. Nan X, He Y, Guan L (2012) Optimization of resource reconfiguration for cloud-based multimediaapplications. In: Proceedings of IEEE visual communications and image processing (VCIP)

17. Nan X, He Y, Guan L (2013) Optimal task-level scheduling for cloud based multimedia applications.In: Proceedings of IEEE international conference on acoustics, speech, and signal processing (ICASSP),pp 3771–3775

Multimed Tools Appl

18. Nan X, He Y, Guan L (2013) Optimization of workload scheduling for multimedia cloud comput-ing. In: Proceedings of IEEE international symposium on circuits and systems (ISCAS), pp 2872–2875

19. Nan X, Zhang Z, Zhang N, Guo F, He Y, Guan L (2013) vdesign: toward image segmentation andcomposition in cave using finger interactions. In: 2013 IEEE China summit & international conferenceon signal and information processing (ChinaSIP), pp 461–465

20. Nancel M, Wagner J, Pietriga E, Chapuis O, Mackay W (2011) Mid-air pan-and-zoom on wall-sizeddisplays. In: Proceedings of ACM conference on human factors in computing systems, pp 177–186

21. Pensieri C, Pennacchini M (2014) Overview: virtual reality in medicine. Journal For Virtual WorldsResearch 7(1)

22. Prachyabrued M, Ducrest D, Borst C (2011) Handymap: a selection interface for cluttered vr environ-ments using a tracked hand-held touch device. In: Proceedings of springer international symposium onadvances in visual computing, pp 45–54

23. Riva G (2003) Applications of virtual environments in medicine. Methods Inf Med 42(5):524–53424. Roberts DJ, Wolff R, Otto O (2012) Supporting a closely coupled task between a distributed team: using

immersive virtual reality technology. Comput Inform 24(1):7–2925. Schaeffer B, Flider M, Kaczmarski H, Vanier L, Chong L, Hasegawa-Johnson Y (2003) Tele-sports

and tele-dance: full-body network interaction. In: Proceedings of ACM symposium on virtual realitysoftware and technology, pp 108–116

26. Shah G, Liang W, Akan O (2012) Cross-layer framework for qos support in wireless multimedia sensornetworks. IEEE Trans Multimedia 14(5):1442–1455

27. Shu MKT, Vrudhula S (2006) Joint optimization of transmit power-time and bit energy efficiency incdma wireless sensor networks. IEEE Trans Wirel Commun 5(11):3109–3118

28. Song P, Goh WB, Fu C-W, Meng Q, Heng P-A (2011) Wysiwyf: exploring and annotating volumedata with a tangible handheld device. In: Proceedings of the SIGCHI conference on human factors incomputing systems, pp 1333–1342

29. Song P, Goh WB, Hutama W, Fu C-W, Liu X (2012) A handle bar metaphor for virtual object manip-ulation with mid-air interaction. In: Proceedings of ACM conference on human factors in computingsystems, pp 1297–1306

30. Sutcliffe A, Gault B, Fernando T, Tan K (2006) Investigating interaction in cave virtual environments.ACM Trans Comput Hum Interact (TOCHI) 13(2):235–267

31. Vogel D, Balakrishnan R (2005) Distant freehand pointing and clicking on very large, high resolutiondisplays. In: Procedings of the 18th annual ACM symposium on User interface software and technology,pp 33–42

32. von Kapri A, Rick T, Feiner S (2011) Comparing steering-based travel techniques for search tasks in acave. In: Proceedings of IEEE virtual reality conference, pp 91–94

33. Vote E, Feliz DA, Laidlaw DH, Joukowsky MS (2002) Discovering petra: archaeological analysis in vr.IEEE Comput Graph Appl 22(5):38–50

34. Wijayasekara D, Linda O, Manic M (2011) Cave-som: immersive visual data mining using 3d self-organizing maps. In: Proceedings of IEEE international joint conference on neural networks (IJCNN),pp 2471–2478

35. Wu Q, Boulanger P, Kazakevich M, Taylor R (2010) A real-time performance system for virtual theater.In: Proceedings of ACM workshop on Surreal media and virtual cloning, pp 3–8

36. Yates RD, Goodman DJ (2004) Probability and Stochastic Processes, a friendly introduction for electricaland computer engineering. Wiley, New York

37. Zhang D, Li G, Zheng K, Ming X, Pan Z-H (2014) An energy-balanced routing method based onforward-aware factor for wireless sensor networks. IEEE Trans Ind Inf 10(1):766–773

Multimed Tools Appl

Dr. Yifeng He is an Adjunct Professor at Ryerson University, Canada. He is the recipient of 2008Canada Governor General’s Gold Medal. Dr. He’s research interests include multimedia communicationsand cloud computing for 3D immersive communication, in which he has published 60 research papers.Dr. He is the Associate Editor of two international journals and was the lead Guest Editor for the Spe-cial Issue on Cloud Computing and Dynamic Resource Allocation for Multimedia Applications in theInt. J. of Digital Multimedia Broadcasting in 2011. He is the editor/author of two books in multimediaprocessing/communications.

Ziyang Zhang received his MASc degree in 2014 in electrical and computer engineering from RyersonUniversity, Toronto, Canada where he is a recipient of Ontario Graduate Scholarship. Currently he works asa research engineer at Ryerson Multimedia Lab at Ryerson University.

Multimed Tools Appl

Xiaoming Nan received his M.S. degree in Telecommunication Engineering from Beijing University ofPosts & Telecommunications, China, in 2010. He is currently a Ph.D. candidate of the Ryerson MultimediaLaboratory (RML) at the Ryerson University, Toronto, Canada. His research interests include cloud resourceoptimization, virtual reality, human computer interaction, face recognition, and content-based video retrieval.

Dr. Ning Zhang is currently a Postdoctoal fellow at Ryerson Multimedia Laboratory, Ryerson University.His research interests are image/video analysis and processing, content-based image/video retrieval, andaugmented/virtual reality. He received his Ph.D in Electrical Engineering, 2012, from Ryerson UniversityCanada; and B.A.Sc in 2006, from University of Toronto, Canada. He is currently a member of IEEE andACM.

Multimed Tools Appl

Fei Guo received the B.S. degree from Zhengzhou University, Zhengzhou, China, in 2012. Currently, heis pursuing the Ph.D. degree in the Department of of Electrical and Computer Engineering at RyersonUniversity, Toronto, Canada. His current research interest includes 3D reconstruction and registration.

Edward Rosales is a M.A.Sc student in the Electrical and Computer Engineering department at RyersonUniversity. He received his B.Eng in Electrical Engineering in 2012 from Ryerson University, Toronto,Canada. His research interests include multi-view television, stereoscopic rendering, and facial detection andrecognition.

Multimed Tools Appl

Ling Guan received his Ph.D. Degree in Electrical Engineering from the University of British Columbia,Canada in 1989. He is currently a professor and a Tier I Canada Research Chair in the Department of Elec-trical and Computer Engineering at Ryerson University, Toronto, Canada. Dr. Guan held visiting positionsat British Telecom (1994), Tokyo Institute of Technology (1999), Princeton University (2000), National ICTAustralia (2007), Hong Kong Polytechnic University (2008–09) and Microsoft Research Asia (2002, 2009).He has published extensively in multimedia processing and communications, human-centered computing,machine learning, adaptive image and signal processing, and immersive 3D computing. Dr. Guan is a Fellowof the IEEE, an Elected Member of the Canadian Academy of Engineering, an IEEE Circuits and SystemSociety Distinguished Lecturer, and the recipient of 2014 IEEE Canada C.C. Gotlieb Computer Medal andthe 2005 IEEE Transactions on Circuits and Systems Best Paper Award.