mobile real-time collaboration for semantic multimedia
TRANSCRIPT
Mobile Netw ApplDOI 10.1007/s11036-013-0453-z
Mobile Real-Time Collaboration for Semantic MultimediaA Case Study with Mobile Augmented Reality Systems
Dejan Kovachev · Petru Nicolaescu · Ralf Klamma
© Springer Science+Business Media New York 2013
Abstract Advanced mobile applications that enable newways of interaction with digital objects become increas-ingly important for on-site professional communities. Thesenew ways of interaction, e.g. in Mobile Augmented Real-ity (MAR) via position and 3D movement, are real needsfor fieldwork domains such as cultural heritage manage-ment and the construction industry. In addition, on-siteprofessional communities generate shared knowledge baseswith multimedia content and semantic annotations throughcollaboration. However, current MAR applications lackreal-time collaboration features. In practice, blending mul-timedia semantics in mobile real-time collaboration is chal-lenging due to the limitations of mobile devices, the lack ofmature dedicated designed communication infrastructuresand the constraints of the remote environments. This paperpresents a mobile real-time collaboration system for seman-tic multimedia annotations with augmented reality features.We use XMPP as a real-time protocol for the secure, scal-able and interoperable processing of XML-based semanticmultimedia metadata described in MPEG-7. Our prototypewas evaluated in the digital documentation of historical sitesfor cultural heritage management. The evaluation resultsindicate potential for increased productivity and enhancedmutual awareness in on-site professional communities.
D. Kovachev (�) · P. Nicolaescu · R. KlammaInformation Systems and Databases, RWTH Aachen University,Ahornstr. 55, 52056 Aachen, Germanye-mail: [email protected]
P. Nicolaescue-mail: [email protected]
R. Klammae-mail: [email protected]
Keywords Real-time collaboration · Augmented reality ·Mobile multimedia · Multimedia metadata · MPEG-7 ·XMPP
1 Introduction
Mobile devices are altering our work and private livesin many ways. For instance, commodity smartphones andtablets have proven to do a better job than custom-developed(and expensive) devices in many professional fields such asdisaster management or military.1 Mobile devices are alsoideal tools for convenient, anytime anywhere productionand sharing of multimedia content. Besides that, smart-phones equipped with handful sensors provide a platformfor context-aware interactions with digital objects.
However, mobile multimedia applications lack supportfor real-time collaborative work. Typically, mobile devicesusage is limited to creating and sharing content, whereas thecollaborative operations are performed asynchronously ondesktop computers or laptops. Yet real-time collaboration isnecessary in many use cases for both on-site professionaland amateur communities. These on-site communities arecharacterized by a high degree of collaborative work, mobil-ity and integration of data coming from many members.Collaboration facilitates peer production [3]. Real-time col-laboration provides the ability to iterate quickly by permit-ting members to work in parallel. In addition, mobile real-time collaboration (MRTC) enables users to benefit fromlocation awareness and work on spatially distributed tasks.Capturing and using the context by multiple collaborators inreal-time increases productivity.
1http://abcnews.go.com/Technology/smartphones-military-pentagon-tests-apps-androids-iphones-ipads/story?id=14615595
Mobile Netw Appl
To illustrate this concept, we consider the use caseof mobile augmented reality browsers which are typicalexamples that provide rich multimedia user experience,but fail to provide collaborative features. Mobile aug-mented reality becomes increasingly feasible on inexpen-sive hardware at mass-market effect [9]. Layers [37] andWikitude [38] are example MAR browsers which displaythird party digital content. However, such MAR browsershave limited support for MRTC, i.e. to enable mobileusers to collaborate and communicate with each other inreal-time. This is due to the lack of genuine two-waypeer-to-peer communication within such systems. More-over, the semantics of digital objects (i.e. multimedia) inMAR is barely considered beyond location parameters. Weargue in this paper that collaborative actions on multi-media semantics with the assistance of augmented real-ity can enhance the user experience in many professionaldomains.
We aim to reduce the barrier that hinders the devel-opment of MRTC systems for multimedia by creating anopen, customizable lightweight infrastructure. It provides aset of services so that mobile clients can perform multime-dia acquisition, sharing and real-time collaborative semanticannotation. While there are other infrastructures that supportcollaborative features, our infrastructure differs from themby focusing on multimedia semantics that is “prosumed”by multiple collaborators in real-time. Access to multime-dia content and metadata are fundamental to our approachto support collaborative mobile applications.
Concretely, the infrastructure relies on the IETF stan-dard eXtensible Messaging and Presence Protocol (XMPP)[26]. In our XMPP-based Mobile Multimedia Collaboration(XmmC) infrastructure, we exploit several XMPP proto-col extensions (XEPs) to fulfill effective communicationbetween mobile clients and to exchange multimedia con-tent, semantics, annotations. Moreover, XmmC integrateswith existing MPEG-7 multimedia services [4] to interop-erate with the MPEG-7 multimedia metadata standard [16].In this way, XmmC takes advantage of existing large multi-media metadata repositories. We also provide a Web-basedfrontend for collaborative metadata editing, which can beused on both mobile and desktop browsers. The Web fron-tend completes the real-time collaboration multimedia lifecycle.
In the rest of the paper, we first review the relatedresearch work Section 2. Then in Section 3 we describeour approach to MAR with real-time collaboration and mul-timedia semantics. We explain the prototype system andits design in Section 4, followed by the description ofsome communication interactions in Section 5. In Section 6we introduce the evaluation setup and the correspondingresults. Finally, we draw conclusions and refer to the futurework.
2 Related work
The work related to this paper can be grouped into fourparts: multimedia prosumers, collaboration techniques, real-time collaborative applications and MAR browsers.
2.1 Multimedia prosumers
Prosumers are users which “actively participate in the cre-ation and modification of products they consume” [34]. Thedecentralized, self-driven prosumer community processes,in which members are creating, sharing and promoting con-tent are named peer production [3, 27, 39]. Many systemsare available to store, consume and annotate multimedia.YouTube is an example of the prosumer paradigm expan-sion, where users can share their own videos in certaincommunities or open on the Web. However, there is lim-ited support for peer production in on-site communities ofpractice by means of multimedia annotations combined withMAR, through real-time collaboration. In this paper, weprovid an infrastructure for real-time collaboration for anno-tating multimedia, available for mobile devices and Webapplications. Examples of on-site professional communitiesfor which our approach can offer support are communi-ties of experts (e.g. architects, historians) in the culturalheritage domain, which have to document historical artifact-s/sites or communities of experienced construction workers,which can document their processes for various purposes(e.g. informal learning for novice workers). In these exam-ples, the architects and the experienced construction work-ers are prosumers that contribute to the shared knowledgeof their community of practice. To achieve interoperabil-ity and interchange between such shared knowledge based,the metadata has to be based on standards. We, therefore,use the (XML-based) MPEG-7 standard for managing thesemantic annotations [1]. In addition, sound tools for groupawareness, participation, and coordination also supportedfor successful collaborative outcomes [18].
2.2 Collaboration techniques
Groupware technology can assist a group of people to com-municate, manipulate and modify shared digital objects ina coherent manner [7]. Popular groupware software appli-cations are based on simultaneous writing of a documentby different authors, also known as shared file editing orcooperative design. Collaborative editing is the practice ofparticipants within a group which work together to pro-duce simultaneously a common output using a set of definedoperations [19].
Currently, Operational Transformation (OT), first intro-duced by Ellis and Gibbs [6], then simplified in the JupiterSystem [23] is the most popular technique behind collabo-
Mobile Netw Appl
ration features. The core of the algorithm is to execute thelocally generated operations without delay and transformremote editing operations into a new form according to theeffects of previously executed concurrent operations. Afterevery operation, the consistency of the artifact at all partici-pant clients has to be ensured [40]. The changes propagationmechanisms can be based on a peer-to-peer model, on acentralized model or can use a hybrid approach.
There is much research to improve the OT approach.dOPT [6], Jupiter [23], SOCT2 [31], GOT [32] focus on lin-ear structured text documents, whereas OT for SGML [5]treeOpt [12] and P2P Editing on XML-like Trees [19] aresome examples that perform OT on tree like structures. Fur-thermore, COT [33] and adOpted modified the algorithm byadding undo functionality. Google Operational Wave Trans-formation assures concurrency control of XML documents[36].
Since XML is the de-facto standard interchange and dataformat, systems that support real-time collaborative edit-ing of XML documents could lead to a general solution formany application areas especially real-time collaboration.Furthermore, XML is widely used for storing and exchang-ing multimedia metadata because its tree like hierarchicalstructure can describe the complexity of metadata descrip-tors which can be interchanged with different systems [2].
The Consistency Maintenance Algorithm for XML(CMAX) [8] is a lightweight approach that focuses onensuring collaborative editing on structural XML-baseddocuments. The algorithm is mainly inspired from oper-ational transformation for structural documents. CMAXfollows the same principles of OT for solving divergenceand causal violation. However, for solving the intention vio-lation problem a new approach is applied. The OT updateoperation is not used since Gerlicher [8] showed that theprobability of modification of the same tree node concur-rently is relatively small when the XML tree is averaged sizeor big size. Therefore, CMAX delegates conflict resolutionto the user instead of combining the attribute values of OT.
2.3 Real-time collaborative applications
The right set of underlying communication protocol is cru-cial in mobile real-time collaboration. Google Wave proto-col is an excellent example of XMPP-based communicationand collaboration platform for concurrently editable struc-tured documents and real-time sharing between multipleparticipants. Novell Vibe Cloud [24] is a Web-based socialcollaboration platform for enterprises, providing social mes-saging and online document co-editing along with filemanagement, groups, profiles, blogs and wikis, and secu-rity and management controls. Both Google Wave Protocoland Novel Vibe are sophisticated collaborative editing soft-ware, but their reliance on powerful clients (i.e. desktop
Web browsers) limits the usefulness for custom mobileapplications.
Contrariwise, the Collaborative Editing Framework forXML (CEFX) [8] enables lightweight concurrent real-timeediting of XML files using operational transformation algo-rithms. Since XML has a generic and extendable nature,different kind of information can be stored such as graphicfiles (SVG), AR contents (ARML) etc.. Voigt [35] furtherextended the framework by changing the communicationprotocol from JAVA RMI to XMPP. Moreover, the commu-nication data volume reduced significantly. Our work usesthe CEFX+ for providing collaborative editing services andthe communication between the nodes that attend a collabo-ration session via XMPP. Similarly, the Mobilis framework[28] uses CEFX+, too. However, the platform lacks mobileAR browsing and multimedia semantic features. Junction[30] is another XMPP-based communication framework formulti-device applications. Junction provides collaborativefeatures using a XMPP multi-user chat room to route XMPPmessages between devices. However, this does not provideconflict resolution and shared data structures. The CoMa[20] tool provide visual awareness of the spatial locations ofgroup members on their mobile devices. It provides meansto users to find a meeting point or chat with each other.CoMa, however, ignore any collaborative operations onmultimedia artifacts. AR-Sumo [10] considers an AR gamein collaborative edutainment settings where users can inter-act with digital object interactively. AR-Sumo, nevertheless,considers neither an extensible collaboration mechanismnor “prosumer” multimedia content.
2.4 Mobile augmented reality browsers
Mobile Augmented Reality is a natural complement tomobile computing, since the physical world becomes a partof the user interface (e.g. in video streaming). Accessing andunderstanding information related to the real world becomeseasier. This has led to a widespread commercial adoptionof MAR in many domains like education and instruction,cultural heritage, assisted directions, marketing, shopping,gaming. Furthermore, in order to support diverse digitalcontent several popular MAR applications have shifted fromspecial-purpose applications into MAR browsers which candisplay third party content.
Holler and Feiner [11] give a detailed introduction toMAR systems and review some important MAR systemconsiderations. Layar [37] is a mobile outdoor AR plat-form for discovering user’s surroundings. The platformdisplays the physical world by augmenting it with the digi-tal retrieved information called “layers” via a mobile device.The platform provides an API for third party developers.They can create, maintain and publish “layers” that extendthe application for their own purposes. However, the major
Mobile Netw Appl
drawback of Layar is the flow of information which is onlyone way. The client cannot push other data besides thelocation and some additional parameters. Therefore it doesnot support collaborative applications. Wikitude [38] is asimilar popular mobile application which lacks also collab-orative features. You et al. [41] tried to overcome such limi-tations of pull-based MAR browsers in their Mixed RealityWeb Service platform. Their platform provides a REST-ful interface for 3rd party geo-spatially oriented content.However, their workaround solution of a publish-subscribesystem with periodical polling suffers from the same limita-tions. In contrast, our XmmC infrastructure is built aroundXMPP protocol extensions, thus ensuring simple and effec-tive bi-directional XML-based communication. The scopeof this paper is to provide collaboration primitives to a MARbrowser. Therefore, we enhanced the Mixare open-sourceMAR browser [21] with semantic multimedia and real-timecollaborative features.
3 Multimedia collaboration approach
A collaboration environment needs to provide sharedworkspace and conversational support. In addition, mobilecollaboration environments should consider the mobilityof users, i.e. their dynamical spatial context. In our case,XmmC services are tailored to the restrictions and featuresof mobile devices. The workspace facilitates sharing ofmultimedia content and context, access to multimedia store,and maintaining a consistent state of multimedia contentand its semantics. The semantics usually express the con-text of the multimedia content, e.g. location and time wherethis was captured, the creators, low-level semantics (such ashistogram features), high-level semantics like annotationsand tags, etc. The semantics are represented in a metadataformat.
Figure 1 demonstrates XmmC use cases. Users are clas-sified into producer, collaborator and consumer roles. A
user can have any combination of those roles. A produceracquires multimedia with the help of the built-in camera andsensors at his/her mobile device. Then the multimedia isshared via the media repository and a Point of Interest (POI)with spatial context is also added. Collaborators annotatemultimedia by creating and editing annotations in real-time.These annotations are propagated to all other collaboratorsvia a collaborative editing infrastructure. Furthermore, col-laborators use chat functionality to converse about variousreasons such as discussing about content, requesting assis-tance for annotation, etc. Consumers view multimedia viaMAR as POIs either at camera view or map view. They canalso choose a POI in order to see further details.
In the use case of MAR browsers, we exploit a device’sfeatures to demonstrate the feasibility of a collaborativeaugmented reality. For the AR functionality, the multime-dia content uses a POI data type. Users can preform spatialrange queries over the POIs. Every POI has a referenceto a multimedia, longitude, latitude, altitude and precision.Our multimedia artifacts can be any kind of data whichcan be rendered on a mobile device, e.g. video, images,3D objects. Title, description and keywords form the basicmetadata about the multimedia. URIs specify the storage ofthe multimedia content and its representations.
Every multimedia artifact has multiple semantic basetypes which are used for annotation. Integration with ourexisting MPEG-7 services [29] is one of the goals of thiswork. The annotation model is defined according to thesemantic base type definition used by the MPEG-7 stan-dard. This integration provides access to large repositoriesof already semantically-enriched multimedia.
XmmC preserves a consistent state of the metadatabetween the multiple collaborators. Our collaborative ARmetadata editing service uses the CMAX algorithm [8].CMAX provides lightweight mechanism which suit mobilerestrictions. Augmented reality information, basic multime-dia metadata and semantic annotations are stored as XMLdocuments. This design choice eases the interoperability
Fig. 1 XmmC use case diagram
Mobile Netw Appl
and the development of collaboration features. The synchro-nization is done by keeping a copy of the XML documentat every client in the session and ensuring timely updates onthe copies in case of edit operations or conflicts.
Figure 2 shows screenshots of the XmmC mobile client.On the left side of the figure, the AR UI shows nearbyPOIs and related multimedia artifacts. The middle snapshotillustrates a multimedia artifact and its associated MPEG-7 semantic annotations. The annotations can be created,edited and deleted collaboratively with other mobile clients.The right snapshot depicts the chat feature of the mobileclient application which serves as a limited conversationalsupport.
4 Real-time XMPP-based mobile collaboration system
In this section, the design and the realization of the XmmCsystem, i.e. the architecture, system interaction, server andclient prototypes are discussed. Special focus is given on ourXMPP protocol extensions that enable interactions tailoredto collaborative operations.
The XmmC system consists of mobile clients, a serverand Web-based frontend. Figure 3 illustrates the multilayerand modular design approach, which results in a flexible,easy-maintainable platform for developing XMPP-basedmobile multimedia collaborative applications.
The XmmC Server is designed as a service-orientedarchitecture. The rationale behind the choice to use XMPPas main communication mechanism is given in the next sec-tion. The XMPP communication with the clients is enabledby an XMPP server. We use an OpenFire server [13] andextend its functionality with our custom-developed plugin.This simplifies the software development and maintenancesince both the mobile client and XmmC Server can use thesame communication components built on top of the Smacklibrary. XmmC extends the XMPP server with an Open-Fire plugin consisting of three main XMPP modules. Thesemodules are responsible for XMPP communication with theclients:
The AR module provides augmented-reality-related ser-vices such as retrieving POIs satisfying given spatial query.The AR Module uses the AR Service and Media CatalogService in background to provide its functionalities.
The Multimedia module is responsible for transfer ser-vices like uploading, retrieving and deleting multimedia.The module also utilizes the Media Catalog Service that per-sists the basic multimedia-related data in a XML databaseand enables the client applications to obtain this type ofdata. Furthermore, this module notifies all clients sub-scribed to an XmmC PubSub node by publishing multime-dia events. The PubSub node overcomes the usual pollingproblem by providing a publish-subscribe mechanism forevent-based workflow, inspired by the observer designpattern.
The Collaborative annotating module handles metadatarelated services, i.e. metadata management and synchro-nization of annotations in real-time between client appli-cations, by using the Collaborative Editing Service. Col-laborative Editing Service is based on CEFX+. Duringthe synchronization process, a copy is also situated at theMPEG-7 Integration Service so that the service acts as aproxy that synchronizes the metadata XML document andcalls the related MPEG-7 multimedia content and semanticbase type services whenever a XML document is modi-fied. The MPEG-7 services are responsible for managingthe metadata repository.
XmmC client is a mobile application that operates bothas a MAR browser and provides features to users for mediaacquisition and metadata annotation. In our reference pro-totype implementation, the XmmC Client is an Androidnative app. The Android ecosystem has already developedXMPP libraries available to be imported into an Androidsystem. Native apps can also be developed for other mobiledevices types that are equipped with camera, GPS, compass,accelerometer and Internet connectivity.
Smack [14] is one of the most mature open-sourceXMPP client libraries. XmmC uses Smack to connect toan XMPP server (s. Fig. 3). We abstract the XMPP com-munication on mobile clients within the XMPP connection
Fig. 2 XmmC MAR browser
Mobile Netw Appl
Web Client
XmmC Server
XmmC Openfire Plugin
Mobile Client
XMPP Server(Openfire)
Collab.Editing Service
LAS-MPEG7Integration
Service
XMPP
WLAN GPS Compass ACCEL
AR Service
Media Store
Service
XM
PP
M
odules
XMPP Client(Smack)
XmmC ClientMedia Catalog Service
XMPP Connection Layer (XCL)
MultimediaModule
Camera
XML DB
HTTP
Collab.Annotation
Module
ARModule
CEFX+ XPath
Mpeg-7 MetadataRepository
Annotate Metadata
AR Browser
Media Acquisition
XMPP over WebSocketXMPP
Fig. 3 XmmC abstract system architecture
layer (XCL), which is responsible for receiving and send-ing the XML stanzas. XCL is shared between multipleXmmC-based apps executing on a same device in order toreduce the XmmC footprint on a mobile device. The CEFX+Component is used for the synchronization of the XMLmetadata.
The Smack library is Java-based, and therefore, unusablefor Web interactions. XmmC is, however, able to inter-act with Web-based clients using the features of modernWeb browsers, i.e. Web Sockets. In order to offer supportfor managing the multimedia content in Web browsers, weextended XmmC with a Web application as depicted inFig. 4. Apart from using the same annotation typesdescribed for XmmC, we experimented with text annota-tions, where the content can be seen and edited in real-time.
This component uses an Operational Transformation engineand XMPP over Web sockets as a communication protocolfor text editing operations and updates. The Web interface isdeveloped using Web widgets (components with limited, butclear-cut functionality). We use a XMPP publish-subscribemechanism to dispatch messages between the widgets, tech-nique which allows communication of data across browserinstances. Through this Web extension, we offer a bet-ter support for stationary users, which are collaborating toremote on-site users, using XmmC on their mobile devices.Moreover, we test the feasibility of managing the XmmCcontent in a Web browser and the impact of adding thereal-time editing feature to the overall annotation experi-ence. However, due to space reasons, more related technicaldetails are not provided here.
Fig. 4 Web application for collaborative annotation
Mobile Netw Appl
5 XmmC communication interactions
In this section we elaborate on the communication interac-tions between the system components. The section revealshow the XMPP protocol is extended to achieve the desiredXmmC aspects.
The XMPP protocol provides a pure XML foundationfor real-time messaging, opening up tremendous possibili-ties for more advanced real-time applications. XMPP andits extensions demonstrate several advantages beyond tra-ditional HTTP-based Web services (e.g. SOAP and REST),such as decentralized, open and flexible (extensible) com-munication protocol, federation of servers, support for real-time data streaming in two directions, event notifications,remote procedure calls and multimedia session manage-ment. Asynchronous invocation eliminates the need forad-hoc solutions like long-polling.
XMPP connection between two entities is realized byXML streams which are containers for XML elements. Aclient initiates a session with an XMPP server and nego-tiates the session details. Afterwards, information can beexchanged bidirectionally on the top of TCP/IP protocol viaXML snippets (i.e. stanzas). Unlike traditional approaches(e.g. email, Web), XMPP ensures that TCP connection willbe always alive until the XML stream is closed. Mean-while unlimited stanzas can be sent asynchronously. Con-sequently, real-time communication can be assured with an”always-on channel”.
Info/query (IQ) stanzas provide a simple request-response mechanism. Requests and replies are trackedby the id attribute that is generated by the entity mak-ing the request. The type of the IQ stanza can be either
get, set, result or error. The requested entity sends a get-IQ or set-IQ which will be replied by a result-IQ or anerror-IQ.
5.1 Multimedia interaction
Every multimedia collaboration should, at minimum, sup-port functionalities to acquire, share and view multimediacontent. XmmC supports these operations for photos andvideos.
Figure 5 illustrates the sequence diagram on how amobile client application inserts a multimedia element tothe media catalog, using the XMPP protocol. First, basicmedia descriptions like title, description, location etc.,are sent via <mmedia-insert>, a custom set-IQ, but theactual content resides on the client side. Once the multime-dia XMPP module receives a stanza, it forwards the requestto the Media Catalog Service. If no error occurs it respondsback with a <mmedia-transfer> get-IQ which demandsfor the actual file transfer. Immediately, the client initiatesthe transfer of the file via XEP-096 SI File transfer exten-sion [22]. When the actual transfer finishes, the client sendsback a <mmedia-transfer> result-IQ that informs theserver about the successfully completed transfer. Finally,the server side sends back a result-IQ <mmedia-insert>,which verifies that the multimedia insertion was completedsuccessfully. The XMPP module also sends a PubSub pub-lish set-IQ that notifies other mobile clients when a newmedia is inserted.
After a media is inserted to the multimedia catalog, it isaccessible for retrieval. The workflow for retrieving multi-media is similar to the insertion process. Further functions
Fig. 5 Sequence diagram of multimedia insertion
Mobile Netw Appl
such as modification or deletion of multimedia descriptionscan be performed.
Listing 1 shows the content of <mmedia-insert> setand result IQ stanzas. The payload of the set stanza con-sists of basic multimedia description. Id field is usedfor uniquely identifying a multimedia artifact. Name,description and keywords are part of the semanticdescription. Latitude, longitude and altitude (interms of meters) describe the geographical location wherethe multimedia is taken, whereas accuracy refers tothe accuracy of the geographical location. The elementcaptureTime refers to the time point when the multimediais created. The last field, mediaType, distinguishes differ-ent media types like image, video, etc.. The resulting IQ hasonly an id element that represents the multimedia.
Multimedia PubSub notifications A significant part ofXmmC are the notifications sent to users about multime-dia events. Without a push functionality for notifications,the prototype should do polling or the data will not be con-sistent after the changes are applied. However, polling isan (already) obsolete and undesirable solution, whose usein XmmC can lead to bad performance and communicationoverhead. On the other hand, the changes that are generatedon the clients should be propagated, in order to ensure con-sistency of the overall system. As aforementioned, PubSubensures data freshness and less communication traffic withpublish and subscribe mechanisms.
To retrieve notifications, mobile clients have to subscribeto the XmmC PubSub node. Whenever data is modified on
Listing 1 Examples of set-IQ and result-IQ stanzas for Mmedia-insertoperation
<iq id="jOE9e-33"to="[email protected]/Smack"from="[email protected]/Smack"type="set"><mmedia-insert xmlns="http://dbis.rwth-aachen.de/xmmc/mmedia-insert"><id>b48ec4b4-e5ea-493f-acce-16a52a60ffa0.jpg</id><name>Trojan Horse</name><keywords>horse,trojan<keywords><description/><latitude>50.77816836</latitude><longitude>6.061606090000001</longitude><altitude>2</altitude><accuracy>23</accuracy><captureTime>1319125999814</captureTime><mediaType>Image</mediaType>
</mmedia-insert></iq>
<iq id="jOE9e-33"to="[email protected]/Smack"from="[email protected]/Smack"type="result"><mmedia-insert xmlns="http://dbis.rwth-aachen.de/xmmc/mmedia-insert"><id>b48ec4b4-e5ea-493f-acce-16a52a60ffa0.jpg</id>
</mmedia-insert></iq>
the server after <mmedia-insert>, <mmedia-update> or<mmedia-delete> set-IQ stanza, the multimedia XMPPmodule sends a PubSub set-IQ message with the payloadof multimedia id and the event that is associated with therespective multimedia. Afterwards, the published informa-tion is broadcasted to all subscribers by the PubSub node.The sent stanza is a PubSub notification message with apayload similar to the published stanza.
5.2 AR interaction
Browsing surroundings augmented with the multimedia isamong the main use cases for the XmmC prototype. Forexample, in the cultural heritage documentation scenario,
Listing 2 Example of an ar-query get-IQ stanza
<iq type=’get’ id="XcupS-1"from=’[email protected]/Smack’to=’[email protected]/Smack’><ar-query xmls=’http://dbis.rwth-aachen.de/xmmc#services/AR’><latitude>50.77816836</latitude><longitude>6.0616060</longitude><radious>800</radious>
</ar-query></iq>
collaborators can have an interactive 3D view of historicalartifacts shown on their video camera stream on a mobiledevice. They can change the artifact representation and inthe same time observe changes performed by others.
We developed XMPP-based augmented reality services,which exchange content represented in the AugmentedReality Markup Language (ARML) [17]. ARML, originatedfrom KML (Keyhole Markup Language), is a specificationthat enables content production to be displayed on vari-ous mobile AR browsers. An ARML file consists of twosections. The first section provides information about thecontent provider to which the POIs are related to. Here,provider’s name, description, logo, URL, icon and relatedtags are specified.
Figure 6 demonstrates the advantages of XMPP overtraditional Web protocols in collaborative settings. Typi-cal solutions using Web services need to perform regularrequests to display new content. On the contrary, XmmCClients communicate in real-time with an XmmC Server inboth directions.
Listing 2 illustrates a get ar-query custom IQ stanza.The client issues a get-IQ ar-query stanza with the loca-tion restriction parameters latitude, longitude andradius that conforms to a circular geographical area, withthe center given by latitude, longitude.
After the AR Module receives the request, it retrieves thePOIs which are inside the restriction area and sends back aresult IQ stanza with an ARML element composed of POIs.
Mobile Netw Appl
Typical MAR Browsers
XmmC
GPS data
ProviderServer
Get POIsData
SOAP/RESTful Web Service
ProviderWebsite
Wikipedia, Flicker,
Facebook,etc
GPS dataXMPP Server XmmC
Server
Get POIs Get POIs
XMPP XMPP
Create POI Create POI
PubSubNode Data
Edit POI Metadata Edit POI Metadata
Publish EventEvent Message
Fig. 6 Comparison between communication mechanisms in a typical MAR browser and the XmmC system
5.3 Mobile collaborative annotation interaction
XmmC is also capable of annotating multimedia collabo-ratively in real-time based on XMPP. Similar to the ARinteraction example, collaborators can modify metadata andobserve remote changes in real-time.
Figure 7 demonstrates the sequence diagram whichspecifies the workflow of the collaborative multimediaannotation. First, mobile client-1 sends a joinSession<jabber:iq:rpc> set-IQ with parameters of multimedia’ssessionId and the clientId. Then, the Collaborative Annota-tion Module decrypts these parameters and calls joinSessionof the Collaborative Editing Service. This creates a collab-oration session and a multi-user chat room for this sessionand returns the generated SessionData. The SessionDatacarries information like MUC room name, participant clientids, etc. The module decrypts the SessionData and sendsback an RPC result IQ with the return value. As soonas the mobile client receives the SessionData, the clientjoins the room generated at the server side and requests theXML document which carries semantic multimedia annota-tions, by sending loadDocument <jabber:iq:rpc> set-IQwith a parameter of the corresponding documentId. Similarto the previous RPC request, the XMPP module decryptsthe parameter and calls Collaborative Editing Service toretrieve DocumentData which comprises the DOM structure
of shared XML, state vector and history buffer. Once theclient receives the DocumentData from the XMPP module,it can start to annotate the multimedia in the collaborativeediting session.
When mobile client-2 wants to participate to the collab-orative annotation session, it has to go through the sameprocess as mobile client-1. Currently, both clients joinedthe collaborative annotation session. Operations made byeach client are sent to all participants of the sessionvia a <groupchat> message of the XEP-0045: Multi-User chat extension [25]. The XML document at server
Listing 3 Insert operation message stanza
<message id="8Dpbh-26"to="[email protected]/Smack"from="[email protected]/test_1"type="groupchat"><body><basetype type="time" cefx:uid="101"><name cefx:uid="201">Second World War</name><date cefx:uid="301">1997-09-24T00:00:00:000F1000+01:00
</date></basetype></body><x xmlns="jabber:x:cefx#ins"p="100" ci="1" sv="0,1"ba="1" fn="null"/>
</message>
Mobile Netw Appl
Fig. 7 Sequence diagram - collaborative annotation
side is also updated via calling executeOperation()
method of the collaborative editing service. Clients can sendvarious update operations during the collaborative annota-tion. The workflow ends when clients leave the collabora-tion session by sending a leaveSession <jabber:iq:rpc>
set-IQ stanza.
Annotation operations Operation message stanzas repre-sent the needed information to be provided for the CMAXalgorithm. Next, abbreviations that are used within opera-tion messages are explained:
– p: position identifier of the parent node at XML Docu-ment which is defined at cefx:uid attribute
– ci: client ID of the operation sender– sv: state vector at the client which is originated before
operation is locally executed– ba: before or after, 0 for before and 1 for after– fn: position identifier of the fix node at XML Document
which is defined at cefx:uid attribute– nt: node type possible values txt for text node or attr for
attribute– ins: insert operation– del: delete operation– us: update state operation
There are two main XML nodes body and x. Body ele-ment as seen at Listing 3 carries string representation ofa new XML node with sub attributes and nodes. On theother hand, x node consists of operation specific attributes.It is crucial to have minimum payload for various operations
in order to fulfill minimum battery consuming behavior.Therefore, only required information is transmitted with thestanzas. For instance for the update operation, the payload ofthe stanza consists of only atomic modified node instead ofsending modification events with the whole correspondingsemantic annotation.
6 Evaluation
We evaluated the system in terms of technical performanceand user experience in a mobile environment. Contem-porary real-time collaboration systems are well suited forWeb-based or desktop clients. However, ensuring real-timeresponsiveness within mobile network settings is prob-lematic, due to the unstable, low-bandwidth, high-latencymobile network connections. In this part, we evaluate theXMPP-based CEFX framework for collaborative annotationcontext in mobile settings. First, performance tests for send-ing and executing remote updates were conducted and theresults were analyzed. Second, we examined the infrastruc-ture for conflict resolution and consistency maintenance insimulated mobile network settings.
In the context of the performance test, two mobile phoneswere used. We measured the time passed during a mobileclient’s generation of an operation with adding, modifyingor deleting a semantic annotation and sending the gener-ated operation to the other mobile client until correspondingoperation is executed at the client. In the test scenario, theclient prototype has generated 10 insert, 10 update and 10
Mobile Netw Appl
Table 1 XmmC vs. Facebook messaging response time in a WLANsetting
Tool Average time (ms) Standard deviation (ms)
XmmC 412 209
Facebook 3028 458
delete operations. Table 1 presents the average time col-lected during our experiments and a comparison with thesend-receive time of a 15-characters Facebook messagebetween two Android mobile devices. The XmmC valuescan be associated with the close geographical proximityto the XMPP server. As it can be observed, the valuesare acceptable in a mobile real-time collaboration scenario.However, they depend on the network characteristics.
XmmC is also capable of integrating with LAS MPEG-7multimedia services. In the executed tests, XmmC prop-agated all annotation requests to LAS MPEG-7 servicessuccessfully.
6.1 User experience evaluation
This section mainly focuses and evaluates the prototypebased on the user point of view. The role of the prototypewas expressed as a technical tool for digitally documentingthe historical sites with semantic annotations and increasingthe cultural heritage awareness. Since it is significant to getactual feedback from users that tried achieving these goals,we conducted an evaluation session with seven participants.
In the scope of the evaluation session, we came acrosswith certain challenges. First, conducting the evaluation ses-sion at a real historical site would be ideal. However, mostof the participants did not have UMTS/3G connection. Themobile devices used in the evaluation consisted of six smart-phones and a tablet. The technical details of the devices aredepicted in Table 2. Only three devices had access to anUMTS/3G data connection, the rest using Wireless LAN inthe evaluation. Therefore, we tried to simulate a historicalsite by mapping historical artifacts from Aachen’s area intoa map. The number of evaluators is limited to seven, since
we have a restricted number of mobile devices. Even thoughmore users are desirable to demonstrate the scalability of thesystem, seven participants are enough to test the conceptsin the cultural heritage scenario. All evaluation participantshad IT skills and had mixed experience with mobile devices(ranging from novice to expert users).
Evaluation process Similar to the motivating scenario, inthe evaluation session, participants were asked to digitallydocument the historical site together and explore it withAR browser. Instruction guidelines were handed out to theevaluators before the start of the evaluation session. Onceall evaluators logged in to the application, they received ashort briefing about the basic features of the application.After that, the participants were requested to perform thefollowing tasks:
– Multimedia Acquisition: Users were asked to capturecorresponding image and upload it to the system.
– Annotating the Multimedia: After acquisition, theusers were asked to annotate the multimedia togetherwith the help of these documents. They were alsoexpected to make use of chat conference feature in orderto organize annotation process.
– Augmented Reality: The evaluators were divided into2 groups and they situated themselves at different loca-tions. They were told to imagine their surroundingsas a historical site and acquire interesting things andannotate them together as the previous parts. Further-more, the evaluators used AR Camera and Map views tobrowse multimedia. They were also asked to comparegeographical location consistency of the multimedia atMap and Camera View.
At the end of the evaluation session participants were re-quested to fill a questionnaire in order to get feedback. Thecomplete user evaluation study lasted approx. three hours.
6.2 Results and discussion
The evaluation demonstrates that all participants (regardlessof their previous knowledge) can still learn new things and
Table 2 Mobile devices used at evaluation session
Dev. model Android version Network Processor Screen size
Samsung Galaxy Tab 10.1 Tablet 3.1 WLAN 1 GHz Dual Core 10.1”
Motorola Defy 2.2 UMTS 3G 800 MHz 3.7”
HTC Desire 2.2 UMTS 3G 1 Hz 3.7”
Samsung Galaxy S2 2.3.3 WLAN 1.2 Hz Dual Core 4.3”
Motorola Milestone 2.2 WLAN 550 Hz 3.7”
Samsung Galaxy S 2.3.4 WLAN 1 GHz 4.0”
HTC Desire HD 2.3.3 UMTS 3G 1 Hz 4.3”
Mobile Netw Appl
increase their cultural heritage awareness. The question-naire revealed that 57 % of the participants learned some-thing new and 14 % learned many new information. Itused a Likert-type scale (strongly agree = 5; stronglydisagree = 1).
The evaluation participants were mostly satisfied with theuser interface. They found it responsive and user friendly,as seen in Table 3. They sometimes used chat functional-ity to organize themselves for annotation and therefore allevaluators found the chat functionality useful. During thecollaborative annotation, the majority of the participants (5)perceived the real-time updates on the annotations, however,some of them (2) could not. This is understandable becausein order to see real-time updates, at least one other usershould also annotate the same multimedia, at the same time.During outdoor experience, majority of the evaluators (5)perceived real-time updates on the Points of Interest. Usersequipped with better hardware devices reported slightly bet-ter overall user experience (4.7-5) than others with slowerhardware (3.4-3.9). However, the limited number of eval-uators forbids us stating any conclusive correlation. Thestudy revealed complains about the battery impact. Actually,high battery consumption at AR browsing is an expectedresult since AR browser constantly requires live camerapreview for exploring the physical world, accelerometerand compass to calculate pitch and roll of the device,GPS to get current location, extra CPU cycles to renderPOIs in real-time and bright screen to have a better ARbrowsing experience. Moreover, user complained on theoccasional disconnections. Network outages or connectiontype changes in the underlying mobile network connection,e.g. WLAN to GPRS, cause to repeat all network socket andXMPP session establishment process. In addition, XMPPgenerates verbose XML streams, which shortens batterylife in mobile environment by over-generating network traf-fic. Moreover, XMPP requires a constant TCP connectionwhich is very energy consumptive.
The outdoor AR experience is significant to evaluate theaccuracy of the AR browsing using the Camera and Map
Table 3 Multiple choice questions responses
Question mean std. dev.
XmmC UI is user friendly? 4.29 0.45
XmmC UI is responsive? 4.29 0.45
Adapting media accuracy is meaningful? 4.14 0.64
Integrated chat is useful for collab.? 4.86 0.35
I received real-time annotation updates? 4.43 0.73
Provided reverse geocoding is useful? 4.71 0.45
Filtering POI is useful? 4.00 1.07
I have noticed real-time POI updates? 4.14 1.12
How useful is the complete system? 4.71 0.45
view. As expected, the Map view shows the locations atthe right positions. However, the questionnaire feedbackpointed out that the AR camera browser provided less accu-rate geographical location perception in certain cases. Theparticipants gave subjective ratings on how consistent werethe perceived geographical locations of POIs at Camera ARview and the actual Map view. The results showed meansubjective accuracy of 77 % with standard deviation of12 %.
In the context of the evaluation session, the 7 partic-ipants acquired 13 multimedia artifacts and created andmodified 47 semantic annotations collaboratively. All of theevaluators found the application successful and useful ingeneral.
6.3 Evaluation summary
We evaluated the implemented system from technical anduser point of view. In the technical part, we tested the pro-totype for storing and transferring for various media types.Then, the feasibility of XMPP-based CEFX+ framework ina mobile collaborative annotation scenario was evaluatedfor real-time performance and consistency maintenance. Inthe user experience based evaluation part, we conducted anevaluation session to examine XmmC while evaluators usedthe prototype for collaboratively documenting a historicalsite and increasing their cultural heritage scenario. After theevaluation, we compared the acquired multimedia artifactsat XmmC and LAS MPEG-7 multimedia services.
In the light of these results, we can conjecture thatXmmC is serving its purpose. However, a larger scale eval-uation on more historical sites by professionals will allowus to improve the prototype.
7 Conclusions and future work
Mobile technologies allow professional communities totransform their collaborative work practices in the field.Since a lot of scenarios for mobile collaboration demandreal-time functionality, more and more professional com-munities adopt such services. Cultural heritage managementis a practice we accompanied since many years [15]. Here,we presented a collaborative semantic-enhanced multime-dia annotation system. We developed collaborative annota-tion and editing services and integrated them with existingmobile augmented-reality browsers and existing multimediaservices based on the MPEG-7 standard. Real-time supportwas realized by deploying the Extensible Messaging andPresence Protocol (XMPP). These services have been usedin a cultural heritage documentation scenario which exposedmany of the addressed requirements. Evaluation indicatesthat such solutions increase the awareness of community
Mobile Netw Appl
members for activities of co-workers and the productivity inthe field in general.
In future work, we will extend the scenarios in twodirections. First, we will deploy more advanced augmentedreality services which do not just rely on GPS for imageregistration but use state-of-the-art video tracking algo-rithms. Second, we will enlarge the user communities andinclude trust and security management. These improve-ments will enable the collaboration within different profes-sional communities such as medical and construction work-ers, which is an objective set in the Learning Layers (ScalingUp Technologies for Informal Learning in SME Clusters)project,2 funded by the European Commission under theseventh Framework Programme.
References
1. Agius H, Angelides MC, Daylamani ZD (2012) Experimentingwith tagging and context for collaborative MPEG-7 metadata.Multimed Tools Appl 1–35
2. Bailer W, Brunie L, Doller M, Granitzer M, Klamma R, KoschH, Lux M, Spaniol M (2008) Multimedia metadata standards. In:Furht B (ed) Encyclopedia of multimedia. Springer, pp 568–575
3. Benkler Y (2006) The wealth of networks: how social produc-tion transforms markets and freedom. Yale University Press, NewHaven
4. Cao Y, Jarke M, Klamma R, Mendoza O, Srirama S (2009) Mobileaccess to MPEG-7 based multimedia services. In: 2009 tenthinternational conference on mobile data management: systems,services and middleware. IEEE, Taipei, pp 102–111
5. Davis AH, Sun C, Lu J (2002) Generalizing operational transfor-mation to the standard general markup language. In Proceedingsof the 2002 ACM conference on computer supported cooperativework (CSCW’02). ACM Press, pp 58–67
6. Ellis CA, Gibbs SJ (1989) Concurrency control in groupware sys-tems. In: Proceedings of the 1989 ACM SIGMOD internationalconference on management of data. ACM Press, pp 399–407
7. Ellis CA, Gibbs SJ, Rein G (1991) Groupware: some issues andexperiences. Commun ACM 34:39–58
8. Gerlicher ARS (2007) Developing collaborative XML editingsystems. Ph.D. thesis, University of the Arts London
9. Gotow JB, Zienkiewicz K, White J, Schmidt DC (2010) Address-ing challenges with augmented reality applications on Smart-phones. In: Proceedings of the third international conference onmobile wireless middleware, operating systems, and applications(Mobileware 2010), LNCS, vol 48. Springer, pp 129–143
10. Gu YX, Li N, Chang L, Duh HBL (2011) A collaborative aug-mented reality networked platform for edutainment. Springer,pp 117–126
11. Hollerer TH, Feiner SK (2004) Telegeoinformatics: location-based computing and services, chap. Mobile augmented reality.Taylor and Francis Books Ltd
12. Ignat CL, Norrie MC (2003) Customizable collaborative editorrelying on TreeOPT algorithm. In: Proceedings of the eighthconference on computer supported cooperative work. Kluwer,Norwell, pp 315–334
13. Ignite Realtime: Openfire XMPP Server. [Online], http://www.igniterealtime.org/projects/openfire/. Accessed April 2011
2http://learning-layers.eu
14. Ignite Realtime: Smack XMPP Library. [Online], http://www.igniterealtime.org/projects/smack/. Accessed April 2011
15. Klamma R, Spaniol M, Jarke M, Cao Y, Jansen M, Toubekis G(2005) ACIS: intergenerational community learning supported bya hypermedia sites and monuments database. In: Proceedings ofthe 5th international conference on advanced learning technolo-gies (ICALT 2005), Kaohsiung, Taiwan, July 5–8. IEEE ComputerSociety, Los Alamitos, pp 108–112
16. Kosch H (2003) Distributed multimedia database technologiessupported by MPEG-7 and MPEG-21. CRC Press, Boca Raton etal
17. Lechner M (2010) ARML—augmented reality markup language.Tech. rep. Mobilizy GmbH, http://www.perey.com/ARStandards/Mobilizy ARML.eps
18. Lowry PB, Curtis AM, Lowry MR (2004) A taxonomy of collabo-rative writing to improve empirical research, writing practice, andtool development. J Bus Commun (JBC) 41(1):66–99
19. Lugiez D, Martin S (2009) Peer to peer optimistic collaborativeediting on XML-like trees. Online http://arxiv.org/abs/0901.4201.Accessed March 2013
20. Meyer EM, Wichmann D, Busch H, Boll S (2012) Support-ing mobile collaboration in spatially distributed workgroups withdigital interactive maps. Mob Netw Appl 17(3):365–375
21. Mixare: Mixare augmented reality engine. [Online], http://www.mixare.org/. Accessed April 2011
22. Muldowney T, Miller M, Eatmon R (2004) XEP-0096: SI filetransfer. XEP-0096(Standards Track). http://xmpp.org/extensions/xep-0096.html
23. Nichols DA, Curtis P, Dixon M, Lamping J (1995) High-latency,low-bandwidth windowing in the Jupiter collaboration system. InProceedings of the 8th annual ACM symposium on user interfaceand software technology, UIST ’95. ACM, pp 111–120.
24. Novell, Inc: Novell vibe cloud (2011) [Online] Accessed April.https://vibe.novell.com/
25. Saint-Andre P (2008) XEP-0045: multi-user chat,XMPP XEP-0045 (Standards Track), http://xmpp.org/extensions/xep-0045.htm
26. Saint-Andre P (2011) RFC 6121: extensible messaging and pres-ence protocol (XMPP): instant messaging and presence. Tech.rep., XMPP Standards Foundation
27. Schmidt A (2012) At the boundaries of peer production: the orga-nization of internet security production in the cases of Estonia2007 and Conficker. Telecommun Policy 36(6):451–461
28. Schuster D, Springer T, Schill A (2010) Service-based develop-ment of mobile real-time collaboration applications for social net-works. In: Proceedings of IEEE PerCom workshops (PerCol’10),pp 232–237
29. Spaniol M, Klamma R, Janßen H, Renzel D (2006) LAS: alightweight application server for MPEG-7 services in communityengines. In: Proceedings of I-KNOW ’06, 6th international con-ference on knowledge management, September 6–8, 2006. Graz,pp 592–599
30. Stanford Mobile and Social Computing Research Group: Junc-tion Documentation for Application Developers [Online],http://mobisocial.stanford.edu/index.php?page=junction.Accessed March 2013
31. Suleiman M, Cart M, Ferri J (1997) Serialization of con-current operations in a distributed collaborative environment.In: Proceedings of the international ACM SIGGROUP confer-ence on supporting group work (GROUP’97). ACM, pp 435–445
32. Sun C, Jia X, Zhang Y, Yang Y, Chen D (1998) Achieving conver-gence, causality preservation, and intention preservation in real-time cooperative editing systems. ACM Trans Computer-HumanInteract 5:63–108
Mobile Netw Appl
33. Sun D, Sun C (2006) Operation context and context-basedoperational transformation. In: Proceedings of the 2006 20thanniversary conference on computer supported cooperative work(CSCW’06). ACM Press, pp 279–288
34. Tapscott D, Williams AD (2008) Wikinomics: how mass collabo-ration changes everything. Portfolio
35. Voigt M (2009) Erweiterung und Anpassung des collaborativeediting framework for XML (CEFX). Master’s thesis, Universityof Applied Sciences Erfurt
36. Wang D, Mah A, Lassen S (2010) Google wave operationaltransformation. Whitepaper, Google Inc
37. Wang X Layar augmented reality browser. [Online]. http://layar.pbworks.com/. Accessed April 2011
38. Wikitude GmbH (2013) Wikitude mobile augmented reality app.[Online], http://www.wikitude.com/app. Accesses March 2013
39. Wilkinson DM (2008) Strong regularities in online peer produc-tion. In: Proceedings of the 9th ACM conference on electroniccommerce—EC ’08. ACM Press, p 302
40. Xia S, Sun D, Sun C, Chen D, Shen H (2004) Leveragingsingle-user applications for multi-user collaboration: the Cowordapproach. In: Proceedings of the 2004 ACM conference on com-puter supported cooperative work (CSCW’04). ACM Press, pp162–171
41. You Y, Belimpasakis P, Selonen P (2010) A hybrid content deliv-ery approach for a mixed reality web service platform. In Ubiq-uitous intelligence and computing, LNCS, vol 6406. Springer, pp563–576