Employing Tangible Visualisations inAugmented Reality with MobileDevices

Sebastian HubenschmidHCI GroupUniversity of Konstanzsebastian.hubenschmid@uni.kn

Johannes ZagermannHCI GroupUniversity of Konstanzjohannes.zagermann@uni.kn

Simon ButscherHCI GroupUniversity of Konstanzsimon.butscher@uni.kn

Harald ReitererHCI GroupUniversity of Konstanzharald.reiterer@uni.kn

Figure 1: Augmented Reality above the Tabletop (ART) [3] isdesigned to facilitate the collaborative analysis of multidimensionaldata. A 3D parallel coordinates visualisation in augmented reality isanchored to a touch-sensitive tabletop, enabling familiar operation.

Copyright is held by the author/owner(s).AVI’18, May 29 – June 1, 2018, Grosetto, Italy.ACM 978-1-XXXX-XXXX-X/XX/XX.

AbstractRecent research has demonstrated the benefits of mixedrealities for information visualisation. Often the focus lieson the visualisation itself, leaving interaction opportunitiesthrough different modalities largely unexplored. Yet, mixedreality in particular can benefit from a combination of differ-ent modalities. This work examines an existing mixed realityvisualisation which is combined with a large tabletop fortouch interaction. Although this allows for familiar operation,the approach comes with some limitations which we ad-dress by employing mobile devices, thus adding tangibilityand proxemics as input modalities.

Author KeywordsImmersive Analytics; augmented reality; collaboration; mul-timodal interaction; mobile device; data analysis

ACM Classification KeywordsH.5.2 [Information interfaces and presentation (e.g., HCI)]:Input devices and strategies (e.g., mouse, touchscreen)

IntroductionHardware for mixed reality systems has now become widelyavailable, paving the way for widespread adoption of aug-mented reality (AR) and virtual reality (VR). Especially in-formation visualisations can benefit from this development,as mixed reality systems can, for example, facilitate com-

prehension of 3D visualisations and offer a large space forvisualisations. Yet, existing systems are often more con-cerned with exploring new visualisation possibilities, sointeraction opportunities are largely ignored. Interactionin current systems is often restricted to a combination ofspatial movement with either touch or 3D controllers, whileother modalities, such as proxemics or tangibility, are leftunexplored. This restriction often leads to several shortcom-ings concerning the interaction and collaboration of thesesystems. In this work we therefore explore the addition ofsuch modalities to an existing mixed reality visualisationsystem and show how this multimodal approach can ad-dress key shortcomings.

ART – Augmented Reality above the Tabletop

Figure 2: Mock-ups for linkingscatter plots. (a) Each mobiledevice represents a single scatterplot. (b) Links appear when scatterplots are close together. (c) If ascatter plot is too far away, no linksare established. (d) Links are thusbased on the proxemic distancebetween mobile devices.

Augmented Reality above the Tabletop (ART) [3] is a datavisualisation system that combines an immersive AR en-vironment1 with touch input on an interactive tabletop tocontrol a 3D visualisation (see Figure 1). The visualisa-tion is composed of linked 2D scatter plots, creating a 3Dparallel coordinates visualisation that hovers directly abovethe tabletop. A control panel on the tabletop allows users tointeract with the visualisation and individual scatter plots.

The ART system offers both touch input and spatial navig-ation in AR for performing different tasks. For plot arrange-ment tasks (adding, reordering, removing scatter plots) andscatter plot configuration tasks (assign dimensions, defineclusters and filters) the touch input provides a familiar in-terface. Navigation is performed through a combination oftouch input (scrolling on the table) and spatial movementaround the tabletop. Additionally, collaboration is supportedthrough the independent spatial movement of each user.

1The AR environment is realised in Unity3D, using a HTC Vive head-set with front-mounted stereoscopic cameras.

In a preliminary evaluation, the touch input for scatter plotconfiguration tasks was beneficial due to its precisenesswhen defining clusters. Although spatial movement allowedparticipants to navigate the visualisation intuitively, the fixedsetup and large size of the tabletop made it difficult to movearound the visualisation, forcing users to limit themselves toview the visualisation from the front. Still, spatial movementwas appreciated for collaborative tasks, as users were ableto view the visualisation from different angles. Because thevisualisation is shared between users, changes made byone user also affect all other users – which may not alwaysbe intended. Users also wanted to interact more directlywith the visualisation (e.g. grabbing the visualisation), espe-cially for navigation and plot arrangement tasks. Althoughgesture interaction was considered, current technologicallimitations (e.g. detection, accuracy, occlusion with virtualobjects) made gestures too error prone for use within ART.

ARts – Augmented Reality with TabletsAugmented Reality with tablets (ARts) aims to move awayfrom a monolithic tabletop towards multiple location-aware2

mobile devices (e.g. tablets). This is a similar approach asapplied for VisTiles [8]. We thus open up the design spacethrough tangibility and proxemics, while keeping touch in-put for suitable tasks. Instead of concentrating the entirevisualisation on a single device, each mobile device rep-resents a single scatter plot (see Figure 2). Guided by theproxemic dimensions [6], we are able to support several in-dividual and collaborative tasks: Distance as a measure forlinking scatter plots, orientation and location for adjustingthe visualisation (e.g. flipping the visualisation through ro-tation), and movement for indicating possible collaborationopportunities. The following sections describe the shift inmodalities regarding plot arrangement, spatial linking, scat-ter plot configuration, navigation, and collaboration tasks.

2For example, by attaching HTC Vive Trackers to each mobile device.

Plot ArrangementBy distributing individual scatter plots onto different mo-bile devices, the modality for plot arrangement shifts to-wards tangibility. The physical affordance of mobile devicesalso avoids some of the inherent difficulties with gestureinteraction, such as the touching the void [2] issue. Userscan therefore arrange plots by physically arranging mobiledevices, allowing for flexible linked visualisation layouts(see Figure 3) similar to systems such as VisLink [4] orImAxes [5]. This also enables users to fully utilise their sur-roundings, for example by mounting visualisations onto awall (see Figure 4).

Figure 3: Mock-up for arrangingscatter plots. Mobile devices allowfor a flexible arrangement of scatterplots, such as a two-to-one layout.

Figure 4: Mock-up for room usage.Mobile devices can be arranged inany location in the room, forexample placed on a table orattached to a magnetic wall.

Spatial LinkingLinks between scatter plots are automatically establishedbased on the proxemic distance between mobile devices(see Figure 2). This allows for dynamic linking of scatterplots similar to ImAxes [5] and enables us to explore cross-device interaction similar to VisTiles [8].

Scatter Plot ConfigurationConfiguration of individual scatter plots can be performedthrough a familiar touch interface on the mobile deviceitself. This approach allows us to focus the control panel UImainly on interaction with the scatter plot (e.g. see Sadanaand Stasko [9] for scatter plot interaction on a tablet).

NavigationThe small size of the mobile devices allows users to easilynavigate the entire visualisation through spatial movement.Users can also simply move or rotate their arrangement ofmobile devices, allowing for navigation through tangibility.

CollaborationPrevious research [1] suggests that co-located collabora-tion is more efficient when each participant has control overparts of the data. By distributing the visualisation onto differ-

ent mobile devices, users can either work on different plotsof the same visualisation simultaneously, or work on entirelyindependent visualisations (see Figure 5). Especially inthe latter case proxemics can be useful to indicate whentwo collaborators want to share their results (e.g. distanceor movement of collaborators), thus supporting differentcollaboration styles (e.g. as classified by Isenberg et al. [7]).

ChallengesOur design space of a multimodal, mobile-devices-basedvisualisation system in AR brings up several technical andinteraction design challenges that still need to be solved.

Technical ChallengesOne of the main technical challenges with this approachis to correctly model the linking behaviour between scat-ter plots. Systems like VisTiles [8] show that distance-based linking can be prone to accidental activation or sufferfrom hidden functionality. Even though ImAxes [5] alreadyprovides a rule set for linking visualisations in VR, the phys-ical surroundings (e.g. tables, walls) may influence spatiallinking. While users can freely position scatter plots in VR,in AR mobile devices must be placed on a physical sur-face, thus limiting freedom for spatial linking significantly.Furthermore, linking mechanisms should ideally avoid unin-tentional line overlap (e.g. when inserting a new scatter plotin between two linked scatter plots), while still allowing forenough flexibility to not restrict users.

Another challenge is that the size of the visualisation is in-herently limited by the number of available mobile devices.Although users can be provided with a reasonable num-ber of devices, it may be beneficial to provide methods forfreeing up devices: For example, users could decouple visu-alisations from their devices, thus keeping a static versionof the visualisation in the AR environment, which may be

recoupled with a device later on. Another option is to con-dense the visualisation onto a single device, thus freeing upthe other devices.

Interaction Design ChallengesThe physical position of mobile devices in the AR environ-ment may influence the user’s perception of the visualisa-tion. Especially when comparing correlations between twoneighbouring scatter plots, differing heights may lead tofalse conclusions if the user is unaware of the height differ-ences. An indication may be essential to make users awarein such cases. Furthermore, while links between two scatterplots can be determined through proxemics, the links them-selves may lack interactivity. For this, gesture interactionmay be necessary, but exceeds the scope of this work.

Figure 5: Mock-ups forcollaboration. (a) Users workingconcurrently on different scatterplots. (b) Users combining theirresults into a shared visualisation.(c) Users working concurrently onindividual visualisations.

ConclusionWe employ mobile devices to create a tangible visualisationsystem in AR. The initial system applied a combination oftouch input with spatial movement in AR, which allowedfor precise and familiar interaction. However, this narrowfocus on touch interaction with a single device led to severalshortcomings in terms of plot arrangement and collabora-tion. Our proposed approach of replacing the tabletop withmultiple mobile devices opens up the design space by intro-ducing tangibility and proxemics. This could result in a moreflexible and collaboration-friendly visualisation system.

REFERENCES1. A. D. Balakrishnan, S. R. Fussell, S. Kiesler, and A.

Kittur. 2010. Pitfalls of Information Access withVisualizations in Remote Collaborative Analysis. Proc.of CSCW’10 (2010), 411.

2. G. Bruder, F. Steinicke, and W. Stuerzlinger. 2013.Touching the Void Revisited: Analyses of TouchBehavior on and above Tabletop Surfaces. In Proc. ofINTERACT’13, Vol. 8117 LNCS. Springer, Berlin, DE,

278–296.

3. S. Butscher, S. Hubenschmid, J. Müller, J. Fuchs, andH. Reiterer. 2018. Clusters, Trends, and Outliers: HowImmersive Technologies Can Facilitate theCollaborative Analysis of Multidimensional Data. InProc. of CHI’18. ACM.

4. C. Collins and S. Carpendale. 2007. VisLink: RevealingRelationships Amongst Visualizations. IEEETransactions on Visualization and Computer Graphics13, 6 (2007), 1192–1199.

5. M. Cordeil, A. Cunningham, T. Dwyer, B. H. Thomas,and K. Marriott. 2017. ImAxes: Immersive Axes asEmbodied Affordances for Interactive Multivariate DataVisualisation. In Proc. of UIST’17. ACM, New York, NY,USA, 71–83.

6. S. Greenberg, N. Marquardt, T. Ballendat, R.Diaz-Marino, and M. Wang. 2011. Proxemicinteractions. Interactions 18, 1 (2011), 42.

7. P. Isenberg, D. Fisher, M. R. Morris, K. Inkpen, and M.Czerwinski. 2010. An Exploratory Study of Co-locatedCollaborative Visual Analytics around a TabletopDisplay. In Proc. of VAST. IEEE, Los Alamitos, CA,USA.

8. R. Langner, T. Horak, and R. Dachselt. 2018. VisTiles:Coordinating and Combining Co-located MobileDevices for Visual Data Exploration. IEEE Transactionson Visualization and Computer Graphics 24, 1 (2018),626–636.

9. R. Sadana and J. Stasko. 2014. Designing andImplementing an Interactive Scatterplot Visualizationfor a Tablet Computer. Proc. of AVI’14 (2014),265–272.