Towards Utilizing Novel Interactive Displaysfor Information Visualization

Christian Tominski Heidrun SchumannInstitute for Computer Science

University of Rostock{ct|schumann}@informatik.uni-rostock.de

Martin Spindler Raimund DachseltUser Interface & Software Engineering Group

Otto-von-Guericke-University Magdeburg{spindler|dachselt}@ovgu.de

ABSTRACTVisualization research has yielded a number of useful tech-niques to generate visual representations and to allow usersto explore data interactively on computer screens. Yet, theconstantly growing complexity of today’s data calls for newways of visual and interactive data analysis. This is wherenew display and interaction technologies offer promising pos-sibilities yet to be explored. In this work, we identify threegaps to be addressed by future research. (1) The technol-ogy gap is about the lack of a systematic mapping of com-mon interaction tasks onto interaction modalities. (2) Theintegration gap concerns the integration of novel interactiontechniques and technologies with existing information visu-alization approaches to create new powerful visualization so-lutions. (3) The guidelines gap criticizes the shortage of sup-port for users to choose suitable solutions for the task at hand.We outline related challenges and ideas for future work bythe example of our work on tangible views for informationvisualization.

ACM Classification: H5.2 [Information interfaces and pre-sentation]: User Interfaces. - Graphical user interfaces.

General terms: Design, Human Factors

Keywords: Tangible views, information visualization, magiclenses, gestural input, multitouch, pen, gestures, diagrams.

INTRODUCTIONThe goal of visualization is to support people in formingmental models of otherwise difficult-to-grasp subjects, suchas massive data, complex models, or dynamic systems [12].The term forming implies that visual output is not the endproduct of visualization. It is rather the process of adaptingthe visual output and interacting with the data in order to gaininsight.

For several decades, visualization researchers have devel-oped a wealth of visualization and interaction concepts formany different types of data and tasks. What most of theexisting techniques have in common is that they are targetedfor regular desktop workplaces with a computer display, akeyboard, and a mouse. With the advent of new displaytechnologies, such as large high-resolution displays or smallhand-held displays, it became necessary to adapt existing vi-sualization approaches or to devise new ones. Recently, mod-ern interaction technologies, such as multi-touch interactionor tangible interaction have considerably broadened the spec-trum of what’s possible and created a need for rethinking ex-

isting visualization solutions with regard to interaction. Theseamless integration of both display and interaction in a sin-gle touch-enabled device, such as interactive tabletops andtablets, makes direct manipulation [11] truly direct. By ex-ploring information directly under your fingertips, the form-ing of mental models for interactive visualizations seems tobe particularly well supported and promising.

In this paper, we aim to describe several issues concerningthe future development of information visualization in thecontext of new interactive surfaces and interaction technolo-gies. We identify gaps in the state of the art, illustrate themwith our own previous work [3, 13] and motivate possiblenext steps for future research. Here, our main concern is re-lated to the systematic investigation of the possibilities of theclassic as well as the promising new technologies, on the onehand, and the well-justified application of these possibilitiesto solve visualization and interaction tasks, on the other hand.

IDENTIFYING THE GAPSIn the following, we identify three research gaps worth beingaddressed by future research.

Technology GapVisualization research builds upon commonly accepted strate-gies for visualizing data. In his classic book, Bertin [1] intro-duces visual variables and defines how data is to be mappedonto them. Cleveland and McGill [2] investigate the effec-tiveness of visual encodings for data exploration. Thus, ona conceptual and on an experimental level, we have backedknowledge how to transform data D to visual representationsV using the mapping vis : D → V .

However, there is no such commonly accepted mapping interms of interaction. So far, mouse and keyboard have beenthe basic and dominant devices for user interaction. Ad-vances in technology have recently added new modalitiesto the interaction toolbox. Multi-touch interaction, gestu-ral interfaces, pen-based interaction and tangible interactionare only a few examples. What still has to be developed isthe mapping interact : T → M that defines how interac-tion tasks T are effectively and efficiently carried out withthe different interaction modalities M available. Obviously,specifying appropriate sets T and M turns out to be a nec-essary and challenging condition for successfully develop-ing novel interactive visualizations. Therefore, a repertoireof suitable interaction techniques has to be defined and de-scribed in a consistent way, which eventually allows for an

(a) Semantic zooming of node-link diagramsby lifting/lowering a tangible view [13].

(b) Tangible views applied to compare differ-ent parts of a matrix visualization [13].

(c) Manipulating node-link diagrams by us-ing multi-touch and pen input [3].

Figure 1: Utilizing novel interactive displays for information visualization.

easy task mapping. Implementing the new techniques willalso form the basis for future visualization applications.

Integration GapClosing the technology gap will result in a new repertoire ofinteraction techniques. However, by now users of interactivevisualizations mostly apply techniques that are designed forclassic desktop computers. Utilizing novel interactive dis-plays for visualization purposes has not received much atten-tion so far. So there is a gap in terms of promising new pos-sibilities on the one hand, but only little integration of thesepossibilities into visualization research and applications onthe other hand.

Yet, there are first approaches that specifically address the in-tegration of visualization and interactive display technology.For instance, Isenberg et al. [5, 6] utilize interactive table-top displays for collaborative visualization, Voida et al. [14]discuss design considerations for interacting with data on in-teractive surfaces, Spindler et al. [13] contribute the conceptof tangible views for information visualization, Heilig et al.explore multitouch input for interacting with scatterplots [4],and Kosara [7] investigates multi-touch brushing for parallelcoordinates. However, these first visualization approachesusing novel interactive displays primarily address very spe-cific problems. The broad range of possibilities of the newtechnology have by far not been explored sufficiently nor an-alyzed appropriately.

Closing this gap by systematically exploring the design spacefor combining modern visualization approaches with recentinteraction technologies will lead to novel solutions for to-day’s data exploration challenges.

Guidelines GapWith the combination of different visualization techniquesand interaction technologies, a vast body of possible solu-tions becomes available. This immense variety of existingand possible new approaches makes it difficult for users todecide which techniques to use. What is needed in the futureare guidelines or rules for choosing effective approaches forthe data, tasks, and device context at hand.

An excellent example of systematically choosing “good” vi-sualizations is Mackinlay’s [10] pioneering work on automat-

ing the design of visual representations. The beauty of thisapproach is that it enables automatic suggestion of visualvariables based on a given data type (quantitative, ordinal,nominal). This is possible thanks to the well-defined setsof data types and visual variables, which abstract from thesubtle details of real world problems. It is part of ongoing re-search how the details of today’s often complex visualizationapplication scenarios can be integrated.

Wouldn’t it be great if we had a similar system to which weinput our data D and our tasks T , and the system would tellus which visualization techniques V and interaction modali-ties M to use given a particular input and output setup? Ob-viously, the required mapping guide : D × T → V ×Mwill be difficult to define. We consider solving this researchquestion a formidable and rather long term task.

DISCUSSING THE GAPSNarrowing and eventually closing the aforementioned gapswill require much research. It is beyond the scope of this ar-ticle to comprehensively suggest directions for future work.We would rather like to use an example to illustrate possibleavenues of investigation.

We chose the example of tangible views for information vi-sualization [13] for the following reasons. Tangible viewsillustrate quite well the new possibilities of advanced tech-nology with a set of different interactive visualizations (seeFigures 1(a) and 1(b)). They also serve as a good illustrationof what is still missing. Finally, since tangible views are ourown previous work, it is easier to criticize and to envisionresearch goals.

Conceptually, tangible views are spatially aware lightweightdisplays that serve two purposes simultaneously: visual out-put and direct input. Multiple of such tangible views areused together with an interactive tabletop display to builda multi-display multimodal visualization ensemble that sup-ports both interacting on the views (by using touch and peninput) and interacting with the views (by moving them throughthe physical space above a tabletop and performing gestures).An interaction vocabulary (see Figure 2) has been compiledas the basis upon which manifold applications can be devel-oped. Several example visualization cases illustrate how tan-gible views can be utilized to display and to interact with

VerticalTranslation

HorizontalTranslation

(a) Translation

VerticalRotation

HorizontalRotation

(b) Rotation

FreezeVertical Freeze

Horizontal Freeze

(c) Freezing

BACK

FRONT

Flipping Tilting Shaking

(d) Gestures

Figure 2: Extract of the interaction vocabulary provided by tangible views. The figures show with-the-view interactionsonly, on-the-view interactions, such as pen and touch input, can be found in [13].

data. These use cases include scatter plots and parallel coor-dinates plots of multivariate data, node-link representationsand matrix representations of graph data, as well as map-based visualization of spatiotemporal data.

Addressing the Technology GapIn order to arrive at a mapping interact : T → M , wefirst need a specification of the set of interaction tasks T .There are approaches that provide first categorizations of in-teraction. Yi et al. [15] describe a list of general interactionintents in visualization. Besides these general descriptions,more specific categorizations exist. For instance, dedicatedinteraction tasks for exploring graphs are described by Leeet al. [9]. These are valuable starting points for defining acomprehensive set of interaction tasks. Most likely, this setwill contain tasks of different complexity ranging from verybasic selection to common brushing and linking to the morecomplex applications of logically combinable dynamic filter-ing.

Secondly, defining an interaction vocabulary as in [13] is avalid first step for closing the technology gap. Such a vo-cabulary serves as a container that holds technically possiblesolutions to be utilized for interaction tasks. The tangibleviews vocabulary focuses on interaction with spatially awarelightweight displays. However, it is not comprehensively ad-dressing the different classes of interactive displays in gen-eral. So, future work has to systematically extend the inter-action vocabulary with further interaction modalities M .

An example for a successful interact mapping can be givenfor the task of exploring spatio-temporal data with tangibleviews. Such data can be mapped to the virtual volume abovea tabletop, where the XY-dimensions encode spatial loca-tion and the Z-dimension represents time (i.e., space-time-cube [8]). In order to control which part of the geo-spaceand which time step are visible (∈ T ), the user can trans-late the tangible view horizontally and vertically (∈ M ), asshown in Figure 2(a).

Another example is the adjustment of a visualization param-eter, e.g., the distortion factor of a fisheye lens (∈ T ), whichcan be mapped onto rotating the view horizontally (∈M ) asshown in Figure 2(b).

Addressing the Integration GapClosing the integration gap, that is bringing together visual-ization research and new interactive displays, involves manydifferent aspects. To name only a few, integration is nec-

essary on a conceptual level (e.g., utilizing tangible viewsfor focus+context visualization), on a software level (e.g.,combining different visualization and interaction toolkits), aswell as on a hardware level (e.g., integration of lightweightdisplays with touch and pen input and tabletop displays). Be-cause we cannot detail all aspects here, we will resort to il-lustrating the integration of exploration and manipulation ofnode-link diagrams of graphs as an example.

Usually, exploration tasks and manipulation tasks are con-sidered separately from each other. While exploration islargely addressed in the visualization community, manipula-tion tasks are more relevant in the realm of human-computerinteraction. For instance, with tangible views we mainly sup-port the exploration of node-link diagrams by utilizing thewith-the-view interaction modalities (see Figure 1(a)). Otherworks address the authoring and manipulation of the under-lying graph data, e.g., by using multi-touch and pen input fordiagram editing as shown in Figure 1(c) [3]. Taking advan-tage of both worlds by integrating them into a single unifiedsystem would clearly be useful, not only because users couldaccomplish multiple tasks within the same familiar environ-ment, but also because data exploration often involves datamanipulation (at least temporary) for testing different “whatif” scenarios.

However, such integration also implicates several challenges.On a conceptual level, distinctive features of different in-teraction modalities and visualization techniques need to becombined appropriately for different tasks. This could beachieved, for example, by utilizing with-the-view interac-tion for exploration aspects, while the more precise on-the-view interaction could be used for manipulation tasks. Seam-less switching between these tasks could be accomplished bychoosing different tools or even different interaction modal-ities, e.g., touch input for zooming/panning and pen inputfor graph editing. On a software level, different softwareworlds need to be consolidated into a single framework thataddresses issues such as distributed rendering required for amulti-display setup, state synchronization between differentdevices, and most importantly the incorporation and adap-tion of visualization techniques that meet the requirementsof such a setup.

Addressing the Guidelines GapThe developed example cases of tangible views indicate thatthere is much potential in utilizing new technologies for in-formation visualization. Although being interesting exam-

ples, it remains unclear why and how tangible views are usedunder which circumstances and when alternative solutionsmight be better suited (as one reviewer of [13] once pointedout). So, there are often questions like Would you really carryout this task with tangible views? or Wouldn’t this be easierto accomplish with basic mouse interaction?

Even though the introduction of an interaction vocabulary isan important step, there are still no definite rules for its appli-cation. In order to make information visualization on mod-ern interactive displays a viable approach, we should striveto provide concrete answers and guidelines much like in thespirit of Bertin, Cleveland and McGill, and Mackinlay.

However, developing approaches for guiding the user in choos-ing the “right tool” is ongoing research, which is challengingfor the following reasons. First, it is usually more difficult tocategorize the data, because today’s data sets are increasinglycomplex and heterogeneous. Furthermore, one has to takethe users’ tasks and goals into account with regard to both:what the users want to see and how they would like to inter-act. In terms of the output, a step has to be made from simplevisual variables to more complex visualization techniques,and possibly to combinations thereof. The aspect of inter-action is entirely missing in classic works. Given some dataand a suitable visualization, how can the user effectively in-teract to accomplish the tasks and to achieve the goals? Andfinally, it is no longer just a question of which visualizationtechnique to use for which data and task, but rather one ofwhich display and interaction technologies to use for whichvisualization techniques, data, and tasks.

CONCLUSIONFor taking full advantage of novel display and interactiontechnologies for information visualization, several gaps haveto be addressed as identified and illustrated in this paper.First, a categorization of interaction tasks and a repertoireof novel interaction techniques have to be described, whichthen allows for mapping specific tasks to particular tech-niques. Secondly, the design space of combining novel in-teraction concepts and existing visualization approaches hasto be explored appropriately. Thirdly, guidelines have to bedeveloped for choosing appropriate and effective approacheswithin a vast body of possible solutions. Filling these gapsstep by step is a formidable task that can only be accom-plished by a vivid research community bringing together vi-sualization and interaction experts.

REFERENCES1. Jacques Bertin. Semiology of Graphics: Diagrams,

Networks, Maps. University of Wisconsin Press, 1983.

2. William S. Cleveland and Robert McGill. GraphicalPerception: Theory, Experimentation, and Applicationto the Development of Graphical Methods. Journal ofthe American Statistical Association, 79(387):531–554,1984.

3. Mathias Frisch, Jens Heydekorn, and RaimundDachselt. Diagram Editing on Interactive Displays Us-ing Multi-Touch and Pen Gestures. In Proc. of Dia-grams, pages 182–196. Springer, 2010.

4. Mathias Heilig, Stephan Huber, Mischa Demarmels,and Harald Reiterer. Scattertouch: a multi touch rub-ber sheet scatter plot visualization for co-located dataexploration. In Proc. of the ACM International Confer-ence on Interactive Tabletops and Surfaces (ITS), ITS’10, pages 263–264. ACM, 2010.

5. Petra Isenberg and M. Sheelagh T. Carpendale. Interac-tive Tree Comparison for Co-located Collaborative In-formation Visualization. IEEE Transactions on Visual-ization and Computer Graphics, 13(6), 2007.

6. Petra Isenberg and Danyel Fisher. Collaborative Brush-ing and Linking for Co-located Visual Analytics ofDocument Collections. Computer Graphics Forum,28(3), 2009.

7. Robert Kosara. Indirect Multi-Touch Interaction forBrushing in Parallel Coordinates. In Proc. of theConference on Visualization and Data Analysis (VDA),pages 786809–1–786809–7. SPIE/IS&T, 2011.

8. Menno-Jan Kraak. The Space-Time Cube Revisitedfrom a Geovisualization Perspective. In Proceedings ofthe 21st International Cartographic Conference (ICC),pages 1988–1995, Newcastle, UK, 2003. The Interna-tional Cartographic Association (ICA).

9. Bongshin Lee, Catherine Plaisant, Cynthia Sims Parr,Jean-Daniel Fekete, and Nathalie Henry. Task Taxon-omy for Graph Visualization. In Proc. of the AVI work-shop BELIV, pages 1–5. ACM, 2006.

10. Jock Mackinlay. Automating the Design of GraphicalPresentations of Relational Information. ACM Trans-actions on Graphics, 5(2):110–141, 1986.

11. B. Shneiderman. Human-computer interaction. chap-ter Direct Manipulation: A Step Beyond ProgrammingLanguages, pages 461–467. Morgan Kaufmann, 1987.

12. R. Spence. Information Visualization: Design for In-teraction. Prentice-Hall, Inc., Upper Saddle River, NJ,USA, 2nd edition, 2007.

13. Martin Spindler, Christian Tominski, Heidrun Schu-mann, and Raimund Dachselt. Tangible Views for In-formation Visualization. In Proc. of the ACM Inter-national Conference on Interactive Tabletops and Sur-faces (ITS), pages 157–166. ACM, 2010.

14. Stephen Voida, Matthew Tobiasz, Julie Stromer, PetraIsenberg, and Sheelagh Carpendale. Getting practicalwith interactive tabletop displays: designing for densedata, ”fat fingers,” diverse interactions, and face-to-facecollaboration. In Proc. of the ACM International Con-ference on Interactive Tabletops and Surfaces (ITS),pages 109–116. ACM, 2009.

15. Ji Soo Yi, Youn ah Kang, J.T. Stasko, and J.A. Jacko.Toward a Deeper Understanding of the Role of Interac-tion in Information Visualization. IEEE Transactionson Visualization and Computer Graphics, 13(6):1224–1231, 2007.