Supporting Everyday ThinkingPractices in InformationVisualization Interfaces

Jagoda WalnyInnovis Group, Interactions LabUniversity of CalgaryCalgary, AB, Canadajkwalny@ucalgary.ca

Figure 1: Changing a bar chart into a line chart using theprototype created for our Wizard of Oz study [19] forunderstanding how people would approach interactivelysketching visualizations.

http://dx.doi.org/10.1145/2669485.2669491

AbstractPeople commonly sketch externalizations on paper andwhiteboards as part of their everyday thinking processes.While common, this practice is little understood,particularly as it may relate to digital visual representation(such as information visualization) and sketch-basedinteraction. My research aims to better understand thesethinking sketches from the perspective of informationvisualization and pen-and-touch interaction and to applythis understanding to the design of interfaces that canbetter support complex and freeform everyday thinkingpractices.

Author Keywordsinformation visualization; pen-and-touch interaction;sketched externalizations; qualitative evaluation

ACM Classification KeywordsH.5.2 [Information interfaces and presentation (e.g.,HCI)]: User Interfaces.

IntroductionWhen visiting the offices of knowledge workers, it is notuncommon to see diagrams, mind maps, and scraps ofinformation scribbled on whiteboards alongside notebooksfilled with ideas and sketches for problem solving. Thesevisual artifacts are residue of the process of externalization

ITS 2014 • Doctoral Symposium November 16-19, 2014, Dresden, Germany

479

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Copyright is held by the author/owner(s). ITS 2014, November 16–19, 2014, Dresden, Germany. ACM 978-1-4503-2587-5/14/11.

- the reification of internal thought to support cognition.Externalizations support cognition in a variety of ways,including providing a way to offload memory or to solveproblems more efficiently [8]. They are often ad-hoc,created on the fly to suit a given context. Thisspontaneous practice becomes more difficult when theobjects of thought are dynamic representations of digitaldata that would be tedious to reproduce manually butwhose manipulation is restricted by the design of theinterface containing it. The availability of pen and touchhardware provides an opportunity to better support suchthinking practices with digital information.

The goal of my research is to further our understanding ofhow to better support externalization for digitalinformation. The information visualization community hastackled its goal of aiding cognition of digital information[5] by shaping data into more manageable and interactiverepresentations. The approaches to this have been mainlydata-centric [14] (shaped by the data) or task-centric [1](shaped by the tasks to be accomplished with the data).However, while these approaches make the information -or select aspects of the information - easier to understand,this still leaves room for a complementary human-centricapproach, in which people’s common thinking processesare supported.

My broad approach to this research consists of two phases:

• 1. Exploration: Deepen our understanding ofexternalization from an information visualizationperspective by performing qualitative exploratorystudies. In particular, I look at sketchedexternalizations.

• 2. Application: Apply this deepened understandingto the design of interfaces and/or interaction

techniques that support externalization, with aparticular focus on using interactive pen- andtouch-enabled surfaces.

In my research, I aim to leverage the similarity of pen andtouch surfaces to conventional whiteboards to supportthinking with digital information. As such, I build on priorwork on pen and touch interfaces as well as work done onsupporting paper- or whiteboard-based practices digitally.Sketch-based interfaces date back to Sutherland‘sSketchpad [15], and have been used, among others, formodeling 3D objects [12], for interactive mathematicalsketching [2, 9, 10], for creating visual queries of data[20, 13], and creating information visualizations [6, 4].Typically, in these systems, sketches are interpreted ascommands or inputs to the system. In contrast, myresearch is concerned with creating interfaces that supportfreeform sketched externalizations, which may not alwayshave a pre-defined or expected form that can beinterpreted. Pen and touch interfaces provide a richerpotential set of tools [7], and recent work suggests thatbimanual pen and touch interactions can be effective [3].

Prior research into supporting knowledge workers’ thinkingpractices in general has resulted in systems that emulateand extend conventional whiteboards with the advantagesof the digital environment [11]. My focus is not oncreating digital whiteboards, but on supporting the kindsof behaviours one finds on conventional whiteboards forthose working with digital information. One approach thatallows for some externalization practices when workingwith digital information has involved integrating digitaland analog technologies. For example, Brandl et al.created NiCEBook [3], a paper notebook thatsynchronizes with its digital representation. This approachhas the advantage of preserving the affordances of paper,

ITS 2014 • Doctoral Symposium November 16-19, 2014, Dresden, Germany

480

but may not be sufficient when working with dynamicvisualizations of data.

Exploratory StudiesI have completed three qualitative exploratory studies ofexternalization through sketching, summarized here.

Visualizations on WhiteboardsTo arrive at a better understanding of the nature ofvisuals created spontaneously during everyday thinkingtasks, we studied a collection of 82 snapshots of 69knowledge workers’ whiteboards collected at a largeresearch institution [17]. As these whiteboards were inindividuals’ offices, they contained the residue of variouskinds of thought, from small pieces of information toremember, to sketches used to work out problems, tothose used in small-group discussions. We analyzed thesequalitatively from an information visualization perspective,examining the types of visual constructs used (bothrecognizable and novel) as well as the relationshipbetween words and diagrammatic constructs. Follow-upinterviews with a subset of the participants validated ourcategorization of constructs and gave us valuable insightinto the nature of the visuals.

This study resulted in a taxonomy of the kinds ofconstructs and visual strategies people use to express theirthoughts, including communicative symbols, layering,spatial organization, and information visualizationconstructs. It also demonstrated the words-to-diagramsspectrum, which describes the variable nature of howwords are used in relation to diagrammatic constructs.This highlighted the restrictive nature of software that,when providing support for making diagrams orvisualizations, considers words only in the role of labels.

Figure 2: An example of a complex, freeform externalizationon a whiteboard. c©2011 IEEE. Reprinted, with permission,from [17].

Sketching DataSketching concepts and ideas on whiteboards may not beentirely analogous to sketching data. We asked people touse colored pencils to sketch a small, relatable, and easilyunderstandable dataset in whatever way they chose and toreport on what they learned about the data [16]. Ouranalysis has focused on qualitatively classifying the widevariety of representations collected and relating the typesof representations to the types of insights gained.

Lifecycles of Diagrams and SketchesWe have also examined the context of externalizationpractices on a broader level by interviewing computerscientists about sketches and diagrams that wereimportant to a recent project [18]. From these interviews,we reconstructed the lifecycles of these sketches in terms

ITS 2014 • Doctoral Symposium November 16-19, 2014, Dresden, Germany

481

of the context (personal, group, or public), the mediumused (e.g. paper, tablet, whiteboard), and the transitionsthat a sketch underwent (e.g. creation, refinement,sharing). See Figure 3 for a sample. This studyhighlighted the variety of habits and strategies used bypeople in their thinking processes, suggesting that aone-size-fits-all interface for supporting thinking may notbe feasible.

debug

draw take photoredraw refine

remember formalize

write

publish

#5

implementdraw

Time

discard revisitarchiveno value

change environment

rememberfuture reference

Per

son

Gro

up

Figure 3: A lifecycle from of a single sketch significant to a computer scientist’s project.c©2011 IEEE. Reprinted, with permission, from [18].

Studying Application ChallengesThis phase of my research involves discovering how theinsights gained in the exploratory studies can be appliedto information visualization tools that supportexternalization. The purpose of this is to highlight thechallenges inherent in the design of such tools and tosuggest some design strategies moving forward. As such,my plan is to use as a basis standard interactivevisualizations typical of what is commonly seen on theweb, such as interactive bar charts and graphvisualizations. I am focused in particular on the context ofpen-and-touch surfaces due to their being the closest innature to an analog whiteboard or piece of paper. Todate, we have completed one study; the next steps aredescribed in the following section.

Understanding Pen and Touch Interaction for Data Explo-rationOur first foray into designing an interactive system thatsupports thinking practices was to design a system forcreating simple visualizations based on real data just bysketching them [19]. We ran a Wizard of Oz study basedon our design in order to see how people would approachinteractively sketching a simple information visualization.During the study, we asked participants to figure out howto use the system on their own in order to see what typesof interactions they tended towards naturally.

Our system, shown in Figure 1, was designed to keep theinterface out of the way as much as possible by supportingspecification of what one wants to see by simply drawingit, supporting manipulation of objects that is as direct aspossible, minimizing explicit mode-switching, andproviding good default behaviours.

Our study results indicated that participants drew onknowledge from both the physical world and from whatthey had already learned about the system in order toinfer how to perform new tasks. Our participants had aclear idea of when to use pen and when to use multitouchinteractions, which was drawn from their experience in thephysical world; they also used a variety of postures whenholding the pen and performing combined pen and touchinteractions. They used multiple interaction paths for thesame task and also tended to perform integratedinteractions, where interaction with the data and otherelements on the screen occurs in proximity to the objectsbeing acted upon.

The Next StepsMy initial inquiry into applying the results of the abovestudies to information visualization software has

ITS 2014 • Doctoral Symposium November 16-19, 2014, Dresden, Germany

482

highlighted some of the particular challenges of designingthinking interfaces. The design of these interfaces goesbeyond usability and learnability into supporting complexand often freeform and rapid thinking practices. Onechallenge is in knowing how to create interfaces that donot interfere with thought processes. Another challenge isin supporting externalization practices for dynamicrepresentations of information. My current project,following from our study of whiteboards [17], explores thespace where words and diagrammatic elements are tightlyintermingled in a single thinking diagram. This posesseveral interesting design challenges such as knowingwhen to distinguish between words and drawn elements.

The exploratory portion of my research is largelycomplete, and my aim is to use the remainder of my PhDto apply the understanding I have gained through thesestudies to the creation of software, particularly asmentioned above, making use of the words-to-diagramsspectrum. However, I believe that an important part ofmy PhD contribution will remain the new understandingof how people use sketched visuals as externalizationsduring their thinking processes.

AcknowledgementsI would like to thank my supervisor, Sheelagh Carpendale,and my collaborators and colleagues in both theInteractions Lab and Microsoft Research for their support.This research was supported in part by MicrosoftResearch, NSERC, SMART Technologies, AlbertaInnovates Technology Futures, SurfNet, and GRAND.

References[1] Amar, R., and Stasko, J. A knowledge task-based

framework for design and evaluation of information

visualizations. In Information Visualization, IEEESymposium on (2004), 143–150.

[2] Bott, J. N., and LaViola, Jr., J. J. A pen-based toolfor visualizing vector mathematics. In Proceedings ofthe Seventh Sketch-Based Interfaces and ModelingSymposium, SBIM ’10, Eurographics Association(Aire-la-Ville, Switzerland, Switzerland, 2010),103–110.

[3] Brandl, P., Forlines, C., Wigdor, D., Haller, M., andShen, C. Combining and measuring the benefits ofbimanual pen and direct-touch interaction onhorizontal interfaces. In Proceedings of the WorkingConference on Advanced Visual Interfaces, AVI ’08,ACM (New York, NY, USA, 2008), 154–161.

[4] Browne, J., Lee, B., Carpendale, S., Riche, N., andSherwood, T. Data analysis on interactivewhiteboards through sketch-based interaction. InProceedings of the ACM International Conference onInteractive Tabletops and Surfaces, ITS ’11, ACM(New York, NY, USA, 2011), 154–157.

[5] Card, S. K., Mackinlay, J. D., and Shneiderman, B.Readings in information visualization: Using vision tothink. Morgan Kaufmann, 1999.

[6] Chao, W. O., Munzner, T., and van de Panne, M.Poster: Rapid pen-centric authoring ofimprovisational visualizations with NapkinVis.Posters Compendium InfoVis (2010).

[7] Hinckley, K., Yatani, K., Pahud, M., Coddington, N.,Rodenhouse, J., Wilson, A., Benko, H., and Buxton,B. Pen + touch = new tools. In Proceedings of the23rd Annual ACM Symposium on User InterfaceSoftware and Technology, UIST ’10, ACM (NewYork, NY, USA, 2010), 27–36.

[8] Kirsh, D. Thinking with external representations. AISociety 25, 4 (2010), 441–454.

ITS 2014 • Doctoral Symposium November 16-19, 2014, Dresden, Germany

483

[9] LaViola, Jr., J. J., and Zeleznik, R. C. Mathpad2: Asystem for the creation and exploration ofmathematical sketches. In ACM SIGGRAPH 2007Courses, SIGGRAPH ’07, ACM (New York, NY,USA, 2007).

[10] Leitner, J., Rendl, C., Perteneder, F., Gokcezade, A.,Seifried, T., Haller, M., Zeleznik, R., and Bragdon,A. NiCE formula editor. In ACM SIGGRAPH 2010Talks, SIGGRAPH ’10, ACM (New York, NY, USA,2010), 55:1–55:1.

[11] Mynatt, E., Huang, E., Voida, S., and MacIntyre, B.Large displays for knowledge work. In Public andSituated Displays, K. OHara, M. Perry, E. Churchill,and D. Russell, Eds., vol. 2 of The KluwerInternational series on Computer SupportedCooperative Work. Springer Netherlands, 2003,80–102.

[12] Olsen, L., Samavati, F. F., Sousa, M. C., and Jorge,J. A. Sketch-based modeling: A survey. Computers& Graphics 33, 1 (2009), 85–103.

[13] Ryall, K., Lesh, N., Lanning, T., Leigh, D.,Miyashita, H., and Makino, S. Querylines:Approximate query for visual browsing. In CHI ’05Extended Abstracts on Human Factors in ComputingSystems, CHI EA ’05, ACM (New York, NY, USA,2005), 1765–1768.

[14] Shneiderman, B. The eyes have it: A task by datatype taxonomy for information visualizations. In

Visual Languages, IEEE Symposium on (1996),336–343.

[15] Sutherland, I. E. Sketch pad: A man-machinegraphical communication system. In Proceedings ofthe SHARE Design Automation Workshop, DAC ’64,ACM (New York, NY, USA, 1964), 6.329–6.346.

[16] Walny, J., and Carpendale, S. Data sketches: Anexploratory study. InfoVis 2014 Posters Compendium(to appear).

[17] Walny, J., Carpendale, S., Riche, N., Venolia, G., andFawcett, P. Visual thinking in action: Visualizationsas used on whiteboards. Visualization and ComputerGraphics, IEEE Transactions on 17, 12 (Dec 2011),2508–2517.

[18] Walny, J., Haber, J., Dork, M., Sillito, J., andCarpendale, S. Follow that sketch: Lifecycles ofdiagrams and sketches in software development. InVisualizing Software for Understanding and Analysis(VISSOFT), 2011 6th IEEE International Workshopon (Sept 2011), 1–8.

[19] Walny, J., Lee, B., Johns, P., Riche, N., andCarpendale, S. Understanding pen and touchinteraction for data exploration on interactivewhiteboards. Visualization and Computer Graphics,IEEE Transactions on 18, 12 (Dec 2012), 2779–2788.

[20] Wattenberg, M. Sketching a graph to query atime-series database. In CHI ’01 Extended Abstractson Human Factors in Computing Systems, CHI EA’01, ACM (New York, NY, USA, 2001), 381–382.

ITS 2014 • Doctoral Symposium November 16-19, 2014, Dresden, Germany

484