A P R I L 1 3 - ~ s , 1996 C H i 9 6 ,r

Structuring Information With Mental Models: A Tour of Boston

I s h a n t h a Lokuge M I T M e d i a L a b

20 A m e s S t ree t

C a m b r i d g e , M A 0 2 1 3 9

i s h i @ m e d i a . m i t . e d u

Stephen A. G i l b e r t

Bra in & Cogn i t ive Sc i ences

MIT, E 1 0 - 1 2 0

Cambr idge , M A 0 2 1 3 9

s t e p h e n @ p s y c h e . m i t . e d u

Whitman R i c h a r d s

M I T M e d i a L a b

20 A m e s S t ree t

C a m b r i d g e , M A 0 2 1 3 9

whi t @ m e d i a . m i t . e d u

ABSTRACT We present a new systematic method of structuring infor- mation using mental models. This method can be used both to evaluate the efficiency of an information structure and to build user-centered information structures. In this paper we present the method using Boston tourist attractions as an example domain. We describe several interfaces that take advantage of our mental models with an activation spread- ing network. Multidimensional Scaling and Trajectory Mapping are used to build our mental models. Because of the robustness of the technique, it is easy to compare indi- vidual difference in mental models and to customize inter- faces for individual models.

Keywords Cognitive models, multidimensional scaling, visualization, interaction design, evaluation.

INTRODUCTION We all know that a curious person can more efficiently absorb information when it is well structured than when it is arbitrarily scattered. The question that every information architect then asks is, "How might I best organize the infor- mation for that person?" This question contains three issues: what structures are useful for organizing informa- tion in general, what structures are useful for organizing that information, and what structures are useful for orga- nizing information for that person.

To answer these above questions, we propose (i) collecting experimental data from a number of subjects, (ii) analyzing the mental models of those subjects with Multidimensional Scaling (MDS) [7, 15, 16] and Trajectory Mapping (TM) [4, 9, 13], and then (iii) using those models to design infor- mation structures. Such models should lead more quickly to suitable interfaces rather than beginning with trial and error explorations.

Permission to make digital/hard copies of all or part of this material for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copy- right notice, the title of the publication and its date appear, and notice is given that copyright is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers or to redistribute to lists, t~qutr~s specific permission and/or fee. CHI 96 Vancouver , BC Canada ¢ 1996 ACM 0-89791-777-4/96/04. .$3.50

Before continuing on to details, we offer definitions of "information structure" and "mental model." By "informa- tion structure" we mean an arrangement of pieces of infor- mation. The arrangement might be a 2-D array on a table or a screen, as in a card game or a spreadsheet. It might also be a 1-D ordering of items, like a shopping list. It could also consist of a set of information nodes with con- necting association links, such as a hypertext. All the pieces of information should belong to the same conceptu- al type category, such as numbers to add, tasks to do, or words to remember. By "mental model" we mean not the explanatory model offered by Johnson-Laird [5], but rather a more general definition: the cognitive layout that a per- son uses to organize information in his or her memory.

THE EXAMPLE DOMAIN Consider the map of Boston shown in Figure la. We have chosen fifteen different sites of activity taken from a tour guide book (listed in Figure lb). This set of information pieces (or "stimuli," as we will call them) is a very high- dimensional in feature space, and it has a relatively large variance across people. That is, there are many features

Figure la: Geographic layout of activities

413

~,,~| ql~ APRIL 13-18, 1 9 9 6

Stimulus Set: Boston Tourist Attractions

1. Sports Museum 2. Children's Museum 3. Science Museum 4. Aquarium 5. Swan Boats (boat tour of garden) 6. Newbury Street (elegant shopping) 7. Quincy Market (outdoor mall) 8. Trinity Church (historic site) 9. Magic Show

10. Salem (nearby historic town) 11. Harvard University 12. Museum of Fine Arts 13. Zoo 14. Fenway Park (baseball stadium) 15. Arboretum (nature preserve)

to illustrate the capabilities of Multidimensional Scaling (MDS), first used by Torgerson [t6], Shepard [15], and Kruskal [7], we collected data from two subjects for each of these mental models (Figure 3). Note that the judgments based on geographic similarity (3a) are completely different from judgments based on content (3b). The input data for MDS takes the form of pairwise similarity judgments, and the output is an arrangement of the stimuli in a metric space.

To clarify the procedure for obtaining these MDS plots, we offer a simple example; consider a collection of seven black and white circles of different sizes. In Figure 2a, these cir- cles are arranged randomly along the edges of a similarity matrix. The numbers show a subject's similarity rating on a scale of 1 to 7. From these numbers the MDS algorithm arranges the circles as shown in Figure 4, with neighboring circles being the most similar.

Note: Activities 9-15 are outside the scale of the map.

Figure lb: The stimuli for the MDS and TM experiments.

that can be used to describe the stimuli, and people do so in significantly different ways. As a contrasting example, a deck of playing cards contains very few features, namely the number of a card, its suit, and perhaps its color, and differ- ent people describe playing cards similarly.

MD S A N D TM M E T H O D S A person who lives in Boston and knows the sites will have at least two mental models of them, one based on their geo- graphic locations, and one based on their content. In order

The distances between the points in the output is usually a non-linear transformation of the values in the similarity matrix. For Figure 3a, we asked the subjects for every pair of stimuli, "On a scale from 1 to 7, how similar are stimuli X and Y in terms of their distance from each other?" For Figure 3b, we asked, "How similar are stimuli X and Y in terms of their content or theme?" We then used KYST2 [6] to run the MDS and generate the graphs shown.

As one might expect, the distance-based MDS plot is simi- lar to the actual map of Boston, though somewhat warped; the warping could stem from differing familiarity with the sites [11] or from thinking of distance as travel time instead of geographic distance. In the content-based MDS plot,

I I

0 0 qll

0

®

, O o O o o . 2 4

6 5

2

4 6 5

4 3 1

3 5 1

2 5 4

1 2

4

Figure 2a: Typical input data for the MDS algorithm; sim- ilarity data for black and white circles of different sizes. 7 = very similar; 1 = very dissimilar.

Figure 2b: Typical input data for the TM algorithm; extrapolations and interpolations for black and white cir- cles of different sizes. Here are 12 of 21 possible pairs.

414

A P R I L t 3 - ~ 8 , 1996 ~ ' ~ 9 ~

Salem Magic Show @ ................ ~ t ....................... t _ _ _ ..................... ?! . . . .L._. . .............

~ Sports e Science Aquarium Museum Museum •

o • Quincy Harvard Market

Swan

Arboretum ZOO ~ Newbury St,~ ® Trinity Children's e

Church Museum

"Fenway @ • Fine Arts Park

Museum ..... ............................. . . . . . . . . . . . . . . . . . . . . . . . . . 7 ~ ...................... ........................ ....

ENTE

Children's : . Museum

o

Sports ® Museum Science

~ •

Museum Fine Arts O

i Fenway Museum Park Zoo

o

Aquarium

o

~TAINMENT Magic Show o Salem HISTOR

Harvard o

g

Trinity Church

CAL

Swar~Boats Neewbury St.

NATURE ® , HOPPING Arboretum Quincy Market

Figure 3a: MDS plot based on geographic similarity Figure 3b: MDS plot based on content similarity

similar activity sites like the Aquarium and the Zoo appear near each other, as do shopping areas Newbury Street and Quincy Market. One can use the groupings of points in an MDS plot to assign features to clusters of points, as we have done in Figure 3b with the features, "historical," etc. Likewise, many researchers who use MDS would attempt to assign meaning to the axes of the plot, suggesting perhaps that the X-axis runs from "playful to serious" and Y-axis runs from "outdoors to indoors". Such labels or category assignments must be done carefully and should be verified with separate experiments, however, because they are often biased by the experimenter's a priori knowledge of the data.

The reader might wonder what would be the outcome if we had asked subjects to give general similarity ratings without specifying what type of similarity. Would the resulting MDS plot have been an unfortunate mixture of the two plots in Figure 3? This issue raises the question of how we could

have known what types of similarity to separate if we knew nothing about the data to begin with. To answer this ques- tion, we introduce the relatively new Trajectory Mapping (TM) procedure [4, 13, 14].

For high-dimensional feature spaces, often TM can better delineate the features of the data than MDS. Instead of giv- ing similarity judgements as in MDS, the subject's task in TM is to imagine a conceptual feature or property that links a given pair of stimuli. The subject then extrapolates that feature in both directions to pick two stimuli from the remaining set that would be appropriate. The subject also picks an interpolant, i.e. a stimulus that would fit well with- in the pair.

Returning to our black and white circle example, a TM set of input data might appear as in Figure 2b. The members of the original pair are in columns A and B; the extrapolants are

-t

-4 i;

e 4~

w

. . . .

t ) ,0

Figure 4: MDS and TM output for black and white circles. Figure 5: Trajectory map for the tourist site data

415

!~ii!

jil i

i ~ i~

C ~ " ~ @ ~ APRIL t 3 - 1 8 , 1 9 9 6

in the two "ex" columns, and the interpolant in the "int" col- umn. As well as using a stimulus for each slot, the subject may also enter an "X" or a "...". An "X" indicates that the subject did not reel comfortable choosing a stimulus for that spot, and the "..." indicates that the subject could imagine a stimulus that would fit there, but that such a stimulus was not present in the given data set to choose from.

From this set of quintuples, we can now extract a connected graph of the stimuli in which the maximum number of quin- tuples fit. For example, in the top row of Figure 2b, we have an ordered set of white circles ending in a small dot in one column, and a set of black circles ending in the same small dot in the other column. If these quintuples are equally con- sidered as constraints on the graph, the ending trajectory map will contain a path from the large white circle to the large black circle, going through the small dot, as shown in Figure 4.

BOSTON DATA Figure 5 shows a trajectory map for tourist site data gathered from the authors. Note here that the positions of the nodes are not important; the mental model lies in the connections between the nodes, i.e. the topology of the graph. The weights on the links are based not on similarity, but rather on the robustness of that link across a gamut of parameters within the TM algorithm. The weight on a link can thus be thought of as the strength of the connection in the mental model. (A TM algorithm is being fine-tuned for release by Gilbert.)

By pruning the trajectory map to only its strongest links, one can see rough feature clusters emerge (see the heavier links

Mag c Show .8~;4;~ ~4 ~;~, . . . . . . . . . . . . . . . . . Salem

; 4 : ~&

( " i

lildren's mourn

Science , M i i~..11 i m i

Museum ~iiiiii ~

O 0

Newbury St.

Arboretum Quincy Market

,:. ~,C? -0! ( ) ~ , ,

Figure 6: The mental models combined: a rough TM path superimposed over the MDS content plot.

in Figure 5). In our example, the clusters are roughly simi- lar to the estimated clusters in the MDS plot (Figure 2b). In a trajectory map, however, the s~imuli are ordered within each cluster, e.g. Arboretum, Swan Boats, Aquarium/Zoo. Figure 6 shows the combined mental models, a rough TM path drawn over the MDS content-based plot.

Because the tourist sites have several possible features that could be used by subjects to report their TM extrapolations and interpolations, one can discover the most salient fea- tures of the data set by running the algorithm across many

m Fenway Park

Aquarium

O

Z q

Figure 7a: Children's activities energized by the region- al activation spreading network.

m

F ¢nw,~tf Pw~

i~ ::~L ̧

I I tmi

Fine Arts Museum Trinity Church

Z ~

m

Figure 7b: Adults' activities energized by the regional activation spreading network.

416

APRIL 13-18, 1996 ~ | ~

1 l l

M

Figure 8: The network activates the path segments when the user examines the sites Sports Museum, Fenway Park and Aquarium. The activation levels are mapped to image and typographic size, enabling the system to suggest to the user an appropriate next step based on the MDS and TM results.

subjects. It is likely with this tourist site information, for example, that some subjects would do the TM based on geo- graphic distance, while others would do it based on the con- tent of the sites. This feature difference would eventually manifest itself from obviously disparate trajectory maps, and lead to the phrasing of the MDS questions described earlier: similarity in terms of distance and similarity in terms of content. Examples of different trajectory maps for the same domain can be seen in [12].

EXPERIMENTATIO N The mental models from TM and MDS provide an excellent basis for measuring the efficiency of an information struc- ture that has been built from the models. One might arrange the stimuli serially according to the TM paths or distribute them according to the MDS plot. After collecting data from a new pool of subjects for measures of readability, ease of remembering, etc. (see [2, 3] for a typical array of memory measures), the information structure can be systematically varied by changing the parameters that produce the MDS and TM models.

EXAMPLE INTERFACES If one considers the set of tourist attractions as an informa- tion space to be explored, we can also use the mental mod- els to give the user well-founded suggestions as to his or her next step of exploration. As a demonstration of feasibility, we have designed a visualization system that allows the exploration of the Boston tourist attractions. Since we are no longer restrained to the geography of Boston, as an actu- al tourist would be, we must now answer the question, "How should we order the various sites?" To explore the different possible answers, we have built three different interfaces for the system, each of which defines an "atten- tional window" as the current region of interest in the infor- mation space. An activation spreading network based on the mental models defines the size of the attentional window as the user explores. Thus, if the aquarium node of the network is currently activated, and the models suggest that the swan

boats node is closely related, then the activation will spread to swan boats, leading the attentional window to include swan boats as a potential next focus.

In the first two interfaces, the 15 sites are arranged two- dimensionally according to the MDS plot in Figure 3b. Whichever sites fall within the attentional window are ren- dered larger than the others. In one interface, the attention- al window envelops MDS regions (Figure 7), and in the other, the window spreads across TM paths (Figure 8). Figure 7a shows the display of regional activation with emphasis on children's activities. Figure 7b shows the activ- ities in an adult context. In contrast, Figure 8 shows sever- al successive frames of path activation: when the user investigates the Sports Museum, the network suggests that Fenway Park or the Children's Museum might be a good next choice (those two sites are slightly enlarged in Figure 8a). As the user shifts attention to the next event, past activ- ities fade away and activities further along the path become more conspicuous.

The difference between these two styles of exploration can be characterized by the width of the attentional window: narrow in the case of paths, and wide in the case of regions. By smoothly varying the width, one might change smooth- ly between modes of exploration.

The third interface attempts to combine the path and region following ideas of the first two. This interface depicts the sites as semitransparent multimedia cubes in a 3-D space (Figure 9). The cubes are arranged along the TM paths, but at each cube, other cubes within the same region can be seen orbiting the current cube nearby. Thus, the user can "fly" smoothly through the space along the TM routes, or branch off to a similar site within the current region. Also, the floor of each cube displays a geographic map with the site's loca- tion high-lighted, thus incorporating all of the features that we have discussed thus far.

417

~"H ~ APRIL 13-18, 1996

Figure 9: An interface that combines path and region following; the sites are represented by cubes in a 3-D space. The cubes lie along the TM paths, and the other sites from the MDS region encircle the cube within the attentional window.

The activation spreading network was designed by Lokuge and Ishizaki [10], and the visualization system is imple- mented on a Silicon Graphics Onyx workstation. This net- work links nodes in a space (Figure 10) and uses the order- ing in the space to control the activation level of a node [ 1, 12]. When attention "jumps" to a new node, the network partially reduces the activation levels in the original set of nodes, leaving a faint trace as a history of the attentional sequence.

DISCUSSION We have designed this visualization system with its various interfaces both as an existence proof that such a system could be devised and as an illustration of our proposed methodology for structuring information. We mentioned above the fact that a variety of mental maps exist for each data set, both within and across individuals. A further devel- opment would be an intelligent activation spreading net- work, i.e. one which contained the full repertoire of differ- ent mental models for a given data set and chose the best model based on the user's behavior. Such a repertoire could be described as a "hyper-mental-map" and could likely formed by gathering MDS and TM data on the mental mod-

els as stimuli themselves.

We have proposed and implemented a method of organizing information space around cognitive maps. We plan next to explore the degree to which this method can be used with more generalized domains of knowledge, but examples from previous MDS and TM papers lends us hope that the method could become generally applicable. By accommo- dating an individual's search context (e.g. through paths or regions) and his or her particular model of a domain (e.g. a particular MDS or TM model), one can offer both a more personalized tour of the information space and a more easi- ly absorbable mass of information.

A C K N O W L E D G M E N T S This work was in part sponsored by ARPA, JNIDS, NYNEX and Alenia. However, the views and conclusions expressed here are those of the authors and do not necessarily repre- sent that of the sponsors.

REFERENCES 1. Anderson, J .A. Spreading activation theory of memory.

Journal of Learning and Verbal Behavior 22, 261-295, 1983.

Goal: Kids' Tour of Boston

Fenway Park

Children's Museum | Science Museum

MagiclShow / Salem ~

Figure 10: Schematic diagram of a regional activation spreading network. Activation energy spreads in the direc- tion of the arrows initiated by the Goal: Kids' Tour of Boston.

.

.

.

.

.

Atkinson, R.C. & Shiffrin, R.M. The control of short- term memory. Scientific American, 225, 82-90, 1971.

Craik, EI.M. & Lockhart, R.S. Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11,671-684, 1972.

Gilbert, S. A. & Richards, W. Using trajectory mapping to analyze musical intervals. Proceedings of the Sixteenth Annual Conference of the Cognitive Science Society. 1994.

Johnson-Laird, EN. Mental models. Cambridge, MA: Harvard University Press, 1983.

Kruskal, J.B. kyst2a. AT&T Bell Laboratories, Computing Science Research Center, Netlib repository of software, http://plan9.att.com/netlib/mds/index.html.

418

i

APRIL 13-18, 1996 Cl" l | 9 6

7. Kruskal, J.B. Nonmetric multidimensional scaling: A numerical approach. Psychometrika, 29, 28-42, 1964.

8. Lokuge, I. Dynamic magical environments: Engaging interaction based on the art of illusion. Masters Thesis, MIT Media Lab, 1995. To receive a copy email docsOmit.edu or call MITDocument Services at 6171253-5650.

12.

13.

Maes, P. Situated agents can have goals in Designing Autonomous Agents: Theory and Practice from Biology to Engineering and Back, ed. P. Maes, MIT Press/Bradford Books, 1992.

Richards, W. & Koenderink, J.J. Trajectory mapping ("TM"): A new non-metric scaling technique. Proceedings of European Conference on Visual Perception, Edinburgh (Abstract), 1993. In press Perception.

9. Lokuge, I., Gilbert, S.A. & Richards, W. Visualizing information spaces: A tour through Boston. Presented at European Conference on Visual Perception '95, Ttibingen. Abstract in Perception, 24 Supplement, p. 76.

14. Saint-Arnaud, N. Class~cafion of Sound Textures. Masters Thesis, MIT Media Lab, 1995. To receive a copy email d o c s @ m i t . e d u or call MITDocument Services at 617/253-5650.

10. Lokuge, I. & Ishizaki, S. GeoSpace: An interactive visu- alization system for exploring complex information spaces. Proceedings of ACM SIGCHI'95, Denver, Colorado, 1994.

15. Shepard, R.N. The analysis of proximities: Multi- dimensional scaling with an unknown distance func- tion. I. & II. Psychometrika, 27, 125-140, 219-246, 1962.

11. Lynch, K. Image of the City. Cambridge, MA: MIT Press, 1960.

16. Torgerson, W.S. Multidimensional scaling: I. Theory and method. Psychometrika, 4, 401-419, 1952.

419