21. direct manipulation

고려대학교 산업공학과

IND 542 UI Engineering

21. Direct Manipulation2. What is Direct Manipulation?

1. 1. Original Definitions and Claims graphical interfaces operated directly using manual actions than typed instructions1. continuous representation of the object of interest2. physical actions or labeled button presses instead of complex syntax3. rapid incremental reversible operations whose impact on the object of interest is

immediately visible usability benefits learnability; enhanced expert performance; memorability; fewer error

messages; better feedback; reduced anxiety; increased control3. What makes Manipulation Direct?

Directness = Engagement + Distance engagement the perceived locus of control of action within the system distance the mental effort required to translate goals into actions at the interface and then

evaluate their effects – gulf between goals and actions4. Data and Development

1. TestsUncritical Comparative Evaluation measuring the usability of a direct manipulation interface against one or more alternative

interfaces advantages – word processing tasks, file manipulation tasks, database retrieval tasks



no differences – file management, drawing, matching concepts and labels disadvantages – table manipulation, filing and retrieval, browsingCritical Comparative Evaluations measuring the the usability of several different implementations of a direct manipulation

interface against each other and against alternative interfacesNaturalistic Choice Studies measuring interactional preferences for direct manipulation and alternative methods in a

mixed mode interface direct manipulation was often used to avoid typing long object names in the dialog box, and

natural language was used to refer to objects which were not visible on the screen2. Mixed Mode Interfaces how to best mix manual and conversational forms of interaction in hybrid interface design

shift the locus of control from whether to when and how3. TheoryThe Value of Mixed Mode Interaction mode of interaction metaphors (Hutchins, 1989)1. conversation metaphor – the character of utterances in a conversation about the task at

hand – limited because users have to learn a new language, maintain a mental model of the world, converse in a very different manner

2. declaration metaphor – the character of speech acts which magically cause things to happen in the world of interest – limited because it depends on a practice effect in using a conversational interface

3. model-world metaphor – the character of actions taken in the world of interest – limited because of its directness in collapsing abstract descriptions into concrete actions



4. collaborative manipulation metaphor – the character of carrying out a task with someone elses help limitations with principles

Laurel (1990) – reintroduction of a conversational metaphor within a model-world context visible intermediary within a model-world -- interface agent defined as a character enacted by a computer, who acts on behalf of the user in a virtual (computer-based) environment

Whittaker (1990)Principle 1 Continuous representation of the object of interest – the problems of locating objects

that are not visible – manual search inefficientPrinciple 3 Rapid incremental reversible operations – leads to inefficiencies in the execution of

common compound actions which might be done faster through a single commandPrinciple 3 whose impact is immediately visible – does not allow for processes such as

reminders or mail forwardingFurther Insights on Directness the directness of manual interaction (Johnson et al., 1989) with Xerox Star system

seeing and pointing over remembering and typing through the graphical interface don’t be dogmatic about the desktop metaphor and direct manipulation

the role of icons in representing metaphors (Familant and Detweiler, 1993) a sign that shares characteristics with the objects to which it refers icons have lost the shared characteristics with referent objects because they have no

real world counterparts (highly abstract) Desain (1988) equates distance with directness as a measure of interfaces

direct manual interfaces well-known graphical formalism with real-world actions direct conversational interfaces natural language syntax with the jargon vocabulary



cognitive directness (least cognitive effort) and social directness (least collaborative effort) cryptic conversations

5. Two Philosophies, Two Debates1. Separating Directness and Manipulation how and when to utilize manual forms of interaction at the interface how by critical tests comparing different implementations of manual interfaces with each

other and by discussions of directness some features of manual and conversational interaction more effective than others

when by uncritical tests comparing manual with conventional interfaces for the same tasks, and by the work on mixed mode interfaces manual interfaces are not always better than conversational

directness is now equated with distance and manipulation with engagement re-conceptualization of the space of interfaces

directness philosophy relating to what makes an interface easy to use manipulation philosophy relating to why manual interaction is preferable to

conversational 2. A New Philosophy of Directness manual interaction -- addition of real world metaphor principle3. A New Philosophy of Manipulation utilize manipulation selectively!



24. Design of Menus

1. Menu-Driven Interfaces a set of options, displayed on the screen, where the selection and execution of one (or

more) of the options results in a change in the state of the interface

2. Designing a Single Menu Panel1. Three Types of Comparison Operations

identity searching – a specific target that is literally displayed as one of the options class-inclusion matching – at the root or other top-level panels of a hierarchical menu equivalence search – at the leaves of bottom levels of menu systemsIdentity Matching Perlman (1984) – alphabetical vs. random order Card (1982) – alphabetical, categorical, randomEquivalence Matching know the name of the target -- either an alphabetized or a categorized menu. uncertain about the name of the target -- categorized lists better than alphabetized listsClass-Inclusion Matching conceptual overlap between the categories don’t seem to fit well into any of the available categories



2. Aiding the Comparison OperatorAdding Descriptors Lee et al. (1984): effective with limited experience longer search time and space Dumais and Landauer (1983): the examples provide very little info beyond that which

could be inferred from the category name aloneUsing Icons three possible advantages over verbal options1. parallel search and no cost associated with a large number of options2. categorizations of pictures can be faster than of words3. can provide additional info that increases the accuracy of selections distinctive icons vs. words and representational icons icons and pictures should be used very selectively

4. Guidelines for Organizing and Naming the Options on a Single PanelOrganization random, alphabetical, categorical organizations conventional order frequency of use – Zipf’s law (1949) – frequency is a negative power function of their

rank



Naming precise naming adding a descriptor – the magnitude of the benefit understood poorly for large menu-driven interfaces, testing the names on each menu panel with a

sample of end users is very costly, but it is the only technique guaranteed to remove all of the clinkers

4. Organization and Navigation Between Menu Panels1. Depth versus Breadth in a Hierarchical Menu Structure

Factors Favoring More Breadth three reasons for considering a system with greater depth

crowding – the amount of available space on a panel insulation – the opportunity to prompt selections that are likely to be needed

and hide those that are unlikely or illegal funneling – a reduction in the total number of options processed that is

achieved by designing a system with more depth and less breadthA General Framework for the Depth-Breadth TradeoffLee and MacGregor’s Linear Model optimal breadth with exhaustive search – 3 to 8 optimal breadth with self-terminating search – 4 to 13Paap and Roske-Hofstrand’s Linear Model in the range of 16 to 36 and sometimes as high as 78



Varying Depth Miller (1981) -- performance is best at the intermediate levels of depth two levels with

eight options is the best -- Snowberry et al. (1983) Kiger (1984) – 28, 34, 82, 16x4, 4x16 performance decreased as depth increased Varying Breadth Across Levels Norman and Chin (1988) – constant (4x4x4x4), decreasing (8x8x2x2), increasing

(2x2x8x8), convex (2x8x8x2), concave (8x2x2x8) searching for specific targets the increasing menu was slightly superior fuzzy targets – concave>increasing>constant>decreasing>convex more breadth on the bottom – the least amount of uncertainty across the top three levels

the superiority of the concave menu over the increasing menu recommendation – allocate the most breadth to the bottom panels Performance Changes Across Levels of the Hierarchy Snowberry et al. (1983) – higher error rates at the top more abstract and ambiguous than

those at the bottom Semantics and Syntax errors are usually caused by labels that are not natural and precise the semantics of a menu system (the quality of the names for categories and terminal

options) are far more important than the syntax (the depth-breadth structure of the menu system)



28. User-Centered Evaluation

1. Introduction shift from hypothesis testing and statistical analysis to information gathering for iterative

design shift focus to formative rather than summative evaluation

2. The Changing View of Evaluation evaluations – object, process, purpose – subjective or objective

1. Evaluation Processes context independent measurements of experience -- usability evaluations as the result of a process with a purpose in a context focused on an object

3. Evaluating Software for Usability1. Why Evaluate Usability?

usability evaluation – take time and cost money -- function, cost and schedule usability evaluation activities are not yet structurally integrated into most development

processes characteristics of the user, the work environment, the documentation or on-line assistance

4. Usability Information Sources the change from “how good is it” to “identify the problems and solutions” the end goal is a redesigned system that meets users are able to achieve their goals and

are satisfied with the product



1. User-Centered Evaluation evaluations conducted during system development are generally aimed at providing

information for improvements in the existing design – verbal reports, walkthroughs, direct tests of interface alternatives

evaluations conducted on completed systems are often less interested in information about specific details and seek information of a more general nature – surveys, questionnaires, general descriptive studies

2. Verbal Reports/Think-Aloud Evaluations anyone can collect them without specific training, and they can be much more easily

manipulated than other behavioral measures as valid as other behavioral measures considered as an excellent way to gain access to the contents of someone’s short term

memory – time is very important think out loud – mental walk – questions that call for LTM retrieval or that call on highly

automated responses will not provide rich verbal reports difficulties in scoring and analysis -- questionnaires

3. Surveys and Questionnaires easiest and least expensive method questions specific rather then general, actual system experience rather than possible

system changes or extensions selections on a scale with a limited number of points questionnaires are administered after an experience -- recollections may be distorted



4. Use Data Collection other measures (such as errors, task completion times, requests for help, general logging

data) can be useful often used in more formal quantitative analysis – empirical data

5. Design Walkthroughs using low-fidelity, paper and pencil or storyboard mockups of a system under development representative end users, programmers, architectures, usability engineers the team uses a prototype of the system to walk through a series of typical end user tasks data can be collected in the form of expected problems or recommended design

alternatives the key challenge of the design review is to consider all elements of the software system,

including errors conditions and assistance seeking behavior

7. Theory-based Reviews can substitute use based evaluations with analysis alone to provide insight into predicted

use – KLM, GOMS



5. Use Inspection Based Techniques whether to use empirical usability testing or inspection methods or both it is not reliable to obtain user performance data with inspection as compared to testing

methods theory-based techniques more reliable for relative comparisons than for absolute predictions

Karat et al. (1991) the usability problems that an inspection method missed were relatively severe

Wharton et al. (1992) inspection methods can identify more potential problems that do not present themselves in a significant way in real use



Distributed Cognition

1. INTRODUCTION a perspective on all of cognition, encompassing interactions between people and with

resources and materials in the environment1. the boundaries of the unit of analysis for cognition2. the range of mechanisms that may be assumed t participate in cognitive processes

2. A DISTRIBUTED COGNITION APPROACH1. Socially Distributed Cognition

the cognitive properties of societies of individuals – social organization is itself a form of cognitive architecture

social interactions as well as interactions between people and structure in their environments

2. Embodied Cognition the organization of mind is an emergent property of interactions among internal (memory,

attention, executive function) and external resources (the objects, artifacts, and at-hand materials constantly surrounding us)

3. Culture and Cognition the study of cognition is not separable from the study of culture, because agents live in

complex cultural environment distributed cognition returns culture, context, and history to the picture of cognition



4. Ethnography of Distributed Cognitive Systems cognitive ethnography in 60’s and 70’s focused on the meanings of words – sought in the

contents of individual minds – extends to the material and social means of the construction of action and meaning

event-centered ethnography – not only what people know, but how they go about using what they know to do what they do

not enough to know how the mind processes info – necessary to know how the info to be processed is arranged in the material and social world

3. AN INTEGRATED FRAMEWORK FOR RESEARCH

4. CONCLUSIONS AND FUTURE DIRECTIONS HCI exclusively focused on single individuals interacting with applications derived from

decompositions of work activities into individual tasks distributed cognition – understanding interactions among people and technology

ethnographic studies of the phenomena f interest and with natural histories of the representations



23. Ubiquitous Computing1. INTRODUCTION

the accomplishments and remaining challenges for three themes1. natural interfaces2. context-aware computing3. automated capture and access for live experiences everyday computing

2. COMPUTING WITH NATURAL INTERFAES natural actions can and should be used as explicit or implicit input to ubicomp system speech-related interfaces, perceptual interfaces, pen-based or free-form interactions, grasp-

able or tangible interfaces1. First-Class Natural Data Type

natural interfaces – audio, video, ink, and sensor input structured gestures, ability to merge independent strokes together as they form letters,

words, and other segments of language2. Error-Prone Interaction for Recognition-Based Interaction

Error reduction – improving recognition technology in order to eliminate or reduce errors Error discovery – thresholding of confidence measures, historical statistics, explicit rule

specification Reusable infrastructure for error correction -- toolkits



3. CONTEXT-AWARE COMPUTING location – GPS-based car navigation systems, hand-held tour guide individual objects – barcode or identifying tag, vision-based recognition

1. What is Context? who, what, where, when, why

2. Representations of Context how to represent context

3. The Ubiquity of Context Sensing – Context Fusion few truly ubiquitous, single-source context services assemble context information from a combination of related context services – sensor fusion

4. Coupling Context-Aware and Natural Interaction – Augmented Reality “probing the world with a tool” metaphor

4. AUTOMATED CAPTURE AND ACCESS TO LIVE EXPERIENCES1. Challenges in Capture and Access

1. Capture wished we had a camera, difficult to find the picture or film informal setting -- not documented properly; formal meetings – poorly captured raw streams of info that are captured mainly for the purpose of direct playback2. Access playback in real time – pinpoint a particular topic, summarization of experience synchronization of multiple captured streams during playback -- foreshadowing



5. TOWARD EVERYDAY COMPUTING results from considering the consequences of scaling ubicomp with respect to time they rarely have a clear beginning or end interruption is expected multiple activities operate concurrently time is an important discriminator associative models of information are needed

1. Synergy Among Themes2. Research Directions in Everyday Computing

Design a continuously present computer interface wearables – continually worn interface, but limited by the current input and display

technologies and are text-based interfaces Presenting info at different levels of the periphery of human attention

generic peripheral backdrop with no mechanism for the user, or the background task, to move the peripheral info into the foreground of attention

Connecting events in the physical and virtual world Modifying traditional HCI methods to support designing for informal, peripheral, and

opportunistic behavior

6. ADDITIONAL CHALLENGES FOR UBICOMP1. Evaluating Ubicomp Systems

1. Finding a Human need reliable system to evaluate



build a compelling story, from the end-user’s perspective, on how any system or infrastructure to be built will be used – the basis for evaluating the impact of a system on the everyday life

feasibility study – how a system is being used, what kinds of activities users are engaging in with the system, whether the overall reactions are positive or negative

2. Evaluating in the Context of Authentic Use3. Task-Centric Evaluation Techniques Are Inappropriate

2. Social Issues for Ubiquitous Computing who can access and modify the contents – security and encryption schemes the lack of knowledge of what some computing system is doing – invisibility control the distribution and use of the info when and what to capture privacy

7. CONCLUDING THOUGHTS



6. Cognitive Models1. Advantages for HCI

cognitive models in three main ways in HCI1. to help examine the efficacy of different designs to predict task performance2. to provide assistance – embedded assistant3. to substitute for users difficulty of connecting the cognitive models to their task environment

2. A Route to Supporting Models as Users1. Artifacts of the Cognitive Modeling Process

produce a cognitive model that performs like a human can be viewed as producing three artifacts1. the cognitive model itself, which simulates the cognitive performance and behavior of a

human performing the task2. task application or its simulation3. mechanism that supports interaction between the model and the task simulation

simulates human perception and action2. Role of User Interface Management Systems

tools used to develop user interfaces – a toolkit to support the cognitive modeling process1. a tool to create interface2. a run-time mechanism that lets the cognitive model interact with the task simulation3. a communication mechanism between the cognitive model and the task simulation



3. Cognitive Model Interface Management Systems1. extend the cognitive model by adding a simulated eye and a simulated hand for perception

and action2. link the cognitive model to the simulation

4. A Functional Model Eye and Hand most important parts of the CMIMS – visual acuity and the speed of motor movements

3. Example Cognitive Models That Perform Interactive Tasks1. A Simplified Air Traffic Control Model

explore how to create a general eye and investigate what a model would do with an eye the basic task involves learning how to direct a single aircraft to land at an airport located

at the center of the display a crucial element of the task is that change of heading command must be issued at the

appropriate time, which requires that the cognitive model be able to detect when an aircraft is approaching a way marker (appear on the screen as crosses)

3. Summary a simple but functional Sim-eye (Soar-IO) could be created using an existing UIMS the eye is not just the fovea; the periphery is needed even for simple search

2. Tower of Nottingham Model Sim-eye and a pair of Sim-hands (grasp, release, rotate)3. Summary the Sim-hand and Sim-eye are generally applicable and that they can be a different

cognitive architecture – ACT-R



4. Related Systems EPIC (Kieras and Meyer, 1997) – used to make accurate predictions of interaction

behavior capabilities and regularities used by Soar and ACT-R/PM ACT-R/PM is more concerned with detailed psychological predictions, but it is not yet

positioned to be a tool that can be widely used to develop user interfaces

4. Cognitive Models as Users in the New Millennium1. Implications for Models

interaction – where to look and what to do with what they see peripheral vision after focusing on functional capabilities, the next is to incorporate more experimental

regularities to model human performance2. Implications for Interfaces

two ways CMIMSs facilitate the improvement of Uis1. cognitive models can be used to evaluate user interfaces2. the ability to tie more accurate user models to interfaces opens up a range of

applications, such as more accurate intelligent assistants to help novices



GOMS1. GOMS AS COGNITIVE MODELING

Card et al. proposed a framework for building analytic models of human performance with computers – two key components

1. a general characterization of the human information-processing system, in term of both a system architecture and quantitative parameters of component performance Model Human Processor (MHP)

2. the GOMS model – four cognitive components of skilled performance in tasks strength – its ability to predict the time it takes a skilled user to execute a task based on

the composite of actions of retrieving plans from LTM, choosing among alternative available methods, keeping track of what has been done and what needs to be done, and executing the motor movements necessary for the keyboard and mouse routine cognitive skills can be described as a serial sequence of cognitive actions and motor activities

Card et al. found parameters that were very consistent across tasks:1. a keystroke (k) for a midskilled typist -- 280 msec.2. a single mental operator (M) from LTM to WM – 1.35 sec3. pointing (P) – average 1.1 sec (Fitts’ law)4. moving the hand (M) from keyboard to the mouse – 400 msec

1. Limitations of the GOMS Approach



2. ADVANCES IN MODELING SPECIAL SERIAL COMPONENTS1. Motor Movements

Keying depending on the skill level of the typist, the frequency with which the particular key is used,

and the predictability and continuity of the text to be typed average typist – 280 msec per keystrokeMoving a Mouse distance and target size – 1.1 sec (Card et al., 1983), following Fitts’ lawAn Example of the Application of GOMS and MHP to Design Generation the time to select an item was far shorter when the menu popped up to the right of the

cursor than when the menu appeared below it Fittsized menus (the target size grows as the distance from the cursor’s starting position

increases) vs. putting a virtual border on the top, right, and bottom edges of the pop-up menu 1.9 sec vs. 450 msec

Hand Movements the time needed to move from the space bar of the keyboard until the pointing control

begins to move the cursor – 360 msec2. Perception

the perceptual processor at 100 msec and a saccade at 230 msec perception of words includes recognition, some verbal encoding, or retrieval of meaning in

addition to simple perception scanning, storing, and retrieving – 2.3 sec



3. Memory and Cognitive ProcessesMemory Retrieval retrieval of well-known units from LTM for placement in WM, ready then to be either

executed by a motor processor or further decomposed by subsequent retrieval from LTM – 1350 msec

Executing Steps in a Task the execution of each procedural step – 70 msecChoosing Among Methods the more choices for a response, the longer the expected response time (Hick’s law) – 620

msec4. Predicting Composite Performance From These Parameters

3. EXTENTION OF THE BASIC FRAMEWORK1. Learning and Transfer

Time to Learn Kieras and Polson developed an extension of GOMS they called Cognitive Complexity

Theory provides a basis for making quantitative predictions about the time to learn and the amount of transfer NGOMSL

time to learn each step – 30 sec; best guess is 25 sec per productionTransfer of Training From One System to the Other the number of productions the two systems share provides a good metric for predictions of

the amount of transfer



2. The Analysis of Errors: Forgetting From Working Memory



2. Definitions and a Notation for GOMS Models1. Goals

A goal is something that the user tries to accomplish. a goal description is an action-object pair in the form: <verb noun>

2. Operators operators are actions that the user executes a goal is something to be accomplished, while an operator is just executed

3. Methods a method is a sequence of steps that accomplishes a goal

4. Selection Rules the purpose of a selection rule is to route control to the appropriate method to accomplish a

goal

3. General Issues in GOMS Task Analysis1. Judgment Calls

in order to do a useful task analysis, the analyst must make judgment calls on these issues how users view the task in terms of their natural goals how they decompose the task into subtasks what the natural steps are in the user’s methods



2. Pitfalls in Talking to Users people have only a very limited awareness of their own goals, strategies, and mental

processes in general what users actually do can differ a lot from what they think they do what users actually do with a system may not in fact be what they should be doing

3. Bypassing Complex Processes the approach presented here is to bypass the analysis of a complex process by simply

representing it with a “dummy” or “placeholder” operatorRepresenting Bypass Processes a bypassed process could be represented just by defining a complex mental operator and

using it wherever it is needed I the methods the user’s task is only to interact with the system in order to carry out this completely described

task, referring to the yellow pas as necessary to obtain the required information4. Analyze a general Set of Tasks, Not Specific Instances

task scenarios trace – the list of specific actions that the user would perform for a specific task the goal of GOMS task analysis is a description of the general methods for accomplishing a set

of tasks, not just the method for executing a specific instance of a task



4. A Procedure for Constructing a GOMS Model top-down, breadth first expansion of methods

21. direct manipulation

Documents