User Interface Design Toolsfor the Future

Multimodal UI Research in the HCC

James A. Landay

Jason Hong, Scott Klemmer,Jimmy Lin, Mark Newman, & Anoop Sinha

HCC RetreatJuly 5, 2000

7/5/2000 2

Several Projects on Design UIs of the Future

Designer’s Outpost SUEDE Multimodal Design Assistant

7/5/2000 3

The Future: Information-centric UIs

7/5/2000 4

Designer’s Outpost: Tangible Tools for Information Design

7/5/2000 5

Designer’s Outpost: Virtual Feedback on Physical Layouts

7/5/2000 6

Designer’s Outpost: Virtual Feedback on Physical Layouts

7/5/2000 7

Designer’s Outpost: Virtual Linking of Related Information

7/5/2000 8

Designer’s Outpost: Export to Next Stage Tools (DENIM)

7/5/2000 9

Designer’s Outpost: Infrastructure for Next Prototype

Two video/still cameras* still camera above desk/wall

+captures ink / IDs* video camera below desk/wall

+occlusion free+captures structure

SmartBoard* 1280 x 1024 resolution

+looking for a projector donation* direct pen input

Cross iPen tablet* high-res ink capture

7/5/2000 10

One Future: Our Access to Information will be via Speech

Read my important email

7/5/2000 11

SUEDE: Low-fidelity Prototyping for Speech-based UIs

7/5/2000 12

A Better Future: Our Access to Info will be via Multimodal UIs

Take advantage of the best mode of input/output for the task/context at hand

How do we design so that these new devices will not seem impoverished relative to desktop computers?

7/5/2000 13

A Better Future: Our Access to Info will be via Multimodal UIs

How do we combine speech, gesture, etc. into a UI design?* rapid production of “rough cuts”

+ informal (sketching / “Wizard of Oz”)+ iterative design (user testing/fast mods)

* generate initial code+ UIs for multiple devices+ designer adds detail / improve UI

Study* uses of these novel modes alone

+ e.g., SUEDE* construction of multimodal apps

+ e.g., a multimedia notebook

Early stages so far* based on inferring models

Model

More information on these design tools at

http://guir.berkeley.edu