Transcript
Page 1: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 1

Jeffrey Nichols and Brad A. MyersCarnegie Mellon UniversityOctober 30, 2002

Generating Remote Control Interfaces for Complex Appliances

The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France

Page 2: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 2

The Problem

Appliances are too complex

Page 3: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 3

The Problem, cont.

Each complex appliances has its own idiosyncratic interface! Stereo systems Telephones VCRs Alarm Clocks …

Increasingly Computerized

Low Usability

Page 4: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 4

Our Solution

Separate the interface from the appliance!

Handheld becomes personal universal controller (PUC)Key Features

Interface-independent appliance specification Automatic generation of GUI and speech interfaces

Specifications

Control

Feedback

Page 5: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 5

Automatic Generation of UIs

Benefits All interfaces consistent for the user

With conventions of handheld Even from multiple manufacturers

Addresses hotel alarm clock problem!

Multiple modalities (GUI + Speech UI)

Can take into account user preferences

Will work on special purpose devices (for disabled)

Page 6: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 6

Important Work By Others INCITS V2 Standardization Effort

Alternative Interface Access Protocol (AIAP)[Zimmermann, CHI 2002]

User Interface Modeling Language (UIML)http:/www.uiml.org/

Xweb (now ICE)[Olsen Jr., UIST 2000]

Stanford iRoom, iCrafter[Ponnekanti, Ubicomp 2001]

Speakeasy[Newman, UIST 2002]

Page 7: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 7

- Comm. Protocol- Interface Generators- Specification Lang.

Architecture

XML-based

- Appliance Adaptors

Page 8: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 8

Language Design

Approach Create reference interfaces

AIWA Shelf Stereo AT&T Telephone/Answering

Machine

Test interfaces with subjects Users twice as fast and made

half the errors with reference interfaces as compared to manufacturers’ interfaces

Analyze interfaces for functional information

Page 9: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 9

Language Elements

State Variables and Commands Represent functions of appliance State variables have types

Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all

“seek” button on a Radio

Label Information One label not suitable everywhere

The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and

text-to-speech information

Page 10: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 10

Language Elements, cont.

Group Tree Specify

organization of functions

We use n-ary tree with variables or commands at leaves

Page 11: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 11

Language Elements, cont.

Dependency Information Formulas that specify when a variable or

command is active in terms of other state variables

Equals, Greater Than, Less Than Linked with logical operators (AND, OR)

For example, <and>

<equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>

Page 12: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 12

Interface Generators

Generators for Two Modalities Graphical

Implemented for PocketPC in Java 1.1 Uses dependency information to generate panel

structure of interface

Speech Implemented using Universal Speech Interface

(USI) techniques [Rosenfeld 2001] Uses dependency information to disambiguate

shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)

Page 13: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 13

Graphical Interface Generator

Focuses on panel structure of user interface

Small groups of controls have basic layouts

Complexity comes from structure of groups

Structure can be inferred from dependency info!

Page 14: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 14

Inferring Structure

Find sets of variables that are “mutually exclusive” Every variable in a set will

never be active at the same time as a variable in another set

Create structure with sets, using overlapping panels

Page 15: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 15

Choosing Panel Typesa)

b)

c)

full screen

tabbed

partial screen

Page 16: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 16

Making the Interface Concrete

Finish conceptual layout Choose controls (decision

tree) Choose row layouts

(one column, two column, etc.)

Allocate space Examine panel contents and

choose sizes

Instantiate and place controls

Page 17: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 17

Generating Speech Interfaces

Automatically build USI tree from dependencies Allows verbal navigation of functional groups

Automatically generate grammar for parser Phrases for query and control

“What is playmode?”“Set playmode to play”“play”

Automatically generate language model and pronunciation for recognizer

Page 18: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 18

Controlling Appliances

We have built adaptors for many actual appliances

Sony Digital Camcorder X10 Lighting Audiophase Shelf Stereo AudioReQuest MP3 player Software Media Players (WinAmp, WMP)

Written specifications for others Elevator Telephone/Answering Machine

Page 19: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 19

Demonstration

Page 20: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 20

Future Work Improve quality of generated user interfaces

Back-tracking, etc.

Specification and generation of list interfaces Necessary for many appliances

Adaptive interface generation New interfaces modeled on past interfaces

General purpose appliance adaptor for HAVi and UPnP protocols

Page 21: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 21

Acknowledgements

PUC Project Members Brad A. Myers Michael Higgins Joseph Hughes Thomas K. Harris Roni Rosenfeld Mathilde Pignol Stefanie Shriver Peter Lucas Kevin Litwack

Funding National Science Foundation Pittsburgh Digital

Greenhouse Microsoft

Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent

Page 22: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 22

User Interface Software & Technology (UIST)October 30, 2002

Thanks!

http://www.cs.cmu.edu/~pebbles/puc/http://www.cs.cmu.edu/~jeffreyn/

Page 23: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 23

Page 24: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 24

Determining Panel Structure

1. Find states that others depend upon

2. Search for the group that contains this state

3. Determine mutually exclusive sets within group children

4. Assign sets to panels Based upon types, tree

properties

Power

Mode

Band

PlayState

CDTrack

FMStation

FMPresets

AMStation

AMPresets

Radio

CD

Mode == Any

Mode == Radio

Mode == CD

Page 25: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 25

Choosing Panel Types

Rules for Choosing Panels If…

state has boolean type all controls are one set

Create… Two full-screen overlapping

panels.

Page 26: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 26

Choosing Panel Types, cont.

Rules for Choosing Panels

If… state has enumerated type

there is a set for every value of the state

Create… a tabbed panel

Page 27: Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #� � � 27

Choosing Panel Types, cont.

Rules for Choosing Panels If…

there are mutually exclusive sets of controls

Create… overlapping panels a control for the state not on

the overlapping panel


Top Related