Creating a Lightweight User InterfaceDescription LanguageAn Overview and Analysis of the Personal Universal Controller Project

Jeffrey NicholsIBM Research – Almaden

ToCHI Track – CHI 2010April 15, 2010

2

Motivation

April 29, 1991

CNN – Jan 2004

Economist – Oct 2004

Nov 27, 2006

3

Personal Universal Controller

Personal MobileDevice

AbstractSpecificatio

n

Control

State Feedback

Automatically

GeneratedInterface

PUC[Nichols, UIST 02]

4

Automatic Generation of UIs

Benefits• Generated interfaces can be

better than today’s appliance interfaces

As shown by user study

>

5

Automatic Generation of UIs

Benefits• Generated interfaces can be

better than today’s appliance interfaces

As shown by user study

• All interfaces personally consistent for each user

Across functionally similar appliances

Original

Uniform[Nichols, CHI 06]

Consistent

6

Automatic Generation of UIs

Benefits• Generated interfaces can be

better than today’s appliance interfaces

As shown by user study

• All interfaces personally consistent for each user

Across functionally similar appliances

• Aggregate interfaces for systems of multiple appliances

Functions from multiple appliances combined into task-based interfaces

Aggregate Interface

Huddle[Nichols, UIST 06]

Flow-Based Interface

7

Today

The PUC Specification Language

•Informative enough to allow generation of high-quality interfaces

•Easy to learn and use

•Descriptive enough for a wide-range of complex appliances

•Supports inclusion of high-level design conventions

•Formal enough to allow algorithms to reason about appliance behavior

Personal MobileDevice

AbstractSpecificatio

n

Appliance

8

Design Principles

• Descriptive enough for any appliance

• Sufficient detail to generate a high-quality interface

• No specific layout information

• Support generation for different devices and modalities

• Short and concise

• Only one way to specify anything

9

Design ProcessStarted by exploring appliance user interfaces

• Created handheld interfaces for two appliances

• Found these interfaces to be better than manufacturers’ interfaces

• Analyzed the interfaces for functional information

Designed appliance specification language

Built interface generators

Iterated on the specification design• Wrote additional specifications for

increasingly complex appliances• Modified language as necessary to

support additional complexity

10

Specification LanguageLanguage

• Functional information • XML-based

Elements• State variables & commands• Group tree• Multiple labels per object• Dependency information

Full documentation available at: http://www.pebbles.hcii.cmu.edu/puc/

11

Dependency InformationUseful for determining structure of user interface

• Some groups are never available simultaneously

• Nondependent controls can be split into more accessible side-panels

• Mutually exclusive groups can be placed on overlapping panels

Improves usability and robustness of specification language

• Reduces emphasis on group tree for structuring interfaces

• Dependency information is objective

12

Smart Templates

Designed Smart Templates technique [Nichols, IUI 04]

• Allows generated interfaces to include high-level design conventions

• Flexible parameterization specified using primitive elements of specification language

Without Smart Templates

With Smart Templates

13

Evaluation

Dimensions• Completeness

What appliances can we write specifications for?What specifications can we generate interfaces for?

• Usability of Specification Language

• Usability of Generated InterfacesHow good are the generated interfaces?

14

CompletenessWe have written 33 complete specifications for a wide variety of appliances

• Home theater appliancesInFocus Television, VCRs, DVD players, Sony receiver, Audiophase shelf stereo,

Sony camcorder, Windows Media Player

• Copiers, All-in-one printers

• Lighting controls (X10, Lutron, UPnP)

• AT&T telephone/answering machine

• Alarm clocks

• Elevator

• Desktop computer applicationsPowerPoint, Photo browser, To-do list

• In-vehicle systems for GMC Yukon DenaliDriver Information Console, Climate Controls, Navigation System

PUC can generate interfaces for all of these specifications

15

Completeness: Navigation Interface

16

Completeness: Printer Interfaces

HP printer Canon printer

17

Usability of Specification

Study of Authoring Appliance Specifications

• Three subjects wrote specifications for alow-end Panasonic VCR

All subjects instructed to: • Include all functions• Be faithful to the existing appliance design

Results:• Subjects were able to learn the language in a

1.5 hour session• About 6 hours was required to create the VCR

specification• Specifications had some problems, but

subjects were not allowed to test their designs

Low End: Panasonic VCR

What consistency issues arise in our domain?

18

Usability of Generated InterfacesEvaluation of Generated Interfaces

• Users perform tasks with two all-in-one printer interfaces (HP and Canon)

• Two comparisonsPhysical interface to PUC interface

Without consistency and with consistency

• 48 participants (28 male, 20 female)

Divided into 6 groups, 8 per group

Recruited through CBDR web page

• MetricsCompletion timeFailures

HP printer Canon printer

19

Usability Results#1 - HP #2 - Canon

Built-In Physical Physical

AutoGen PUC PUC

Consistent PUC Uniform

#1 - Canon #2 - HP

Built-In Physical Physical

AutoGen PUC PUC

Consistent PUC Uniform

* = p < 0.05† = p < 0.1

* *

Failures per subject also significantly less for PUC than Physical(Fisher’s Exact Test, p < 0.05)

HP: 1.125 Built-In vs. 0.125 PUCCanon: 2.0 Built-In vs. 0.625 PUC

* * * * * * * * *† †

20

Retrospective

What Worked:• Short and concise principle• Use of workarounds for particularly difficult functions (e.g. Smart Templates)

What Didn’t Work:• “Only one way to specify anything” principle• Some features turned out not to be useful• Versioning

21

Takeaways

• Iterative design is important, even for modeling languages

• Need to choose a work balance between the specification author and interface generator

• Balance between subjective and objective information

• High-quality user interfaces can be automatically generated

22

AcknowledgementsThesis Committee

• Brad A. Myers (chair)• Scott Hudson• John Zimmerman• Dan Olsen Jr.

Funding• National Science

Foundation• Microsoft• General Motors• Intel• Pittsburgh Digital

Greenhouse

Equipment Grants• Mitsubishi (MERL)• VividLogic• Lucent• Lutron• Lantronix• Nokia

PUC Project Members• Brandon Rothrock• Duen Horng Chau• Kevin Litwack• Thomas K. Harris• Michael Higgins• Joseph Hughes • Roni Rosenfeld• Rajesh Seenichamy• Pegeen Shen• Htet Htet Aung• Mathilde Pignol• Suporn Pongnumkul• Stefanie Shriver• Jeffrey Stylos• Peter Lucas

Collaborators & Friends• Naomi Ramos• Desney Tan• Daniel Avrahami• Gaetano Borriello• Laura Dabbish• Andrew Faulring• James Fogarty• Krzysztof Gajos• Darren Gergle• Andy Ko• Amy Nichols• Mick Nichols• Sally Nichols• Trevor Pering• Fleming Seay• Irina Shklovski• Rachel Steigerwalt• Roy Want• Jake Wobbrock• and many others…

Thanks for listening!

For more information…http://www.jeffreynichols.com/http://www.pebbles.hcii.cmu.edu/puc/

24

ToCHI Papers in ProcessN

umbe

r of S

ubm

issi

ons

Months since Submission

Comparison to other TransactionsTotal

ArticlesTotal

DownloadsTotal

Cited ByAve. Downloads

per ArticleAve. Citesper Article

ACM Letters on Programming Languages and Systems (LOPLAS) 42 20121 762 479 18ACM Transactions on Accessible Computing (TACCESS) 28 5497 835 196 30ACM Transactions on Algorithms (TALG) 228 76689 5536 336 24ACM Transactions on Applied Perception (TAP) 164 58555 5498 357 34ACM Transactions on Architecture and Code Optimization (TACO) 113 51945 3998 460 35ACM Transactions on Asian Language Information Processing (TALIP) 146 59699 3775 409 26ACM Transactions on Autonomous and Adaptive Systems (TAAS) 74 34888 2540 471 34ACM Transactions on Computation Theory (TOCT) 7 1297 165 185 24ACM Transactions on Computational Logic (TOCL) 236 74990 8103 318 34ACM Transactions on Computer Systems (TOCS) 362 341130 11969 942 33ACM Transactions on Computer-Human Interaction (TOCHI) 261 372520 10506 1427 40ACM Transactions on Computing Education (TOCE) 21 2133 669 102 32ACM Transactions on Database Systems (TODS) 726 504504 21880 695 30

ACM Transactions on Design Automation of Electronic Systems (TODAES) 494 195495 14079 396 29ACM Transactions on Embedded Computing Systems (TECS) 260 175947 8346 677 32ACM Transactions on Graphics (TOG) 1321 1145631 39623 867 30ACM Transactions on Information Systems (TOIS) 471 441142 17095 937 36ACM Transactions on Information and System Security (TISSEC) 205 254079 6918 1239 34ACM Transactions on Internet Technology (TOIT) 150 217521 5794 1450 39ACM Transactions on Knowledge Discovery from Data (TKDD) 62 30515 2212 492 36ACM Transactions on Mathematical Software (TOMS) 1254 630979 19279 503 15ACM Transactions on Modeling and Computer Simulation (TOMACS) 315 156620 8691 497 28ACM Transactions on Multimedia Computing, Communications, and Applications (TOMC 124 64142 4330 517 35ACM Transactions on Programming Languages and Systems (TOPLAS) 914 436568 29429 478 32

ACM Transactions on Reconfigurable Technology and Systems (TRETS) 50 7991 1241 160 25ACM Transactions on Sensor Networks (TOSN) 127 87315 4265 688 34ACM Transactions on Software Engineering and Methodology (TOSEM) 253 259823 10251 1027 41ACM Transactions on Speech and Language Processing (TSLP) 30 16705 1175 557 39ACM Transactions on Storage (TOS) 84 45645 3225 543 38ACM Transactions on the Web (TWEB) 50 38943 1944 779 39

#2 in Ave. Downloads per Article

#2 in Ave. Citations per

Article