1 past, present and future of user interface software tools brad a. myers (work with scott e. hudson...

1

Past, Present and Future ofUser Interface Software ToolsPast, Present and Future ofUser Interface Software ToolsBrad A. MyersBrad A. Myers(work with Scott E. Hudson(work with Scott E. Hudsonand Randy Pausch)and Randy Pausch)

Human-Computer Interaction InstituteHuman-Computer Interaction InstituteSchool of Computer ScienceSchool of Computer ScienceCarnegie Mellon UniversityCarnegie Mellon University

http://www.cs.cmu.edu/~bamhttp://www.cs.cmu.edu/[email protected]@cs.cmu.edu

MyersMyersCMU - HCI InstituteCMU - HCI Institute 2HFES’00HFES’00

Introduction

User Interface Software Tools Help developers design and implement user interfaces

Focus on Tools, but influenced by future UIs Today’s tools are highly successful

Window Managers, Toolkits, Interface Builders ubiquitous Most software built using them Are based on HCI research

Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

Future tools must be different

User Interface Software Tools Help developers design and implement user interfaces

Focus on Tools, but influenced by future UIs Today’s tools are highly successful

Window Managers, Toolkits, Interface Builders ubiquitous Most software built using them Are based on HCI research

Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

Future tools must be different


What Worked

Window Managers and Toolkits Event Languages Graphical, Interactive Tools Component Architectures Hypertext Object Oriented Programming

Window Managers and Toolkits Event Languages Graphical, Interactive Tools Component Architectures Hypertext Object Oriented Programming


Window Managers

Multiple (tiled) windows in research systems of 1960’s: NLS, etc.

Overlapping introduced in Alan Kay’s thesis (1969) Smalltalk, 1974 at Xerox PARC Successful because multiple windows help users

manage scarce resources: Screen space and input devices Attention of users Affordances for reminding and finding other work

Multiple (tiled) windows in research systems of 1960’s: NLS, etc.

Overlapping introduced in Alan Kay’s thesis (1969) Smalltalk, 1974 at Xerox PARC Successful because multiple windows help users

manage scarce resources: Screen space and input devices Attention of users Affordances for reminding and finding other work


Toolkits

A collection of widgets (“controls”) Menus, scroll bars, text entry fields, buttons, etc.

Toolkits help with programming Help maintain consistency among UIs

Key insight of Macintosh toolbox

Successful partially because address common, low-level features for all UIs

A collection of widgets (“controls”) Menus, scroll bars, text entry fields, buttons, etc.

Toolkits help with programming Help maintain consistency among UIs

Key insight of Macintosh toolbox

Successful partially because address common, low-level features for all UIs


Event Languages

Create programs by writing event handlers Many UIMSs used this style

Univ. of Alberta (1985), Sassafras (1986), etc.

Now used by HyperCard, Visual Basic, Lingo, etc. Toolkits with call-backs or action methods are related

Advantages: Natural for GUIs since generate discrete events Flow of control in user’s hands rather than programmer’s

Discourages moded UIs

Won’t work well in future

Create programs by writing event handlers Many UIMSs used this style

Univ. of Alberta (1985), Sassafras (1986), etc.

Now used by HyperCard, Visual Basic, Lingo, etc. Toolkits with call-backs or action methods are related

Advantages: Natural for GUIs since generate discrete events Flow of control in user’s hands rather than programmer’s

Discourages moded UIs

Won’t work well in future


Graphical Interactive Tools

Create parts of user interface by laying out widgets with a mouse Examples: Menulay (1983), Trillium (1986), Jean-Marie

Hullot from INRIA to NeXT Now: HyperCard, Interface Builders, Visual Basic’s

layout editor, resource editors, “constructors”

Advantages: Graphical parts done in an appropriate, graphical way Accessible to non-programmers

Create parts of user interface by laying out widgets with a mouse Examples: Menulay (1983), Trillium (1986), Jean-Marie

Hullot from INRIA to NeXT Now: HyperCard, Interface Builders, Visual Basic’s

layout editor, resource editors, “constructors”

Advantages: Graphical parts done in an appropriate, graphical way Accessible to non-programmers


Component Architectures

Create applications out of components which are separately developed and compiled In UI software, each component controls an area of the

screen Example: drawing component handles picture inside a

document

Invented by Andrew research project at CMU (1988) Now: OLE, OpenDoc, ActiveX, Java Beans

Create applications out of components which are separately developed and compiled In UI software, each component controls an area of the

screen Example: drawing component handles picture inside a

document

Invented by Andrew research project at CMU (1988) Now: OLE, OpenDoc, ActiveX, Java Beans


Hypertext

Ted Nelson named it in 1965 and developed a Hypertext system at Brown University

Important systems: NLS (1967), Hyperties (1986) World-Wide Web

Phenomenal success due to: Ease of use of Mosaic browser Support for embedded graphics Support for easy authoring

Ted Nelson named it in 1965 and developed a Hypertext system at Brown University

Important systems: NLS (1967), Hyperties (1986) World-Wide Web

Phenomenal success due to: Ease of use of Mosaic browser Support for embedded graphics Support for easy authoring


Object Oriented Programming

Success of OO owes much to UI software field Popularized by Smalltalk GUI elements (widgets) seem like objects

Have state, accept events (messages)

Rise parallels GUIs C++ with Windows 3.1

Success of OO owes much to UI software field Popularized by Smalltalk GUI elements (widgets) seem like objects

Have state, accept events (messages)

Rise parallels GUIs C++ with Windows 3.1


What Hasn’t Caught On

User Interface Management Systems Separation of UI and application is harder than expected

Formal Language-Based Tools Grammars and Transition networks are not appropriate

for direct manipulation

Constraints Declare once, system maintains Get to be unpredictable and hard to control

Model-Based and Automatic Techniques Results are not as good as hand-created Sometimes, no savings of effort

User Interface Management Systems Separation of UI and application is harder than expected

Formal Language-Based Tools Grammars and Transition networks are not appropriate

for direct manipulation

Constraints Declare once, system maintains Get to be unpredictable and hard to control

Model-Based and Automatic Techniques Results are not as good as hand-created Sometimes, no savings of effort


Future Prospects and Visions

Important Trends Ubiquitous Computing Move to recognition-based interfaces End-user customization and scripting

Important Trends Ubiquitous Computing Move to recognition-based interfaces End-user customization and scripting


Ubiquitous Computing

Computation embedded in many kinds of devices Digital pagers and cell phones, Palm Pilots,

CrossPads, laptops, wall-size displays, “smart” rooms Next wave: easy communication with radio

E.g., BlueTooth: www.bluetooth.com

Significant Implications for tools Moving target problem Tools for coordinating multiple, distributed,

communicating devices “Multi-computer” user interfaces

Computation embedded in many kinds of devices Digital pagers and cell phones, Palm Pilots,

CrossPads, laptops, wall-size displays, “smart” rooms Next wave: easy communication with radio

E.g., BlueTooth: www.bluetooth.com

Significant Implications for tools Moving target problem Tools for coordinating multiple, distributed,

communicating devices “Multi-computer” user interfaces


More problems: Varying Input and Output Today’s Desktop screens vary by a factor of 2.5 in

size and a factor of 4 in pixels Tomorrow’s screen will vary by factors of 100 in

size and a factor of 625 in pixels Cell phone to Stanford’s wall (3796 x 1436 pixels)

Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels

Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels Cell phone to Stanford’s wall (3796 x 1436 pixels)


Varying I/O Capabilities, cont.

Interaction techniques for desktop will not work No room on small devices Can’t reach menubar on wall-size devices

Want to run same application on different devices

Interaction techniques for desktop will not work No room on small devices Can’t reach menubar on wall-size devices

Want to run same application on different devices


Need for Prototyping Devices

User interface will be in hardware Rapid design and prototyping needed for hardware Pragmatics and usability cannot be evaluated from

a simulation on a screen

User interface will be in hardware Rapid design and prototyping needed for hardware Pragmatics and usability cannot be evaluated from

a simulation on a screen


Move to recognition-based interfaces

Speech, gestures, camera-based vision Multimodal interaction

User will pick which modality to use Use multiple modalities at same time

Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. Need tools to hide these complexities

Speech, gestures, camera-based vision Multimodal interaction

User will pick which modality to use Use multiple modalities at same time

Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. Need tools to hide these complexities


Fundamental Differences ofRecognition-based UIs

Input is uncertain Recognition can make errors Requires monitoring, feedback, correction

Do not produce discrete events GUI event model no longer works

Interpreting input requires deep knowledge of data Context of the application “Move the red truck to here” Need a architecture based on accessible application

data structures “Reflection”, “Open Data Model”

Input is uncertain Recognition can make errors Requires monitoring, feedback, correction

Do not produce discrete events GUI event model no longer works

Interpreting input requires deep knowledge of data Context of the application “Move the red truck to here” Need a architecture based on accessible application

data structures “Reflection”, “Open Data Model”


More Assumptions of Today’s Tools

Skill and Dexterity of users Older users Makes single, fixed library of widgets untenable

Non-overlapping and opaque components Preclude translucency, magic lens interactions

Skill and Dexterity of users Older users Makes single, fixed library of widgets untenable

Non-overlapping and opaque components Preclude translucency, magic lens interactions


Operating Systems Considerations

What is in the OS? Window Manager? Toolkit? Communication? Scripting

facilities?

Need ever increasing services for applications Need more access to low-level information

E.g., hardware buttons, whether on network

Ideally, API to support competition and research into these components

What is in the OS? Window Manager? Toolkit? Communication? Scripting

facilities?

Need ever increasing services for applications Need more access to low-level information

E.g., hardware buttons, whether on network

Ideally, API to support competition and research into these components


Conclusions

Research in tools necessarily trails innovation in UI design

Due to consolidation on desktop metaphor, significant progress in tools

UI design poised for radical changes New opportunities and challenges for tools

Research in tools necessarily trails innovation in UI design

Due to consolidation on desktop metaphor, significant progress in tools

UI design poised for radical changes New opportunities and challenges for tools

22

Past, Present and Future ofUser Interface Software ToolsPast, Present and Future ofUser Interface Software ToolsBrad A. MyersBrad A. Myers(work with Scott E. Hudson(work with Scott E. Hudsonand Randy Pausch)and Randy Pausch)

Human-Computer Interaction InstituteHuman-Computer Interaction InstituteSchool of Computer ScienceSchool of Computer ScienceCarnegie Mellon UniversityCarnegie Mellon University

http://www.cs.cmu.edu/~bamhttp://www.cs.cmu.edu/[email protected]@cs.cmu.edu

Talk given withSlideShow Commander

Result of the Pebbles research projecthttp://www.cs.cmu.edu/~pebbles

SlideShow Commander to be commercialized by:

Available soon!

Result of the Pebbles research projecthttp://www.cs.cmu.edu/~pebbles

SlideShow Commander to be commercialized by:

Available soon!

Synergy Solutions, Inc.Synergy Solutions, Inc.http://www.synsolutions.com/

Synergy Solutions, Inc.Synergy Solutions, Inc.http://www.synsolutions.com/

Brad Myers 23

1 past, present and future of user interface software tools brad a. myers (work with scott e. hudson...

Documents

n toolkits

n smalltalk

n overlapping

style n

n future tools

programming n help

event handlers n

n screen space