dialog design how do we communicate with computers?
Post on 21-Dec-2015
220 views
TRANSCRIPT
![Page 1: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/1.jpg)
Dialog design
How do we communicate with computers?
![Page 2: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/2.jpg)
Agenda
Exam review Project Part 3 Presentation: Moo Nam Dialogue styles
Command line WIMP Direct Manipulation Speech/Natural Language Pen/Mobile Device
![Page 3: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/3.jpg)
Exam review
Top score: 97 Average: 79 Median: 86
![Page 4: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/4.jpg)
Project Part 3
Due in 3 weeks Choose one design
Go more in depth on details Create testable prototype(s)
Demo for me Create evaluation plan
Discuss in class on 3/30
![Page 5: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/5.jpg)
Dialog Design
How does a user interact with the interface?
![Page 6: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/6.jpg)
Dialog Styles
1. Command languages 2. WIMP - Window, Icon, Menu,
Pointer 3. Direct manipulation
4. Speech/Natural language 5. Pen & PDA
![Page 7: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/7.jpg)
General Issues in Choosing Dialogue Style Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required
![Page 8: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/8.jpg)
Command Languages
Earliest UI interaction paradigm Examples: MS-DOS shell, UNIX, Linux Little or nothing is visible so…
Work primarily by recall, not recognition Heavy memory load
Poor choice for novices but...
![Page 9: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/9.jpg)
CL Advantages
Advantages for experts Speed, conciseness
• % ls (hard to beat)
Can express actions beyond a limited set• Flags, piping one command to another
Repetition, extensibility• Scripting, macros
Easier implementation, less overhead Power
• Abstraction, wild cards
![Page 10: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/10.jpg)
CL Dangers
With added power, comes added responsibility and dangerUNIX
• % rm -r *• Deletes every file that you have, and you
can’t get them back
![Page 11: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/11.jpg)
CL Design Goals
ConsistencyHave options and arguments
expressed the same way everywhere Good naming and abbreviations
UNIX fails here because commands were developed by lots of different people at different organizations
• No guidelines provided
![Page 12: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/12.jpg)
General Issues - CL
Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required
![Page 13: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/13.jpg)
Direct Manipulation
Definition: 1) Continuous visibility of the objects
and actions of interest 2) Rapid, reversible, incremental
actions whose effect is immediately noticeable
3) Replacement of command language syntax by direct manipulation of object of interest (physical actions, buttons, etc.)
Shneiderman ‘82
![Page 14: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/14.jpg)
DM Essence
Representation of reality that can be manipulated
The user is able to apply intellect directly to the task
The tool itself seems to disappear
Related to Directness Engagement
![Page 15: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/15.jpg)
Direct Manipulation
See pgs 171-174 in DFAB Examples
WYSIWYG editors and word processors
VISICALC - 1st electronic spreadsheetCADDesktop metaphorVideo games
![Page 16: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/16.jpg)
Example: Homefinder
![Page 17: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/17.jpg)
DM Issues
Advantages Flexible, easily
reversible actions helps reduce anxiety in users
WYSIWYG provides context & instant visual feedback
Exploits human use of visual spatial cues
Limits types of errors that can be made
Disadvantages visual representation
not self-explanatory Mouse ops may be
slower than typing Not good at:
Repetition History keeping Certain tasks
(Change all italics to bold)
Abstract elements (variables)
![Page 18: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/18.jpg)
General Issues - DM
Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required
![Page 19: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/19.jpg)
WIMP
Windows, Icons, Menus, PointersFocus: Menus, Buttons, Forms
Predominant interface paradigm now (with some direct manipulation added)
Advantages:?
![Page 20: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/20.jpg)
Menus
Advantages: 1 keystroke or mouse operation vs. many No memorization of commands Limited input set
Disadvantages: Less direct user control - have to find correct
menu / menu item Not so readily extensible Slower than keyboarding for experienced
users, at least without accelerators
![Page 21: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/21.jpg)
Menu Items
Organization strategiesCreate groups of logically similar
itemsCover all possibilitiesEnsure that items are non-overlappingKeep wording concise,
understandable
![Page 22: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/22.jpg)
A Good Example
Logical grouping Visual separation of
groups Disabled items
“grayed out” Shortcuts shown … indicates leads to
dialogue
![Page 23: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/23.jpg)
Presentation Sequence
Forms, dialogue boxes, menus Use natural if available
Time• e.g. Breakfast, Lunch, Dinner
Numeric ordering• e.g. Point sizes for font
Other possibilities: Alphabetical Group related items Frequently used first Most important first
![Page 24: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/24.jpg)
Other WIMP issues
Windows managementHow to locate, move, findTransfer information between
IconsNeed graphic design attention
Toolbars
![Page 25: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/25.jpg)
General Issues - WIMP
Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required
![Page 26: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/26.jpg)
Natural input
Universal design Take advantage of familiarity,
existing knowledge Alternative input & output Multi-modal interfaces Getting “off the desktop”
![Page 27: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/27.jpg)
When to Use Speech
Hands busy Mobility required Eyes occupied Conditions preclude use of keyboard Visual impairment Physical limitation
![Page 28: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/28.jpg)
Speech Input
Speaker recognitionTell which person it is (voice print)Monitoring, recording
Speech recognitionIdentify wordsIBM ViaVoice, Dragon Dictate, ...
Natural language understanding
![Page 29: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/29.jpg)
Recognition Dimensions
Speaker dependent/independent Parametric patterns are sensitive to
speaker With training (dependent) can get better
Vocabulary Some have 50,000+ words
Isolated word vs. continuous speech Continuous: where words stop & begin Typically a pattern match, no context
usedDid youvs.Didja
![Page 30: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/30.jpg)
Recognition Example:
My flight experience uh, I started off as a private pilot and then I spent 20 years flying as a navigator on C130s in the air force and I currently fly as a first officer on the SAAB 340 turboprop.
my flight experience a and I started off as a private pilot and I spent 20 years flying is a navigator on C-130s in the Air Force and I currently fly as a first officer on the sound 340 turboprop.
![Page 31: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/31.jpg)
And another example
Here here's one of the
thoughts I'd throw in on
that. I'd agree with
Howard that the
combination of checklists
and flows is the way way
to manage things best.
hair and here's one of the
outside troodon and
an eye to agree with
power that the
domination checklists
and flows is the way
that managed things best
![Page 32: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/32.jpg)
Errors
Systems make four types of errors:Substitution - one for anotherRejection - detected, but not
recognizedInsertion - addedDeletion - not detected
Problems with recovery
![Page 33: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/33.jpg)
Natural Language Understanding
Putting meaning to the words
Input might be spoken or typed
Holy grail of Artificial Intelligence problems
![Page 34: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/34.jpg)
Eliza - a “Faked” Understanding Person : Men are all alike.
ELIZA : In what way?Person : They're always bugging us about something or other.ELIZA : Can you think of a specific example?Person : Well, my boyfriend made me come here.ELIZA : Your boyfriend made you come here?Person : He says I'm depressed much of the time.ELIZA : I am sorry to hear that you are depressed.Person : It's true. I am unhappy.ELIZA : Do you think coming here will help you not to be unhappy?Person : I need some help, that much seems certain.ELIZA : What would it mean to you if you got some help?Person : Perhaps I could learn to get along with my mother.ELIZA : Tell me more about your family
http://www-ai.ijs.si/eliza/eliza.html Weizenbaum, J., "ELIZA -- A computer program for the study of
natural language communication between man and machine", Communications of the ACM 9(1):36-45, 1966
![Page 35: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/35.jpg)
NL Factors/Terms
SyntacticGrammar or structure
ProsodicInflection, stress, pitch, timing
PragmaticSituated context of utterance, location,
time Semantic
Meaning of words
![Page 36: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/36.jpg)
SR/NLU Issues
Advantages Easy to learn and
remember Powerful Fast, efficient (not
always) Little screen real estate
Disadvantages Assumes domain
knowledge Doesn’t work well enough
yet Requires confirmation And recognition will
always be error-prone Expensive to implement Unrealistic expectations
can generate mistrust
![Page 37: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/37.jpg)
Speech Output
Tradeoffs in speed, naturalness and understandability
Male or female voice? Technical issues (freq. response of phone) User preference (depends on the application)
Rate of speech Technically up to 550 wpm! Depends on listener
Synthesized or Pre-recorded? Synthesized: Better coverage, flexibility Recorded: Better quality, acceptance
![Page 38: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/38.jpg)
Speech Output
Synthesis Quality depends on software ($$) Influence of vocabulary and phrase choices http://www.research.att.com/projects/tts/demo.php
Recorded segments Store tones, then put them together The transitions are difficult (e.g., numbers)
![Page 39: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/39.jpg)
Designing Speech Interaction Constrain vocabulary
Limit valid commandsStructure questions wisely (Yes/No)Manage the interactionExamples?
Slow speech rate, but concise phrases Design for failsafe error recovery Visual record of input/output Design for the user – Wizard of Oz
![Page 40: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/40.jpg)
Speech Tools/Toolkits
Java Speech SDK FreeTTS 1.1.1
http://freetts.sourceforge.net/docs/index.php
IBM JavaBeans for speech Microsoft speech SDK (Visual Basic, etc.) OS capabilities (speech recognition and
synthesis built in to OS) (TextEdit) VoiceXML
![Page 41: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/41.jpg)
General Issues – Speech/NL
Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required
![Page 42: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/42.jpg)
Non-speech audio
Good for indicating changes, since we ignore continuous sounds Traditionally used for warnings, alarms or
status information Provides secondary representation
Supports visual interface Provides information that helps reduce error
Tradeoff in using natural (real) sounds vs. synthesized noises.
![Page 43: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/43.jpg)
Non-speech audio examples
Error ding Info beep Email arriving ding Recycle Battery critical Logoff LogonOthers?
![Page 44: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/44.jpg)
Pen & Mobile dialog
Stylus or finger Tradeoffs of each?
Pen as a standard mouse (doubleclick?) Variety of platforms
Desktop touch screens or input pads (Wacom)
Tablet PCs Handheld and Mobile devices Electronic whiteboards
Platforms often involve variety of size and other constraints
![Page 45: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/45.jpg)
![Page 46: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/46.jpg)
Mobile devices
More common as more platforms available PDA Cell phone Ultra mobile tablets
Smaller display (160x160), (320x240) Few buttons, different interactions
Free-form ink Soft keyboard Numeric keyboard => text Stroke recognition Hand printing / writing recognition
![Page 47: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/47.jpg)
http://www.oqo.com/
http://www.blackberry.com/
http://www.intel.com/design/mobile/platform/umpc.htm
Ultra-Mobile PC (Samsung)Palm Z22 handheldhttp://www.palm.com
![Page 48: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/48.jpg)
Soft Keyboards
Common on PDAs and mobile devicesTap on buttons on screen
![Page 49: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/49.jpg)
Soft Keyboard
Presents a small diagram of keyboard You click on buttons/keys with pen QWERTY vs. alphabetical
Tradeoffs?Alternatives?
![Page 50: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/50.jpg)
Numeric Keypad -T9
Tegic Communications developed You press out letters of your word, it matches the
most likely word, then gives optional choices Faster than multiple presses per key Used in mobile phones http://www.t9.com/
![Page 51: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/51.jpg)
Cirrin - Stroke Recognition
Developed by Jen Mankoff (GT -> Berkeley CS Faculty -> CMU CS Faculty)
Word-level unistroke technique UIST ‘98 paper Use stylus to go
from one letterto the next ->
![Page 52: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/52.jpg)
Quikwriting - Stroke Recogntion
Developed by Ken Perlin
![Page 53: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/53.jpg)
Quikwriting Example
p l
http://mrl.nyu.edu/~perlin/demos/Quikwrite2_0.html
e
Said to be as fast as graffiti, but have to learn more
![Page 54: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/54.jpg)
Hand Printing / Writing Recognition Recognizing letters and numbers and
special symbols Lots of systems (commercial too) English, kanji, etc. Not perfect, but people aren’t either!
People - 96% handprinted single characters Computer - >97% is really good
OCR (Optical Character Recognition)
![Page 55: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/55.jpg)
Recognition Issues
Boxed vs. Free-Form input Sometimes encounter boxes on forms
Printed vs. Cursive Cursive is much more difficult
Letters vs. Words Cursive is easier to do in words vs individual
letters, as words create more context Usually requires existence of a dictionary
Real-time vs. off-line
![Page 56: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/56.jpg)
Special Alphabets
Graffiti - Unistroke alphabet on Palm PDAWhat are your
experienceswith Graffiti?
Other alphabets or purposesGestures for commands
![Page 57: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/57.jpg)
Pen Gesture Commands
-Might mean delete
-Insert
-Paragraph
Define a series of (hopefully) simple drawing gesturesthat mean different commands in a system
![Page 58: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/58.jpg)
Pen Use Modes
Often, want a mix of free-form drawing and special commands
How does user switch modes?Mode icon on screenButton on penButton on device
![Page 59: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/59.jpg)
Error Correction
Having to correct errors can slow input tremendously
StrategiesErase and try again (repetition)When uncertain, system shows list of
best guesses (n-best list)Others??
![Page 60: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/60.jpg)
Free-form Ink
Ink is the data, take as is
Human is responsible forunderstanding andinterpretation
Often time-stamped Applications
Signature verification Notetaking Electronic whiteboards Sketching
![Page 61: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/61.jpg)
Electronic whiteboards
Smartboard and Mimio Can integrate with projection Large surface to interact with
Issues?
http://www.smarttech.com/http://www.mimio.com/
![Page 62: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/62.jpg)
Real paper
Anoto digital paper and pen technology (http://www.anoto.com/)
Issues?
http://www.logitech.com/
Logitech io Digital Writing System
![Page 63: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/63.jpg)
General Issues – Pen input
Who is in control - user or computer Initial training required Learning time to become proficient Speed of use Generality/flexibility/power Special skills - typing Gulf of evaluation / gulf of execution Screen space required Computational resources required
![Page 64: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/64.jpg)
Other interesting interactions
Gesture inputSpecialized hardware, or tracking
3D interactionStereoscopic displaysVirtual reality
• Immersive displays such as glasses, caves
Augmented realityHead trackers and vision based
tracking
![Page 65: Dialog design How do we communicate with computers?](https://reader038.vdocuments.net/reader038/viewer/2022103005/56649d615503460f94a42905/html5/thumbnails/65.jpg)
What’s coming up
Upcoming related topicsMultimodal UIs: Ted3D user interfaces: AmyConversational agents: Evan
Next week: evaluation