post-wimp interaction with desktop computersdragice.fr/papers/postwimp.pdfpost-wimp sketching-based...
TRANSCRIPT
1
Post-Wimp Interaction With Desktop Computers
Pierre Dragicevic
New Tools and Techniques
2
WIMP Interaction
n Standard Interaction Techniques:
n Windows, widgets, iconn Text input, click, double-click, d&d
+
n WIMP = Windows, Icons, Menus, Pointer
n Standard Devices:
n Mousen Keyboard
3
What Next and Why?
n WIMP system(Xerox Star, 1979)
4
Post-WIMP Interaction Paradigms
Better exploit our motor abilities:
n Bimanual Interaction
Map Zooming [Hinckley et al 98]
SmartSkin [Rekimoto 2002]
n « Multifinger » Interaction
5
Post-WIMP Interaction Paradigms
Better exploit our perceptual abilities:
n Information visualization
TreeMaps [Schneiderman et al]
Multiple screens & large displays
n Enhanced displays
6
Post-WIMP Interaction Paradigms
Make interaction more « direct »:
n Gestural interaction
Toolglasses [Bier et al 93]
n See-through tools
7
Two Research Directions
n Explore Post-WIMP interaction further:
n Propose innovative solutions to known problemsn Investigate alternative interaction paradigms
n Disseminate existing research:
n Integrate Post-WIMP paradigms into GUI toolkitsn Provide new design guidelines
8
My Research
n Explore Post-WIMP interaction further:
n Propose innovative solutions to known problemsn Investigate new interaction paradigms
n Disseminate existing research:
n Integrate Post-WIMP paradigms into GUI toolkitsn Provide new design guidelines
n Support the exploration of new approaches:
n Study more usable and more flexible toolsn Advocate a vision of Post-WIMP with sound examples
9
My Research
New tools
New techniques
10
New techniques
Introduction
Conclusion
II. Artistic Resizing
IV. Fold n’ Drop
I. Input Configurator
III. SpiraClock
New tools
11
I. Input Configurator
Pierre Dragicevic
Jean-Daniel Fekete
An Input Interaction Model forHighly Configurable Multi-Device
Interactive Systems
Introduction
Conclusion
II. Artistic Resizing
IV. Fold n’ Drop
I. Input Configurator
III. SpiraClock
12
Goal: Support Alternative Input
n WIMP system(Xerox Star, 1979)
n Input-Open System
13
Why Using Alternative Input?
the task the user the environnement
n Augmenting the bandwith
n Optimizing interaction according to:
14
Current State of the Art
n Very poor support for alternative input devices in:
n Current operating systemsn Current applicationsn Current toolkits
n Little research on input management
n Post-WIMP tools are specialized towards:
n A fixed set of input devicesn A fixed set of interaction techniques
n No approach deals with:
n Multiple and heterogeneous input (enriched / impoverished)n Flexibility and configurability
15
The Input Configuration Model
n Approach:
n Fully separate physical devices from applications
n Provide a flexible way to map devices to applications
16
The Input Configuration Model
n Device:
n Parametrizable black boxn Processes input into output
n I/O Slots:
n Simple typesn + compound types
n Implicit I/O:
n User inputn Feedback
Inputslots
Outputslots
Implicit input
Implicit output
17
The Input Configuration Model
n Device types:n System
(hardware ressources)n Adapter
(software processes)n Application
(specific to each app)
n Input Configuration:n A set of system and
application devicesconnected throughadapter devices
System devices
Application devices
Adapter devices
Input Configuration
18
The ICON Toolkit
n System Devices:
19
The ICON Toolkit
n Adapter devices:
n Data processing
20
The ICON Toolkit
n Adapter devices:
n Data processingn Graphical feedback
21
The ICON Toolkit
n Adapter devices:
n Data processingn Graphical feedbackn Interaction techniques
22
The ICON Toolkit
n Application Devices:n Example: IConDraw tools
23
The ICON Toolkit
n Graphical Toolkit Devices:
n Example: Swing devices
24
The ICON Toolkit
n The input configuration editor:
Availabledevices
Availabledevices
Editingpane
Editingpane
25
The ICON Toolkit
n The input configuration editor
26
The ICON Toolkit
n The input configuration editorn Dynamic wire drawing algorithm
27
The ICON Toolkit
n The input configuration editorn Fluid expand/collapse
28
The ICON Toolkit
n The input configuration editorn Fisheye-drag
29
Examples
1. Exploiting rich input
n Pressure-sensitive drawing inside IConDraw
30
1. Exploiting rich input
Examples
31
Examples
1. Exploiting rich input
n Symmetric bimanual interaction in IConDraw
32
Examples
1. Exploiting rich input
33
Examples
1. Exploiting rich input
n Integral control of the Responsive Face [Perlin 97]
34
Examples
1. Exploiting rich input
n Integral control of the Responsive Face [Perlin 97]
35
Examples
1. Exploiting rich input
36
Examples
2. Improving Accessibility
n Keyboard ? Pointing
2nd order control
37
Examples
2. Improving Accessibility
n Pointer ? Text editing
« Floating QuikWriting » [Perlin]
38
Examples
2. Improving Accessibility
n Pointer ? Text editing
« Floating QuikWriting » [Perlin]
39
Examples
3. Augmenting WIMP applications
n « Augmented pointing »
Filter Events
40
Examples
3. Augmenting WIMP applications
n « Augmented pointing » : smoothed cursor
41
Examples
3. Augmenting WIMP applications
n « Augmented pointing » : throwable cursor
42
Examples
3. Augmenting WIMP applications
n « Augmented pointing » : speech cursor
rightrightNoise…Noise… downdown
stopstoppresspressupupslowerslowerreleaserelease
43
Examples
3. Augmenting WIMP applications
n « Generalized Drag and Drop »
44
Projects using ICON
n Marina II (2001)Photograph-based architectural modelling tool
n Svalabard [Huot et al 2003]Post-WIMP sketching-based 3D modelling interface
n MaggLite [Huot et al 2004]Post-WIMP UIMS based on ICON
n LRI, University of Orsay [Appert et al 2003]Experimental comparison of WIMP / Post-WIMP techniques
n Smart Home and Impairment (INT/ENST/EDF 2005)Configurable software architecture for accessibility
n Student projects
45
II. Artistic Resizing
Pierre Dragicevic
Stéphane Chatty
David Thevenin
Jean-Luc Vinot
A Technique for RichScale-Sensitive Vector Graphics
Introduction
Conclusion
II. Artistic Resizing
IV. Fold n’ Drop
I. Input Configurator
III. SpiraClock
46
n Graphic designers are given more and more importance in GUIs
n Exploitation of vector graphics(e.g, Scalable Vector Graphics)
n IntuiKit [Chatty et al 2004]
Graphically Rich User Interfacs
47
IntuiKit: Example of Team Work
Low-fi prototyping
Graphic design
Programming
Integration
48
The Resizing Problem
n Fixedsize
n Naivescaling
n Common resizing
49
Expressing Resizing
n Resizing is commonly described using formulae
n These formulae are:
n Translated into code by the programmern Or used as an input to constraint-solving systems
w
h
wL
hL
yL
xLr
hB
wB
• xL = (w-wL) / 2• yL = (h-hL) / 2• wL = 20• hL = 10
• wB = 5• hB = 5
• r = 20
50
Expressing Resizing
n But designers think visually
n They are used to produce visual variants
n Variants are also useful to convey general laws
On whitepaper
On blackpaper
On redpaper
Mediumsize
Smallsize
51
Artistic Resizing
1. Designers produce variantsusing their authoring tool
2. IntuiKit interpretsthe example set
52
Artistic ResizingA scenario
53
Artistic ResizingA scenario
54
Artistic ResizingA scenario
55
Artistic ResizingA scenario
56
Artistic ResizingA scenario
57
Artistic ResizingHow does it work?
n Based on local interpolation of transformations
T1 T1’
58
Artistic ResizingHow does it work?
n Each variant of T1 is associated withthe example’s bounding box
T1 T1’
?
T1’’ T1’’’
59
n Orthogonal Interpolation hypothesis:
n Width only impacts the X contributionn Height only impacts the Y contribution
Artistic ResizingOrthogonal Interpolation
a11 a12 a13
a21 a22 a23
0 0 1
X
Y
1
X0
Y0
1
=
width
height
60
n Piecewise linear interpolations are performed along width and height separately:
Artistic ResizingOrthogonal Interpolation
width
height
a11a12 a13
a21a22
a23
61
Artistic ResizingProperties of Orthogonal Interpolation
n Preserves algebraic measures:
n If horizontal or vertical distances are the same on the examples they will remain the same
62
Artistic ResizingProperties of Orthogonal Interpolation
n Preserves relative ratios:
n Ratios that are invariant on the exampleswill remain invariant
63
Artistic ResizingProperties of Orthogonal Interpolation
n Preserves contacts:
n Two points that coincide on the exampleswill always coincide
64
Artistic ResizingProperties of Orthogonal Interpolation
n Preserves parallelism:
n Two lines that are parallel on the exampleswill remain parallel
65
Artistic Resizing
n Supports invariants but also finely tuned behaviors
w
h
wL
hL
yL
xLr
hB
wB• xL = (w-wL) / 2• yL = (h-hL) / 2• wL = 20• hL = 10
• wB = 5• hB = 5
• r = 20
66
Artistic ResizingBlending with traditional layout managers
n Containment hierarchies:
67
Artistic ResizingBlending with traditional layout managers
n Inside an « aqua-dock » layout manager:
68
n Demonstrational layout specification systems:
n Peridot [Myers 90]
n Chimera [Kurlander & Feiner 93]
n Constraint inference:
n Extensive searchfor invariants
n Sensitive to combi-natorial explosion
n Over-constrained orunintended rules
Artistic Resizingvs. Constraint Inference
n Orthogonal interpolation:
n Ensures the preservation of « interesting » invariants
n Fast computation, even with complex vector graphics
n Limitations are known, results are easy to predict
69
III. SpiraClock
Pierre Dragicevic
Stéphane Huot
Introduction
Conclusion
II. Artistic Resizing
IV. Fold n’ Drop
I. Input Configurator
A Continuous and Non-IntrusiveTechnique for Upcoming EventsIII. SpiraClock
70
Time Management Tools
n Calendars displays:n Good for long-term time
managementn But useless for
upcoming events
n Reminders and Alarms:n Prevent forgettingn But too intrusiven Don’t allow
continuousanticipation
71
SpiraClock
n Analog desktop clock
n Events are displayedas sectors
72
SpiraClock
n Event are aligned withminute marks
n Relative times can beread directly
73
SpiraClock
n One example: viewing bus departures
It’s 18h50.What busescan I catch ?
74
SpiraClock
n One example: viewing bus departures
75
SpiraClock
n One example: viewing bus departures
76
Uses of SpiraClock
n BusClock used in Ecole des Mines de NantesSince 2002
n Other applications:
n ShuttleClock - shuttles of University of Marylandn OutClock - synchronized with MS Outlookn CalendarClock - uses VCal exchange format
77
IV. Fold n’ Drop
Combining Paper-Based and
Crossing-Based Interaction Paradigms
for Dragging and Dropping betweenOverlapping Windows
Pierre Dragicevic
Introduction
Conclusion
II. Artistic Resizing
IV. Fold n’ Drop
I. Input Configurator
III. SpiraClock
78
Drag & Drop Scenario
n Moving a file into a folder
79
Drag & Drop Scenario
n Problem: the target is hidden
80
drag & drop
Drag & Drop Scenario
n Solution 1: adjust windows
adjust
81
cut
bring to the front
Drag & Drop Scenario
n Solution 2: cut & paste
82
paste
Drag & Drop Scenario
n Solution 2: cut & paste
83
Using Window Navigation
n Dual representation → Mapping problems (XP taskbar)
n Activation costs (mouse-over timeout on Exposé)
84
Using Window Navigation
n Activation is a superfluous concept for D&D tasks
85
The Fold n’ Drop Technique
n Folding a window
n The mouse leaves the window
n The mouse leaves then returns back
86
The Fold n’ Drop Technique
n Other gestures
n Leafing through
n Discarding a window
n Unfolding
87
Video
88
Fold n’ Drop builds upon…
n Paper-based window metaphors
n The « Folded » effect [Beaudouin-Lafon 01]
Ametista Toolkit [Roussel 03] AttrActive Windows [Denoue 03]
89
Fold n’ Drop builds upon…
n Gestural interfaces
n Natural, « physical » gestures
n Do not need a recognition engine
n Naturally blend with direct manipulation[see e.g., Pook et al 00]
n Little interference with other interactions
90
Fold n’ Drop builds upon…
n The crossing-based interaction paradigm[Accot & Zhai 02]
n Fn’D uses double-crossing and n-crossing
n Extends the paradigm:
• Point & Click → Cross boundaries• Drag → Push boundaries• Double-Click → Timed double-crossing
n Timing ismade visible:
91
Conclusion
Introduction
Conclusion
II. Artistic Resizing
IV. Fold n’ Drop
I. Input Configurator
III. SpiraClock
92
Conclusion
n Common desktop computing activities canstill be improved:
n Management of multiple tasks• Task switching• Data transfer between applications
n Management of personal data• Viewing, searching• Organizing
n Time management• Long-term• Short-term
n But how to design Post-WIMP techniques ?
93
ConclusionPost-WIMP GUI Design as a Creative Activity
n The success of a technique is difficult to predict:
n Efficiency + pleasingness
? Importance of « trial and error » (iterative design)
n But techniques also require fine tuning:
n Timings, input filtering, graphical effects, color design,…
? Need for closer experimentation-evaluation loops
n « Creative tools » allow to explore and test by ourselves:
n Flexible, openn High degree of control, finely tunablen Fast experimentation-evaluation loopn Simple and predictable
94
Conclusion5 characteristics of Post-WIMP Techniques
1. Specificity
2. High I/O bandwidthn Multiple input channelsn Advanced output devices or rich graphics
3. Directnessn Stimulus/Response compatibility
4. Reactivity and Fluidityn Responsive and continuous feedback
5. Straightforwardnessn Familiar metaphorsn Simple, predictable system behavior
95
ConclusionAbout Predictability
n Example: gestures in Fold n’ Drop
« I missed the fold »
« I returned backtoo late »
96
ConclusionAbout Predictability
n Another example: freehand polyline drawing
n Using a smart tool (stroke beautifier):
97
ConclusionAbout Predictability
n Another example: freehand polyline drawing
n Using a smart tool (stroke beautifier):
98
ConclusionAbout Predictability
n Another example: freehand polyline drawing
n Using a dumb tool (spring-loaded cursor):
99
ConclusionFuture Work
n Continue to explore new interaction paradigmsn Desktopn Non-desktop
n Generalize and theorize:n Intelligence vs predictabilityn Explicit gesturesn Augmented WIMPn …
n Papers, videos and demos:http://www.intuilab.com/~dragice