Input Devices

CS160: User Interfaces

John Canny

Reminders

Midterm is next Weds.

Sample midterms are online, with solutions.

Review session in class on Monday, also go to

discussion sections this week.

Reminders

Next project checkpoint (interactive prototype)

assignment is out, due 11/5/12.

PA III is out, due 10/29/12. Uses CometD, but server

code is provided.

Input Devices: Important Tasks

• Text Entry

• Pointing/Marking

– Target acquisition

– Steering / positioning

– Freehand drawing

– Drawing lines

– Tracing and digitizing

– …

• Voice: like text entry but limited

– better to be “dialogic”

• Array of Discrete Inputs

• Many variants of form and key layout

– Can be one-handed or two

– Wide range of sizes

– Two-hand full keyboard is relatively standardized, Less standardization on others: Command keys, generic function keys, cursor movement, numeric keypad,...

• Use procedural memory

– Power law of practice

Text Entry: Keystroke Devices

5

1a

nT T n c

Keyboards

Ergonomics

Ergonomics

Key Layouts

Mobile Text Entry: Keypads

11

• Multi-tap mappings

– Multiple presses per letter

• Ambiguity resolution

– One press per letter, dictionary lookup

Mobile Text Entry: Keypads

12

• Chording

– Multiple keys pressed

simultaneously

– (n choose 4)

combinations = 495

• Can use for words!

– Use fewer keys for

common words

– Covers perhaps 80% of

typical words.

Twiddler2, HandyKey

Mobile Text Entry: Touch / Stylus

Soft Keyboards

• Benefits?

Drawbacks?

Mactoids.com

Mobile Text Entry: Handwriting

Recog.

Mobile Text Entry: Touch / Stylus

• Custom symbol sets

• Graffiti

• improve recognition accuracy; appropriate for indirect (eyes-free) input

Mobile Text Entry: Touch / Stylus

• Stroke Entry Methods (e.g., Swype, ShapeWriter)

Which is fastest?

Beyond text input…

• Even with keyboards, there is more information gathered

than the text.

• Every person types differently: characters are grouped

together into “chunks” in different ways.

• Biometrics: you can partially identify a person from the

timing of their keystrokes. e.g. timing of password typing has

shown accuracies as high as 99%:

Novelty Detection Approach for Keystroke Dynamics Identity

Verification, Enzhe Yu and Sungzoon Cho, Intelligent Data Engineering and

Automated Learning Lecture Notes in Computer Science, 2003, Volume

2690/2003

What about Speech Recognition?

• Dictation is faster than typing (~100 wpm)

What about Speech Recognition?

• Dictation is faster than typing (~100 wpm), BUT:

– Speech is different from written language:

Speaking in well-formed, complete, print-ready sentences

is cognitively challenging

– High cost of correcting errors through speech channel

alone

– Social awkwardness?

• Then came Siri, and…

AT&T Watson service

Pointing Devices

22

(cc) Flickr photo by Mike fj40

Mouse. Engelbart and English ~1964

Source: Card, Stu. Lecture on Human Information Interaction. Stanford, 2007.

24

IR emitter IR detector slotted wheel

(between emitter & detector)

Uses two out-of-phase detectors

25

High

High

26

Optical mice

In effect use computer vision with a tiny image sensor.

Optical mice

Use computer vision with a tiny image sensor.

Other device properties:

• Indirect vs. Direct

– Direct: Input and output space are unified

• C:D Ratio

– For one unit of movement in physical space, how far does

the cursor travel in display space?

– Q: What is the C:D ratio for direct touch screen input?

• Device Acquisition Time

30

Trackball, Trackpad

31

Trackpoint

• Indirect, force sensing, velocity control

• Nonlinear transfer function

Force

Velo

city

(cc) Image by flickr user tsaiid

Mobile Pointing

• D-Pad (see: arrow keys)

• Trackball

• Direct touch (see: Trackpad)

• Stylus

What else can you detect with a Mouse?

33

What else can you detect with a Mouse?

Stress affects muscle tension, which will change the dynamics of

mouse movements. Analyze “reference” movements

34

MouStress

Analyze the arm/hand as a simple second-order mechanical

system, and find two poles using LPC.

Actual tasks were pointing and dragging on variably-spaced

targets:

35

MouStress

Phases of the tests:

36

MouStress

Mouse dynamic tests (mCalm and mStress states)

p < 0.002 p < 0.002

37

MouStress

Mouse sensing was able to reliably distinguish the

calm and stress states, even when standard measures

(Heart-Rate Variability) were not.

i.e. it appeared to be even more sensitive than the

best existing biometric measure.

38

Next steps

Analyze arm/hand dynamics while holding a cell phone, and

performing screen gestures.

Several different dynamic modes,…

Impulse gestures (tapping) and dragging,…

39

Everything is best for something and worst for something else.

- Bill Buxton

Multi-Touch: Strengths

• Direct input allows maximal screen space for mobile devices (ocular centrism).

• More degrees of freedom.

• “Virtual input devices” are adaptable.

• No extra pieces to lose or break (styli!)

Challenges

• No tactile feedback.

• Requires free use of (both) hands and eyes.

• “Fat Finger” problems – precision & occlusion

The “Fat Finger” Problem

touch

Graphics: Patrick Baudisch, nanoTouch

A Software Solution

Graphics: D. Vogel, P. Baudisch - Shift

A Hardware Solution: Use the Backside

touch

pointer

Graphics: Patrick Baudisch, nanoTouch

Terminology

• Touch-tablets vs Touch screens

• Single-finger vs multi-finger

• Multi-person vs multi-touch

• Points vs Postures

• Hands and fingers vs Objects

(from Buxton)

Wobbrock, J., Morris, M.R., and Wilson, A. User-Defined Gestures for Surface Computing. Proceedings of CHI 2009, 1083-1092.

Multi-point Gestures

Posture-based Interaction

Golan Levin, Zach Lieberman – The Manual Input Sessions Workstation

Hybrids: Keyboards on Interactive Tables

http://www.youtube.com/watch?v=qIASBXG3-Sk

B. Hartmann, M. Morris, H. Benko, A. Wilson: Augmenting

Interactive Surfaces With Keyboards and Mice. UIST 2009

Hybrids: Multi-touch on Mice

Mouse 2.0: Multi-touch Meets the Mouse

Nicolas Villar, Shahram Izadi, Dan Rosenfeld, Hrvoje Benko, John Helmes, Jonathan Westhues, Steve Hodges,

Eyal Ofek, Alex Butler, Xiang Cao and Billy Chen.

Proceedings of UIST 2009.

Input modes for Tablets

• Accelerometer (usually), gyroscope, compass

(sometimes) move the tablet as input (Holotoy)

• Camera: track head, eyes, limbs move yourself,

OpenCV toolkit (Computer Vision).

Microsoft Kinect

• Huge number of applications in games etc.

• Fast calculation of body skeleton supports easy

recognition of gestures.

• Some very compelling special

apps, e.g. in surgery

Skinput

http://www.youtube.com/watch?v=g3XPUdW9Ryg

Skinput

Humantenna

http://www.youtube.com/watch?v=7lRnm2oFGdc

Humantenna

Summary

Keyboards, mice, touchscreens, speech. But lots of

innovations:

• Rich gesture vocabulary

• Using local context

• Dialogicality

• Secondary cues: identity, stress, emotion

New input modes:

• 3D remote sensing (Kinect)

• Body sensing

• RF environment sensing