Thomas Lövgren

Interactive Developer, designer & programmer

thomas.lovgren@humlab.umu.se

Umeå Institute of Design, 2012-10-15

Programming for Interactivity

Terminology, techniques & design/interaction-concepts

(self-study)

Self-study outline

In this self-study the student will have a chance to study the following topics:

• Touch-input categories

• TUIO

• Touch technologies

• Resistive and capacitive screens

• Multi-touch ”All point screens” (The Future of User Interface Design)

• Gestures & Recognition

• Adobe Flash CS5/CS5.5 & AIR Multi-touch API

• GestureWorks

• Links & Resources

Next lecture will be:

ActionScript 3.0 Multi-touch pt.2

Designing, coding & publishing for a touch-device

Introduction to Multi-touch pt. 1 with ActionScript 3: Terminology & techniques

Multi-touch

(Technique 1)

(Technique 2)

(Technique 3)

What is Multi-touch?

Multi-touch is an enhancement to touch-screen technology, which provides the user the

ability to apply multiple finger-gestures simultaneously onto an electronic visual-display to

send complex commands to the device

Multi-touch

Touch Event: A term used to describe when a system knows that an object has touched the touch

device

Single Touch: Single Touch occurs when a finger or stylus creates a touch event on a touch-

device so it is detected by the touch controller and the application can determine the X,Y

coordinates of the event

Multi-touch: An interactive technique that allows single or multiple users to control graphical

displays with more than one simultaneous finger (+2 fingers)

Gesture: A physical movement that can be sensed, and often an action assigned to it. Some

common gestures are single finger panning, swipe, and two finger zoom-pinching

Gesture-enhanced single-touch: Also known as "Dual Control", "Gesture Touch" and often

"Dual-Touch" describes the ability of a touchscreen to register certain two-finger gestures, even

though the display hardware does not have full Dual-Touch capabilities

Touch-input categories

Multi-touch

TUIO (Tangible User Interface Objects) is an open framework that defines a common protocol

and API for tangible multi-touch surfaces

The protocol encodes control-data (position, size, and relative velocity of blobs) from a tracker

application and sends it to a client application

TUIO is based on Open Sound Control: An emerging standard for interactive environments

The Flash ActionScript 3 TUIO library supports true Multi-touch (TUIO client)

• Positive: Can be used for large custom made Multi-touch installations

• Negative: Requires in general a lot of ’tweaking’ to get working properly

TUIO: overview

http://www.tuio.org

Multi-touch

Technique 1

Bending Wave

Infrared (Grid / Cameras) (IR)

Optical (Camera)

Projected Capacitive (Matrix)

Resistive

Surface Acoustic Wave (SAW)

Surface Capacitive

Touch technologies

http://solutions.3m.com/wps/portal/3M/en_US/TouchTopics/Home/Technologies/

Multi-touch

Resistive touch-screens are composed of two flexible sheets coated with a resistive

material and separated by an air gap or microdots

When contact is made to the surface of the touch screen, the two sheets are pressed

together, registering the precise location of the touch

Advantages

• Most widely used touch technology

• Can be activated by bare finger, gloved hand, or stylus

• Low cost

Disadvantages

• Top sheet is highly susceptible to scratches, cuts and cigarette burns

• Transmission typically in 80% to 85%

Touch functionality

• Single touch

Examples

• Nokia 5800 and Samsung Omnia

Resistive screens

Multi-touch

In Capacitive screens a small voltage is applied to the four corners of the screen

The human body also conducts electricity so when a person touches the screen with their

finger, there is a change in the capacitance of the screen where the person touched it

The computer then registers the exact location of the persons touch

Advantages

• Very light finger touch required for activation

• Technology with fastest touch response time

• Transmission typically 88% to 92%

Disadvantages

• Severe scratch can affect operation within the damaged area

• Supports only finger or tethered pen

• Cannot be used in all weather scenarios

Touch functionality

• Multi-touch

Examples

• Apple iPhone, Samsung Galaxy S, Tablets etc

Capacitive screens (Surface Capacitive Touchscreens)

Multi-touch

The iPhone touch-sensitive screen includes a layer of capacitive material

The iPhone's capacitors are arranged according to a coordinate system

Every point on the grid generates its own signal when touched and relays that signal to the

iPhone's processor

This allows the phone to determine the location and movement of simultaneous touches in

multiple locations

The processor uses software to analyze the raw data and determine the features of each

touch (size, shape, location…)

How the iPhone works (Capacitive touch-screen)

http://electronics.howstuffworks.com/iphone.htm

Multi-touch

Single-touch touch-screens, which optimize design space and

provide unlimited numbers of buttons, still only allow users

to select one function at a time

With multi-touch gesture touch-screens, users can perform

more complicated inputs like sizing photos, rotating and

adjusting web pages

Multi-touch all-point touch-screens tear down the boundaries

of user input, which is only limited by the designer’s

imagination (design your own complex gestures, 10 fingers

etc)

Multi-Touch All-Point Touch-screens: The Future of User Interface Design

http://mobiledevdesign.com/tutorials/multitouch-touchscreens-0723/

Multi-touch

Gestures are the synthesis of multiple touch events into a single event

Gesture recognition is a topic in computer science and language technology with the goal

of interpreting human gestures via mathematical algorithms

Gesture recognition can be seen as a way for computers to begin to understand human

body language (humans to interface with the machine)

Basic gesture recognition can be done in three steps:

• Capture user input

• Homogenize and normalize input

• Match to a predefined pattern

The Touch Gesture Reference Guide is a unique set of resources for software designers

and developers working on touch-based user interfaces

Gestures: Gestures & Recognition

http://blog.sqrtof5.com/?p=173 http://www.lukew.com/ff/entry.asp?1071

Multi-touch

The latest Flash Multi-touch API supports Single-touch, Multi-touch and Gestures on various

platforms, browsers and devices (Windows 7, Flash Player +10.1, AIR +2.0, Android & iOS etc)

The technique can handle as many touch points as the touch-device and underlying operating

system supports

All Touch-inputs are very similar to the standard AS3 Mouse/keyboard - events, which makes

it easy and intuitive to work with

The AS3 classes that handles Touch-inputs are:

TouchEvent, TransformGestureEvent, GestureEvent, PressAndTapGestureEvent, GesturePhase

Supported Gestures on various platforms: PAN, ROTATE, ZOOM, TWO_FINGER_TAP, GESTURE_PRESS_AND_TAP

The Adobe AIR technology also support Multi-touch for

publishing on: Desktop, Android, iOS and TV

Note! In this course we are going to focus on

Flash/Air for Android (See Multi-touch pt2 for more info)

Adobe Flash CS5/CS5.5 & AIR Multi-touch API: Overview

http://www.adobe.com/devnet/flash/articles/multitouch_gestures.html

Multi-touch

Technique 2

GestureWorks is the most advanced solution for creating true multi-touch applications with

Adobe Flash, AIR and Flex (AS3 programming)

Open source framework that provides a simplified and standardized approach to multi-

touch application development

Supports over 200 gestures, multiple & simultaneous gestures and ”custom gestures”

Multi-touch authoring (Flash and Mac) solution for Windows 7-compatible multi-touch

devices (Android and iOS)

Multi-touch simulation functionality

GestureWorks is developed by IDEUM

GestureWorks: Overview

http://www.gestureworks.com

Multi-touch

Technique 3

NUI group (Natural User Interface group)

http://nuigroup.com

Multi-touch terminology

http://wiki.nuigroup.com/Multi-Touch_Terminology

Adobe Flash CS5/5.5 & AIR Multi-touch API

http://www.adobe.com/devnet/flash/articles/multitouch_gestures.html

GestureWorks (Flash & AIR MT with AS3)

www.gestureworks.com

DreamTouch

http://www.dreamtouchblog.com/

Microsoft Surface

http://www.microsoft.com/surface/en/us/default.aspx

Designing for Multi-Touch, Multi-User and Gesture-Based Systems

http://drdobbs.com/cpp/216402697?pgno=1

IDEUM: Building a High-Resolution Multi-touch Wall

http://www.ideum.com/blog/2010/11/building-a-high-resolution-multitouch-wall-part-1/

Links & Resources

Multi-touch

NUITEQ (Snowflake - suite)

www.nuiteq.com/

TUIO

www.tuio.org