multi touch
TRANSCRIPT
MULTI TOUCH
A SEMINAR REPORT
Submitted by
ALIKUTTY K A
in partial fulfillment for the award of the degree
of
BACHELOR OF TECHNOLOGY
in
COMPUTER SCIENCE & ENGINEERING
SCHOOL OF ENGINEERING
COCHIN UNIVERSITY OF SCIENCE & TECHNOLOGY,
KOCHI-682022
NOVEMBER 2008
DIVISION OF COMPUTER ENGINEERING
SCHOOL OF ENGINEERING
COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY
KOCHI-682022
CertificateCertified that this is a bonafide record of the seminar entitled
“MULTI TOUCH”
done by the following student
ALIKUTTY K A
of the VIIth
semester,Computer Science and Engineering in the year 2008 in
partial fulfillment of the requirements to the award of Degree of Bachelor of
Technology in Computer Science and Engineering of Cochin University of
Science and Technology.
Ms.Shekha Chenthara Dr.David Peter S
Seminar Guide Head of the Department
Date:
ACKNOWLEDGEMENT
I thank my seminar guide Ms Shekha Chenthara , Lecturer, CUSAT, for her
proper guidance, and valuable suggestions. I am indebted to Mr. David Peter, the
HOD, Computer Science division & other faculty members for giving me an
opportunity to learn and do this seminar. If not for the above mentioned people my
seminar would never have been completed successfully. I once again extend my
sincere thanks to all of them.
ALIKUTTY K A
ABSTRACT
Multi-touch technology is an advanced human-computer interaction
technique that recogonises multiple touch points and also includes the
hardware devices that implement it, which allow users to compute
without conventional input devices . Multi-touch consists of a touch
screen (screen, table, wall, etc.) or touchpad, as well as a software that
recognizes multiple simultaneous touch points, as opposed to the
standard touchscreen which recognizes only one touch point at a time.
Multi touch using Frustrated Total Internal Reflection is a simple,
inexpensive, and scalable technique for enabling high-resolution multi-
touch sensing on rear-projected interactive surfaces. Different
applications for multi-touch interfaces both exist and are being proposed.
Some uses are individualistic eg iPhone, iPod touch, MacBook Pro,
MacBook Air. The use of multi-touch technology is expected to rapidly
become common place.
i
TABLE OF CONTENTS
CHAPTER NO TITLE PAGE NO
LIST OF FIGURES ii
LIST OF SYMBOLS,ABBREVIATIONS AND iii
NOMENCLATURES
1. INTRODUCTION 1
1.1 TOUCH SCREEN 1
1.1.1 TECHNOLOGIES 2
1.1.2 DEVELOPMENT 4
2. MULTI TOUCH 5
2.1 HISTORY 5
2.2 RECENT DEVELOPMENTS 6
2.3 FUTURE 7
3. FTIR 8
3.1 PRINCIPLES 8
3.2 DESIGN
3.3.1 REQUIREMENTS 10
3.3.1.1 HARDWARE 10
3.3.1.2 SOFTWARE 11
3.4 WORKING 13
3.5 ADVANTAGES 15
3.6 APPLICATIONS 16
4 CONCLUSION 17
5 APPENDICES 18
5.1 APPENDIX I 18
6 REFERENCES 20
ii
LIST OF FIGURES
FIG NO DESCRIPTION PAGE NO
3.1 Total Internal Reflection 9
3.2 Acrylic with led’s at its edges 11
3.3 IR rays obtained on a camera 12
3.4 Frustrated Total Internal Reflection 13
3.5(a) IR view of a touch 14
3.5(b) Pixel Positions located on the screen 14
5.1(a) Prototype setup 18
5.1(b) Video output w/o diffuser 18
5.2 Photo sharing and multimedia 18
5.3 Wonders of multi touch-transfering music between Ipods 19
5.4 A roadside wall at Helsinki 19
iii
LIST OF SYMBOLS,ABBREVIATIONS AND NOMENCLATURES
NO ABBREVIATION EXPANSION
1 FTIR Frustrated Total Internal Reflection
2 TIR Total Internal Reflection
3 IR Infra Red
4 LED Light Emitting Diode
5 PP Perceptive Pixel
6 SAW Surface Acoustic Wave
7 PDA Personal Digital Assistance
8 MS Microsoft Surface
9 FW Finger Works
10 API Application Interface
11 LCD Liquid Crystal Display
12 CRT Cathode Ray Tube
Multi Touch
Division Of Computer Science ,SOE,CUSAT 1
1.INTRODUCTION
1.1 TOUCH SCREEN
A touchscreen is a display which can detect the presence and location of a
touch within the display area. The term generally refers to touch or contact to the
display of the device by a finger or hand. Touchscreens can also sense other passive
objects, such as a stylus. However, if the object sensed is active, as with a light pen,
the term touchscreen is generally not applicable. The thumb rule is: if you can interact
with the display using your finger, it is likely a touchscreen - even if you are using a
stylus or some other object.
Up until recently, most touchscreens could only sense one point of contact at a time,
and few have had the capability to sense how hard one is touching. This is starting to
change with the emergence of multi-touch technology - a technology that was first
seen in the early 1980s, but which is now appearing in commercially available
systems.
The touchscreen has two main attributes. First, it enables you to interact with what is
displayed directly on the screen, where it is displayed, rather than indirectly with a
mouse or a touchpad. Secondly, it lets one do so without requiring any intermediate
device, again, such as a stylus that needs to be held in the hand. Such displays can be
attached to computers or, as terminals, to networks. They also play a prominent role in
the design of digital appliances such as the personal digital assistant , satellite
navigation devices and mobile phone
1.1.1 TECHNOLOGIES
There are a number of types of touchscreen technology
Resistive
A resistive touchscreen panel is composed of several layers. The most important are
two thin metallic electrically conductive and resistive layers separated by thin space.
When some object touches this kind of touch panel, the layers are connected at certain
Multi Touch
Division Of Computer Science ,SOE,CUSAT 2
point; the panel then electrically acts similar to two voltage dividers with connected
outputs. This causes a change in the electrical current which is registered as a touch
event and sent to the controller for processing.
Surface acoustic wave
SAW technology uses ultrasonic waves that pass over the touchscreen panel. When
the panel is touched, a portion of the wave is absorbed. This change in the ultrasonic
waves registers the position of the touch event and sends this information to the
controller for processing. Surface wave touchscreen panels can be damaged by
outside elements. Contaminants on the surface can also interfere with the functionality
of the touchscreen.
Capacitive
A capacitive touchscreen panel is coated with a material, typically indium tin oxide
that conducts a continuous electrical current across the sensor. The sensor therefore
exhibits a precisely controlled field of stored electrons in both the horizontal and
vertical axes - it achieves capacitance. The human body is also an electrical device
which has stored electrons and therefore also exhibits capacitance. When the sensor's
'normal' capacitance field (its reference state) is altered by another capacitance field,
i.e., someone's finger, electronic circuits located at each corner of the panel measure
the resultant 'distortion' in the sine wave characteristics of the reference field and send
the information about the event to the controller for mathematical processing.
Capacitive sensors can either be touched with a bare finger or with a conductive
device being held by a bare hand. Capacitive touchscreens are not affected by outside
elements and have high clarity. The Apple iPhone is an example of a product that uses
capacitance touchscreen technology.
Infrared
An IR touchscreen panel employs one of two very different methods. One method
uses thermal induced changes of the surface resistance. This method is sometimes
slow and requires warm hands. Another method is an array of vertical and horizontal
IR sensors that detect the interruption of a modulated light beam near the surface of
the screen.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 3
Optical imaging
A relatively-modern development in touchscreen technology, two or more image
sensors are placed around the edges (mostly the corners) of the screen. Infrared
backlights are placed in the camera's field of view on the other sides of the screen. A
touch shows up as a shadow and each pair of cameras can then be triangulated to
locate the touch. This technology is growing in popularity, due to its scalability,
versatility, and affordability, especially for larger units.
Dispersive signal technology
Introduced in 2002, this system uses sensors to detect the mechanical energy in the
glass that occur due to a touch. Complex algorithms then interpret this information
and provide the actual location of the touch. The technology claims to be unaffected
by dust and other outside elements, including scratches. Since there is no need for
additional elements on screen, it also claims to provide excellent optical clarity. Also,
since mechanical vibrations are used to detect a touch event, any object can be used to
generate these events, including fingers and stylus. A downside is that after the initial
touch the system cannot detect a motionless finger.
1.1.2 DEVELOPMENT
Virtually all of the significant touchscreen technology patents were filed during the
1970s and 1980s and have expired. Touchscreen component manufacturing and
product design are no longer encumbered by royalties or legalities with regard to
patents and the manufacturing of touchscreen-enabled displays on all kinds of devices
is widespread.
The development of multipoint touchscreens facilitated the tracking of more than one
finger on the screen, thus operations that require more than one finger are possible.
These devices also allow multiple users to interact with the touchscreen
simultaneously.
With the growing acceptance of many kinds of products with an integral touchscreen
interface the marginal cost of touchscreen technology is routinely absorbed into the
products that incorporate it and is effectively eliminated. As typically occurs with any
Multi Touch
Division Of Computer Science ,SOE,CUSAT 4
technology, touchscreen hardware and software has sufficiently matured and been
perfected over more than three decades to the point where its reliability is
unassailable. As such, touchscreen displays are found today in airplanes, automobiles,
gaming consoles, machine control systems, appliances and handheld display devices
of every kind. The ability to accurately point on the screen itself is taking yet another
step with the emerging graphics tablet/screen hybrids.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 5
2.MULTI TOUCH
Multi-touch is a human-computer interaction technique and the hardware devices that
implement it, which allow users to compute without conventional input devices e.g.,
mouse, keyboard. Multi-touch consists of a touch screen like screen, table, wall or
touchpad, as well as software that recognizes multiple simultaneous touch points, as
opposed to the standard touchscreen i.e. computer touchpad, ATM, which recognizes
only one touch point. This effect is achieved through a variety of means, including but
not limited to: heat, finger pressure, high capture rate cameras, infrared light, optic
capture, tuned electromagnetic induction and shadow capture.
2.1 HISTORY
Multi-touch technology dates back to 1982, when the University of Toronto
developed the first finger pressure multi-touch display. The same year, Bell Labs at
Murray Hill published what is believed to be the first paper discussing touch-screen
based interfaces.
Bell Labs
In 1984 Bell Labs engineered a multi-touch screen that could manipulate images with
more than one hand. The group at the University of Toronto stopped working on
hardware and moved on to software and interfaces, expecting that they would have
access to the Bell Labs work.
A breakthrough occurred in 1991, when Pierre Wellner published a paper on his
multi-touch “Digital Desk”, which supported multi-finger and pinching motions.
Fingerworks
In 1998, FW, a Newark-based company run by University of Delaware academics
John Elias and Wayne Westerman, produced a line of multi-touch products including
the iGesture Pad and the TouchStream keyboard. Westerman published a dissertation
Multi Touch
Division Of Computer Science ,SOE,CUSAT 6
in 1999 on the subject. In 2005, after years of maintaining a niche line of keyboards
and touchpads, Fingerworks was acquired by Apple Computer.
2.2 RECENT DEVELOPMENTS
Various companies expanded upon these discoveries in the beginning of the twenty-
first century. Mainstream exposure to multi-touch technology occurred in the year
2007, when Apple unveiled the iPhone and Microsoft debuted surface computing. The
iPhone in particular has spawned a wave of interest in multi-touch computing, since it
permits greatly increased user interaction on a small scale. More robust and
customizable multi-touch and gesture-based solutions are beginning to become
available, among them TrueTouch, created by Cypress Semiconductor. The following
is a compilation of notable uses of multi-touch technology in recent years.
Microsoft Surface
In 2001 Steve Bathiche and Andy Wilson of Microsoft began work on an idea for an
interactive table that mixes both physical and virtual worlds. Research and
Development expanded rapidly in 2004, once the idea caught the attention of
Microsoft Chairman Bill Gates. In 2007 Microsoft introduced MS, a functional multi-
touch table-top computer based on a standard PC platform including an Intel Core 2
Duo processor, Windows Vista, and 2 GB of RAM
Perceptive Pixel
PP is a company founded by New York University consulting research scientist
Jefferson Y. Han that creates wall displays and tables that can accommodate up to 20
fingers. Han introduced the FTIR technique to multi touch screens. The displays use
light emitting diodes along with infrared light to determine the point of contact. Han
envisions large collaborative spaces that will allow multiple users to work and
interact. PP’s technology is currently being utilized, in the form of the Multi-Touch
Collaboration Wall, by CNN and an unspecified government contractor everyday
programs with ease and most importantly more than one user can operate the system
at any given time.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 7
Apple iPhone, iPod touch, MacBook Air, and MacBook Pro
In 2005, Apple acquired Fingerworks. In 2007 they introduced the iPhone, marking
the first time multi-touch technology was used on a phone. The iPhone includes such
components as a web browser, music player, video player, and a cell phone without
the use of a hard keypad or stylus.
Following the release of the iPhone, Apple also expanded its use of multi-touch
computing with the new iPod Touch, as well as the new MacBook Air. Multi-touch
was later added to the 2008 MacBook Pro line in the form of a trackpad. Apple is
currently in the process of trying to patent its “Multi-touch” technology and to
trademark the term "multi-touch".
2.3 FUTURE
The use of multi-touch technology is expected to rapidly become common place. For
example, touch screen telephones are expected to increase from 200,000 shipped in
2006, to 21 million in 2012. Developers of the technology have suggested a variety of
ways that multi-touch can be used including:
Enhanced dining experience
Concierge service
Governmental use
Concept mapping
Collaboration and instruction on Interactive Whiteboards
Multi Touch
Division Of Computer Science ,SOE,CUSAT 8
3.FRUSTRATED TOTAL INTERNAL REFLECTION
3.1 PRINCIPLES
Total internal reflection is an optical phenomenon that occurs when a ray of light
strikes a medium boundary at an angle larger than the critical angle with respect to
the normal to the surface. If the refractive index is lower on the other side of the
boundary no light can pass through, so effectively all of the light is reflected. The
critical angle is the angle of incidence above which the total internal reflection occurs.
When light crosses a boundary between materials with different refractive indices, the
light beam will be partially refracted at the boundary surface, and partially reflected.
However, if the angle of incidence is greater (i.e. the ray is closer to being parallel to
the boundary) than the critical angle — the angle of incidence at which light is
refracted such that it travels along the boundary — then the light will stop crossing
the boundary altogether and instead be totally reflected back internally. This can only
occur where light travels from a medium with a higher refractive index to one with a
lower refractive index. For example, it will occur when passing from glass to air, but
not when passing from air to glass.
The critical angle is the angle of incidence above which total internal reflection
occurs. The angle of incidence is measured with respect to the normal at the refractive
boundary. The critical angle is given by:
where n2 is the refractive index of the less dense medium, and n1 is the refractive
index of the denser medium.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 9
Fig 3.1 Total Internal Reflection
An important side effect of total internal reflection is the propagation of an evanescent
wave across the boundary surface. Essentially, even though the entire incident wave is
reflected back into the originating medium, there is some penetration into the second
medium at the boundary. Additionally, the evanescent wave appears to travel along
the boundary between the two materials. This wave can lead to a phenomenon known
as frustrated total internal reflection.
Under "ordinary conditions" it is true that the creation of an evanescent wave does not
affect the conservation of energy, i.e. the evanescent wave transmits zero net energy.
However, if a third medium with a higher refractive index than the second medium is
placed within less than several wavelengths distance from the interface between the
first medium and the second medium, the evanescent wave will be different from the
one under "ordinary conditions" and it will pass energy across the second into the
third medium.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 10
3.2 DESIGN
3.2.1 REQUIREMENTS
Multi touch is designed using the FTIR technology. FTIR describes the internal
reflection of light .It is force-sensitive, and provides unprecedented resolution and
scalability .Large enough to accommodate both hands and multiple users.This
phenomenon is also used in fingerprint and robot sensors.Allows us to create
sophisticated multi-point widgets for applications
3.2.1.1 HARDWARE
The basic design has a hardware and software part. Hardware requires basically IR
led’s,acrylic,camera,projector and a computer. Infra red light has a higher wavelength
than that of visible light .Therefore it has more intensity and will be felt everywhere
inside the denser medium. Acrylic is the denser medium .It is a synthetic fiber having
half the density of glass.An infra red camera or a webcam is used to catch IR light . A
IR block filter of the camera has to be removed since it blocks IR light. It also consists
of a projector and a computer. The object has to be projected on top of the acrylic
from a computer . Thus acrylic is a virtual display. The IR LED’s about 10-20 are
arranged on both sides of the acrylic along its edges .The camera below the acrylic
and the projector located behind to acrylic.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 11
Fig 3.2 Acrylic with led’s at its edges
3.2.1.2 SOFTWARE
Computer applications are necessary to communicate between a multi touch display
and a computer.These applications can be developed within several languages /
programming environments.For example: Processing (P5), Flash, C, C++, Java and
others.
Because of the usability of certain API’s and the relatively simple visualisation
possibilities, P5 or Flash 9, in combination with Actionscript 3.0 will make a great
combination .
Multi Touch
Division Of Computer Science ,SOE,CUSAT 12
Fig 3.3 IR rays obtained on a camera
Multi Touch
Division Of Computer Science ,SOE,CUSAT 13
3.4 WORKING
FTIR describes the internal reflection of light, inside a certain material. In our case, it
will be infrared light, that internally reflects inside is a piece of acrylic, also known as
plexiglas.This way, infrared light is beamed inside the acrylic and reflects internally.In
a simple way, you can say that, IR-light bounces inside the acrylic, from one side to
another.As soon as a finger touches the acrylic surface, the internal reflection of the
IR-light, is interrupted.The infrared light scatters on the finger tips.Infrared light is
invisible to the human eye, but by placing an infrared camera behind the acrylic your
fingertips will be visible on the infrared camera.The images that are generated by the
camera, contain white blobs (caused by the fingertips).
These blobs will be analyzed by software. Every blob corresponds to certain
coordinates. Software can by analyzing these coordinates perform certain tasks, for
example move, resize or rotate objects.
Fig 3.4 Frustrated Total Internal Reflection
Multi Touch
Division Of Computer Science ,SOE,CUSAT 14
(a) (b)
Fig 3.5 (a) IR view of a touch(b)Pixel positions located on screen
Multiple points are obtained on the camera .Each point is a pixel position. Either a
single pixel or a group of pixel . Each point locations are identified and all operations
are performed .Suppose if we want to zoom a picture we use 2 fingers and move it in
or out to perform zoom in and zoom out respectievely . Two coordinates will be
located on the camere .The difference is found which is put as the offset and it is
either added or subtracted with the locations to zoom out and zoom in respectievely.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 15
3.5 ADVANTAGES
Multi touch based on FTIR is a simple and inexensive technique .It
constructs a multi touch display with the available and less costly
materials .
Scalable technique that enables high-resolution graphics .It provides
support to any resolution possible as all multiple points could be
generated on a camera
It acquires true touch image information at high spatial and temporal
resolutions.The actual finger print of the touch is obtained .This could
be used to determine the force sensitivity on displays , either too hard
or soft touches can be analysed.
It is scalable to large installations.Any kind of applications can be
made to suit multi touch using FTIR . Allows us to create sophisticated
multi-point widgets for applications
Larger shared-display systems ie it is well suited for use with rear-
projection like wall screens,table tops .All this lead to high resolution
graphics.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 16
3.6 APPLICATION
A myriad different applications for multi-touch interfaces both exist and are being
proposed. Some uses are individualistic e.g., iPhone, iPod touch, MacBook Pro,
MacBook Air, HTC Diamond . However, multi-touch technology is mainly used to
incorporate collaboration into the computing experience .
A multi touch display can be used in
Personal computers,Laptops,Tabletops,Graphics Tablets .
It supports both LCD and CRT monitors .
Telephones ,Watches ,PDAs, Mobile phones.
Advanced multi touch Gaming with high graphics support
Governmental,office and business purposes
An enhanced multimedia experience including audio,video and photo sharing
Enhanced dining experience
Applications for a multi touch display are never ending . We can even convert a
computer to a mere piece of display attached to a wall or a table
Multi Touch
Division Of Computer Science ,SOE,CUSAT 17
4.CONCLUSION
Touch screens are the interface for the 21st century. Touch screens address the
conflicting demands for smaller portable electronics with larger displays, by
eliminating traditional buttons without sacrificing screen size. The recent release of
the iPhone has created a buzz around touch screen interfaces and its multi-touch
acrobatics have caught the eye of many industry leaders.
There are many ways to make a multi-touch screen. Some of the early designs
measured the change in electrical resistance or capacitance on a surface when fingers
touched it. But these devices have limited resolution, are relatively complex, and don't
easily and inexpensively scale up to large dimensions. Multi-touch technologies have
a long history. This technique using FTIR is simple and easy to implement . It
provides any resolution displays supported with high graphics .The applications being
both made and proposed are plenty in number .
A drawback of the approach is that, being camera-based, it requires a significant
amount of space behind the interaction surface, though we primarily expect
application scenarios where rear-projection would have been employed anyway (e.g.
interactive walls, tables). Also, as an optical system, it remains susceptible to harsh
lighting environments.
Multi Touch
Division Of Computer Science ,SOE,CUSAT 18
5.APPENDICES
5.1 APPENDIX I
(a) (b)Fig 5.1 (a) Prototype setup (left) (b) Video output w/o diffuser (right)
Multi Touch
Division Of Computer Science ,SOE,CUSAT 19
Fig 5.2 Photo Sharing and multimedia
Fig 5.3 Wonders of multi touch ,Transfering music between Ipods
Fig 5.4 A roadside wall at Helsinki
Multi Touch
Division Of Computer Science ,SOE,CUSAT 20
6.REFERENCES
Low-Cost Multi-Touch Sensing through FTIR by Jefferson Y. Han
Buxton, W., Hill, R., and Rowley, P. 1985. Issues and Techniques in
Touch-Sensitive Tablet Input
Donald Hearn ,M Pauline Baker, Computer Graphics C version, 2/E
Pearson Education ,2003
Buxton, Bill. 2008. Multi-Touch Systems that I Have Known and Loved.
http://www.billbuxton.com/multitouchOverview.html
How to build a multi touch by Harry Vaan Der
Opensource,MultitouchDisplay,http://www.technologyreview.com/Infotec
h