The Music Sphere

Name: Andreas Papadopoulos

Degree: MA in Music Technology

Supervisors: Oliver Larkin and Dave Malham

Date: 31-08-2010

Abstract

This project report describes the concept and the development process of a multi-

touch musical interface, the Music Sphere, an application made for the iPad. The

main means of interaction between the user and the produced sounds is a Sphere.

As the user performs gestures on the screen, various visual and audio effects are

applied to the sphere and the sounds respectively, resulting in an interactive

experience. The project researches into existing multi-touch technology and musical

interfaces that rely on that technology. It also explores gestural interactivity as a way

of composing and manipulating electronic music in real time. Such multi-touch

interfaces present a more natural way of control and can consequently help in crucial

elements of a live performance such as expressiveness and spontaneity.

Y4796421

2

Table of Contents

Glossary 5

1. Introduction 6

1.1 History of Multi-touch Technology in HCI 6

1.2 Overview of this work 8

2. Multi-touch Technologies 9

2.1 Optical-based 9

2.2 Resistive 11

2.3 Capacitive 12

3. Multi-touch Technology in Music 13

3.1 The reacTable* 13

3.2 Jazzmutant Lemur 17

3.3 Apple iPhone/iPad 20

3.4 Subcycle 26

4. The Music Sphere 27

4.1 The Concept 27

4.2 Scene One - Pre-Recorded Material 29

4.3 Scene Two - FM Synthesiser 31

Y4796421

3

4.4 Other Synthesis Methods 32

5. Development Process 33

5.1 Program Structure 33

5.2 The C++ Sphere Method 35

5.3 Gestures 38

5.4 Communication Protocol 40

6. Conclusions 44

6.1 Evaluation 44

6.2 Further Development 45

7 References 47

8 Appendices 52

8.1 Sphere C++ Function 52

8.2 Apple SDK Gesture Recognisers 53

8.3 Custom Gesture Recogniser 55

8.4 Custom Gesture Recogniser Responders 57

8.5 OpenGL ES 58

8.6 OSC 58

Y4796421

4

Glossary

AM - Amplitude Modulation

CPU - Central Processing Unit

DAW - Digital Audio Workstation

DI - Diffused Illumination

DIY - Do It Yourself

DSI - Diffused Surface Illumination

FFT - Fast Fourier Transformation

FM - Frequency Modulation

FPU - Floating Point Unit

FTIR - Frustrated Total Internal Reflection

GPU - Graphics Processing Unit

GUI - Graphical User Interface

HCI - Human Cmputer Interaction

LFO - Low Frequency Oscilator

LLP - Laser Light Plane

MaxMSP - Object oriented audio programming environment

MCU - Mackie Control Universal

MIDI - Music Instrument Digital Interface

NI - Native Instruments

OpenGL ES - OpenGL Embedded Systems

OSC - Open Sound Control

PD - Pure Data

PDA - Personal Digital Assistant

RGBA - Red Green Blue Alpha

SDK - Software Development Kit

Y4796421

5

1. Introduction

1.1 History of Multi-touch Technology in HCI

In 1997 Matsushita and Rekimoto unveiled the HoloWall, a multi-touch sensitive

interactive wall. The “wall” consisted of a sheet of glass coupled with a sheet of rear-

projection material. Behind it, a projector was used to display material while an

infrared light source and a camera were used for the touch detection. Anything that

would touch the glass wall would reflect the infrared light back to the camera

(Matsushita, 1997:209).

In 2004, Wilson presented TouchLight, a technology suited for installations and

wall interaction. It uses two cameras placed behind a semi-transparent sheet of

acrylic, which acts as the interactive surface. The outputs of these cameras are

combined and further processed with various image processing techniques to result

in touch detection. This technique also uses a projector to display material on the

acrylic surface (Wilson, 2004:69).

Recently, multi-touch systems have become more affordable. Han (2005:115)

presented such a low cost multi-touch technique in 2005 called Frustrated Total

Internal Reflection. Infrared light is emitted within a sheet of acrylic which forms the

touch screen surface. Contact with objects or fingers on the screen will cause the

Infrared light to scatter towards a camera. After analysis on the captured data is

possible to track the spots that are being touched.

In June 2007 Apple Computer releases the iPhone, a cellular phone device

that features a 480x320 multi-touch screen. It achieves multi-touch sensitivity by

using a technology where tiny electrical fields are spread across the multi-touch

surface. These fields will change their value when they are in contact with a finger.

In 2007 Microsoft Research Cambridge presented ThinSight, an optical

sensing system that can be used in conjunction with standard LCD displays to form

Y4796421

6

multi-touch panels. ThinSight’s operational principle relies on the properties of a

optosensor, a device that can both emit and detect amounts of infrared light. When

an object or finger touches the surface, some of the emitted light is reflected back to

the optosensor and that is how detection is achieved. Multiple optosensors are used

in a grid configuration to cover the display surface and transform it into multi-touch

system (Hodges, 2007:259).

There is also a number of interfaces that demonstrated multi-touch

functionality and were oriented towards musical expression and composition. The

AudioPad works by placing electromagnetically tagged objects on a surface. Each

object includes a simple circuit with a capacitor that resonance at different frequency.

A specifically designed antenna can measure the amplitudes of the resonant

frequencies and, as a result, track the object. Feedback is provided by a projector

which is placed above the user (Patten, 2002:1).

The SoundScape Renderer is a software system aimed to reproduce spatial

audio. The current version can render acoustic scenes by either Wave Field

Synthesis (WFS) or Vector Based Amplitude Panning (VBAP). The multi-touch

SoundScape renderer is a multi-touch controller for the SoundScape Renderer

software. It is basically a FTIR multi-touch table (Han, 2005, 115) that controls

various parameters of the software in real-time, allowing multiple musicians to

collaborate in creating acoustic music (Brown, 2009:446).

DrawSound, introduced in 2008, combines multi-touch input technology with

the expressive characters of drawing on the ordinary paper in a sound performance.

It uses a DiamondTouch multi-touch interface for touch sensitivity and data extraction

while MaxMSP is used for mapping the data from DiamondTouch to different sound

synthesis techniques. DrawSound has been used in two projects, the Sine Wave

Quartet and aeo (Jo, 2008:59).

Y4796421

7

1.2 Overview of this work

Multi-touch interaction both in Human Computer Interaction (HCI) and in music is a

area with particular technological interest. The initial goal for this project was to

develop a multi-touch interface targeted for musical interactivity. The first and most

important principle in this work was to avoid conventional and widely used control

concepts such as sliders, knobs and buttons. Instead, the interface explores and

evaluates the scenario where a 3D shape like a sphere could serve as the main

means of interaction between the performer and the music.

Despite the fact that a custom build multi-touch interface could be possible,

the development of this project is targeted towards the Apple iPad multi-touch device

for a number of different reasons. First of all, the iPad’s multi-touch technology is

proven to work effectively without issues, eliminating any errors and instability that a

custom interface might have. In addition, there is a growing community of developers

who provide support and help to new and inexperienced developers that wish to take

the first steps into programming. There are also numerous examples bundled in the

iPhone SDK which make the development process easier. Finally, developing

applications for such multi-touch mobile devices is an area of particular interest for

me, as it open the possibility of developing commercial applications in the future.

The objectives that were set for this project where very straightforward: to

create a multi-touch musical prototype interface where the user could interact with

audio. The project also evaluates the level of success of using the 3D concept for

controlling music through gestural interactivity. In addition, the development process

would enable me to investigate how multi-touch interfaces work as well as make me

familiar with developing apps for the iPhone, which is something I would like to do in

the future.

Y4796421

8

2. Multi-touch Technologies

Different inventors have taken different technological approaches to achieve multi-

touch functionality, depending on factors such as cost, ease of construction, size of

the interface and more. There are three main categories of technology that evolved

over the years which are: optical-based, resistive and capacitive. Each one has its

own weaknesses and strengths while which one is the best depends on the desired

functionality of the final interface. As a general rule, optical-based multi-touch

technology is preferable for installations and large multi-touch surfaces while the

remaining technologies are suited for smaller surfaces and commercial devices.

2.1 Optical-based

One very commonly used method is achieving touch sensing via optical techniques

and IR light sensing. These approach to multi-touch sensing requires an optical

device (camera), IR light generator (IR LEDs or other) and an visual feedback device

(LCD display or digital projector). There is a number of variations in the construction

when utilising optical multi-touch sensing methods each having its positive and

negative aspects.

Generally, all the optical techniques share the same operational principle. The

camera scans the touch surface at a fixed amount of frames per second and the

resulting images are processed with various image rectification algorithms in order to

eliminate any errors that might occur. The resulting clearer view of the interaction

points, or else “blobs”, is fed to the tracking algorithm. The tracking algorithm starts

here and exports data as lists of XY coordinates which are used by the software that

runs on the computer to perform tasks (e.g. make sounds). These XY coordinates

are also used to crate visuals that the projector displays on the touch surface to

provide feedback to the user.

Y4796421

9

Frustrated Total Internal Reflection (FTIR)

This multi-touch sensing technique relies on Total Internal Reflection, a well-

known optical phenomenon. In FTIR, an acrylic sheet (about 1 cm thick) is run

through IR light which is emitted by IR LEDs that are placed along the perimeter of

the sheet. The IR light is trapped inside the acrylic surface and whenever the user

interacts (a finger comes into contact with the surface), the IR light finds a way to

escape the acrylic surface as it is scattered by the skin (the light is frustrated). This

light is then scanned by the camera (Han, 2005:115).

Diffused Illumination (DI)

The Diffused Illumination technique comes in two variations, Front Diffused

Illumination and Rear Diffused Illumination. Both methods rely on the shadow that is

created when a finger touches a surface that is projected with light. The Front

Diffused Illumination method uses visible light which comes from above while the

Rear Diffused Illumination utilises IR light that is place underneath the touch surface

(NUI Group Authors, 2009:13).

Laser Light Plane (LLP)

The LLP technique differs from the others in terms of the light source, as it

uses laser technology instead of IR light. The light is shed just above the surface so

when something touches the surface the camera will pick up a blob.

Diffused Surface Illumination (DSI)

DSI combines characteristics from the FTIR and DI methods. Physically, it is

very similar to FTIR technique (a frame of IR LEDs is needed) apart from the fact that

a special type of acrylic is needed, the EnLighten Acrylic. Due to the nature of this

special acrylic, the light get more even distribution than the DI method (NUI Group

Authors, 2009:15).

Y4796421

10

All of the aforementioned techniques have their strengths and weaknesses

and choosing which one is the most suitable depends largely on the applications and

features that are needed. Also, some of these techniques might be affected by

external factors such as the ambient light and they might only be suitable for

particular applications (installations etc).

Since the camera is a very important aspect of this touch sensing technology,

the reliability and usability of such a system will depend largely on the quality of the

camera. First of all, the camera needs to be sensitive to IR light, since this is used for

the finger tracking. Many consumer USB cameras utilise an IR filter to filter out the IR

information as this band is not visible and not needed in an average consumer

device. The camera has to be modified in order to be sensitive to the IR light by

taking the IR filter out. Also, other technical characteristics (specifications) of the

camera device are important such as the resolution and the frame rate. Typically, a

standard VGA 640x480 pixels frame size is adequate but more resolution will

improve precision. Also, the frame rate plays an important role in the responsiveness

of the system and largely determines the latency of the system. Finally, it would be

sensible to go through the technical specifications of the camera to see the exact

response of the lens to IR light and find IR LEDs or other IR light source that emit IR

light in the IR band that the camera is most sensitive at (Muller, 2008:11).

2.2 Resistive

Resistive multi-touch sensing is achieved by implementing a structure of two

electrically conductive layers coated with indium tin oxide (ITO) separated by a thin

insulating layer. A specified amount of electric current is sent to one layer while the

current of the other layer is being monitored. When the user touches the screen, the

two layers are connected and that makes some electrical current to pass through the

Y4796421

11

two layers. By measuring the amount of that current it is possible to determine the

exact position of the touch (Schöning, 2008:2).

The production of resistive touch screens is cheap and as a result such touch

screens have been very successful commercially. Numerous mobile devices, PDAs

and portable gaming devices like the Nintendo DS feature such a touch screen not

only because its affordable, but also because it is durable and robust under normal

use conditions.

However, providing additional protection to the screen by adding a protective

layer is not an option because the touch sensing responsiveness decreases. As a

result, the touch sensor is vulnerable to damage by sharp objects. Moreover, adding

a resistive touch screen layer on top of an existing LCD or TFT screen reduces the

clarity of the screen to approximately 75-85%, resulting in a blurry image. (Schöning,

2008:2) Finally, resistive touch sensing is considered as a low-resolution technology

because of the fact that the screen responds only when a touch is made at the exact

point of the resistors.

2.3 Capacitive

Electrodes are placed around the edges of the screen surface, establishing an

electrical field of a specified amount. Whenever a finger comes in touch with the

screen, an amount of the electrical charge is passed through, resulting in an

alteration of the original electrical field. By measuring the amount of current that is

passed through the adjacent object the position of the touch is determined

(Schöning, 2008:3).

Capacitance based multi touch technology does not reduce the clarity of the

screen like the resistive technology and that is an important advantage. This

advantage in combination with features like robustness and durability makes this

Y4796421

12

technology more suitable for a much wider application area. Capacitive multi-touch

surfaces can be used equally well in installations and in commercial products where

reliability is crucial.

This technology has its own disadvantages, such as the high construction

cost. In addition, capacitive sensing can only respond touches made with fingers or

other objects with similar electrical properties, making use of regular stylus pens not

possible. However, special stylus pens are becoming available.

3. Multi-touch Technology in Music

As seen in the introduction, multi-touch technology has not left the music community

unaffected. Various projects have been undertaken either by researchers or

commercial companies in quest of new and innovative ways to incorporate this

promising new technology in musical interfaces. Several projects have been

successfully completed and presented to the public.

This section covers examples of multi-touch musical interfaces that use the

three different types of multi-touch technology presented in the previous chapter. The

fact that each of these projects achieve multi-touch functionality by utilising different

type of multi-touch technology, proves that musical expression is not affected by the

technology itself but rather, by the principle of multi-touch interaction.

3.1 The reacTable*

The reacTable is an modern musical instrument that features a circular multi-touch

surface. In addition, it is a collaborative instrument, meaning that more than one

users can operate the instrument at the same time. The users provide input to the

device by placing, moving and rotating pre-determined objects on the illuminated

multi-touch surface. Each object represents a link in the audio signal chain which

Y4796421

13

affects the sonic outcome in different ways. Various parameters of this signal path

are controlled by rotating the objects or moving them to be either closer or further

from each other.

The reacTable is the perfect example of FTIR multi-touch technology,

presented by Han in 2005. Physically, the reacTable is a round table of one meter

diameter, space that in most cases will be enough for two to four performers

simultaneously. Infrared emitters provide the necessary infrared illumination which

runs through the acrylic surface. Whenever an object of a finger comes in contact

with the surface, the infrared light is scattered towards in the inside of the reacTable’s

body and picked up by the infrared-sensitive camera. The images captured by the

camera are processed and analysed by the reacTIVision software, which produces

data based on that analysis (Kaltenbrunner, 2009:9). Examples of such data are the

the exact position of a finger touch or shape recognition. The visuals are

accomplished with a digital projector which projects the generated images on the

acrylic surface.

As described earlier, computer vision systems tend to have several

disadvantages like occlusion issues, latency and instability over other more

sophisticated technologies. However, the reacTable utilises a camera based

computer vision system to get input. By using higher quality components such as

cameras sampling 60 frames per second or more, the inherited technical limitations

of the system are not noticeable to the end user, making FTIR preferable in terms of

cost and ease of construction. Furthermore, choosing FTIR technology presented

some advantages like support for virtually unlimited touches, support for recognition

of several hundreds markers and the ability to detect the orientation of an object.

Furthermore, this approach improves the aesthetics of the device, by hiding the

components (camera, projector, IR source) within the reacTable’s body.

Y4796421

14

The audio generation engine is made in Pure Data (PD) and is based on the

principle of modularity. Basic sound synthesis and processing concepts like wave-

table oscillators and granular synthesis were implemented in PD patches which were

encapsulated in order to fit in the reacTable’s modular structure. Each of these

abstractions is represented, instantiated and controlled by a particular object that the

user might choose to use in the composition (Jordà, 2004:20).

The visual aspect of the reacTable is a crucial part of the instrument as it

serves as the only means of user feedback. As the users interact with the reacTable,

various lines, shapes and abstract graphics are drawn on the screen which reflect the

way audio is synthesised and processed. These graphics derive from the audio part

of the reacTable, giving instant and comprehensive visualisation of the whole signal

path. (Jordà, 2004:20) In addition, part of the visual generation provides information

about the state of the objects, how these objects are connected and various other

essential information about the composition. For example, the circles around It is

worth noting that numeric values and characters are intentionally avoided in order to

let the users interact in a more natural way.

The idea of collaboration has been a serious objective in the reacTable’s

development, and can take the form of either Local or Remote Collaboration (Jordà,

2006:408). The reacTable is designed to host up to four users at the same time.

These users can choose to have separate workspaces or to share the the whole

reacTable surface. In the first case, each user produces sounds individually which

are mixed together to produce the final musical outcome. In the second case all

users contribute to the same patch, each affecting various parameters of the same

signal path. Musical instrument players can also contribute to the session by

providing the output of their instruments as a sound source for the reacTable

performers.

Y4796421

15

The reacTable development team has taken it a step further, incorporating the

Remote Collaboration feature. In Remote Collaboration, two or more reacTables can

be linked for a performance. The two tables can be placed in different rooms, cities or

countries, as they communicate via a the internet. The actions, and consequently the

sounds, that a user performs in one location are reflected and reproduced to other

location, giving the performers of both locations the ability to interact musically and

creatively.

Finally, there is also the Remote Participation scenario, where people from the

audience can contribute to the composition. This form of collaboration does not

require a physical reacTable but rather, a reacTable simulator application that can run

on mobile devices like PDAs, net-books and mobile phones. Even though the

simulator is fully integrated into the reacTable setup and theoretically provides the

same flexibility and features, it is obviously not capable of delivering the same feeling

and level of creative freedom as the real reacTable.

The reacTable development team has also been working on the scoreTable,

an interface very similar to the reacTable that will be primarily used for sequencing

purposes. The two tables can interact with each other by sending and receiving pitch

and tempo values, expanding the creative potential of the interface.

The objects that a user can use are divided into seven groups. Each group

has a different functionality and affects the musical result in a different way. The first

group of objects includes all the Generator objects. As their name suggests, these

objects are used to instantiate a sound synthesis engine or playback of pre-recorded

material. They also have the ability to interconnect with control objects (explained

later) in order to let the user access various parameters of the generator. The second

category is called Audio Filters and consists of all the objects that are capable of

changing and manipulating audio streams in real time. The Audio Filters objects do

Y4796421

16

not produce sound on their own, but they require the output of a Generator object (or

the output of another Audio Filter) in order to function. Each object from this group is

associated with a particular audio processing algorithm like low-pass filter and delay.

Controlling the algorithm’s parameters in greater detail is achieved by using control

objects.

The next two groups are Controllers and Control Filters and are auxiliary

objects that work with data rather than audio streams. The output of these objects

can feed the control inputs of the aforementioned Generator and Audio Filter object

groups. A Controller object generates data based on an algorithm. Example.

The next group consists of a single object, the Mixer object. Unsurprisingly, this

object sums many audio streams into one single audio stream. There is also the

ability to use the Mixer object inverted in order to create a Splitter object which

outputs duplicates of the incoming audio stream. The Clock Synchronisers form the

sixth group of objects and have a higher level role in the reacTable’s workflow. Clock

Synchronisers control multiple objects simultaneously in order to fine tune the

parameters of these objects in terms of time (tempo) and pitch. The last group

includes all the Container objects, objects that encapsulate the functionality of two or

more combined objects. These objects are particularly useful for recall of complex

effect chains and synthesis engines. The Container objects need to be pre-

configured, meaning that a user cannot make one in real time (Jordà, 2004:21).

3.2 Jazzmutant Lemur

The Jazzmutant Lemur is an interface that features a 12” multi-touch TFT screen.

The screen has a resolution of 800x600 pixels and supports up to 10 simultaneous

finger touches. The device achieves multi-touch functionality by utilising the resistive

technology. The purpose of the Lemur is to provide real-time control over different

Y4796421

17

types of musical hardware and software in an intuitive way. The user can directly

control and even modulate various parameters of the available sound processors by

touching and changing UI controls on the screen. Due to its ease of use, robustness

and mobility, the device is very popular among live performers such as DJs.

The Lemur operates as a stand-alone unit, meaning that can run

independently from the computer, without relying on the computer’s CPU. The device

draws the required computational power from its own CPU, GPU and FPU

processors. The GPU is responsible for rendering the graphics on the multi-touch

screen while the CPU and FPU undertake other operations like touch detection. The

communication between the Lemur and the computer is accomplished via an

Ethernet cable. The Ethernet 100-baseT protocol has more than sufficient Data

Transfer Rate (DTR) for transmitting and receiving OSC messages and,

consequently, guarantees that even large numbers of OSC data will successfully

stream through the cable without any noticeable artifacts like data loss or latency

(Jazzmutant, 2010).

Apart from musicians, the Lemur has also gained popularity among other

types of performers who want to control their equipment in real-time such as VJs and

lights engineers in concerts and night clubs. These creative professionals rely more

and more on computer software for their performances, and the Lemur has proved to

be the perfect companion to accomplish great performances. In fact, the Lemur will

communicate with any OSC and/or MIDI compliant piece of software, giving the

potential for new and interesting control concepts not only for music, but also for

human computer interaction. In addition, the device is capable of communicating with

many computers as well as many different software applications at the same time,

meaning that performers can easily control their entire setup from a single device.

Y4796421

18

One of the Lemur’s main strengths is the ability to create customised User

Interfaces (UIs) that suit a particular application. Users can create a custom UIs

(patches) on the computer using the JazzEditor application, and then upload these

patches to the hardware. It is worth noting that JazzEditor also works as an OSC to

MIDI converter and forms the way the Lemur interfaces with MIDI applications. There

is a number of pre-determined modular UI elements (objects) that a user can choose

from, ranging from simple sliders to more sophisticated and interactive components.

Starting with conventional objects, the Lemur offers the following types of objects: a

fader (horizontal and vertical), knob (endless and normal), switches (arrays of on/off

toggles), multi-slider and pad. In most cases these objects act like their physical

counterparts, occasionally offering enhanced functionality. For example, the multi-

slider object reacts to finger swipes (physically difficult to do with real faders) and

also offers a gravity mode, which is particularly useful in cases where the performer

wants not only to control but also to modulate a target parameter.

There are also objects that are used only to provide complementary user

feedback and do not react when they are touched. These objects are Signal Scope,

Monitor and SurfaceLCD. These objects can be set to display either the incoming

OSC/MIDI data or various parameters from Lemur’s internal operations. Some

objects are made with a specific use in mind and have special features like displaying

the history of a parameter’s state or number formatting for timecode display.

The Lemur also offers two unconventional UI elements, the Ring Area and the

MultiBall. The Multiball object is a 2D controller and is basically an enhanced version

of an XY pad. The object can host up to ten balls, one for each finger, and apart from

XY functionality each ball also has a Brightness property which can be used to

expand the object’s functionality even further. The Ring Area object is visually similar

to the multiball object where a ball is confined within a circular space. There is the

Y4796421

19

option to add attractors to the Ring Area so the ball returns to a pre-determined spot

after the object is released.

Jazzmutant also offer Dexter, which is a software add-on to the Lemur

hardware. Dexter is oriented towards Digital Audio Workstation (DAW) control,

bringing the benefits of multi-touch technology to studio engineers and producers

(Jazzmutant, 2010). It is worth noting that such technology was not previously

available for dedicated studio equipment. Similarly with the Lemur, Dexter’s UI is

completely customisable by using the JazzEditor application, meaning that a user

can map any UI control to any parameter of a DAW project. The creative potential of

this approach is obvious, as it allows DAW based audio professionals to explore and

experiment with new control concepts and possibilities in typical DAW workflows.

Finally, sound engineers can also reduce the required time for tedious tasks like

editing automation by incorporating Dexter in their workflow. Apart from the common

mixer view (volume faders, pan pots etc), Dexter also offers a wide range of

innovative controls. The user can change the panorama position of many tracks at

the same time by using a special multi-XY pad.

The Lemur has had a huge success around the world and that is reflected by

the fact that well known and established artists like Bjork have been using the Lemur

in live performances as well as in the studio. Many artists also take advantage of the

visuals on the Lemur’s screen to supplement the performance and give the audience

an idea of how the sounds are created (Jazzmutant, 2010).

3.3 Apple iPhone/iPad

In June 2007 Apple Computer releases the iPhone, a cellular phone device that

features a 480x320 multi-touch screen based on the capacitive technology. The

touch screen acts as the main means of interaction between the user and the device,

Y4796421

20

but the user can also take advantage of the accelerometer and the built-in

microphone to provide input, depending on the application. In addition, the bluetooth

and wifi capabilities of the device provide an effective way to networking with

computers or other mobile devices (Apple, 2010). The latest models of the iPhone

also have a built-in camera which can be used as input for applications. One of the

key features that contributed to the iPhone’s commercial success is its software and

hardware expandability. Regarding software, the iPhone SDK, which is freely

available from the Apple web site, enables developers to code iPhone specific

applications, also known as apps. These apps are available to download from a

special internet location called the App Store either for free or for a relatively small

amount of money. There is also a vast variety of hardware accessories that plug into

the iPhone to expand its functionality. Due to the iPhone’s popularity, Apple released

a very similar multi-touch device called the iPod Touch, which is essentially an

iPhone lacking the cellular phone capabilities.

In April 2010 Apple released iPhone’s successor, the iPad, a device that aims

to bridge the gap between a cellular phone and a laptop computer. Externally, the

iPad looks like a scaled up iPhone with a 9.7” multi-touch screen. In fact, the two

devices have more similarities than differences in terms of both hardware and

software. The iPad includes most of the iPhone’s hardware components such as the

multi-touch screen, accelerometer, built-in microphone and wifi connectivity. It also

runs the same operating system as the iPhone, called iOS, which is capable of

running the same apps and uses the same SDK for software development.

Apart from the screen size, the only difference between the two Apple devices

is the available processing power. While Apple has refused to disclose any

information about the iPhone’s CPU, it has been reported that it uses a 600 MHz

ARM processor, which, depending on the device generation and firmware, might run

Y4796421

21

on a speed range from 400 MHz to 667 MHz (Ryan, 2008). The iPad features a

custom Apple A4 1GHz CPU, which significantly boosts the performance and

capabilities of the device when compared to the iPhone (Apple, 2010). This

improvement has enabled developers to deliver more sophisticated apps that can

undertake more tasks than was previously possible. The latest incarnation of the

popular phone, iPhone 4 released in June 2010, incorporates the same A4 CPU as

the iPad (Apple, 2010), fact that is expected to eliminate the performance difference

between the two devices.

Since the launch of the first generation iPhone in 2007, many developers

embraced the idea of using a mobile device for artistic purposes. It is worth noting

that the iPhone has become a rapidly expanding platform where developers around

the world make new musical applications very often, adding new possibilities for

experimentation and musical expression. Apps developed for musicians as well as

audio professionals range vastly in functionality and complexity. A metronome and an

instrument tuner are invaluable tools for most music performers, and there are

numerous apps that bring these tools to the iPhone user. At the time of writing there

are hundreds of music oriented apps in the App Store, and their number keeps

growing. While it would be almost impossible to cover all of them, a basic

categorisation of these apps gives a complete understanding of the capabilities of

this device as well as what functionality is available.

The first category of apps are the ones that can record incoming audio. As

mentioned earlier, the Apple devices have the advantage of being expandable with

software and hardware. Particularly useful hardware accessories to musicians for the

iPhone are external microphones which, paired with an app that has audio recording

capabilities, effectively transforms the iPhone into a portable field recorder. At the

time of writing there is a number of such apps that can take advantage of this

Y4796421

22

external hardware while some of them even provide multi track playback with basic

audio editing features and effects, resembling a DAW.

Along with the iPad, Apple presented a series of accessories for this new

interface. One of the most interesting and eagerly anticipated accessories is the

Camera Connection Kit, which adds USB and SD card expandability for the iPad

(Apple, 2010). As its name suggests, the Camera Connection Kit’s main objective is

to connect a digital camera via USB and download photos on the device. Despite the

fact that Apple does not officially support other types of USB devices with Camera

Connection Kit and the iPad, it has been reported that some USB microphones and

USB audio interfaces work (Rogerson, 2010). This non-documented functionality

paired with an app capable of recording audio can make the iPad a very powerful

and robust solution for recording audio either on-location or in the studio.

Unfortunately, audio recording on the iOS platform is restricted to standard CD

quality (44.1 kHz sample rate, 16 bit resolution), fact that may discourage many

audio professionals who seek better audio quality to use the iPad in a professional

project.

Another category of music-oriented apps available for iPad users are apps that

send data messages to synthesis engines. Such apps do not produce sound on their

own, they just control and communicate with other musical equipment in a very

similar way to how a MIDI controller communicates with hardware synths, computers

or any other devices equipped with MIDI. These apps take advantage of the iPhone’s

networking capabilities to send their messages over a wireless local network. The

connection is established via a server/client configuration where the server is a

computer (could be any other device with wifi connectivity and capable of running the

server app) and the client is the iPad. In most cases communication is bi-directional,

Y4796421

23

meaning that both the server and the client update their values when a parameter

changes on either device.

Recently, there have been made a few additional hardware accessories

available that are attached to the iPad and provide wired MIDI functionality. This

configuration allows the iPhone to be plugged directly into a MIDI setup and instantly

become an active controller.

There are currently two major protocols that musicians use for control

purposes, MIDI and OSC. Both are very popular in the music making community and

have been used in professional as well as amateur projects. The fact that both

protocols can be carried through a network by sending UDP messages makes it

possible to implement MIDI and/or OSC functionality into an iPad app.

There are numerous advantages of using MIDI and OSC apps on a device like

the iPad. First of all, the iPad offers wireless connectivity with the computer, making it

a truly mobile controller for an existing rig. It also brings multi-touch functionality that

was not previously available at this price tag. The potential of future software

expandability is also a strength of the iPhone platform. However, the device has its

own weaknesses compared to similar devices. The wireless connection might slow

down the responsiveness of the system. Also, the integration in a existing setup

might be time-consuming, especially when the OSC protocol is used.

At the time of writing, there are numerous apps available for download that

provide MIDI and OSC functionality. Most available MIDI apps so far tend to simulate

various elements of a hardware MIDI controller, such as the keyboard, knobs and

sliders. Other MIDI apps are dedicated to controlling DAWs by using the Mackie

Control Universal (MCU) MIDI mapping.

There are also a few OSC apps. Most of them use the same control concepts

as the MIDI apps, providing various UI elements that resemble hardware

Y4796421

24

components like knobs and sliders. However, there are a few apps that do not follow

the same functionality principles, enabling the user to modify various parameters in a

non-conventional manner. These apps communicate with the computer through a

wireless network, making the iPad one of the first wireless devices widely available to

musicians.

One of the most popular apps for the iPhone platform is hexler’s TouchOSC

app. Instead of utilising an UI with a fixed number of controls, TouchOSC enables the

user to choose from a list of available UI elements the ones that are needed and

assemble a customised user interface. The range of available UI elements consists

of pushbutton, toggle, slider, text, XY pad and accelerometer as well as user

feedback objects such as title and textview. (hexler, 2010) As with the Lemur, the

user creates the customised UI on an editor application on the computer and then

uploads it to the device.

Features like robustness, stability and active community have made

TouchOSC very popular among iPhone users. People around the world create new

UIs that are customised to work with a particular software application like NI Traktor

Pro and Ableton Live and share them so other users get the functionality instantly

(Osculator Forum, 2010).

One last category of music apps consists of the apps that produce offer multi-

touch control while at the same time produce sound on the iPad, making the iPad an

individual and autonomous instrument. Unlike control apps like TouchOSC, such

apps rely exclusively on the iPad’s CPU to produce the sound. Some of the available

apps tend to emulate hardware electronic instruments like synthesisers, drum

machines, samplers and sequencers. Other apps provide more sophisticated UIs that

do not resemble any hardware instrument. There is a plethora of such apps available

while some of them are made by established software and hardware manufacturers,

Y4796421

25

reflecting that the iPhone/iPad platform despite its comparatively low cost is capable

of delivering very goo results.

3.4 Subcycle

Subcycle is a combination of pre-existing hardware and software technologies rather

than an individual system and is a perfect example of DIY multi-touch music

interface. It also proves that multi-touch technology is not limited to laboratories and

sophisticated products, but it can be successfully implemented by individuals for use

in a particular project and for a particular purpose.

Because of the fact that the project is still under development, there is lack of

detailed documentation while the only available information come from the

developer’s blog. In addition, as the project progresses to higher build versions the

developer presents different and often contradicting interaction concepts, which

makes an accurate description of the system a difficult task.

Even though the project uses an optical based technology, the details of the

multi-touch sensing setup are not yet available. The core of the project is a computer

than runs a number of different programs, each one having its own role in the whole

interactive outcome. First of all, the computer provides its DSP power to run the

audio synthesis engine which in this project is Logic Pro hosting Native Instruments’

Massive virtual instrument plugin. The instrument aims to provide control over

various parameters of this Wave Scanning synthesis engine. (Native Instruments,

2010) Equally important, the computer hosts the visual generation software which in

this case is the open source Processing. The visuals are fed to the screen through a

digital projector. The last required piece of software is a utility made in MaxMSP that

converts and maps MIDI messages to OSC messages. This last bit of software is

essential in order to interconnect Logic Pro with Processing (Bannister, 2010).

Y4796421

26

It is important to note that no audio analysis takes place. The visuals that are

presented on the screen are composed in Processing and modulated by various

parameters that derive from Logic Pro. For example, the LFO that is used in a

synthesiser for sound generation is also used to modulate various shapes and

graphics in Processing. Similarly, the grain time of a granulation effect will also

control the respective graphic representation.

4. The Music Sphere

4.1 The Concept

The Music Sphere project is a multi-touch interface for creating and processing

music. Unlike many similar multi-touch interfaces that are developed for musical

purposes, the Music Sphere’s user interface does not utilise the conventional

elements like sliders, buttons and knobs. Instead, it uses a new concept for creating

and processing sounds and music, a 3D object.

The Music Sphere is made with a very specific idea in mind, to serve a

particular musical purpose. As Bown suggests (2009:190), these instruments are

described as composed instruments:

In electronic and digital instruments, controllers, generators and the sets of

mappings that link them are independent units that can be combined with great

structural and functional flexibility. There is no longer a fixed and direct

correspondence between the interface and sound production mechanism. This

leads to several distinct (but not mutually exclusive) senses in which an instrument

can be composed: a software instrument can combine a control interface with a

generative algorithm, such that explicit temporal structure can be predefined; the

instrument itself can be composed, in the sense that modules (controllers, mappings

and generators and their subcomponents) are arranged with specific musical intent;

in both these cases, the software system embodies some aspect of the maker’s

Y4796421

27

musical intent, and acts (like a score) as a vehicle for sharing musical ideas across

culture. (Bown, 2009:190)

The interface operates by performing gestures on the sphere that alter both

the sonic outcome and the sphere’s geometrical properties, resulting in a

combination of audio/visual real time manipulation. These alterations in both the

audio and the visual aspect have to be perceptually similar and complementary in

order to achieve a great level of interactivity and also help in crucial aspects of a live

performance such as expressiveness and precision. In addition, matching audio and

visual effects can enable non-musicians or people that have no background

knowledge on digital audio synthesis and processing to get an idea about how sound

is made on a digital system.

The audio aspect in this particular project concentrates in manipulating

parameters in two different sound generators, or scenes. The user can select the

desired scene by tapping the corresponding coloured spheres at the bottom right

corner of the screen. As soon as a scene is selected, the background colour changes

to signify the event of scene change and provide feedback to the user. The first

scene deals exclusively with pre-recorded material while the second scene controls

an FM synthesiser. This selection of sound sources might seem somehow limiting in

terms of possible sounds that can be created, however, the same operational

principles can be easily applied to different kinds of synthesis techniques, expanding

the sonic outcome of the interface.

Each scene consists of a sphere placed in the middle of the screen, the

sphere in focus, and four more spheres revolving around the central sphere, each

having its own different colour. The sound generator and processing algorithms are

divided into four smaller individual parts, for example LFO, distortion, filter and

Y4796421

28

modulation effect while each of these parts is represented by a revolving sphere.

Tapping on a revolving sphere makes that sphere the sphere in focus and the user

can then access various parameters associated with that sphere/sound generator

part by performing gestures on the screen.

4.2 Scene One - Pre-Recorded Material

The first sphere controls various aspects of the audio file’s playback. The playback of

the audio files is accomplished by using the elasticx~ MaxMSP external object

developed by Joe Jarlett and Simon Adcock, mainly because of its high quality and

low latency pitch shifting algorithm. (Jarlett, 2008) The pitch parameter is easily

accessible by the simple two-finger pinch gesture. The sphere deformation effect that

is used as visual feedback is egg for pitch shifting to a higher note and disk for pitch-

shifting to lower note.

Gestures with more than two fingers will control more complex audio

processes like the slicer processor. The slicer processor is a MaxMSP patch that was

developed for use in this particular project. The processor randomly choses a small

part of an audio file and plays it repeatedly until the user ends the gesture. As soon

as the gesture starts, an algorithm detects the areas with the . These values are set

as the looping points for the elasticx~ object. The corresponding visual effect for this

audio process is a combination of the extrude and sculpt effects.

The second sphere responds only to more than two finger gestures and is

dedicated to an audio process that utilises the resonators~ MaxMSP external object

made and maintained by CNMAT. This objects is essentially a resonant filter. (Freed,

1999) The process includes four instances of the resonators~ object and a simple

reverb algorithm. The numeric values that derive from the gesture analysis are

scaled and mapped to the resonator~ object inputs as well as the reverb Size, Decay

Y4796421

29

Time, Damping and Diffusion parameters. By performing a multi-finger rotation

gesture the user can change the pitch of the resonator~ object and produce a range

of interesting sounds.

The visual feedback for this audio process is a combination of many sphere

alterations. As the resonance of the resonator~ object increases, the density and

sculpt (set to form a star with seven points) effects increase as well.

The third sphere enables the user to control a flanger effect and it works in a slightly

different way compared to the other ones, is as it provides visual feedback even

without the user’s interaction. The sphere distorts and gradually changes shape

reflecting the LFO modulation. The deformation that is used for this purpose is

reshape because of it looked very appropriate for this purpose. When the LFO

reaches its minimum value the sphere remains unaltered while at the maximum value

the sphere is fully deformed. The sphere responds to multi-touch gestures by

becoming bigger in size as the wet/dry ratio increases. By performing the five finger

rotation gesture the user can change the modulation rate to pre-determined values.

These pre-determined values are calculated based on the tempo of the composition,

so the modulation rate will always be on time. Spreading the fingers away from each

other will increase the feedback of the flanger along with the depth and the wet/dry

output ratio.

The fourth sphere controls a combination of distortion effects. The combination

of distortion effects consists of an audio quality degrader and an FFT noise gate.

Touching the sphere with five fingers will cause reduction in the playback sampling

rate and bit depth depending on the spread property of the gesture. Moreover,

spread will also control the threshold of the FFT noise gate. The visual feedback for

this effect is a combination of density, completeness and extrusion effects that it

reveals certain parts of the sphere.

Y4796421

30

4.3 Scene Two - FM Synthesiser

The second scene controls an FM synthesiser. Similarly with the first scene, the user

taps on the coloured spheres to select which part of the available synthesis and

audio processing parameters will be modified by the gestures. The FM synthesis

implementation is rather basic, but adequate to demonstrate the operational

principle.

The first sphere provides the synthesiser with pitch information to produce sounds.

The pitch is provided by a multi-touch pinch gesture. When the gesture spreads the

synthesiser produces a higher pitched note while when the gesture shrinks the

synthesiser produces a lower pitched note. This is accomplished by mapping the

spread property of the gesture to MIDI values. The interface also supports recording

and playback of the MIDI sequences by performing simple long press gestures

anywhere on the screen. Finding appropriate visual representation for this

functionality was a very difficult task and this detail of the project could be

significantly improved.

The second sphere controls four distortion units and a XY pad which

determines the volume of each distortion unit. Each corner of the XY pad represents

a different distortion technique (overdrive and phase distortion variations). The XY

pad is controlled by calculating the average coordinates of all touches and extracting

a single touch equivalent. The visuals are a combination of completeness, reshape,

sculpt and density effects. These effects result in a deformation that is particularly

suited for distortion effects, as they make the sphere look grainy and intermittent.

The third sphere represents a low pass filter. The user controls the resonance

of the filter with multi-touch pinch gestures while multi-touch rotation gestures change

the cutoff frequency. The visual feedback for this sphere is a combination of egg,

Y4796421

31

density and extrude effects. Unfortunately, the resulting deformation is not very suited

for the low pass filter effect but the limited amount of available choices made this

choice unavoidable.

The last sphere controls the synthesiser's LFO in exactly the same fashion as

the flanger LFO found in the first scene. The user controls the rate by multi-touch

rotation while other parameters are controlled with the multi-touch pinch.

4.4 Other Synthesis Methods

The development process also included a Waveshaping synthesis engine.

Waveshaping synthesis uses a sine wave as the primitive sound and distorts it in

order to create a new shape and, consequently, a new sound. The waveshaper is a

fundamental part of a waveshaping synthesis engine which distorts the signal

depending on the signal’s amplitude. The amount of distortion depends on the

waveshaper’s transfer function which usually is a small audio file that serves as the

distortion index. In the case where the distortion index is linear, no distortion is

introduced and the sound passes through the waveshaper unaltered. (Dodge,

1997:140)

The development of the Music Sphere interface explored the idea of

manipulating the distortion index in real-time instead of using an audio file. This

functionality would involve drawing the distortion index using touches on the screen.

By moving the fingers the user would also be able to change the distortion index and

create different sonic timbres instantly. A MaxMSP implementation of this idea was

achieved by extracting the coordinates of each touch on the screen, scaling the

numeric values to the appropriate value range and mapping them into a function

object.

Y4796421

32

Even though this approach did work and the contents of the distortion index

were successfully updated in real time according to the user’s input, the calculation of

the distortion index proved to be very CPU intensive, resulting in serious issues and

instability of the system. More specifically, the MaxMSP DSP monitor reported 100%

CPU usage as soon as the distortion index was modified, while audible artefacts

such as clicks, drop outs and latency appeared.

5. Development Process

The project is a combination of a computer running the MaxMSP programming

environment and an Apple iPad serving as the multi-touch controller for the MaxMSP

patch. These two parts communicate via a USB cable, exchanging Open Sound

Control (OSC) messages. The audio synthesis and processing is done in MaxMSP

mainly because of the ease of use and the availability of a wide range of objects. The

original idea was to code the audio synthesis engine and the effect algorithms to run

natively on the iPad but, even though it is possible, it would have taken considerably

more time to develop. However, using an object oriented programming environment

like MaxMSP or Pure Data is a very good alternative and serves well for prototyping

and demonstrating the concept.

5.1 Program Structure

The user provides input on the iPad by touching the screen. Essential algorithms for

touch recognition, sphere selection and basic gesture recognition are performed on

the iPad’s CPU. These algorithms can extract the exact location (XY coordinates) of

each touch on the screen as well as various other parameters that correspond to

multi-touch gestures such as velocity (the speed at which a multi-touch gesture is

performed) and spread (how close/far away are the touches). The calculated data

Y4796421

33

that derive from the touches are scaled to an appropriate the range, formatted into

the OSC messaging style and then sent to MaxMSP through the USB cable.

The data received by MaxMSP is rescaled and mapped to many synthesis and effect

controls simultaneously. For example, the spread property of a multi-touch gesture

could be mapped to a filter’s resonance parameter as well as another effect’s wet/dry

control. Of course, some gestures are exclusive to some processes. In some cases,

additional and more complex gesture recognition algorithms are performed in

MaxMSP in order to provide more control and flexibility over the desired parameters.

For instance, the user could control the filter’s resonance with the spread gesture

property and, while still touching the screen, start a multi-touch rotation gesture which

will affect the filter’s cutoff frequency, leaving the resonance set to its last state.

The MaxMSP patch is made in a modular manner in order to simplify the

development process and allow future expandability. The signal path consists of

individual processing blocks of effects that are placed one after the other.

As the audio processing is performed, certain parameters that were affected

by the user input are re-scaled to an appropriate value range, formatted into the OSC

style and sent back to the iPad through the USB cable. The incoming OSC

messages trigger certain methods in the code which alter and deform the spheres in

ways that could describe the audio precesses. For example, pitch-shifting an audio

file could be represented on the screen with stretching the sphere.

The user interface is made by using the OpenGL ES framework. OpenGL ES is

OpenGL’s equivalent for mobile devices such as the iPhone and Android. Even

though OpenGL ES lacks of many features that are present in the full OpenGL

framework, the available capabilities are sufficient for basic shape drawings. Open

source developers have also been contributing to expand the standard OpenGL ES

Y4796421

34

library by either porting existing OpenGL code to mobile platforms or writing new

code to suit the specific needs of a project.

Figure 1: Data flow of the Music Sphere project

5.2 The C++ Sphere Method

Typically, drawing a sphere in an OpenGL project is done by calling the gluSphere

function. However, OpenGL ES does not include this functionality, meaning that a

separate function which will be responsible for calculating the sphere geometry will

have to be used. In this particular project, an external to OpenGL ES function was

used in order to draw the spheres. The source code for the sphere is written by Jeff

LaMarche and published on his personal blog while a large part of it was modified in

order to be successfully used for this project (Appendix 8.1). This method calculates

a approximation of a sphere based on a mathematical procedure. The level of

approximation depends on the amount of stacks and slices of the sphere. As the

developer explains:

Y4796421

35

These basically just define how many vertices there are latitudinally and

longitudinally. Think of an orange. If you slice it with a knife vertically, those are

"slices". If we cut it horizontally, those are "stacks". (LaMarche, 2009)

A high stacks and slices number will result in a smoother and rounder circle but will

also require more CPU power. A value of 50 stacks and slices results in a smooth

circle with no obvious dents while the required calculations are kept within the iPad’s

CPU threshold.

In order to create the effects on the sphere, various variables were inserted

into the code. By trial and error, all the spots in the code where additional variables

could be inserted were identified. As these variables change values, they affect the

procedural drawing of the sphere resulting in various deformations of the sphere.

These basic deformations can also be combined in order to make more complex

effects that seem more suitable for a particular audio process.

There are eight basic effects that derive from arithmetic manipulation in the

sphere method. These primitive effects were named after the result they have on the

sphere and are: scale, density, completeness, egg, disk, extrude, reshape and sculpt.

Some of them are self-explanatory while others affect the sphere in a more

unpredictable way. Figure 2 depicts the these effects. Some of these effects were

used for direct manipulation for the sounds while others were used as user feedback

only. All of the effects were equally important to achieve the interactivity and

responsiveness required for the project.

There is a number of very simple effects like the scale effect which even though are

not particularly impressive on their own, they are heavily used in combination with

other effects. The scale effect just scales the sphere, but when used in combination

Y4796421

36

with another effect it gives the impression of fading in the second effect. The egg and

disk effects affect how symmetrical is the sphere towards an axis, stretching the

sphere to look like an egg, or shrinking it to resemble a disk. Similarly to the egg

effect, the extrude effect stretches the sphere but, unlike the egg effect, it also

creates ripples on the surface of the stretched sphere.

The density effect affects the sphere in a very interesting way as it only draws

horizontal slices of the sphere rather than the whole sphere surface. The

completeness effect, as its name suggests, allows only a portion of the sphere to be

drawn. This effect was used to provide feedback over the pre-recorded loop playback

percentage, allowing the user to instantly figure out how much of the loop has been

played.

The reshape and sculpt effects are more sophisticated effects compared to the

other effects, as their results can be varying and unpredictable. These effects depend

largely on the value range of the input and exceeding these value ranges might

cause instability or even crash. The reshape effects stretches or shrinks certain parts

of the sphere surface and it results is weird and unusual shapes. The sculpt effect

gradually transforms the sphere into other shapes like a star or a hexagon. Similarly

to the reshape effect, the resulting shape depends on the range of the input. There is

a wide range of resulting shapes which in combination with other effects can produce

very interesting deformations on the sphere.

Y4796421

37

Figure 2: the eight basic effects (a) scale, (b) completeness, (c) density, (d) egg, (e) disk, (f) extrude, (g) reshape, (h) sculpt to hexagon, (i) sculpt to a stat with seven points, (j) the original unaltered sphere

5.3 Gestures

Apple has provided support for a few common multi-touch gestures in the iPhone

SDK, making gesture recognition a fairly easy task (Appendix 8.2). The standard

gesture recognition system found in the iPhone SDK supports six different gestures:

tap, swipe, pinch, rotate, pan and long press. (Apple, 2010:49) While all these

gestures are heavily used throughout iOS and are guaranteed to work efficiently for

the most tasks, there is also the ability to implement additional user-definable

gestures.

Y4796421

38

The tap gesture similar to a “mouse click” in the iOS. The users tap and

instantly release the screen to push buttons on the screen and more. The tap gesture

also has properties where the it can be specified how many sequential taps are

required (for example double-tap), or how many touches are required (for example

two-finger tap). A special kind of tap is the long press gesture that is performed by

tapping and holding the finger for a certain amount of time. Swipe is a gesture that is

usually mapped to the “scroll” function of a computer mouse. The user touches the

screen and quickly moves the finger towards one direction to scroll there. Similar to

the tap gesture, swipe can be specified to respond only for swipes on an axis (up-

down or left-right) or towards a pre-determined direction (only for right swipes).

The pinch gesture is a multi-touch gesture which requires two touches on the screen

and is performed by moving the two fingers away from each other, both following a

straight line. Many applications use the pinch gesture for zooming and scaling

purposes. The rotation gesture is used, as expected, for rotating images and other

elements in iPad apps. Is performed by touching the screen with two fingers and

moving them in a circular manner. Finally, the pan gesture is performed by touching

the screen and simply moving the finger around. The pan gesture can be also

constructed to respond for more touches. For the last two described gestures,

rotation and pan, there is also the option to access the gesture’s velocity, the speed

in which the gesture is performed.

Figure 2: Common gestures in the iPhone SDK: (a) pinch gesture outwards, (b) pinch gesture inwards and (c) rotation gesture

Y4796421

39

(a) (b) (c)

Apart from the gesture recognition bundled with the Apple SDK, there was the

need to implement custom gesture recognisers specifically for this project. These

additional gesture recognisers work in a very similar fashion with the Apple ones,

providing a consistent way of controlling the interface (Appendix 8.3 and Appendix

8.4). The need for extra gesture recognition derives from the fact that Apple only

supports gesture recognition for gestures that use two fingers at most. Even though

such simple gestures are adequate for controlling the iOS and the vast majority of the

iPad applications available to date, the need for a more sophisticated gesture

recognition system was apparent in this project.

The custom gestures that can be performed are multi-finger pinch and multi-

finger rotation. These gesture recognition algorithms present a significant advantage

compared to the Apple system: the user can make a gesture, for instance a multi-

finger pinch, and without releasing the fingers from the screen start another gesture,

for example a multi-finger rotation. This feature plays a very important role in the

spontaneity and the expressiveness of a performance.

Figure 3: Custom Gestures (a) multi-touch rotation and (b) multi-touch pinch

5.4 Communication Protocol

As described earlier, the networking features of the iPad allow both the MIDI and

OSC protocols to be integrated into iPad apps. Dealing with music oriented tasks has

Y4796421

40

(a) (b)

been the primary objective in the creation of both protocols, and their success is

proven by the fact that both have been embraced by the music making community,

allowing musicians to interconnect many pieces of hardware and software. In

addition, features like simplicity, robustness and unnoticeable latency were

appreciated by non-musicians, bringing MIDI and OSC support to many different

types of creative applications like robotics, VJing and more.

MIDI, defined in 1982, is a communication protocol that allows MIDI

compatible devices to control and synchronise with each other by sending data.

Since 1982, countless hardware synthesisers, effect units, drum machines and audio

interfaces are equipped with MIDI DIN connectivity, achieving instant interoperability.

The most common MIDI message is a 7-bit message which has a range of 128 steps

(integer values ranging from 0 to 127). These messages can represent Velocity,

Volume, Pan and numerous other parameters. Other supported MIDI messages are

14-bit messages which have a range of 16383 discrete steps (often used as integer

values ranging from -8192 to 8191 to represent pitch bend information), Note On,

Note Off messages and more.

Partly because of its age, the MIDI protocol is a popular and well established

method of interconnecting music equipment. The MIDI protocol is maintained by the

International Standards Association [ref], fact that has numerous advantages to MIDI

users like standardisation of hardware-specific MIDI mappings and plug-and-play

functionality of any MIDI equipped device.

OSC is a protocol developed and maintained by CNMAT. Its purpose is to

allow communication between synthesisers, computers and other multimedia

devices by taking advantage of the today’s advanced networking technology. One of

the most prominent advantages of OSC over MIDI is that OSC can deal with a wide

range of data types such as integers, floating point numbers and strings instead of

Y4796421

41

just integer values. (Freed, 2008) Recent OSC implementations also include support

for boolean values, RGBA colour values and more. While MIDI still offers the

possibility to carry other data types (float numbers etc) by using System Exclusive

messages, this approach has several disadvantages like difficulty to set up. In

addition, MIDI has pre-determined binary message fields such as note, velocity, pitch

etc while OSC allows the user to easily create customised message structures that

suit the needs of a particular application. Also, this message structure is in a human

readable URL-style, which eliminates the complexity of binary data and enables

users to easily understand and re-configure an OSC system. Finally, OSC supports a

high timing resolution timestamp (picosecond) that allows OSC messages and

performances to be scheduled, recorded and re-produced. While the same

functionality is achievable with MIDI, the resolution is several times lower

(millisecond) and that can result in inaccuracies in capturing and re-producing MIDI

performances. There is also the possibility to implement OSC communication over

the internet, fact that can help in collaborative musical projects. However, such a

scenario would most probably be vulnerable to latency and instability.

Despite its many strengths, the OSC protocol has its own weaknesses that

have mostly to do with the fact that it is a relatively new technology. First of all, major

music hardware manufacturers have been slow to implemented support for OSC,

narrowing down the practical capabilities of this protocol. Moreover, and due to the

fact that OSC is completely customisable, there is no standardisation of control

messages as opposed to MIDI, leading in interoperability issues.

Both MIDI and OSC are Hardware Transport Independent, meaning that no specific

hardware is required in order to achieve transmission and receipt of messages. Even

though the MIDI 1.0 Specification includes the MIDI DIN transport specification,

realising MIDI through this transport is optional as it is possible to implement the MIDI

Y4796421

42

protocol through other transports like USB, Firewire and Ethernet. The transport

determines the data rate of the system and, consequently, plays a very important role

in the system’s stability and responsiveness. Since OSC was introduced there has

been the false impression that OSC is a faster protocol than MIDI. This is only true

when MIDI is implemented through a low data rate transfer such as the original MIDI

DIN. In the case where another transfer like Ethernet is used, the MIDI protocol does

not show any difference to the OSC protocol in terms of speed (MIDI Manufacturers

Association, 2008).

The Music Sphere project implements the OSC protocol instead of MIDI for a

number of different reasons. First of all, there is existing OSC open source code for

the iPhone platform (rayculter, 2009) which is reliable and works efficiently, fact that

made the development process fast and easy. Moreover, the typical 0-127 value

range provided by MIDI could be proved insufficient in terms of resolution and ease

of dealing with the values. The customisability of the OSC protocol was also a very

significant reason for choosing OSC, as the code is human readable instead of

binary numbers, fact that helped for a better understanding of the programming

structure that had to be followed. Finally, future development will benefit from the

OSC support, allowing expandability with RGBA values and character strings.

During initial development process, many tests were performed in order to

make sure that the OSC implementation works and its speed is adequate for the

project’s needs. Initially, OSC communication was implemented over a wireless

network, taking advantage of iPad’s wifi connectivity. Unfortunately, this method

resulted into very poor results in terms of responsiveness and accuracy, especially

when values changed rapidly. This drawback is caused by the iPad’s poor wifi

hardware, which presents flaws and malfunctions even in simple tasks like browsing

Y4796421

43

the internet. The solution to this problem was to use the wired USB transfer for

exchanging OSC messages instead. The responsiveness of the system was radically

improved up to an acceptable level, even though in some cases issues like latency

and sluggishness still are apparent.

6. Conclusions

There are numerous advantages that the rise of multi-touch technology has brought

to HCI, and that is demonstrated by the fact that more and more companies and

organisations research and incorporate multi-touch devices. Compared to traditional

input methods like mouse and keyboard, multi-touch interaction enables the users to

control the computer in a more comfortable and natural way, which increases

productivity. This technological development has not left the music community

unaffected, as many research projects (reacTable*) as well as commercial products

(Lemur) have incorporated multi-touch technology and successfully reached their

goal which is to create and manipulate sounds and music in real time.

6.1 Evaluation

The project achieves its primary objective which was to create an interactive multi-

touch musical interface. It also accomplishes, in different levels of success, many of

its secondary objectives such as gesture recognition and mapping gesture values to

audio processes. Even though the customised gesture recognition system presents

flaws, it is a very good first step into understanding the underlaying principles and

also developing gesture recognisers for future projects. Due to the lack of more multi-

touch gesture recognisers the mapping is very basic, as in some cases a single

gesture maps to many parameters at the same time, which results in limited control.

Y4796421

44

! In many cases the deformations made on the sphere are aesthetically similar

to the alterations made to the sound. However, not all sphere deformations are

suitable for the corresponding audio processes. This is because the range of

available sphere graphics was limiting, while effect combinations in many cases

would result into very similar visuals, which made it difficult to achieve individual and

instantly distinguishable deformations for each sphere/audio process.

Finally, the steep learning curve of programming affected this project greatly,

due to lack of previous programming experience. Unfortunately, a lot of time and

effort were spent on matters that were not directly related to the musical or artistic

aspect of the project. For instance, managing to stream OSC data through USB,

drawing 3D objects in OpenGL and fine-tuning the settings in XCode were crucial

parts of the development process and took a considerable amount of time to

implement, without contributing directly to the aesthetic and musical outcome.

However, this project was an excellent opportunity to learn more about programming

and develop programming skills, which will be invaluable in the future.

6.2 Further Development

The Music Sphere project concentrates on controlling sounds and music by means of

a 3D shape instead of other common elements like virtual keyboards, sliders and

buttons. It successfully achieves its goal in terms of compositional freedom and

expression. However, there is the potential of future improvement and development.

Many different aspects of the project could undertake improvement, while the

modular structure of the program allows the addition of new features.

One of the first things that could improve the project would be to translate the

MaxMSP patch into code that runs natively on the iPad’s CPU in order to make Music

Sphere a complete and independent instrument. Not only that would eliminate the

Y4796421

45

need for a computer running MaxMSP, but it also would mean that the instrument

would not require OSC support and the USB cable, making the interface truly

autonomous. Another area of improvement would be the graphics rendering. An anti-

aliasing filter could considerably improve the edgy graphics and present smoother

image. At the time of development, anti-aliasing for the iOS 3 was not supported by

Apple and, consequently, was not implemented. Even though there are ways to apply

anti-aliasing filters, the development process would take much more time without

contributing enough to the final result.

Probably the most limiting area of this project is the lack of many different

sound generators. Even though the purpose of this project is to research into modern

music multi-touch interface and demonstrate a new operational concept, the

available sound generators are not enough to make the project a complete

instrument. In the future, more synthesis engines could be incorporated in the project

to produce a wider palette of sounds that can be created with the interface. The most

prominent feature would be an AM synthesiser which would also be very easy to

incorporate, due to the operational similarities with FM synthesis. Additional sound

processors like modulation effects (tremolo, chorus, auto-pan) could be a nice

complementary addition to the current flanger effect while spatial processors like

reverb and delay could provide more options to the user. More sophisticated audio

processes like granulation could also be added in order to expand the sonic

possibilities of the interface.

Apart from sound generation using common synthesis techniques, the

interface could also utilise the built-in microphone to either record audio files and load

them in the pre-recorded material scene or process the incoming audio in real time.

This feature would greatly improve the mobility of the device, making it a portable

instrument capable of creating and manipulating sounds instantly.

Y4796421

46

There is also a lot of room for improvement in the gestural interactivity of the project.

The Music Sphere app currently supports up to five fingers even though the iPad

platform supports much more. The reason for this limitation is that the 9.7” screen is

just big enough to accommodate a hand with the fingers fully spread. However,

support for more touches could be possible to implement in the case that gesture

recognisers are made that can respond to six or more touches. Alternatively, a

system where multiple gestures are recognised simultaneously could also expand

the controllability of the system. For example, the user could perform a five finger

rotation gesture with one hand and a pinch gesture with the other in order to control

multiple parameters at the same time.

7 References

Benko, H., Wilson, A.D. & Balakrishnan, R. (2008) ‘Sphere: multi-touch interactions on a spherical display’ (Proceedings of the 21st annual ACM symposium on User interface software and technology). Monterey, CA, USA: ACM

Bown, O., Eldridge, A. & Mccormack, J. (2009) ‘Understanding interaction in contemporary digital music: From instruments to behavioural objects’ (Organised Sound, vol.14 no.2) Bredies, K. et al. (2008) ‘The multi-touch SoundScape renderer’ (Proceedings of the working conference on Advanced visual interfaces). Napoli, Italy: ACM

Couturier, J. & Arfib, D. (2003) ‘Pointing fingers: using multiple direct interactions with visual objects to perform music’ (Proceedings of the 2003 conference on New interfaces for musical expression). Montreal, Quebec, Canada: National University of Singapore[, pp. 184-187.

Dodge, C. & Jerse, T. (1985/R1997) Computer Music: Synthesis, Composition and Performance. Boston, Schirmer

Essl, G. & Rohs, M. (2009) ‘Interactivity for mobile music-making’ (Organised Sound, vol.2 no14), 197-207.

Freed, A., Dudas, R. & Jehan, T. (1999) Musical Applications of New Filter Extensions to Max/MSP.

Y4796421

47

Geiger, G. (2006) ‘Using the touch screen as a controller for portable computer music instruments’ (Proceedings of the 2006 conference on New interfaces for musical expression). Paris, France: IRCAM — Centre Pompidou, pp. 61-64.

Han, J.Y. (2005) ‘Low-cost multi-touch sensing through frustrated total internal reflection’ (Proceedings of the 18th annual ACM symposium on User interface software and technology). Seattle, WA, USA: ACM, pp. 115-118.

Hodges, S., Izadi, S., Butler, A., Rrustemi, A., and Buxton, B (2007) ‘ThinSight: versatile multi-touch sensing for thin form-factor displays’. (Proceedings of the 20th annual ACM symposium on User interface software and technology). Newport, Rhode Island, USA: ACM, pp. 259-268.

Izadi, S., Hodges, S., Butler, A., Rrustemi, A., and Buxton, B. (2007) ‘ThinSight: integrated optical multi-touch sensing through thin form-factor displays’ (Proceedings of the 2007 workshop on Emerging displays technologies: images and beyond: the future of displays and interacton). San Diego, California: ACM, p. 6.

Jo, K. (2008) ‘DrawSound: a drawing instrument for sound performance’. (Proceedings of the 2nd international conference on Tangible and embedded interaction). Bonn, Germany: ACM, pp. 59-62.

Jordà, S. (2010) ‘The reactable: tangible and tabletop music performance’. (Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems). Atlanta, Georgia, USA: ACM, pp. 2989-2994.

Jordà, S. & Alonso, M. (2006) ‘Mary had a little scoreTable* or the reacTable* goes melodic’. (In Proceedings of the 2006 conference on New interfaces for musical expression). Paris, France: IRCAM — Centre Pompidou, pp. 208-211.

Jordà, S., Geiger, G., Alonso, M., and Kaltenbrunner, M. (2007) ‘The reacTable: exploring the synergy between live music performance and tabletop tangible interface’s’. (Proceedings of the 1st international conference on Tangible and embedded interaction). Baton Rouge, Louisiana: ACM, pp. 139-146.

Jordà, S., Kaltenbrunner, M., Geiger, G., and Alonso, M. (2006) ‘The reacTable: a tangible tabletop musical instrument and collaborative workbench’. (ACM SIGGRAPH 2006 Sketches). Boston, Massachusetts: ACM, p. 91.

Kaltenbrunner, M., Geiger, G. & Jordà, S. (2004) ‘Dynamic patches for live musical performance’. (Proceedings of the 2004 conference on New interfaces for musical expression). Hamamatsu, Shizuoka, Japan: National University of Singapore, pp. 19-22.

Kaltenbrunner, M., Jordà, S., Geiger, G. & Alonso, M. (2006) ‘The reacTable*: A Collaborative Musical Instrument’. (Enabling Technologies, IEEE International Workshops on). Los Alamitos, CA, USA: IEEE Computer Society, pp. 406-411.

Kaltenbrunner, M. (2009) ‘reacTIVision and TUIO: a tangible tabletop toolkit’. (Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces). Banff, Alberta, Canada: ACM, pp. 9-16.

Y4796421

48

Matsushita, N. & Rekimoto, J. (1997)’ HoloWall: designing a finger, hand, body, and object sensitive wall’. (Proceedings of the 10th annual ACM symposium on User interface software and technology). Banff, Alberta, Canada: ACM, pp. 209-210.

Maximo, A., Saba, M.P. & Velho, L. (2009) ‘collecTable: a natural interface for music collections’. (SIGGRAPH '09: Posters). New Orleans, Louisiana: ACM, pp. 1-1.

NUI Group Authors (2009) Multi-Touch Technologies. New York: NUI Group

Patten, J., Recht, B. & Ishii, H. (2002) ‘Audiopad: a tag-based interface for musical performance’. (Proceedings of the 2002 conference on New interfaces for musical expression). Dublin, Ireland: National University of Singapore, pp. 1-6.

Schöning J., Brandl P., Daiber F., Echtler F., Hilliges O., Hook J, Löchtefeld M, Motamedi N., Muller L., Olivier P, Roth T, & on Zadow U. (2008) ‘Multi-Touch Surfaces: A Technical Guide’ (Technical Report TUM- I0833). Munich, Technical Reports of the Technical University.

Shaer, O. & Hornecker, E. (2010) ‘Tangible User Interfaces: Past, Present, and Future Directions. Found’. Trends Hum.-Comput. Interact., 3(1–2), 1-137.

Wilson, A.D. (2004) ‘TouchLight: an imaging touch screen and display for gesture-based interaction’. (Proceedings of the 6th international conference on Multimodal interfaces). State College, PA, USA: ACM, pp. 69-76.

Wright, M., Freed, A. & Momeni, A. (2003) ‘OpenSound Control: state of the art 2003’. (Proceedings of the 2003 conference on New interfaces for musical expression). Montreal, Quebec, Canada: National University of Singapore, pp. 153-160.

Internet Sources

Apple. iPad Programming Guide. Internet WWW page, at URL: <http://

developer.apple.com/iphone/library/documentation/General/Conceptual/

iPadProgrammingGuide/GestureSupport/GestureSupport.html> (version current at

September 2010).

Apple. iPhone 4 Technical Specifications. Internet WWW page, at URL: <http://

www.apple.com/iphone/specs.html> (Accessed 10 August 2010).

Bannister, Christian. Subcycle Labs. Internet WWW page, at URL: <http://

www.subcycle.org/> (Accessed 18 July 2010).

Y4796421

49

Freed, Adrian. The Open Sound Control 1.1 Encoding Specification. Internet WWW

page, at URL: <http://cnmat.berkeley.edu/user/adrian_freed/blog/2008/10/06/

open_sound_control_1_1_specification> (last updated 06 October 2008).

hexler. TouchOSC. Internet WWW page, at URL: <http://hexler.net/software/

touchosc> (26 July 2010).

Jarlett, Joe & Adcock, SImon. elastic~ in more detail Internet WWW page, at URL:

<http://www.elasticmax.co.uk/elastic~details.php> (Accessed January 2010).

Jazzmutant. Lemur Hardware Features. Internet WWW page, at URL: <http://

www.jazzmutant.com/lemur_features_hard.php> (Accessed August 2010).

Jazzmutant. Hands on Control. Internet WWW page, at URL: <http://

www.jazzmutant.com/dexter_intuition.php> (Accessed August 2010).

Jazzmutant. Damian Taylor (Björk Volta Tour). Internet WWW page, at URL: <http://

www.jazzmutant.com/artists_bjork.php> (Accessed August 2010).

Khronos Group. OpenGL ES - The Standard for Embedded Accelerated 3D

Graphics. Internet WWW page, at URL: <http://www.khronos.org/opengles/>

(Accessed June 2010).

LaMarche, Jeff. Procedural Spheres in OpenGL ES. Internet WWW page, at URL:

<http://iphonedevelopment.blogspot.com/2009/05/procedural-spheres-in-opengl-

es.html> (version current at 5 May 2009).

MIDI Manufacturers Association. White Paper: Comparison of MIDI and OSC.

Internet WWW page, at URL: <http://www.midi.org/aboutmidi/midi-osc.php>

(Accessed November 2008).

Native Instruments. Massive Feature Details. Internet WWW page, at URL: <http://

www.native-instruments.com/#/en/products/producer/massive/?page=675>

(Accessed 10 August 2010).

Y4796421

50

Osculator Forum. Mobile Apps (iOS, Android). Internet WWW page, at URL: <http://

www.osculator.net/forum/forum/7> (Accessed August 2008).

rayculter. vvopensource. Internet WWW page, at URL: <http://code.google.com/p/

vvopensource/> (version current at 19 July 2009).

Rogerson, Ben. iPad USB audio interface compatibility confirmed. Internet WWW

page, at URL: <http://www.musicradar.com/news/tech/ipad-usb-audio-interface-

compatibility-confirmed-250703> (12 May 2010).

Ryan. iPhone CPU Speed. Internet WWW page, at URL: <http://macosrumors.com/

articles/iphone-cpu-speed/> (Accessed 2 January 2008).

Y4796421

51

8 Appendices

8.1 Sphere C++ Function

- (void) drawSphere1: (GLfloat) radius ! ! ! slcs:(GLfloat) slices! // slices number! ! ! stcks:(GLfloat) stacks! // stacks number! ! ! tSVH:(Vertex3D **)triangleStripVertexHandle! ! ! tSNH:(Vertex3D **)triangleStripNormalHandle! ! ! tSVC:(GLuint *)triangleStripVertexCount

! ! ! tFVH:(Vertex3D **)triangleFanVertexHandle! ! ! tFNH:(Vertex3D **)triangleFanNormalHandle! ! ! tFVC:(GLuint *)triangleFanVertexCount

! ! ! nsgn:(GLfloat) _nsign! // disk variable! ! ! scale:(GLfloat) _scl! // scale variable!! ! ! density:(GLfloat) _dnst! // density variable! ! ! extrude:(GLfloat) _extrd! // extrude variable! ! ! makeegg:(GLfloat) _egg! // egg variable! completeness:(GLfloat) _cmplnss!! // completeness variable! ! ! reshape:(GLfloat) _rshp! // reshape variable! ! ! sculpt:(GLfloat) _sclpt! // sculpt variable! ! auxiliary:(GLfloat) _sth;{! GLfloat rho, drho, theta, dtheta; GLfloat x, y, z; GLfloat s, ds; drho = M_PI / ((GLfloat) stacks*_rshp);! // RESHAPE dtheta = 2.0 * M_PI / (GLfloat) slices; // Calculate the Triangle Fan for the endcaps *triangleFanVertexCount = slices+2; triangleFanVertices = calloc(*triangleFanVertexCount, sizeof(Vertex3D)); triangleFanVertices[0].x = 0.0; triangleFanVertices[0].y = 0.0; triangleFanVertices[0].z = _nsign * radius; int counter = 1; for (int j = 0; j <= slices; j++) { theta = (j == slices) ? 0.0 : j * dtheta; x = -sin(theta) * sin(drho); y = cos(theta) * sin(drho); z = _nsign * cos(drho); triangleFanVertices[counter].x = x * radius; triangleFanVertices[counter].y = y * radius; triangleFanVertices[counter++].z = z * radius; } // Normals for a sphere around the origin are darn easy - just treat the vertex as a vector and normalize it. triangleFanNormals = malloc(*triangleFanVertexCount * sizeof(Vertex3D)); memcpy(triangleFanNormals, triangleFanVertices, *triangleFanVertexCount * sizeof(Vertex3D)); for (int i = 0; i < *triangleFanVertexCount; i++) Vector3DNormalize(&triangleFanNormals[i]); // Calculate the triangle strip for the sphere body *triangleStripVertexCount = (slices + 1) * 2 * stacks; triangleStripVertices = calloc(*triangleStripVertexCount, sizeof(Vertex3D)); counter = 0; for (int i = 0; i < stacks*_scl; i++) {! ! // SCALE rho = i * drho*_dnst;! ! ! ! // DENSITY! ! s = 0.0; for (int j = 0; j <= slices*_cmplnss; j++)! // COMPLETENESS {

Y4796421

52

theta = (j == slices) ? 0.0 : j * dtheta*_sclpt;// SCULPT x = -sin(theta) * sin(rho); y = cos(theta) * sin(rho)*_egg;! ! ! // EGG z = _nsign * cos(rho)*_extrd;! ! ! // EXTRUDE

triangleStripVertices[counter].x = x * radius; triangleStripVertices[counter].y = y * radius; triangleStripVertices[counter++].z = z * radius; x = -sin(theta) * sin(rho + drho); y = cos(theta) * sin(rho + drho); z = _nsign * cos(rho + drho); // TXTR_COORD(s, t - dt); s += ds; triangleStripVertices[counter].x = x * radius; triangleStripVertices[counter].y = y * radius; triangleStripVertices[counter++].z = z * radius; } } triangleStripNormals = malloc(*triangleStripVertexCount * sizeof(Vertex3D)); memcpy(triangleStripNormals, triangleStripVertices, *triangleStripVertexCount * sizeof(Vertex3D)); for (int i = 0; i < *triangleStripVertexCount; i++)! ! Vector3DNormalize(&triangleStripNormals[i]); *triangleStripVertexHandle = triangleStripVertices; *triangleStripNormalHandle = triangleStripNormals; *triangleFanVertexHandle = triangleFanVertices; *triangleFanNormalHandle = triangleFanNormals;}

8.2 Apple SDK Gesture Recognisers

This the source code that handles the gesture recognisers provided by Apple in the iPhone SDK.

// ----------- S T A N D A R D S D K G E S T U R E S ---------- //// ---------- tap, swipe, pinch, pan, rotation, long press -------- //!! // * TAPS * TAPS * TAPS * TAPS * TAPS * TAPS * TAPS !! // SINGLE TAP singleTap = [[UITapGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleSingleTap)]; singleTap.numberOfTapsRequired = 1; [self addGestureRecognizer:singleTap]; [singleTap release];

! // DOUBLE TAP doubleTap = [[UITapGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleDoubleTap)]; doubleTap.numberOfTapsRequired = 2; [self addGestureRecognizer:doubleTap]; [doubleTap release];!! // TRIPLE TAP tripleTap = [[UITapGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handleTripleTap)]; tripleTap.numberOfTapsRequired = 3; [self addGestureRecognizer:tripleTap]; [tripleTap release];!! // QUADRUPLE TAP quadrupleTap = [[UITapGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handleQuadrupleTap)]; quadrupleTap.numberOfTapsRequired = 4;

Y4796421

53

[self addGestureRecognizer:quadrupleTap]; [quadrupleTap release];! // TWO FINGER SINGLE TAP twoFingerSingleTap = [[UITapGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleTwoFingerSingleTap)]; twoFingerSingleTap.numberOfTapsRequired = 1; twoFingerSingleTap.numberOfTouchesRequired = 2; [self addGestureRecognizer:twoFingerSingleTap]; [twoFingerSingleTap release];!! // distinguish between single and multiple taps so it doesnt trigger single taps in case of double tap and so on//! [singleTap requireGestureRecognizerToFail:doubleTap];//! [doubleTap requireGestureRecognizerToFail:tripleTap];//! [tripleTap requireGestureRecognizerToFail:quadrupleTap];//! [twoFingerSingleTap requireGestureRecognizerToFail:doubleTap];!! // * SWIPES * SWIPES * SWIPES * SWIPES * SWIPES * SWIPES * SWIPES!! // SWIPE FROM RIGHT TO LEFT swipeLeft = [[UISwipeGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handleSwipeLeft:)]; swipeLeft.direction = UISwipeGestureRecognizerDirectionLeft;! swipeLeft.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeLeft]; [swipeLeft release];!! // SWIPE FROM LEFT TO RIGHT swipeRight = [[UISwipeGestureRecognizer alloc]! ! ! ! ! ! ! ! ! !! initWithTarget:self action:@selector(handleSwipeRight:)]; swipeRight.direction = UISwipeGestureRecognizerDirectionRight;! swipeRight.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeRight]; [swipeRight release];!! // SWIPE FROM DOWN TO UP swipeUp = [[UISwipeGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handleSwipeUp:)]; swipeUp.direction = UISwipeGestureRecognizerDirectionUp;! swipeUp.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeUp]; [swipeUp release];!! // SWIPE FROM UP TO DOWN swipeDown = [[UISwipeGestureRecognizer alloc]! ! ! initWithTarget:self action:@selector(handleSwipeDown:)]; swipeDown.direction = UISwipeGestureRecognizerDirectionDown;! swipeDown.numberOfTouchesRequired = 1; [self addGestureRecognizer:swipeDown]; [swipeDown release];!! // * MULTI-TOUCH GESTURES * PINCH * PAN * ROTATION!! // PINCH! pinchGesture = [[UIPinchGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! ! initWithTarget:self action:@selector(handlePinchGesture)];! [self addGestureRecognizer:pinchGesture];! [pinchGesture release];!! // ROTATION! rotationGesture = [[UIRotationGestureRecognizer alloc]! ! ! ! ! ! ! ! ! ! ! ! !initWithTarget:self action:@selector(handleRotationGesture)];! [self addGestureRecognizer:rotationGesture];! [rotationGesture release];!! // PAN

Y4796421

54

! panGesture = [[UIPanGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handlePanGesture)];! panGesture.minimumNumberOfTouches = 1;! panGesture.maximumNumberOfTouches = 1;!! [self addGestureRecognizer:panGesture];! panGesture.delegate = self;! [panGesture release];

! [panGesture requireGestureRecognizerToFail:pinchGesture];! [panGesture requireGestureRecognizerToFail:rotationGesture];!! /*! // PAN TWO FINGERS! panGestureTwoFingers = [[UIPanGestureRecognizer alloc]! ! ! ! initWithTarget:self action:@selector(handlePanGestureTwoFingers)];! panGestureTwoFingers.minimumNumberOfTouches = 2;! panGestureTwoFingers.maximumNumberOfTouches = 2;!! [self addGestureRecognizer:panGestureTwoFingers];! panGestureTwoFingers.delegate = self;! [panGestureTwoFingers release];!! [panGesture requireGestureRecognizerToFail:pinchGesture];! [panGesture requireGestureRecognizerToFail:rotationGesture];! */!! // LONG PRESS! longPressGesture = [[UILongPressGestureRecognizer alloc]! ! ! ! ! ! initWithTarget:self action:@selector(handleLongPressGesture)];! //! longPressGesture.numberOfTapsRequired = 1;! [self addGestureRecognizer:longPressGesture];! [longPressGesture release];! // ***************** Standard SDK Gestures END ******************* //

8.3 Custom Gesture Recogniser

In order to maek custom gesture recognisers, the developer has to implement the following methods from the iPhone SDK:

- (void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {!

}- (void)touchesMoved:(NSSet *)touches withEvent:(UIEvent *)event{! UITouch *touchA, *touchB, *touchC, *touchD, *touchE;! CGPoint pointA, pointB, pointC, pointD, pointE;!! NSSet *touchEnum = [event allTouches];! // get all touches on the screen!! // No point in making 2 finger recognisers cause they are covered in the SDK Gesture Recognisers!! // ------- THREE FINGERS ------- //! if ([touchEnum count] == 3)! {! ! // Disable pinch, pan and rotation recognisers so data is not passed to the pinch and rotation responders! ! pinchGesture.enabled! = NO;! ! rotationGesture.enabled = NO;! ! panGesture.enabled! ! = NO;! !! ! touchA = [[touchEnum allObjects] objectAtIndex:0];! ! touchB = [[touchEnum allObjects] objectAtIndex:1];! ! touchC = [[touchEnum allObjects] objectAtIndex:2];! !! ! pointA = [touchA locationInView:self];

Y4796421

55

! ! pointB = [touchB locationInView:self];! ! pointC = [touchC locationInView:self];! !! ! float currDistanceAB = [self distanceFromPoint:pointA toPoint:pointB];! ! float currDistanceAC = [self distanceFromPoint:pointA toPoint:pointC];! ! float currDistanceBC = [self distanceFromPoint:pointB toPoint:pointC];! !! ! float abcSpread = [self threePointsSpread:pointA andPoint:pointB andPoint:pointC];! ! float abcAverage = [self threePointsAverage:pointA andPoint:pointB andPoint:pointC];! ! float averagex = (pointA.x + pointB.x + pointC.x)/3.f;! ! float averagey = (pointA.y + pointB.y + pointC.y)/3.f;! !! ! [self drawView];! !! ! NSLog(@"%@", [NSString stringWithFormat:@"%0.2f", abcSpread]);! }!! // ------- FOUR FINGERS ------- //! if ([touchEnum count] == 4)! {! ! // Disable pinch, pan and rotation recognisers so data is not passed to the pinch and rotation responders! ! pinchGesture.enabled! = NO;! ! rotationGesture.enabled = NO;! ! panGesture.enabled! ! = NO;! !! ! touchA = [[touchEnum allObjects] objectAtIndex:0];! ! touchB = [[touchEnum allObjects] objectAtIndex:1];! ! touchC = [[touchEnum allObjects] objectAtIndex:2];! ! touchD = [[touchEnum allObjects] objectAtIndex:3];! !! ! pointA = [touchA locationInView:self];! ! pointB = [touchB locationInView:self];! ! pointC = [touchC locationInView:self];! ! pointD = [touchD locationInView:self];! !! ! float abcdSpread = [self fourPointsSpread:pointA andPoint:pointB andPoint:pointC andPoint:pointD];! ! float abcdAverage = [self fourPointsAverage:pointA andPoint:pointB andPoint:pointC andPoint:pointD];! ! float averagex = (pointA.x + pointB.x + pointC.x + pointD.x)/4.f;! ! float averagey = (pointA.y + pointB.y + pointC.y + pointD.y)/4.f;! !! ! [self drawView];! !! ! NSLog(@"%@", [NSString stringWithFormat:@"%0.2f", abcdSpread]);! }!! // ------- FIVE FINGERS ------- //! if ([touchEnum count] == 5)! {! ! // Disable pinch, pan and rotation recognisers so data is not passed to the pinch and rotation responders! ! pinchGesture.enabled! = NO;! ! rotationGesture.enabled = NO;! ! panGesture.enabled! ! = NO;! !! ! touchA = [[touchEnum allObjects] objectAtIndex:0];! ! touchB = [[touchEnum allObjects] objectAtIndex:1];! ! touchC = [[touchEnum allObjects] objectAtIndex:2];! ! touchD = [[touchEnum allObjects] objectAtIndex:3];! ! touchE = [[touchEnum allObjects] objectAtIndex:4];! !! ! pointA = [touchA locationInView:self];! ! pointB = [touchB locationInView:self];! ! pointC = [touchC locationInView:self];! ! pointD = [touchD locationInView:self];! ! pointE = [touchE locationInView:self];! !

Y4796421

56

! ! float abcdeSpread = [self fivePointsSpread:pointA andPoint:pointB andPoint:pointC andPoint:pointD andPoint:pointE];! ! float abcdeAverage = [self fivePointsAverage:pointA andPoint:pointB andPoint:pointC andPoint:pointD andPoint:pointE];! ! float averagex = (pointA.x + pointB.x + pointC.x + pointD.x + pointE.x)/5.f;! ! float averagey = (pointA.y + pointB.y + pointC.y + pointD.y + pointE.y)/5.f;! !! ! [self drawView];! !NSLog(@"%@", [NSString stringWithFormat:@"%0.2f", abcdeSpread]); // debug line! }}

- (void)touchesEnded:(NSSet *)touches withEvent:(UIEvent *)event{! // Re-enable pinch, pan and rotation gesture recognisers so they are ready for use! pinchGesture.enabled! ! = YES;! rotationGesture.enabled ! = YES;! panGesture.enabled! ! = YES;}

8.4 Custom Gesture Recogniser Responders

These methods are called in order to calculate the spread of the multi-touch pinch and the equivalent point of the gesture.

- (float)distanceFromPoint:(CGPoint)pointA toPoint:(CGPoint)pointB {! float xD = fabs(pointA.x - pointB.x);! float yD = fabs(pointA.y - pointB.y);!! return sqrt(xD*xD + yD*yD);}

- (float)fivePointsAverage:(CGPoint)pointA andPoint:(CGPoint)pointB andPoint:(CGPoint)pointC andPoint:(CGPoint)pointD andPoint:(CGPoint)pointE {! float xD = fabs(pointA.x + pointB.x + pointC.x + pointD.x + pointE.x)/5.f;! float yD = fabs(pointA.y + pointB.y + pointC.y + pointD.y + pointE.y)/5.f;!! return sqrt(xD*xD + yD*yD);}

- (float)fivePointsSpread:(CGPoint)pointA andPoint:(CGPoint)pointB andPoint:(CGPoint)pointC andPoint:(CGPoint)pointD andPoint:(CGPoint)pointE {

! float ab = [self distanceFromPoint:pointA toPoint:pointB];! float ac = [self distanceFromPoint:pointA toPoint:pointC];! float ad = [self distanceFromPoint:pointA toPoint:pointD];! float ae = [self distanceFromPoint:pointA toPoint:pointE];! float bc = [self distanceFromPoint:pointB toPoint:pointC];! float bd = [self distanceFromPoint:pointB toPoint:pointD];! float be = [self distanceFromPoint:pointB toPoint:pointE];! float cd = [self distanceFromPoint:pointC toPoint:pointD];! float ce = [self distanceFromPoint:pointC toPoint:pointE];! float de = [self distanceFromPoint:pointD toPoint:pointE];!! float spread = (ab+ac+ad+ae+bc+bd+be+cd+ce+de)/10.f;!! return spread;}

Y4796421

57

8.5 OpenGL ES

The following code indicates OpenGL which vertices to use in order to draw the spheres. This code is called for every frame of the animation which is set to 60 frames per second. However, the actual frame per second rate is not as high. The real frame rate is about 30 frames per second as measured with the performance tool Instruments.

! // OpenGL code to draw the sphere on the screen! // This code is called for every frame of the animation!! glPushMatrix();!! /* color the sphere */! GLfloat s2Ambient[] = {RGBToFloat(82), RGBToFloat(46), RGBToFloat(4), 1.0}; glMaterialfv(GL_FRONT_AND_BACK, GL_AMBIENT, s2Ambient); GLfloat s2Diffuse[] = {RGBToFloat(29), RGBToFloat(50), RGBToFloat(71), 1.0}; glMaterialfv(GL_FRONT_AND_BACK, GL_DIFFUSE, s2Diffuse);!! /* rotate the sphere */! _x3 = 3.000f*sin(_rotationTemp+SphreSpacing(numberOfSpheres))*_diaspora;! _y3 = 3.000f*cos(_rotationTemp+SphreSpacing(numberOfSpheres))*_diaspora;

! /* place the sphere into the drawing */! glTranslatef((_x3*_vathos), (_y3*_vathos), (-11*_vathos));! glRotatef(_rotMid,0.1f,0.1f,0.1f);!! /* OpenGL commands */ glVertexPointer(3, GL_FLOAT, 0, sphereTriangleFanVertices3); glNormalPointer(GL_FLOAT, 0, sphereTriangleFanNormals3); glDrawArrays(GL_TRIANGLE_FAN, 0, sphereTriangleFanVertexCount3); glVertexPointer(3, GL_FLOAT, 0, sphereTriangleStripVertices3); glNormalPointer(GL_FLOAT, 0, sphereTriangleStripNormals3); glDrawArrays(GL_TRIANGLE_STRIP, 0, sphereTriangleStripVertexCount3);! glPopMatrix();

8.6 OSC

This is the code that sends and receives OSC messages. The receiver method also works as a parser.// Send OSC message Method- (void)sendOSCMessage:(float)floatMessage label:(NSString*)label {! OSCMessage *msg = [OSCMessage createWithAddress:label];! [msg addFloat:floatMessage];! [outPort sendThisPacket:[OSCPacket createWithContent:msg]];}

- (void) receivedOSCMessage:(OSCMessage *)m! {!!! NSString *address = [m address];! OSCValue *value = [m value];! float! incomingOSC = [value floatValue];! int! incomingIntegerOSC = [value intValue];!! if ([address isEqualToString:@"/something"]) {! !! ! // do stuff here...! !! } else if ([address isEqualToString:@"/something_else"]) {! !! ! // do some other stuff here...! !! }NSLog(@"%@ %@", address, value);!// debug line!}

Y4796421

58

Y4796421

59