Aural LimboSpace as a sonic interactive interface

A Thesis Submitted to Parsons School of Design, a division

of New School University, New York in Partial Fulfillment

of the Requirements for the degree of Master of Fine Arts

in Design and Technology.

Mateo Zlatar

2003

Thesis Supervisors: Golan Levin, Mark Stafford

Thesis Advisor: Stephanie Owens

Aural LimboSpace as a sonic interactive interface

1

Abstract

Aural Limbo is an interactive installation that uses the

physical relationship between sound and space as a poetic

theme. In this work I seek to create opportunity for

spontaneous interaction in the context of the public space,

engaging passersby in an inhabitable sonic instrument that

uses the body presence and location as variables for the

dynamic transformation of sound.

The physical space acts as host for an aural space composed

of everyday sounds of the cityscape. These sounds represent

the heterogeneity of our environment, the disregarded

result of hundreds of activities happening at the same time,

which cognitively don't have particular significance and

usually is filtered as meaningless acoustic information

( noise ).

The play of this work is based on the idea that we can

discover meaningful sound patterns in noise, in this case

through an interface that reveals them, making us aware

of listening itself, as a creative act.

2

Acknowledgements

I am profoundly grateful to my wife Paz Guzman and my

Family for their support during this process. I also feel

beholden to the people who provided me with advice and

inspiration. Thanks to: Golan Levin, Stephanie Owens, Josh

Goldberg, David Rokeby, Mark Stafford, Colleen Macklin,

Sven Travis, Marko Tandefelt, Camille Utterback, Barbara

Morris, Jose Miguel Tagle, Elaine Castillo Keller, Fang-Yu

Lin, Mimi Chan, Eduardo Matamoros, Juan Herrera, Matias

Martinez, Ian Szydlowsky, Matthew Mohr, and all the DT

Community at Parsons.

3

Table of Contents

Preliminaries

Title

Abstract

Acknowledgements

Table of Contents

Chapter 1. Introduction

1.1. Motivations

1.2. Overview of the Thesis

1.3. Contribution of this Thesis

Chapter 2. Background

2.1 Introduction

2.2 Spaces for sound

2.3 The Aural Perception of Spaces

2.3.1 Space as Sound Points

2.3.2. Sound as Feedback of Space

2.3.3. Spaces with aural directional

messages

2.3.4. Dislocated Perception

2.3.5. Sound as Inner Space

2.4 Sound as Space

2.5 Patterns in Noise

Chapter 3. Methodology

3.1. Introduction

3.2. Preliminary Experiments

3.2.1. Wearable Synth (Fall 2002)

3.2.2. Trigger Space (Spring 2003)

3.2.3. Sonic Arena (Fall 2002)

3.3. Thesis Prototypes

3.3.1. System Overview

3.3.2. Traffic Report (Spring 2003)

3.3.3. Spatial Scrub (Spring 2003)

4

3.3.4. Accumulation

3.3.5. Panning Sound

3.4. Summary of Implementations

3.5. Thesis Installation

3.5.1 Revealing Patterns in Noise.

3.5.2. The Context of the Public Space.

3.5.3. The users experience

Chapter 4. Discussion and Analysis

4.1. The Music of Sound

4.2. Challenges and Pitfalls

Chapter 5. Conclusion

5.1. Conclusions

5.2. Future Directions

Chapter 6. Bibliography

Appendices

Appendix A. Custom Interfaces Screenshots

A.1. Program for Accumulation

A.2. Program Interface for Aural Limbo

A.3. Program Interface for Traffic Report

Appendix B. Supplementary Sketch

Colophon

5

Chapter 1. Introduction

1.1. Motivations

“Whereas colors are present “naturally” in nature, there are

no musical sounds in nature, except in a purely accidental and

unstable way; there are only noises.” Claude Levi Strauss,

“The Raw and the Cooked”

The phenomenon of sound has always intrigued me as a

physical experience. Our bodies absorb the acoustical

stimuli not only by the ears; they can perceive the vibration

of the sound waves traveling through the air and bouncing

between the walls, augmenting our perception, modifying

our mental and physical environments.

Sound as an invisible medium needs to be embodied

through physical interfaces in order to be produced and

manipulated, like the piano or the violin in traditional music,

but these instruments represent just a slice of the entire

audible spectrum, and moreover, they represent the culture

of music, which uses discrete units –timbre, pitch and

duration, as semantic units of the language of music. But

sound itself precedes language and culture as a natural

physical phenomenon.

Years ago, a couple of friends and I went in a field trip to

an abandoned water tank in the middle of the forest. The

tank was three stories tall and about 50 feet in diameter

and was completely empty. Once inside we started throwing

stones at the walls, discovering a huge reverberation of

these percussive sounds bouncing between the walls. We

also tried our voices and all possible sound sources within

the space. Sounds were trapped in the space for several

minutes mounting one on top of the other, giving the

possibility of building chords and creating complex

rhythmical patterns. Suddenly I was aware of how soundFig.1 Sketches of the Watertank Experience.

6

propagates through the air and the importance of the space

in this phenomena. I realized that space can act and be

treated as a sonic instrument.

In the present thesis, I explore intrinsic relationships

between Sound and Space; from their concreteness as

primary elements of perception to the representational

aspects they may reflect from culture. In this work I seek

to create a non-conventional sound instrument, which goal

is not only to serve as medium of expression, but ultimately

to make us aware of the act of listening itself.

In large urban cityscapes, such as New York, our acoustic

environment is highly heterogeneous and saturated by

hundreds of activities happening at the same time. In the

impossibility of paying attention to every single stimulus,

we filter out what we consider as meaningless acoustic

information (noise). In this context, the concept of noise

becomes highly relative, because we determine what sounds

are significant in a particular moment. The same sound

may be considered meaningful or meaningless, depending

on the focus of attention.

In the process of this work, I explore this idea of a polarized

noise, using different spaces in where the presence of the

body unchains a series of dynamic modifications in non-

musical sounds, revealing hidden rhythmical patterns

emanating from them. The actual space remains the same

while sound reshapes a virtual space, one that can change

its form, texture and size.

The discovery of this “other” space happens in a similar

way as we may see in digital imagery where things

seamlessly morph from one form to another, for example

an abstract object that resolves in a human head. The

surprise of it lies in the change of meaning that the object

suffers along the way. The moment of the transformation

7

is far more suggestive than the final result.

The word Limbo in its common definition is: An intermediate

place or state, An imaginary place for lost or neglected

things, The state of being disregarded. Most of these

definitions are derived from the theological conception,

which refers to an indeterminate place between earth and

heaven, or in Dante’s words: “the higher state the man can

achieve without god”. The Latin word limbus, refers to “an

ornamental border to a fringe”, or a “band or girdle” that

was chosen by Christian theologians of the Middle Ages to

denote this border region.

In my work, the metaphor of Limbo points to the experience

of a meta-space, one that is revealed through contemplative

and expressive rituals as we may find in sacred spaces,

particularly in rituals where the transformation is driven by

sound. This situation may be found in ancestral rituals,

such as in mantra praying or in African drums, where sound

and space play the role of an interface between man and

his divinities.

1.2. Overview of the Thesis

This Thesis is organized in four chapters. In Chapter 2,

Background, I identify the main variables that may be

involved in the development of this work, which are: The

creation of spaces for sound, The aural perception of spaces,

Sound as Space, and the semantics of sound. For each one

of them I recall the work of modern and contemporary

artists who have addressed such conceptions from a variety

of perspectives, providing documentation and analysis.

Chapter 3, Methodology, documents the prototyping phases

in the development of this Thesis, which is comprised of

five design experiments in interactive sound. The process

8

is exposed through documentation on the design and

programming fronts, as design sketches, flowcharts and

mapping diagrams, explaining functionality, interaction

goals and expectations, that finally are summarized and

evaluated.

Chapter 4 is dedicated to discussion and analysis of the

work outcome, evaluating at all its development stages.

Chapter 5, Conclusions, summarizes the Thesis

contributions, pointing unfulfilled desires and future

directions.

Finally, I provide two technical appendixes, one showing

visual programming diagrams and other for supplementary

sketches.

1.3. Contribution of this Thesis

The development of digital technologies has reduced the

way we process images and sound to the same level of

abstraction and treatment, zeros and ones. Programmability

as an intrinsic computational attribute, allows us to

manipulate and specify our will over several kinds of data.

In this common ground, digital designers have the chance

to approach sonic design problems in the same way as they

do visual information. However, the contemporary practice

of multimedia design has remained centered on visual

communication. The use of sound is generally treated as

support for visual information, but not usually in the

opposite way.

While doing research on Sound in New Media, I noticed

how contemporary art practice has addressed sound in a

completely new way, separating it from the musical tradition,

creating a whole new field for conceptual/concrete

expression which is somehow known as Sound Art. The

choice of sound, as medium for this project, represents for

me an opportunity to incorporate the vocabulary of Sound

9

Art into the realm of computer multimedia.

This thesis may be considered an Art project because it has

aesthetic and philosophical goals, but also represents an

inquiry on human-computer interaction, because it

undertakes the responsibility of adapting an information

system to human needs. The use of ‘invisible computing’

in my work, can be seen as a particular example and possibly

extrapolated to the wide variety of tasks we all do with

computers, such as planning, analyzing, visualizing or

entertaining ourselves, which can happen in an individual

or collective basis, collaborating with physically present or

non-present others. All these tasks don’t require the

sedentary behavior of the desktop. In this regard, this thesis

proposes the use of computational augmented spaces as

a contemporary form of computer interaction.

10

Chapter 2. Background

2.1 Introduction

Through the history of architecture it is interesting to notice

how humans have always modified space to host particular

sonorities; for example small offices wrapped in double

panels absorb sounds, creating perfect spaces for private

conversations, and big temples result in perfect

reverberating spaces for elevated chants and prayers. The

constant association between type of spaces and kinds of

sounds contributes to our cultural responses to places.

Our notion of reality leans in each one of our senses. What

if in a sunny day we enter our homes and instead of what

is expected you hear a storm? How would this intervention

change our perception and behavior in that space? In this

thesis I seek to work on the manipulation of sounds and

space at both physical and semantic levels.

In this inquiry, for me was important to understand how

contemporary art practice has addressed sound in a

completely new way, separating it from the musical tradition,

creating a whole new field for conceptual/concrete

expression, giving form to what we know as “Sound Art”.

This chapter uses works of installation or architectural

intervention and performance that depict one or more of

the three key aspects my work wants to address, which I

summarize as:

› The creation of spaces for sound.

› The aural perception of spaces. (Sound as Space)

› Sonic Representation and the semantics of sound.

11

2.2 Spaces for sound

Through the history of architecture it is interesting to notice

how humans have always modified space to host particular

sonorities; for example small offices wrapped in double

panels absorb sounds, creating perfect spaces for private

conversations, and big temples result in perfect

reverberating spaces for elevated chants and prayers. The

constant association between type of spaces and kinds of

sounds contributes to our cultural responses to places. By

looking at the form and size of a space, one may infer what

kind of sonority it will have, but the visual perception does

not prepares us to the actual aural experience they convey.

Some spaces that may seem to be acoustically neutral or

inert, may surprise us with unexpected sound properties,

the sonic quality of spaces influence our perception of

them, modifying its significance and our behavior in that

space. Is in that discovery that we become aware of listening,

paying attention to the subtle variations and underlying

patterns that sound may reveal.

As an example of this kind of experience I came across a

particular work of the Brazilian Architect Oscar Niemayer,

who during the construction of Brasilia in the sixties,

designed a curved structure that served as cover for the

rain in a public square. This construction that seems to be

very functional also was acoustically designed to amplify

the sounds of people’s footsteps,

Fig.2 Public building in Brasilia by Oscar Niemayer (1960).

12

creating percussive patterns by the reverberation of its

sounds. A friend of mine, who brought me pictures of the

building, described the experience as an unexpected gift,

because she just walk through the space without knowing

what was going to happen. As she discovered these

rhythmical patterns, she also noticed how other people

stayed in the space to play with those sounds, as if the

building were a large-scale sound instrument. A building

that is a sonic instrument.

A different kind of strategy for sound spaces can be found

in the work of the Austrian artist Bernhard Leitner, who

created an architectonic intervention as a way to redirect

natural sounds. In 1997 he was commissioned to produce

a sound work for the Donaueschinger music days, a yearly

contemporary music festival in Donaueschingen Germany.

Leitner choose a public park by a river, which had a pergola

under which the water flows. He did a simple intervention

by hanging up a curved metal sheet, which reflects the

natural sound of water to the inhabitant’s ears. This surface

acts as an amplification device for natural sound, which

can be perceived just by inhabiting the space.

Among Leitner’s work we also can find hybrid sound

techniques that combines physical construction with

electronic amplification to redirect and highlight natural

sounds. As an example, Le Cylindre Sonore is a cylindrical

Fig.3 Bernhard Leitner: Wasserspiegel (Water Temple).

13

structure built by Leitner in Parc de la Villette, a public park

in Paris. A double ring made of concrete host 24

loudspeakers between the walls; microphones in the

surrounding bamboo forest capture and amplify its sound

to the interior of the ring. The electronic augmentation of

the natural sounds creates a paradoxical perception: an

indistinguishable blending of actual and virtual sound

spaces, which give the space an almost magical property,

a “listening building” in its double connotation.

2.3 The Aural Perception of Spaces

"If we were trained to turn mentally towards everything we

hear, we would achieve a sense of spatial correspondence

comparable to visual perception." Bill Fontana.

In the interrelation of spaces and the sounds they produce

or host, we can also reverse the picture by studying how

sounds may reassure or neglect the space that contains

them, modifying our perception of spaces. The work of

Bernhard Leitner provides a good example of the design of

spaces based on the vocabulary of sound. The perception

of these spaces unfolds in time with the movement of lines

or points of sound, which perceptually modify our spatial

Fig.4 Bernhard Leitner: Le Cylindre Sonore, Parc de la Villette, Paris. 1987

14

experience, making the whole body a permeable acoustic

depository, a “big ear” for three-dimensional space.

Here I use the work of Leitner as a way to exemplify and

discuss some strategies for building spaces with sound.

In the spring of 1971, Leitner started a series of practical

investigations in a large hall in New York. These

investigations were based on theoretical projects and

concepts, originally published in Artforum (March 1971).

The following lists the investigations and experiments

conducted by Leitner during the 70’s in chronological order:

Soundcube: Borders of a corridor, Swinging space.

Raum-Wiege (Sound Swing)

Leit-Räume (Guiding Spaces): Sound Gate, Sound Slopes

Gang-Variationen (Corridor Variations)

Liege-Objekte (Lying Within Sound)

Trag-Objekte (Portable Objects)

Vertikale Räume (Vertical Spaces)

Erweitern – Verengen von Raum (Expanding-Contracting

Spaces)

These instruments can be broadly classified as general and

specific ones. “Soundcube” and “Spatial Grid” are general

instruments because they are neutral structural matrices

where to test several different sound movement

specifications in space, analyzing their effect on the

perception of space. In the other hand, specific instruments

are the ones designed specifically for certain pre-defined

and particular sound movements such as: “Sound Swing”,

“Guiding Spaces” and “Corridor Variations.”

All these instruments utilized an electronic switching device

built according to Leitner’s specifications, which allow the

programming of sound sequences for any number between

two and forty loudspeakers.

15

“The Soundcube is an instrument for producing space with

sound. It has a grid of loudspeakers on each of its six walls.

It is visually speaking “neutral”, i.e., without any specific

spatial message. The sound is programmed to travel from

loudspeaker to loudspeaker. The dimensions of the cube

depend on the particular situation. An infinite number of

spaces or spatial sensations can be created. The Soundcube

is a laboratory for studies in the definition and character

of space and for investigation into the relationship between

motions of sound and their audio-physical experience. At

the same time it is a place for demonstrations to the public.”

(Leitner, 18.)

Soundcube and Spatial Grid were never built, although

Leitner designed the sound movement specifications in

custom scores, and punch-card programming.

These instruments served as hypothetical situations were

to develop his theoretical framework. The hypothesis he

was trying to depict, was “Rhythm as Space” referring to

accentuation of individual points in space by the pulsing

of individual speaker units across the six walls. A sequence

of pulses describes direction, creating the illusion of lines

and circles of sound traveling in space, in which intensity,

tempo and duration became critically significant.

2.3.1 Space as Sound Points

Across the diverse experiments conducted by Leitner, we

can distinguish specific elements and variables he used as

a strategy to build spaces with sound, which I summarize

as:

› The sequence of loudspeakers. (Direction, trace, line.)

› The speed of traveling sound. (Timing)

› The intensity of sound at each point.

› The quality of sound (Timbre.)Fig.5 Spatial grid of speakers andmovement of sound in SoundCube, Leitner 1971.

16

The choice of sounds for most of his experiments was

percussion and simple electronic tones. They were used as

ways to demarcate points in space, the message of these

sounds was tied to the perception of dimensions in space,

rather than conveying a message in sounds themselves.

As an example of this, I recall Leitner description of sounds

in “Corridor Variations” (1973)

“The arching example shows how fast, soft beats (small

drum) start forte (T) in A2, lift off with a decrescendo, move

slowly (Q) and piano (V) through the two top loudspeakers

(A0,A7) and continue to A6, increasing in both traveling

speed and intensity before ending in A5 with forte (T). The

perception of the ceiling’s curvature depends upon the

decrescendo/ritardando of the rising line and the

crescendo/accelerando of the falling line (steep arch, flat

arch).” (Leitner, 52.)

2.3.2. Sound as Feedback of Space

“Borders of a corridor” (fig.6), is a sequence of sound

movements describing three inclined planes in half of the

cube, mirroring the exact same situation in the other half,

creating a central corridor in the middle of these two

movements. In “Swinging space” (fig.2), there are seven

circles in the horizontal axis gradually changing their

inclination angle. In these two early experiences, individual

accentuation of points in space creates the illusion of a

sound line describing movement, establishing dimensions

of height and length, which gives an aural perception of

boundaries of space. Although these boundaries cannot

be experienced at once, they are transformed, repeated

and developed in the dimension of time.

“Sound Swing” (1975) is an actual installation conveying

Fig.6 Borders of a corridor, Leitner 1971.

Fig.7 Swinging Space, Leitner 1971.

17

the idea of a pendulum of sound. (fig.8) This instrument

consists of four loudspeakers, two of them placed at the

ends of mirroring diagonal platforms, and the other two at

the bottom of the structure, leaving a central space for the

body to walk by or stay. The diagonal wooden panels act

as an acoustical resonance link between the upper and

lower loudspeakers, which distance between them was

empirically determined to convey the perception of

continuous pendulum-like motion.

The program for Leitner instruments was recorded on

punched tape in three different codes: one determining the

sequence, one the intensity and one the speed of motion.

“Working with a visually readable program permits one to

introduce directly corrections such as adding or taking out

a loudspeaker in a particular sequence, or modifying the

intensity of each loudspeaker.” (Leitner, 14.).

In particular cases, his instruments allowed the users to

manually select the speed and intensity of the sequences,

which he thinks as “self-adjustable feedback between

person and space.” I think this idea can be extrapolated

to the dynamic possibilities offered by contemporary

computer capabilities. How these experiences may be

enhanced if we give users the control of sound variables

on the fly? For example in “Borders of a Corridor” and

“Sound Swing”, the speed of sound motion (rhythm) may

be coupled to the speed of users motion in the space,

increasing or decreasing according to the different users,

Fig.9 An example of Leitner’spunched card programming.

Fig. 8 Sound Swing, Leitner 1975.

18

or a single user behavior in the space.

Among Leitner’s experiments the one that is closer to the

idea of interactive sound is “Spiraling Space” (1972). In a

hypothetical tube-like corridor, successive rings of speakers

render the space. Sound moves along the ring in a 4-second

revolution and all rings move at the same time. When

somebody stands still under a ring, he can feel the circling

sound around, as he moves forward, the sound becomes

to describe a spiraling effect, mapping walking speed to

the rate of the spiral wavelength.

In this work, there is a direct correspondence between the

users movement and the unfolding of the spiral in a way it

can be considered “reactive” because users perceive

according to their own individual displacement in space,

with the possibility of having several people experiencing

their own spiral at the same time.

2.3.3. Spaces with aural directional messages

“Sound Gate” (1971), is a vertical square structure, which

holds 17 speakers along his perimeter and is the building

block for a series of them, assembled in a way to create a

passageway of sound. The structure is 4 X 4 meters and

has two extensions on the floor. Leitner created two different

programs for it; in one the sound moves from one

loudspeaker to the following next loudspeaker along the

structure, giving a directional message to the gate (fig.11).

In the other program (non-directional) criss-crossing motions

of sound accentuates the gate’s vertical plane, creating aFig.11 Sound Gate, Leitner 1971.

Fig.10 Model for Spiraling Space. Leitner 1972.

Fig. 14 Sound Slopes, Leitner, 1972.19

dynamic sound layer, which is crossed by the body in motion

in any given direction, perceiving the dimension of the

gates physically (fig.12) Leitner made a sketch of how he

sees a sequence of gates implemented in a public space

(passageway of hallway) A directional message in this case

will have a polarized meaning; one encouraging the body

movement if walks in the same direction, and the other

moving against your direction which may be analogized to

the experience of walking against the “wind direction”,

which is not wrong or right, but simply adds this sense of

impulse.

2.3.4. Dislocated Perception

“Ascending and descending lines of sound are

superimposed on a slope. Their angles of inclination

enhance or negate each other. The downward movement

of a person is emphasized by a descending line of sound.

Ascending and descending lines of sound bio-

psychologically influence the descending person’s reading

of the slope’s actual angle of inclination.” (Leitner, 47)

This kind of distortion of spatial sensation happens in the

match or mismatch between sound movement and spatial

forms, as we may find in “Sound Slopes” (1972). This

phenomena can be used to demonstrate differences

between visual and aural perception, perhaps in the design

Fig.12 Sound Gate, Leitner 1971.Non-directional program.

Fig. 13 Sketch for Implementationof Sound Gates in a public space,Leitner, 1971.

20

of a space that can be perceived completely different aurally.

As an example, I imagine a small cubicle in where sounds

reverberate as if were in a big cathedral. Any given sound

in that space will actually be referring to another space.

The encounter of such contradictory elements allow for the

dislocation of perception.

2.3.5. Sound as Inner Space

One of Leitner’s most interesting observations about the

influence of sound in the body is the fact that we absorb

sound with the entire body and not merely by the ears.

“Lying within sound “ are experiments in which sound is

applied directly or close to the body in a static position.

“In sound objects one listens to the sound, one feels it

wherever the vibrations enter the body and one retraces

the movements mentally. Body position and spatial

movements of sound must relate to each other. Lying down

implies a particular readiness to perceive sound motions

around, along and through the body.” (Leitner, 58.)

Among these experiments there is “Platform for horizontal

motions” (1976), which consists of a bench with two

loudspeakers underneath. Staccato-like electronic beats

move in the upper part of the body producing a stretching

effect. The same bench is used but with a set of

loudspeakers positioned underneath the chest and above

in two layers in “Platform for vertical motions” (1976). A

soft, medium fast electronic beat moves between them

provoking the persons breathing to match the sound

movement. (Fig.16)

In these experiments, the motion of sound through the

body can influence biological functions such as breathing.

After a time under the influence of rhythmical patterns,

Fig. 15 Sketches for Platform for HorizontalMotions. Leitner, 1976.

Fig. 16 Sketches for Platform for VerticalMotions. Leitner, 1976.

21

breathing falls in synch with them, which can be thought

as an expansion of the body in space through sound, similar

as we may found in the ancestral practice of mantra praying,

a form of meditation where sound leads the ritual as its

“score”.

Sound tends to create a feeling of awareness of an inner

space, which goes beyond the body space. Leitner was

aware of this as he recalls in an interview during the eighties:

“Hearing experiences not only enable us to have a special

spatial experience but also an internal space” (Bernhard

Leitner interviewed by Wolfgang Pehnt during Documenta

1984, Cologne.)

I believe that sound has a strong influence at the spiritual

level, independent of individual religious believes or ritual

practices. David Rokeby, multimedia artist creator of Very

Nervous System, a video tracking software and performance

instrument, comments about this issue in the context of

his interactive sound installation “VNS” (1986-1991).

“The diffuse, parallel nature of the sound interaction and

the intensity of the feedback loop can produce a state that

is almost shamanistic. The self expands, and loses itself to

fill the space and by implication, the world.” (Rokeby)

2.4 Sound as Space

“Like modernism itself the phonograph represented a new

day in aurality through its ability to return virtually any sound

back again and again into the sensorium and into the historical

register.” (Kahn, 5)

As different musical instruments produce different sounds,

spaces have also their own sounds, based on their physical

characteristics. The reproduction of sound by

electromagnetical means allow us to recreate a given

moment in time and space in another time and another

Fig.17 David Rokeby performing with hisVery Nervous System.

22

space, which suggests the possibility of remote presence.

Recording or transmitting live sounds from one environment

to another, translates a spatial situation, carrying the

representation of one space to another. In this sense,

artificially recreating a space can be seen as the creation

of a virtual space.

An elegant example of this can be found in the work of the

American Sound Artist Alvin Lucier. In his piece “I am Sitting

in a Room” (1970), Lucier reads aloud the following text,

recording his voice in a tape recorder:

“I am sitting in a Room different to the one you are in now.

I am recording the sound of my speaking voice and I am going to play

it back into the room again until the resonant frequencies of the room

reinforce themselves so that any semblance of my speech, with perhaps

the exception of rhythm, is destroyed.

What you will hear, then, are the natural resonant frequencies of the

room articulated by speech.

I regard this activity not so much as a demonstration of a physical fact,

but more as a way to smooth out any irregularities my speech might

have.”

The recording is played back into the room through a

loudspeaker, while simultaneously re-recorded using a

microphone. The new recording is played back and re-

recorded in a successive series of generations. In each new

recording the natural resonance of the room is captured

and reinserted, amplifying it until we can no longer

distinguish the original text, only retaining the pure

resonance of the room.

By listening the recording of this piece, one can “see” the

space, not in form but as function. “I am sitting in a room”

uses space as a sonic instrument that speaks about itself

as space.

One key aspect about Lucier’s piece is the fact that the

Fig.18 Lucier performing“I am Sitting in a Room”

23

room in which he is “… is different from the one you are”

as he reads. I think the translocation of spaces is possible

not only because of the technologies of reproduction, but

because our experiences of space can be recalled by our

imagination.

Aural reproduction suffers of transformations that would

make sounds mere representations of other ones. As an

example, a photographic image can be easily understood

as a representation, because the changes in scale,

perspective, lighting, depth, etc. In an approximation to

achieve realism we can find the example of the trompe

l'oeil in painting, which creates an illusion of reality by

coupling dimension and spatiality.

A particular work of the Canadian artist Janet Cardiff, works

out this conception with sound; “Forty Part Motet” is a

sound installation that replaces the voices of forty choir

voices by forty loudspeakers. This work uses a piece of

secular music created in the Sixteenth century by the English

composer Thomas Tallis. Visitors can perceive the choral

music as a whole while standing in the middle of the space;

as they approach to individual speakers they can distinguish

individual voices, being able to walk through and climb

between the different harmonies and layers of sound. This

piece translates the physical inaccessible inner space of

choral music to an accessible other space, as Cardiff

explains: “Even in a live concert the audience is separated

from the individual voices. Only the performers are able to

hear the person standing next to them singing in a different

harmony. I wanted to be able to 'climb inside' the music

connecting with the separate voices. I am also interested

in how the audience may choose a path through this physical

yet virtual space.”

The translocation of sounds can also contradict or neglect

the space which contains it, as in the work of Bill Fontana.Fig.19 Installation of Forty Part Motet inEngland, by Janet Cardiff.

24

Fontana transmits sounds from one location to another

creating a re-presentation of a distant space, which in the

change of context suffers of an intriguing permutation,

which evokes another physical space. An example of this

is "Sound Island" a sound intervention made by Fontana

at Arc de Triomphe, Paris in commemoration of the 50th

anniversary of the D-day. In this work loudspeakers were

placed in the four façades of the monument, which

transmitted live sounds from the Normandy Coast,

transforming the visual and aural experience of the constant

traffic around the Arc. The harmonic complexity of the

natural sounds of the ocean and crushing waves has the

psycho-acoustic ability to mask other sounds, directing our

attention to them over the overwhelming noise of traffic.

It is interesting to notice how an intervention like this, with

no visual or physical alteration, has the power to completely

transform the notion of a space.

2.5 Patterns in Noise

“As a visually oriented culture our essential responses to the

everyday world are semantic. Everyday sounds are regarded as

not having semantic significance (noise). Noise pollution (with

the exception of sounds that are dangerously loud: close

proximity to a jet aircraft or heavy machinery) can be explained

as a semantic problem. Because sounds must be semanticized

in order to be meaningful, our main aural concerns as a culture

have been language and music. Sounds in themselves have

not been regarded as having communicative effectiveness.”

(Sonic Ecology, Fontana.)

In large urban cityscapes, such as New York, our acoustic

environment is highly heterogeneous and saturated by

hundreds of activities happening at the same time. In the

impossibility of paying attention to every single stimulus,

we filter out what we consider as meaningless acoustic

Fig.20 Sound Island, Sound interventionat the Arc d’Triumph by Bill Fontana. Paris,1994.

25

information falling in the category of noises. In this context,

the concept of noise becomes highly relative, because we

determine what sounds are significant in a particular

moment. The same sound may be considered meaningful

or meaningless, depending on the focus of our attention.

In 1970, the Canadian composer R. Murray Schafer started

his “World Soundscape Project” in which he collects and

analyses the sounds of a particular environment to examine

interrelations in the biosystem and the techno spheres;

identifying noise pollution sources, and sounds that may

be preserved or encouraged.

In his research he embraces the idea of a sound ecology,

which have influenced many other sound artists and

composers such as Bill Fontana and the Australian

radiophonic artist Paul Carter who states: “[…] Degraded

environments will be sparsely orchestrated and badly tuned,

while relatively undisturbed habitats will be harmonically

subtler and rhythmically more various […]” (Madsen, 2).

For Schafer, the degradation of our soundscapes is the loss

of “resonant wilderness”, the loss of the sacred. His essay

“Radical Radio” suggests the idea of injecting fresh sounds

to the heart of the cities from remote and wild locations.

Schafer’s ecology is practiced in an almost scientific way,

even though the problem is based on an aesthetic

parameter, the one of harmonic degradation.

John Cage was one of the first musical composers who

neglected the traditional western affinity for harmony and

tonality as a means of structuring compositions. One of his

most famous pieces, entitled: 4’33’’ is a score containing

the equivalent time of 4 minutes and 33 seconds of just

silence. Cage thinks that absolute silence doesn’t exist,

which may be inspired by his experience in an anechoic

chamber in 1952, when instead of the expected silence, he

heard two constant sounds; his nervous and circulatory

systems at work. Cage started seeking sound in every single

Fig.21 Cover Diagram for “The WorldSoundcape Project’s Handbook for AcousticEcology” R. Murray Schafer 1978.

Fig.22 John Cage and oneof his “Prepared Pianos”

26

object he found, which can be reflected in his “prepared

pianos”. From them it can be said that Cage wanted to

incorporate musical meaning to commonly unsemanticized

sounds, which in Bill Fontana’s words, is the ultimate

contribution of Sound Art.

“The semantic ambiguity of sound will change when society

develops a capacity to perceive patterns or qualities that

are recognizable as part of a context of meaning, such as

the sound vocabularies of contemporary music and acoustic

art […] The task of acoustic art and acoustic design is to

fundamentally challenge all of the old historical definitions

of noise and the resulting preconceptions that most people

have about the sounds they live with”. (Fontana, 3)

For Cage and Fontana the problem of noise may be

understood as a lack of listening awareness; in the case of

Fontana by creating works that uses the natural environment

as living source of musical information. He assumes that

“at any given moment there will be something meaningful

to hear” and “that music - in the sense of meaningful sound

patterns - is a natural process that is going on constantly.”

A fantastic extrapolation of this idea can be seen in the

movie “Dancing in the Dark” of the Danish filmmaker Lars

von Trier. This movie tells the story of a nearly blind worker

of an industrial factory (Bjork), who discovers in the

machinery sounds “hidden” music that she dances and

sings to. This music is in her mind, but we can hear those

musical compositions made from machine sounds as she

thinks it.

Our memories not only store images, words or smells; they

can recall sounds and music as well. In the same way we

mentally envision images we have never seen before, we

can mentally listen to unheard music.

27

Chapter 3. Methodology

3.1. Introduction

The development of the present Thesis has been supported

by a series of experiments and small projects in physical

interaction with sound. In my intend to incorporate physical

space as an interface, I started from simple electronic

components and sensors, using a chip as the sound source.

For the following stages I switched to prerecorded and live

video as input and MAX/MSP as the programming

environment. The present Chapter expose five design

experiments through documentation on the design and

programming fronts, as design sketches, flowcharts and

mapping diagrams, explaining functionality, interaction

goals and expectations, that finally are summarized and

evaluated.

3.2. Preliminary Experiments

During the Fall of 2002 I was involved in a collaboration

studio called Musical Interfaces which first was conceived

as a fundamental electronics workshop for building sound

interfaces. In this studio I had the opportunity to experiment,

design and build three projects, which helped me

understand the basic issues of physically interacting with

sound.

3.2.1. Wearable Synth (Fall 2002)

In this project I intended to map my own body movements

to basic sound parameters in an Invisible Interface, which

allows the body to move freely without the constraints of

being tied to a device by cables or any other gadget. My

goal was to create a synesthetic relationship between body

language and sound, testing different mappings of

28

movements to sound and observing the expressive ranges

I obtain by using the most minimal interface.

To accomplish this I used a 555- timer chip, a small electronic

chip that produces electrical impulses, which can be easily

controlled and modified by varying resistance. In my setup

I implemented two of these chips to produce FM synthesis,

where one chip acts as the modulator of the other. (Fig.X)

The chip setup was attached to the belt in a soft case that

contains the speaker as well. Bend sensors attached to the

elbows, knees or shoulders were connected to the chips,

allowing continuous change in sound by flexing the main

body articulations. In this project I spent a big portion of

the timeframe finding an interest sound in the combination

of fixed resistors and capacitors used. The resulting sound

was a constantly warbling tone.

Although the sound was rich and interesting, the expressive

range was limited and required a great amount of practice

to learn how to control the output in an interesting way.

Additionally, the continuous sound was a problem; I wanted

to use silence as another controllable variable, which I later

implemented using a photocell attached to the inner part

of the leg, so when the legs were close to each other, the

photocell was covered, silencing the synth.

Sometimes the body would adopt awkward positions to

accomplish a particular sound, making the performance

humorous at some points. I realized that the body was

trying to adapt to the instrument requirements to produce

interesting sonorities, rather than using the natural body

inflections to produce a synesthetic sound parallel.

3.2.2. Trigger Space (Spring 2003)

An experiment similar to Wearable Synth was done later in

time, using video input instead of sensors attached to the

body.

The video-input is processed in the computer using Cyclops,

Fig.23 Configuration of FM Synthesisin Wearable Synth.

Amplifier

Speaker

555 Timer Chip

555 Timer Chip

Fig.24 Sensors positions and chiplocation in Wearable Synth.

Bend SensorBend Sensor

Bend Sensor

Timer Chipsetup

29

and MAX/MSP as the computer vision and sound processing

software respectively. Cyclops allows for the specification

of a grid of custom rows and columns over the image,

outputting integer numbers according to the occlusion of

any of the cells in the grid.

I assigned a different sound to each cell. The visual field of

the camera is converted in a kind of "minefield", in that

way the body can trigger those sounds by just crossing a

cell.

In this case, there was no restriction in the kind and variety

of sound I can use, but due to the way in which I specified

the sounds, that is "cell-specific", the space becomes fixed

and specific positions in space always trigger the same

sound. This constrain made the interaction a “memory”

game, because once you hear a sound at a specific position

of hands, head or legs, you can repeat the movement or

sequence of them to repeat a series of sound events again.

I plan to develop this prototype further for a live dance-

sound performance, maybe using two video inputs to map

horizontal and vertical motions.

3.2.3. Sonic Arena (Fall 2002)

Users are confronted with a frame containing fine sand rose

to the level of the hand. As users move the sand, trails are

left behind; this action is captured by a video camera placed

inside the pedestal, pointing to the bottom of the frame,

which is made of transparent acrylic. The trails create

difference in the depth of sand, allowing the light to pass

in different intensities and forms. The sand movements are

mapped to a set of prerecorded samples of abstract sounds,

which fade in and out seamlessly, giving an aural feedback

in real-time to the patterns created in the sand.

This project can be seen as a scaled-down version for my

thesis installation because its input and output are the

same but in an inverted situation. The patterns in the sand

Fig.25 Each cell represent a sampled sound

Fig.26 Preliminar sketch for Sonic Arena.

30

being observed from underneath will be finally extrapolated

to people movements being observed from above.

The development of this project went through a series of

stages, because I was learning MAX at the same time, so

it had several instances of programming before getting to

the final interface and its inputs and outputs. This process

started with a mock up of the video input with a basic

“paint” object, which is a simple canvas window in MAX

that returns x and y values for the current mouse position.

The first mapping I tried was using these values to control

a set of cycle~ objects, (MSP built-in tone generator) which

was implemented as LFO (low frequency oscillator). The

parameters being controlled were the LFO rate and LFO

depth, perceived aurally as an oscillating pitch in variable

intensities.

In the following stage I switched to video as the input, using

the boundary dimensions of the light blurbs as the

meaningful values to manipulate sound, so in this case the

size of the trails was the modifier parameter rather than

the form or position. I built a patch using the Jitter object

jit.findbounds to get these values. This object analyses the

video input returning numerical coordinates of the size of

a particular color blurb. Here I mapped these values to a

tone generator. The values cross through a series of

mathematical transformations to be received by a series

of cycle~ and line~ functions. In this case, the

correspondence between sand trails and the sound output

was less obvious, but the sound always had the same

timbre with variations only in pitch and depth similar to

Aural Paint. I realized that would be more interesting to

allow users to cross through different sounds along the

sand field.

A third stage was built using Cyclops a computer vision

extra designed by Erik Singer, which has three modes of

image analysis. The one I found interesting to explore is

Fig.28 Video input from the bottom ofthe frame as received in the computer

Fig.27 Sonic Arena in use.

Fig.29 Boundary dimensions oflight trails in the sand box.

31

called “difference” which measures differences in the image,

frame by frame in a matrix grid which can be specified in

columns and rows that returns integer values per each grid

unit for each captured frame.

I specified a 8 X 8 grid and connected each output to a

different sample player object (sfplay~), which reads stored

samples in the hard drive. These sound units were assigned

to a specific grid unit, acting as triggers. The samples where

edited so that they faded in and out in relatively long

phases, and they crossfade seamlessly when the different

units were exited. In this case, the approach chosen was

completely different, because I was using sampled sounds

as a way to convey abstract sounds which were also

synthetic in origin, but hard to specify in MAX using sound

synthesis techniques. The result was clearly richer in terms

of sonority enhancing the experience and making it

interesting for a longer time.

In another version I used another set of sampled sounds

that created a semantic relation between the sand in the

interface and the sand by the sea. The sounds were

recordings of the ocean: the shore water cycle, crushing

waves, birds and wind.

The use of samples, compared to synthesized sounds,

represent its advantages and disadvantages. The synthesis

of sounds allows an infinite degree of form manipulation

because it works in the specification of sine waves, which

are the basis of aural perception. At the other hand, sampled

sounds opens the possibility of manipulation at the semantic

Fig.30 Sonic Arena in use.

32

level (sound narrative) because one can use sounds that

convey linguistic messages such as speech or natural

phenomena like weather, birds, or music. In Sonic Arena I

experimented with sampled sounds in both ways, in an

abstract and figurative way. In this particular piece, the

figurative approach was easily understood and made the

experience more successful from the users point of view,

mostly because the interface reinforced the ocean metaphor,

however I think is still possible to play in the boundaries

of both conceptions, a fluctuation between the recognizable

and the indecipherable, the morphing between noise and

meaning.

3.3. Thesis Prototypes

3.3.1. System Overview

During the development of this project I had to research,

find and learn a series of system software that would allow

me to control sound based on a video input. After a short

period of experimentation with simple electronics I started

using MAX/MSP, which is a visual programming language

developed by Miller Puckette. MAX was originally developed

to manipulate MIDI data; with the addition of MSP, (MAX

Signal Processing), it was possible to specify and manipulate

sinewaves and samples. After trying the software for a

couple of months I felt comfortable enough to start

developing my own tools. Especially important for me was

the possibility to have a video input, through the addition

of computer vision extras.

During the prototyping stages I have worked with three

different computer vision extras. The first one was Cyclops,

designed by Erik Singer. Secondly I tried Jitter, and finally

David Rokeby’s SoftVNS 2.1, which I found to be the best

video tracking – image analysis external for MAX, because

it can combine different analysis modes using a single

33

input.

All the prototypes for this Thesis share the same input –

output system, which is based on single video Input,

MAX/MSP programming, and loudspeaker output, the only

variation is in the Computer Vision externals used in each

one of them.

3.3.2. Traffic Report (Spring 2003)

In the process of learning the different objects for sound

manipulation in MAX, I came across a playback speed

control of sampled sounds. I built a patch that used a loop

of about 40 seconds long, which I previously edited in

Protools. The same day I went to the Brooklyn Library and

while waiting to cross the street, I saw how the traffic of

vehicles flows in rhythmical cycles. Using a video camera

I recorded a full cycle from no traffic to full circulation to

no traffic again, which later I used as the input for a program

I wrote in MAX.

Fig.31 Single video Input,MAX/MSP programming, andloudspeaker output.

Fig.32 The surroundings of the Brooklyn Library

34

From the video image I analyzed the amount of overall

movement occurring in the image frame by frame. I

connected this value to the speed of the sampled sound;

as a result there was a proportional increase in pitch and

speed of sound according to the traffic cycle. I thought that

this translation can be implemented as a simple feedback

a public building gives to its surrounding environment.

The avenue in a side of the Library has four lanes and many

people wait for several minutes to access the Children’s

section entrance, which may give them time enough to

make the association of traffic and sound if loudspeakers

are mounted in the lateral façade.

3.3.3. Spatial Scrub (Spring 2003)

For this prototype I chose the space of a corridor, in which

there’s only two ways to go, back and forth. Using this

space situation, I created a short segment of sound that

contained a large amount of sound events; chords,

percussion, segments of voice, one after another. This

soundtrack loops in very short cycles, about 100

milliseconds. A video input was used to “track” position in

that space. That position was connected to the main audio

cursor, the one indicating the current position in the sample,

in this way, when people walk through the corridor, they

can “scrub” through the sample as if they were a scanner

or playback head.

The speed of motion determined the size of the loops in

milliseconds, so the faster the speed, smaller portions were

looped, creating a finer “granularity” in sound. The repetition

of the loops, create a rhythmical pattern of sound, which

degree of musical harmony and beauty depends on the

type of sounds in the soundtrack. For this experiment I

used subtle chords, basses and hi-hats that are continuous

tones, without abrupt cuts, so when looped, they seem to

Fig.33 The pattern of the traffic cycle.

Fig.34 The body as the playback head of a sound.

35

interweave in a rhythmical progression.

As experience, the connection between the size of the

corridor and the size of the sound, is not immediately clear,

but as soon as users stop to listen, the sound keeps looping

in the same position, so when resuming motion, the

connection becomes evident.

3.3.4. Accumulation

In this prototype I wanted to experiment with a feedback

whose changes can only be perceived in a long timeframe.

I placed a video camera observing the Brooklyn Library

main hall from above, (almost plain vertical), so I could see

a large space and the traffic of people happening in it.

The sound used was the ambient sound of the same space.

The video image was analyzed in terms of amount of

movement happening each 15 frames. The number obtained

from that reading was used to control a “delay feedback”

in the sound. A delay feedback in sound is similar to a

“video feedback” as we may find in early video works of

Nam Jun Paik, where the camera points to a monitor

displaying the same image being captured, creating a

spiraled image because of the closed circuit between input

and output. In sound such effect can be accomplished by

reinserting a portion of the input to the output, with a slight

delay. The effect is a constant tone emerging from the

original sound, which changes also according to the input.

The amount of movement in the space was connected to

the amount of feedback in the sound, reflecting the physical

situation. In this way, we can obtain an aural “portrait” of

the space in a given moment. If we compare snapshots

along a day, we would see significant changes in the sound.

This experiment used an extremely long “exposure”, so the

changes of sound may not be perceived instantly, but along

the course of a day.Fig.36 The increases in the delay areimplemented in a logarithmic curve, making thefeedback more “sensible” to small changes inthe environment, and more “stable” at the peak.

Fig.35 The interior hall of the Brooklyn Library

36

3.3.5. Panning Sound

This prototype was an experiment on tracking positions of

multiple users in space, and also the amount of movement

within zones of the space. Tracking people’s positions

requires certain conditions of light in the space in order to

differentiate people from the background. Also as the

lighting conditions may change along the day, there is the

need for periodic revision of the thresholds that determine

people’s presence. This task is hard, so I spent a great

amount of time figuring out a way to dynamically take

periodic measures of lighting conditions to readjust the

thresholds. There is also the problem of occlusion. When

two people are close to each other, it is almost impossible

for the computer program to differentiate them as two

individuals. Nevertheless, the position and distribution of

users in space is valuable information that can be used in

a variety of ways, for example the panning of sounds

between output channels. As I am currently working with

two output channels, Left and Right, I divided the video

input field in 16 vertical zones, each one reporting movement

in its own space, I assigned to each one of these zones a

numerical value from 0 t o16, each extreme representing

an output channel. The numbers in between represent

intermediate values in the amplitude of each channel. In

this way, if one person is present in the space, its location

pans the sound to the opposite side of which he/she is. As

the user moves around, sound moves accordingly.

Fig.37 Custom tracking interface

37

In the case of a greater number of users, their positions

are averaged for determine the movement of sound. The

movement of sound in space would give a dynamic response

to peoples displacement within the space, creating an

attraction–repulsion relationship between sound and

listeners.

I decided to move sounds in the opposite direction because

this makes the movement more evident, because mapping

to the same position can be easily confused with sounds

being louder by being closer to a channel side, which

happens naturally in the proximity of a sound source.

38

3.4. Summary of Implementations

For this summary I considered the five most relevant

experiments to my final Thesis project, which are: Trigger

Space, Spatial Scrub, Traffic Report, Panning and

Accumulation.

These Prototypes represent a wide range in physical size,

which has helped me understand and consider particular

details relevant for each particular scale. In each one of

them I am using a different mode of Image Analysis, what

was suggested in part by the physical situation itself and

in part for the kind of mapping to sound I wanted to try.

While doing this experiments, I could see how each one of

them had a different timeframe requirement, also due to

the nature of the event, and the type of mapping, that will

define the time of the experience.

Required Temporality

Milliseconds Seconds Seconds / Minutes Minutes Hours

Mapping to Sound

Trigger Samples Pointer to file time Panning / Reverb Speed of soundtrack Feedback DelayAmount

Image Analysis

Zones Horizontal Position x, y position /Movement Amount

MovementAmount

MovementAccumulation

Physical Scale

Body Corridor Hall Building Facade Large Interior Plaza

Fig.38 Comparison Table

39

3.5. Thesis Installation

3.5.1 Revealing Patterns in Noise.

Aural Limbo is a physical space, which act as host for and

aural space composed of everyday sounds of the cityscape.

These sounds represent the homogeneity of our

environment, the disregarded result of hundreds of activities

happening at the same time, which cognitively doesn't

have particular significance, and usually are filtered as

meaningless acoustic information (noise).

A ring printed on the floor demarcates the boundaries of

the sensible space. Within this zone, the presence of the

body, its location and amount of movement unchains a

series of dynamic modifications in non-musical sounds,

revealing hidden rhythmical patterns emanating from them.

These patterns are revealed by the digital manipulation of

sound parameters such as: Speed, Reverb, Pan, Delay

Feedback, and Convolve. These parameters are controlled

by a combination of data input from the physical space,

such as presence in the space, position in the space and

difference between frames (motion).

The actual space remains the same while sound reshapes

a virtual space, one that can change its form, texture and

size. The discovery of this other space happens in a similar

way as we may see in digital imagery where things

seamlessly morph from one form to another, for example

an abstract object that resolves in a human head. The

surprise of it lies in the change of meaning that the object

suffers along the way. The moment of the transformation

is far more suggesting than the final result.

Fig.39 Installation Squeme

40

3.5.2. The Context of the Public Space

Although computers seem to be anywhere present in our

lives, people are not expecting to interact with one in the

middle of the street. What interests me about using public

spaces is that creates opportunity for spontaneous

interaction. This installation takes the form of an

architectonic intervention; in the sense it becomes part of

the building, rather that an Artwork exhibited in a building.

The purpose of this intervention is to engage passersby in

a habitable sonic instrument.

With this goal in mind I searched for possible spaces within

the University facilities, looking for an interior public space

that had pedestrian traffic throughout the day.

After finding and requesting two possible spaces, finally I

got the 2W 13th street access lobby at Parsons. This lobby

is a transitory public space. People are distributed from

here to the gallery, offices and elevators. Its physical

structure as well as its inhabitance and traffic patterns,

makes it a good place to host the interactive experience.

The project is conceived as a non-invasive installation

because it doesn't modify or alter the space in any physical

way. It consists of a demarked circle in the floor, which will

be the interaction zone. (Camera field of view). In the

surrounding space a set of four speakers will output the

sound.

3.5.3. The users experience

The following are desiderata for my installation, in terms

of what I expect to be the users experience from the

aesthetic and functional points of view.

0. The space is empty, people start accessing the building

for class. On their way to the elevator they hear sounds of

the city and other ambient sounds coming from the space.

Fig.40 Access Lobby at 2W 13th Street

Fig.41 Installation Squeme

41

A ring printed in the floor demarcates the center of the

sound projection.

1. While waiting in the elevator line, some of them come

back to the space to see what is this about. Others may

want to try in their way out.

2. When one person enters the space, the sound pans in

the opposite direction of his/her position, as this person

moves around the sound moves from speaker to speaker,

creating an interplay between listener and sound.

3. If more people enter the space, the panning is a result

of the average position in the space. If people are equally

distributed, the pan will stay in the middle. (Equal

amplification in Left and Right Channels).

4. As users move within the area, the sounds are morphed

by adding reverberance, or varying speed. At the peak of

movement the original sounds are not longer recognizable

as such, turning into reverberating patterns of ever changing

rhythms and echoes, behaving as musical elements.

5. When the space is emptied, a new soundtrack is triggered.

There are 4 different soundtracks with different

transformations. This sounds are called randomly so two

users may experience different sounds and its respective

transformations.

42

Chapter 4. Discussion and Analysis

4.1. The Music of Sound

Nowadays, everybody is exposed to the notion of digitally

manipulated images and sounds; they can notice them in

surround cinema, radio commercials, TV and pop music.

As computers penetrated people’s homes with the

availability of digital tools, they have been given the

possibility to store, display and manipulate images and

sound. This is especially common with digital pictures and

movies, where people have started to edit their own home

made movies and pictures. In some cases they also have

become aware of manipulation beyond the basic time-

based alterations, introducing changes such as color

correction or the addition of special effects. Why has the

manipulation of images become so popular and accessible,

while sound has not? I think this is not only due to our

nature as culture, which is highly visually oriented but also

because we associate sound with music, with all the

traditions and suppositions this impose.

In the past, interaction with sound was confined to the use

of very specific interfaces, and the developing of particular

skills for each instrument; therefore sound was confined

as a medium for musicians. For the rest of us, interfaces

such as the Violin, are scary, because the expectation is to

produce “music”. Sound have continued to be out of reach

for common people, maybe because is still seen as a skillful

medium, or simply because the development of sonic

interfaces has not addressed this audience. Moreover, the

experience of sound doesn’t even require to involve music

as we may think in traditional music, which is in the

aesthetical order of harmony and virtuosity. As Bill Fontana

has expressed: “ [...] music - in the sense of meaningful

sound patterns - is a natural process that is going on

43

constantly.” The discovery of such patterns is something

that happens in our minds. In this work I assume that that

state of mind can be encouraged through an interface that

reveals such patterns.

The work presented in this Thesis seeks to give its users

the experience of a literal “sound embodiment”, by attaching

sounds to their physical action, allowing them to modify

sounds, revealing significant patterns emanating from them.

In this sense, this work should not be considered as a

precision tool for the manipulation of sound, neither an

instrument for musical performance, but as a physical

experience, which ultimate goal is making them aware of

the act of listening itself. Is in this sense that this work

should be discussed and evaluated.

4.2. Challenges and Pitfalls

In the Chapter 3, Methodology, I presented five different

experiments, each one of them having a different spatial

situation and a different mapping to sound. They, as a set

of experiments, can be seen as a prototype for the actual

installation, because in each one of them I tested, in an

isolated way, a particular aspect of interaction with sound

in space.

It is difficult to create equivalent comparisons between

them. Also, as they required the use of a particular spatial

situation, they were set up for just short periods of time,

having myself as the only experiencer. I think is extremely

important to have test users, I am still looking forward to

get to that point, which would allow me to observe from

outside and introduce changes on the fly. Nonetheless, as

exposed before, I still can think of my experiments in terms

of experience, and tell in which degree they succeed in

creating a connection between a physical situation and

44

sound in an interesting way.

Although in each one of them such connection was present

in one or another way, I think “Spatial Scrub” did it better

in terms of surprise, because the connection was not

immediately evident, neither imperceptible, creating a

moment of uncertainty, which is resolved after a few seconds

of use. In that sense, “Triggers” is too immediately

understood, and even though playful, the experience is

short, depending on how playful is your mood at the

moment. In the other extreme is “accumulation” in which

the required time to get the connection is too long. “Traffic”

is the only one that doesn’t use body, but act in a similar

way as accumulation in the sense it can be a mirror of what

is happening in Space, but in a shorter period of time. In

a way, these two experiments succeed more as

“sonifications” of a situation in space rather than as

interactive experiences.

At this point none of them individually satisfy my

expectations and goals, but they have been extremely

useful as ways to understand the infinite details that have

to be considered and the limitations that need to be sorted.

After conducting those experiments I was able to identify

the required “temporality” for each situation, and see its

relation to the chosen mapping and spatial situation,

realizing that the ideal for the installation would be to have

two different mappings, with different temporalities each.

For example, it is important that users notice an immediate

response to their actions, but also that they discover further

influence in sound as they explore the space.

At this point I am working with a set of soundtracks, each

one of them will be introduced in the piece as users

incorporate, keeping the count of number of users. Their

positions will determine pan distribution, and their amount

of movement the amount of delay feedback.

It is my hope that this combinatory system of mappings

45

will increment the possibilities of exploration, extending

the experience in time and depth.

Chapter 5. Conclusion

5.1. Conclusions

The work of this Thesis starts in the questioning of the

cultural approaches we have to sound, in an attempt to

expand its notion as a primary element of perception and

especially as a physical phenomenon, using the example

of contemporary artworks that have approached sound

from spatial, cognitive, and semantic levels, finding special

influence in the work of Bill Fontana and his writing on

Sound Ecology. Based on this practical and theoretical

background, this Thesis has attempted to incorporate the

vocabulary of Sound Art into the realm of computer

multimedia, making use of its capabilities to display and

manipulate sounds in real time. This proposal is made

concrete by introducing and documenting a series of

experiments in physical interaction with sound. In such

experiments, aesthetic and philosophical goals have been

considered to give form to an interactive sound installation

that uses the context of the public space as an example of

the use of computer out of the usual context of the desktop

setup.

5.2. Future Directions

The final outcome of this work represents just a scratch in

the surface of the idea of physical interaction with sound.

46

One single piece gives the opportunity to explore one or

two aspects of the inquiry at a time, and the technical

constraints make necessary to reframe the ideas to the

actual possibilities. Amongst the unexplored avenues of

this work, I can mention the implementation of sound

spatialization, which gives the possibility of moving sounds

in space, through the use of grids of loudspeakers, through

which sound is distributed by special hardware controlled

by the computer. Also from the input side, the use of the

available computer vision software imposes its own limits.

There is an increasing number of artists developing their

own computer vision programs for their specific needs. In

my case I am using a general video-tracking tool, which

was designed for a wide range of purposes, and whose

stability and robustness is extremely delicate.

Another aspect I evaluated to implement in my piece was

the use of live sound input, through the use of microphones

located in different locations in the city, but that requires

broadcasting technology that is not easily available. I also

thought to use microphones listening to the space of the

installation itself, but in this case there’s the need to sort

the problem of the feedback happening in the proximity of

sound input and output.

The choice of the space for the installation has been made

based on the available facilities of the school. For sure, the

city is plenty of interesting public spaces that would give

different contexts to the piece and levels of spontaneity to

the interaction.

I also envision the use of this kind of interface for live

performance of simultaneous dance improvisation and

sound composition, in which sounds stay in a suspended

animation state until the performer unchains them by the

use of space.

47

Chapter 6. Bibliography

Books

Ando, Yoichi. Architectural Acoustics: Blending Sound Sources, Sound Fields, and Listeners. AIP

Press, 1998.

Barnes, Ralph M. Motion and Time Study. New York, NY. John Wiley & Sons, Inc. 1950.

Calvino, Italo. Six Memos for the Next Millennium. New York, NY. Vintage Books, 1993.

Lévi-Strauss, Claude. The Raw and The Cooked. New York, NY. Harper & Row, 1964.

Heim, Michael . The Metaphysics of Virtual Reality. OxfordUniversity Press, 1993.

Kaemmer, John E. Music in Human Life: Anthropological perspectives in music. Austin, Univerity of Texas

Press, 1993.

Kahn, Douglas. Noise, Water, Meat: A History of Sound inthe Arts. Cambridge, MA. MIT Press, 1999.

Kahn, Douglas. and Gregory Whitehead. Wireless Imagination: Sound, Radio, and the Avant-Garde.

Cambridge, MA. MIT Press, 1992.

Kobin, Minard . Sound Installation Art. Call number # ML1380. M55. 1996.

48

Leitner, Bernhard . Sound:Space. New York University Press.New York, 1978.

McLuhan, Marshall. Understanding Media: The extensionsof man. Cambridge, MA. MIT Press, 1994.

Moore, F. Richard . Elements of Computer Music. PrenticeHall, Engelwood Clifs, New Jersey, 1990.

Rush,Michael . New Media in Late 20th-Century Art. New York, Thames & Hudson Inc, 1999.

Journals, Magazines, Electronic Sources and OtherTheses.

Davies, Shaun. Sound in Space: Adventures in Australian Sound Art. February, 23. 2003

<http://www.autonomus.org/soundsite/csa/eis2content/e

ssays/p34_lost.html>

Eno, Brian. Interview by John Alderman posted in HotWired,June 5 1996.

<http://hotwired.lycos.com/popfeatures/96/24/eno.tran

script.html>

Fontana, Bill. Sonic Ecology. October,27. 2002<http://www.kunstradio.at/ZEITGLEICH/CATALOG/ENGLISH/

fontana-e.html>

Fontana, Bill. Sound as Virtual Image. December,8. 2002<http://www.resoundings.org>

Fontana, Bill. Resoundings. December,8. 2002<http://www.resoundings.org/Pages/Resoundings.html>

49

Kruger, Myron . Essays.

La Grow Sward, Rosalie . An examination of the MatematicalSystems Used in Selected Compositions of Milton

Babbit and Iannis Xenakis: A Disertation in partial fulfillmentof the requirements for the degree Doctor of Philosophy.Evanston, Illinois, 1981.

Leitner, Bernhard. Bernhard Leitner Website. January, 11. 2002 <http://www.bernhardleitner.at>

Levant, Yvie. Music as Image: Sound and Language.<http://interact.uoregon.edu/MediaLit/wfae/readings/L

evant.html>

Levin, Golan . Painterly Interfaces for Audiovisual Performance. MIT, 2000.

<http://acg.media.mit.edu/golan/thesis>

Rokeby, David . VNS Website.<http://www.interlog.com/~drokeby/vns.html>

Walters, John L. Sound, Code, Image. Published in EYE26,1997. Pages 24-35.

50

Appendices

Appendix A. Custom Interfaces Screenshots

A.1. Program for Accumulation

Stereo DelayFeedback Pacth

51

A.2. Program Interface for Aural Limbo

Audio On/OffControl

AmplitudeControl

Video InputControl

InputMonitor

MotionMonitor

Main AudioAmplifier

Audio RouterControl

Pan Process Effects Mixer ProcessMotion

Mix / Pan Monitor

RandomTrack Picker

Dataconversion

Peak Monitor

TrackSelector

AudioTrack01

AudioTrack02

AudioTrack03

AudioTrack04

Main Audio Mixer

ReverbPatch

ReverbPatch

FeedbackPatch

ReverbPatch

52

A.3. Program Interface for Traffic Report

53

Preliminar Sketch for A.L.

B. Suplemmentary Sketch

54

Colophon

This document was prepared with Macromedia Freehand

and Microsoft Word.The text of this thesis was set in

twelve-point MetaPlus Normal (1993)designed by Erik

Spiekermann. The titles were set in MetaPlus Medium Caps.

Mateo Zlatar

mateo@parsons.edu

2003