the foundations of spatial music
TRANSCRIPT
Music Aided Design:The Foundations of Spatial Music
Felix Faire BA Arch Dissertation - Click to open pdf
Acknowledgements:
I would like to thank Prof. François Penz for his expertise, guidance and humour; Amir Soltani for his enthusiasm and support;
And all the friends who volunteered to be experimented on:
Adam, Sohanna, Matt, Max, Lucy, Song, Jazz, Lauren, Katie, Mica, Iain, Immy, Fiona, Daniel, Elly, Miranda, Charlie, Sam, Freddy, Emma, Phoebe, Sophia, Alice, Kitty, Livvy
and Francois.
2
9353 words
2 3
Abstract.
Key words: Embodied music cognition, sound, space, interaction, association
This dissertation hypothesises that our perceptions of music and space are cognitively connected through embodied interaction with the world. These perceptual connections will be initially revealed by looking at our structures of language and speech, and then examining physical and audible precedents in the work of Iannis Xenakis, Oskar Fischinger and Mark Applebaum. The hypothesis will be further developed by exploring the shared neurological processes in listening to music and the navigation of spatial environments. The theoretical framework developed from the research is tested with a series of spatio-musical interactions tested by 24 volunteers. The results are discussed in relation to their application to musical composition, architecture and urbanism, speculating the potential for a new paradigm in Music Aided Design.
Contents.
Introduction:
Precedents:
Music Signification:
The Musical Score:
Animated Music:
Sound Gestures:
The Psychogeography of Music:
Experimental Research:
Pitch and Height:
Sonic Compass:
Sounds in Space:
Spaces of Sound:
Music and Motion:
Haptic Music:
Experiment Data:
Discussion:
List of Illustrations:
Bibliography:
Appendix:
5
7
9
13
15
18
21
22
23
24
25
26
27
29
30
33
34
36
4
“To most of us music suggests definite mental images of form and colour. The
picture you are about to see is a novel scientific experiment - it’s object is to convey
these mental images in visual form.” - Oskar Fischinger 1938
“It is now becoming clear through scanning technologies that the various senses
also share higher order cerebral networks, or perceptual supramodalities that
engage a crossover of sensory inputs from one sense to another”
- Harry Francis Mallgrave 2010
Over the past hundred years, increasingly refined theories of perception have
emerged. The top down philosophical theories are beginning to coalesce with the
bottom up scientific theories giving us greater insight into the study of experience1.
The relatively recent theory of Embodied Cognition has shed light on how the mind,
the body and the world form a conscious system of dynamic symbiosis. This recent
view of perception implies that experiences originally considered to reside solely
in the human mind such as music are in fact intrinsically connected to our bodies
and environment. Marc Leman describes this concept of embodied cognition:
“In contrast to dualism, the concept of mind is seen as an emerging effect of the
brain perceiving its own actions in relation to a physical environment.2 From that
perspective the subjective world of mental representations is not an autonomous
category but a result of an embodied interaction with the physical environment.”3 He
also suggests how our perception of music is tied to other embodied experiences:
“The multimodal aspect of musical interaction draws on the idea that the sensory
systems – auditory, visual, haptic as well as movement perception – form a fully
integrated part of the way the human subject is involved with music”. 3 This
dissertation will not focus on specific emotional processes involved in making and
listening to music. The primary aim of this discussion is to reveal and test how our
embodied interaction with the world develops perceptions of music and sound that
are fundamentally connected to our perceptions of space.
1 The history of phenomenology through to embodied cognition is described in further detail in Dourish (2004) p.102-1262 Ernelling and Johnson (2005)3 Leman (2008) p.13
Introduction.
5
To develop this hypothesis, I will examine the works of Iannis Xenakis, Oskar
Fischinger and Mark Applebaum. I will observe how these artists have been able to
transpose between sound, image and gesture by utilising and adapting the intrinsic
spatial inferences that music creates. These works will be examined based on the
principles of embodied cognition to find the theoretical basis for the vocabulary and
intersubjectivity of each medium. The discussion will also look briefly towards shared
neurological processes in music and space perception, particularly the narrative
and memory functions of the hippocampus necessary for both spatial navigation
and musical experience.
The research is consolidated with a series of spatio-musical interactions in the form
of volunteer tested experiments. The experiments use the theoretical framework set
out in the first chapters to actively engage the volunteer’s spatio-musical ability and
develop new kinds of intersensory interactions. I will then discuss the potential value
and opportunities given by this frame of mind towards architectural discourse.
For the purposes of this text I will use a broad definition of “music” as: any sequence
of sounds that are arranged within a larger temporal structure, and the term “sound”
to describe any audible form.
6
Listening to music is a vastly complex mental phenomenon that activates many
different parts of the brain. However, we can start to reveal the spatial aspects of
music perception by examining the ways in which we signify musical concepts.
The genealogy of music perception has consistently been related to the evolution
of language.4 Verbal communication, as well as refining the sophisticated pitch,
rhythm and pattern recognition necessary for music appreciation, uses higher level
representations to signify different experiences.
Linguistic descriptions of music (such as Hatten’s texts about Beethoven’s works
1994) rely heavily on metaphors to create intersubjective representations of melodies,
dynamics and timbres. These metaphors (such as Escalating, Swirling, Collapsing
and Sweeping) are not simply means to describe music but actively display how
we perceive music through the language of other experiences. George Lakoff and
Mark Johnson illustrate how our entire language is constructed from cross modal
inferences and that these associations form the essential rudiments in which we can
conceptualise and think about the world.5,6 They describe the metaphor of “affection
is warmth” as emergent from a child’s concurrent experiences of affection and warmth
in the embrace of a parent.6,7 This association between conceptual and sensory
inferences forms what Lakoff and Johnson call an “Embodied concept”.8 Our ability
to think is facilitated by both the brain and the nervous system7,8,9. The prevalence of
physical and spatial metaphors use to describe musical attributes and features, such
as dancing melodies, sweeping and swirling phrases, rising and falling scales etc.
allow us to see that our conceptions of music rely heavily on spatial understanding
and embodied concepts.
4 Brown (1999)5 Mallgrave (2010) p.175-180.6 Lakoff and Johnson (1980) p.255-2567 The neurological basis for metaphors in language is described by Lakoff and Johnson (1999) using Donald O. Hebb’s theory of concurrent sensory information and synaptic growth (Hebb 1949), This is explained fully in Mallgrave (2010) p.175-180.8 Lakoff and Johnson (1999) p.209 Tversky (2008)
Music Signification.
7
The words “high” and “low” used to denote audible pitch as well as vertical
position is a prime example of an audio-spatial metaphor that we can suggest a
possible etymology from an embodied perspective. Perhaps the most direct way
we interact with sound from the moment we are born is through the use of our
vocal chords. In producing or replicating different pitched sounds we must perform
certain biomechanical operations that are coordinated with our auditory sense.
Our vocal production of sound affects our spatial perception of pitch through the
perceived location of vocal noises relative to the ears. The position of the larynx
in the throat moves vertically to allow a higher or lower vocal range, this physical
change in position affects the perceived location of the produced sound. Singers
use the terms “Chest Voice”, “Middle Voice” and “Head Voice” to represent these
different positions as this is where the sound is felt to be coming from, this clearly
reinforces the perception that lower pitches are physically positioned below higher
pitches in space. This embodied explanation displays that the signification of
the words “high” and “low” to audible pitch is not an arbitrary selection of words
but is heavily reinforced by the spatial perception of our voices, catalysing this
association between pitch and height over time. This is one example of how an
embodied action produces cross modal inferences between seemingly disparate
sensory perceptions. This multimodal perception is consolidated and symbolised
as polysemous homonyms in language. This vertical mapping of pitch in space was
initially observed by C.C Pratt in 1930 after observing that the specific succession
of tones in a musical phrase can generate a sensation of vertical movement.10
These ideas have since been experimentally demonstrated by Lidji et al. 2007 and
Rusconi et al. 2006 and show the potential that audible properties can be mapped
successfully into spatial representations.11,12 I have highlighted just one example of
how spatial terms are used to describe audible characteristics in speech; however
the discipline of music has developed other means of signification to represent and
translate musical attributes in visual and spatial terms.
10 Pratt, C.C. (1930) 11 Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007)12 Rusconi E, Kwan B, Giordano B.L, Umilta C, Butterworth B (2006)
8
The musical score is a syntax and grammar used to translate music from a dynamic
temporal form to a static representation in 2 dimensions and vice versa. The 2
dimensions of space used in musical scores traditionally represent time read from
left to right in the x axis, and pitch arranged vertically on the y axis. The traditional
score clearly utilises the aforementioned spatial inference by signifying relative
pitches with vertical positions. The length of the note or rest is symbolised using
conventions of notation that must be learned in order to read the musical text.
Whilst the general form of instrumental music can be effectively transcribed with
these traditional graphical languages, the strict use of both spatial dimensions and
limited syntax of expression are abstracted from the experiential qualities of the
music13. This further asks the question whether there are alternatives to the standard
graphical musical signification that are able to capture the more experiential
qualities of music.
The composer Iannis Xenakis started to experiment with spatial representations of
sounds to reveal wider experiential truths within his music. His work Metastasies
originated with a graphical score that maintained the conventional use of
representational axis yet aimed to become more than just an instructional text for
performance (Fig. 1). Xenakis was particularly interested in the aspect of time in
music, the contemporary Einsteinian view had shown that time was relative to mass
and energy, in that any change in the contents of time would change time itself.
Xenakis aimed to emulate/explore this abstract idea in his music through the act of
capturing it in a static space.14 This negation of real time allowed the comprehension
of the whole in the instance of seeing the score. Xenakis used the example of
gunshots in a battlefield to describe the nonlinearity of these particular musical
13 Linguistic descriptions and conventional notations are critiqued by Bengtsson, I & Eggebrecht, H. (1973), and Leman, M. (2008)14 Hofmann, B. (2005)
The Musical Score.
9
Fig. 1 - Graphical glissandi from Xenakis’ Metastasis.
experiences; the exact order of each bullet fired is irrelevant as the resulting sound
as a whole will undoubtedly be that of “gunfire”. This concept of the musical form
being something other than a sum of its parts is particularly evident in Xenakis’
Mycenes Alpha (1978).The graphical score of Mycenes Alpha is comprised of many
(often straight) lines which collectively define larger, curved undulating masses (Fig.
2). The image that results is a collection of undulating conglomerate forms. The
graphical score was translated into audio by Xenakis’ own UPIC system15, whilst
the system translates each individual line into a note; the resulting cacophony of
all the combined sounds also generates the perception of an undulating dynamic
whole. Xenakis was fascinated by the form of the Hyperbolic Paraboloid (Fig. 3)
which too features this characteristic explored in his music: whilst every element
in a hyperbolic paraboloid is a straight line, the perceived whole is a twisting and
smoothly curved surface. These ideas were manifested spatially in the design of the
Philips Pavilion (Fig. 4). This is an example of how a perceived audial experience
has become effectively formalised to generate similar visual (and then spatial)
experiences. However, Xenakis was not the only proponent of the graphical score
and whilst his methods of representation reveal some ways in which the experience
of music can be perceived in static representations, others began to extend the
boundaries of how 2 dimensional spatial representations in fact become sonic
experiences in themselves.
Mark Applebaum’s Metaphysics of Notation (2010) rejects the standard
use of dimensional axis to represent specific attributes of sound and time.
His work instead relies solely on subjective associations and metaphorical
synaesthetic interpretations. This indeterminate notation does not specify any
instrumentation or temporal structure and thus, the individual performers of The
Metaphysics of Notation produce varying audial results. The musicians’ process
of transforming/analogizing the shapes and forms on the page into temporally
arranged soundscapes displays the brains ability to interchange sensory forms
to produce meaning. This ‘ability’ of multimodal chiasm is not simply a skill of the
conscious brain but is in fact a fundamental property of how the brain works.16
The scores themselves break so far from traditional scores that many of the
interpretations are completely unique, however some aspects of an intersubjective
15 The computerised UPIC system translated the vertical position of a mark into pitch and its horizontal length into duration.16 Mallgrave, H. F. (2010)17 Arnold, R. (2010)
10
Fig. 2Excerpt from Mycenes Alpha.
Fig. 3Example Hyperbolic Paraboloid.
Fig. 4Philips PavilionXenakis & Le Corbusier.
associative language are revealed by intrinsic similarities between the performers’
interpretations of certain passages of shapes.17 Some of these are simply carried
across from the performers’ traditional training of scores; sweeping and curving
lines become long notes changing in pitch respectively, and sections of repetitive
spots seem to unanimously represent individual notes or rhythmic musical events
in time (Fig. 5). Some similar responses that aren’t traditional of standard notation
are that the size of the elements are commonly attributed to volume, this is used in
the standard crescendo symbol but is also a basic perception that larger objects
generally make louder sounds when interacted with. Associations derived from
common perceptual experiences of actions and sounds in the world form a key
aspect of the sensory chiasm that allows the translation of these images into sounds.
Shape, texture and formal characteristics are also passed between our experiences
of sound and sight; they are intrinsic to the legibility of the graphical scores and
have been observed experimentally.
The Kiki/Bouba effect first observed by Wolfgang Kohler 1929 gives great insight
into how shape and formal characteristics of objects produce strong inferences to
types of sound.18 These synaesthetic connections are essential subconscious tools
that the musicians employ when translating the many shapes of “The Metaphysics
of Notation” into music. Kohler tested this idea by presenting subjects with two
shapes, a smoothly curved anamorphous blob and a jagged, angular star-like shape
(Fig. 6). When asked which shape was called “Kiki” and which was called “Bouba”,
95% of subjects assigned “Kiki” to the spiky form and “Bouba” to the rounded blob.
Speculatively this could be connected to the visual graphemes used to represent
the verbal sounds. For example the letters of the word “Kiki” feature sharper, more
angular lines than those in the word “Bouba”, which ubiquitously features rounded
letters. This implies that there may be some aspect of reading the shapes in the
graphical score using the musician’s functions of verbal language, developed in
parallel to their musical training. However, further work on sound-shape correlations
by Daphne Maurer in 2006 observed that the Kiki/Bouba effect was also prevalent
in toddlers as young as 2.5 years old.19 Whilst the effect was slightly less prominent,
the fact that the toddlers had yet to develop reading skills shows that the shape-
sound inference is primarily based on the audible sound rather than inferences back
to a visual representation of the words. Ramachandran and Hubbard hypothesised
18 Köhler, W. (1929)19 Maurer D, Pathman T & Mondloch C.J. (2006)20 Ramachandran, V. S., & Hubbard, E. M. (2001)
11
Fig. 5
Fig. 5 - Excerpt from The Metaphysics of Notation
Fig. 6Kiki. Bouba.
that this association is built through the use and shape of the mouth in creating
the sounds of the nonsense words: the angular figure “mimics the sharp phonemic
inflections of the sound kiki, as well as the sharp inflection of the tongue on the
palate.”20 From this we can see how our ability to translate shapes into sounds
is also derived from an embodied application of using our mouths and lips to
produce different sounds. However, many musicians have refined interactions with
sound (other than using their vocal chords) that they use to interpret Applebaum’s
scores. For these examples we must look at the coupling of body to instrument,
and then to the affordances of the instrument in terms of the corporeal motion used
to produce sounds. The limitations and facilitations of the instrument will invariably
affect the interpretation of the visual form. The bodily motion that changes the
pitch of a trombone for example is very different to that of playing a cello or flute;
as such, associations of movement across the score could yield different audible
results. The empathic association that allows the symbols to be read as sound
generating movements is another factor in interpreting these scores, and can be
generated through the very act of drawing the score. The signification of the musical
expression has emerged from a motion, such as a sweeping brush stroke or flick
that has imprinted its temporal history into a static symbol. The intentionality of the
artist in interpreting the symbol can be read through their empathic comprehension
of how the mark was made, this can then be manifested as corporeal musical
expression.21 We have seen how forms on a 2 dimensional plane can infer audible
responses, through convention, “embodied concepts” and even inferences to
motion and gesture that inform the instrumental output. This inferred motion and
mental animation of the symbols on the score as they are interpreted displays the
limitations of 2D graphical representations. Music is dynamic and it appears that our
spatial representation and cognition of music can be further augmented through a
dynamic spatial medium.
21 Corporeal imitation, articulation and expression are further investigated in Leman, M. (2008) p.103,123,141
12
The medium of film and in particular the animations of Oskar Fischinger address
some of the dimensional abstractions of graphical scores as the visual space and
animation occupy the same temporal dimension as the music itself. Whilst the
visual metaphors between shape and sound such as size to volume and shape
to timbre etc. are still used to great effect in Fischinger’s films, the possibilities of
movement are released by the animated medium. Musical objects now exhibit
speed and characteristics of movement that bring with them their own physical
and aural associations. Objects that exist within the dimension of time (unlike the
images of Applebaum and Xenakis) can swoop, shake, dart, fold in on themselves
and perform a wide variety of dynamic behaviours; as a result the animations get
closer to synthesising these ‘intuitive’ connections between sound and visual form.
Susannah Knights writes of Oskar’s animations: “Both media unfold and engage the
audio and visual senses over time, similarly capable of evoking an automatic sense
of narrative expectation. Secondly, both create an illusion of a gestalt, through the
audience psychologically connecting component parts which run in a temporal
sequence.”22 The dynamic properties of Fischinger’s films reveal how the viewer’s
expectations of imminent musical and visual activity can become synchronised to
further enhance the perception of an audio-visual gestalt. Our ability to expect future
events based on current conditions is based on observing recurring phenomena
to build an understanding of how objects behave. If we continue to use embodied
experiences and interactions as a basis for generating audio-visual associations
Animated Music.
22 Knights, S. (2012) p.13
13
we can start looking at our spatial expectations and see how these interact with our
musical expectations. For example, the resolution of objects in motion is for them to
fall downwards under gravity and come vertically to rest. We use this expectation
and understanding of the mechanics of our environment to be able to perform
actions, such as catching a falling ball. The same kind of resolution also appears
in chord sequences. Sustained chords and perfect cadences also create a tension
that is traditionally expected to be resolved. This effect of expected resolution is
evident in our melodies of speech, for example an upwards inflection at the end of a
sentence or melody creates a musical question that expects an answer. Susannah
Knights observes how Oskar exploits this sense of melodic narrative and physical
understanding: “although the first two strokes arch downwards on the audible
downbeats, the third, marking a quaver leading-note which sparks a short passage
of syncopation, arcs upwards, briefly creating a similar expectation of a resolution.”23
(Fig. 7) This expectation is not only built with the motions that precede it, the visual
and audible upward inflection evokes the same expectation of resolution as a verbal
question or object being flung into the air. We can see that, as static shapes can
elicit audible timbres, we also perceive motions and expected trajectories of both
dynamic physical objects and musical phrases.
23 Knights, S. (2012) p.17 describing Fischinger’s Studie No. 7 00:37 † Images are 20 frames of animation superimposed to show motion paths.
14
Downbeat 00:37
Fig. 7†
Downbeat 00:38
Expectation 00:39
Resolution 00:40
Fischinger’s films visibly correlate to our embodied perception of music’s motions
and dynamic patterns far more effectively than static graphical symbols. However,
both mediums achieve their audio-spatial inferences through embodied concepts
and common sensory experiences. Mark Applebaum’s piece Aphasia (2010)
combines hand gestures with music to create an audiospatial gestalt that is created
around the body itself. The medium of hand gestures not only moves the experience
of the music into a dynamic 3 dimensional realm but immediately generates an
implication of action to the audience. We use our hands as a primary means to
interact with the world, therefore a movement of a hand implies an action is taking
place in which there will be an effect. Unlike the dancing shapes and sprites
of Fischinger’s films, that are simply characterised by motional behaviours that
complement the music, the movements of the hands are perceived to have a direct
causal relationship to the music that they accompany. The electronically altered
vocal samples and abstract audio snippets used to create the audio of the piece
do not remind us of any traditional instrumentation, however the hand gestures
appear so perfectly matched to the audio that it is difficult to imagine which is
derived from the other, or if they were composed in parallel. Applebaum’s advanced
and multidisciplinary instrumental experience has given him a sophisticated
understanding of how motions and hand actions produce sounds with objects. His
search for new musical interactions has led to the invention of bespoke instruments
such as the “Mousketeer” (Fig. 8). Applebaum avoids the motional associations that
accompany traditional instrument sounds (such as the smooth sweeping motions of
the string instruments visualised to great effect in Fischinger’s Studie Nr. 7 01:22).
Instead he creates an original vocabulary of sounds and motions that is strictly
adhered to in the performance of Aphasia. The lack of a perceptible time signature
augments the audience’s perception that the sounds are causally linked to the
actions as they appear spontaneous yet temporally synchronised. By repeating the
sound gestures, the audience signifies the visible motion to the audible response
Sound Gestures.
15
Fig. 8 - Mark Applebaum playing his “Mouseketeer”.
and gradually learns the vocabulary of the piece as it progresses and new sounds
and motions are revealed. By producing the illusion that the motions actively create
the sounds, the actions that Applebaum uses appear to imbue the sounds with a
physical presence and spatial attributes. The piece opens with several disparate
beats to the chest, creating a short percussive knocking sound. After the sixth
instance of this action, instead of returning his hand to his lap his arm opens out
away from his body (Fig. 9 - Aphasia.mp4 - 00:34), the knock appears to echo
around this new space that the gesture has created. This perception localises the
sound of the knock to the space around the body that is perceived to be its source.
Similarly at (Fig. 10 - Aphasia.mp4 - 02:12) Applebaum strenuously pulls his hands
apart emulating a tension that is audibly reflected with a series of strained rubbery
squeaks. The perception that the sound is being tangibly stretched apart is a very
successful example of the sounds assuming a spatial and tangible form in the mind.
It must be noted at this point that a core aspect of embodied cognition is that for
any intentionality or experience to be embodied it must also be situated. “People’s
conceptions of space differ for different spaces and are a joint product of perception
and action appropriate for those spaces”24 This is the idea that behaviour is situated
in it’s architectural environment.25 A person might be more likely to dance if music is
played in a dance studio however would perhaps act differently in a church or office.
The gestures of Applebaum’s Aphasia are not only afforded by the biomechanical
trajectories of the human body but also influenced (or indeed generated) by the
wider spatial context and the perception and action appropriate for the environment
itself. The piece is primarily viewed on a blank stage or in front of a black
background as shown in the filmed version. There are no visual clues to the spatial
context in terms of architecture, however the piece is by nature a performance so
there is an intrinsic factor of ‘Audience’ that informs the spatial representations of the
music that Applebaum creates. This audience is either a real audience in the stage
context or the camera in the filmed context. This directional connection between
audience and performer changes how the gestures are interpreted, it would
certainly not have the same effect if viewed from behind. We can see how the study
of spatio-musical chiasm in a dynamic 3 dimensional space demands embodied
and situated means of analysis; the spatial context and relative perspectives of
people within the system deeply affect the resulting musical interpretation.
24 Tversky, B. (2008) p.20225 Behaviour and language are spatially situated but not environmentally determined Dourish, P. (2004) p89
16
Fig. 9
Echo Gesture.
Fig. 10Stretch Gesture.
By initially looking at examples of prior art we have seen how static and dynamic 2
and 3 dimensional spaces have been used to naturally mediate our perceptions of
sound, music and space. I have also described how these gestalts are formed in
the mind by utilising the approach of embodied cognition26,27 and the action oriented
ontology of music described by Leman. The precedents have not only examined the
spatial inferences of individual sounds, but also touched on how the sounds (and
indeed shapes and gestures) in the music relate to each other in time and generate
expectations, moments and contrasts that are attuned to our embodied experience
of the physical world. This aspect of music distinguishes it from being simply
sequences of sound or noise and deserves greater examination as it displays more
fundamental ways in which space and music are shared in the brain.
26 Tversky, B. (2008) 27 Shapiro, L. (2010)
17
Marvin Minsky’s description of music as an experience of learning begins to illustrate
how the temporal structures of music are experienced and formed in the mind. He
uses the words “sonata as teaching machine”28, describing the traditional musical
sections of ‘exposition’, ‘development’ and ‘recapitulation’ as phases of observation,
assimilation and accommodation in learning how to listen to the rest of the music29.
In a simple sense the gradual exposition of initial rhythms, harmonies and scales
start to build a psychological temporal framework of time signatures and key
signatures to which the rest of the music can develop a relationship. Minsky makes
the distinction that “Learning to recognise is not the same as memorizing”30 and that
whilst we rarely memorise entire pieces of music, something remains in the mind
that allows phrases to trigger memories of the subsequent bars. He illustrates the
similarities between our use of sight and of music in building mental maps: “How do
both music and vision build things in our minds? Eye motions show us real objects;
phrases show us musical objects. We “learn” a room with bodily motions; large
musical sections show us musical “places.” Walks and climbs move us from room
to room; so do transitions between musical sections. Looking back in vision is like
recapitulation in music; both give us time, at certain points, to reconfirm or change
our conceptions of the whole.”31 This musical view of space not only applies to
“rooms and objects” but also has resonances with the views of navigation in larger
scale psychogeography. In navigating a city or country route, we do not memorise
the exact spatial arrangement of our environment, however moments, contrasts
and landmarks along familiar paths cue us in understanding how our current space
relates to other nearby spaces (Fig. 11). Minsky suggests that the parts of the brain
that allow us to map spaces in this way are the same ones used to appreciate
musical form. This leads to the hypothesis that our experience of music uses similar
mental processes of learning and cognition to those of spatial navigation.
28 Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.2929 Stages of learning outlined in Piaget. (1970)30 Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.3031 Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.37-38
The Psychogeography of Music.
18
Fig. 11 - Front cover of Debord’s “Guide Psychogeographique de Paris” illustrates mapping of ‘the city’ based on the relationships between spaces, not actual positions.
Cupchick et al tested the neurological aspects of this hypothesis in their paper
“Shared processes in spatial rotation and musical permutation”.32 The paper found
a direct correlation between a subject’s performance at a spatial rotation task and
their ability to detect whether a tune had been played backwards or inverted etc.
Cupchick’s paper illustrates the mental chiasm used to manipulate physical and
musical objects; this is neurological evidence of a mediating process that facilitates
our experiences of the precedents in the first chapter. However, these tests are
primarily focussed on short length musical phrases and technical cognition such as
pitch processing. They do not illustrate the wider temporal experiences of narrative,
expectation and contrast that are intrinsic to musical experience. The hippocampus
has been described as the essential area of the brain that allows the perception
of temporal and spatial narratives. “In addition to its outstanding role for memory
and spatial navigation,33,34 the hippocampus has been suggested to be involved in
novelty detection 35,36. Hippocampal novelty detection is based on a comparison of
actual sensory inputs with stored stimulus patterns.37,38,40”. 41This is clear evidence
for Minsky’s learning analogy of music. For example listening to a preliminary bar of
music is a “sensory input” that becomes a “stored stimulus pattern” in memory, as
the next bar is heard this new “sensory input” is compared to the growing structure
of the previously heard bars by the hippocampus. This is also a neurological basis
for the aforementioned expectation generated in Oskar Fischinger’s Studie No.
7. This emerging structure grows as we are exposed to more of the music, and
the subtle or substantial harmonic shifts and changes push and pull us in varying
degrees of expectation42. The hippocampus in this sense facilitates our perception
of rhythm, flow, continuity and surprise. This narrative experience separates what I
have called “sound” from “music” in the same ways that a single frame of a film is
different to the film itself, or a photograph of a street is different to walking through it.
It is now known that taxi drivers and musicians both develop larger hippocampi due
32 Cupchik, G. C., Phillips, K., & Hill, D. S. (2001)33 Maguire, E.A. (2001)34 Ekstrom A. D, Kahana M. J, Caplan J. B, Fields T. A, Isham E. A, Newman E. L, Fried I. (2003)35 Knight, R. (1996)36 Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999)37 Gray J.A, Rawlins J.N.P. (1986)38 Strange B.A, Dolan R.J. (2001)39 Vinogradova, O.S. (2001)40 Kumaran D, Maguire E.A. (2007)41 Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B., & Cattapan-Ludewig, K. (2010) p.142 Huron, D. (2006)
19
to their practiced experience and reliance on both spatial and temporal narratives
and memory. 43,44,45,46 The increased hippocampal neuroplasticity clearly illustrates
the essentiality of the hippocampus in perceiving narrative and temporal contrast,
and thus in facilitating our higher level emotional responses to sensitively curated
temporal experiences.
This associative understanding of human navigation through space is currently in
common use by architects and urban planners to generate consistent elements
and extensions of urban form. The reason for this is not neurologically based
but has evolved as a refined human sensitivity to city making. Our experience of
the city is built on a framework of how spaces relate to each other. We learn the
relationships between roads, streets, alleys and open spaces as we move through
the city, these relationships are used as a subconscious mental framework that
we use to comprehend and operate within other cities and spaces. By examining
the processes and functions of the hippocampus, we have seen that music and
dynamic navigation of space share this cognitive process, yet the parallels between
sensitive urban design and musical compositions are rarely consciously used.
Many metaphors such as “harmony” and “rhythm” are used to discuss and design
architectural proposals and their relationships to their context, yet few actively
engage the processes of musical composition to augment the embodied experience
of the city that they are developing. Perhaps by further understanding the deeper
complexities of music and its interaction with mind, body and place we could use
Music Aided Design (MAD) to develop deeper and more emotionally engaging
experiences of architecture and the built environment. This specific area of spatial
musicality deserves a lot more research but we are beginning to see how musical
experiences and spatial experiences are not as distinct as previously thought.
43 Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D. (2000)44 Gaser, C., & Schlaug, G. (2003)45 Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010)46 Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B. & Cattapan-Ludewig, K. (2010)
20
We have examined how recent research and development in neuroscience, music
cognition and embodied cognition provide a framework for a spatial interaction
based ontology of music, this has been illustrated through basic observations
about how our human physiology affords particular experiences of the world and
also by examining similarities in neurological processes. A series of 6 interactive
experiments were developed in order to directly expand upon the multimodal
experiences described in the first chapters. The information and precedents
discussed suggest that we should also be able to form a symbiotic perception of
space and music that uses our knowledge and experience of both to produce an
intuitively interactive gestalt.
The 6 experiments featured 3 focussed on “sound” and 3 focussed on “music”
and were followed by a short questionnaire about the different interactions and the
user’s experiences (Fig. 12-13). The experiments were tested on students ranging
in musical ability and spatial awareness and were placed in a category of either
“Architect” (architecture students with less than 5 years musical experience),
“Musician” (non architecture student with over 5 years of musical experience),
“Musician Architect” (Architecture student with over 5 years musical experience)
or “Non Musician” (does not study architecture and has less than 5 years musical
experience). The main body of the questions required an X on a spectrum between
Agree and Disagree; the data of each questionnaire was calculated by measuring
the distance on the spectrum and dividing it by the length to create a coefficient of
agreement (0 represents strong disagreement and 1 represents strong agreement).
(Film clips of the experiments are on the Data DVD at the back)
Experimental Research.
21
Fig. 12 - Experiment Setup.
Fig. 13 - Speaker placement and Camera field of view.
Body Tracking CameraParticipant
Custom Software
Surround Audio
The first experiment the user was introduced to aimed to test the prevalence of the
embodied ‘Pitch is Height’ metaphor previously examined by C.C Pratt et al.10 The
user was effectively presented with an invisible plane of sound that was consistently
at arm’s length in front of them (Fig. 14) (PitchAndHeight.mp4); as the user extended
their arm out into this field an audible sine wave was produced. The frequency of
the sine wave was directly related to the user’s vertical hand position such that when
the user moved their hand up, a higher pitch was produced and vice versa a lower
position produced a deeper tone. The distance the user reached out correlated to
the sound’s volume, emulating the effect of force or pressure in creating the sound.
The pitch and volume would correlate to their hand’s movement and position in
real time. After exploring the relationships between sound and space for 1 minute
the relationship was reversed such that moving the hand towards the ground now
produced a higher pitched tone. The users were given another minute to play with
this interaction and were asked which interaction they found more intuitive and
natural. 92% of the subjects agreed that the deeper tones belonged below the
higher pitches in space and the average coefficient of agreement was 0.84. One
“Architect musician” subject felt very strongly that the higher pitches intuitively
belonged below the deeper pitches in space, it was soon made evident that this
particular user had played the cello for 12 years from the age of 6, the embodied
action involved in raising the pitch of a cello indeed requires a lower hand position
in space. This demonstrates the hypothesis that relationships between sound and
space are forged through corporeal interaction and experience.
Pitch and Height.Experiment 1
22
Fig. 14 - “High” and “Low” Frequency Sound Plane Visualised.
The second experiment used a similar mechanic of reaching away from the body to
produce a sine wave pitch yet the responsive area was arranged radially around the
center of the body as a ring at arm’s reach. The Highest pitches pointed towards the
front of the lecture room and the lowest pointed towards the rear of the lecture room.
The users were able to tap into this audial sound compass by extending their hands
out and feeling for direction with the sound (Fig. 15). The users were not told how
the sounds were oriented but were given 2 minutes to explore their affective audible
environment. The users were then blindfolded and disorientated by revolving on
the spot a number of times. The blindfolded, disoriented users were asked to
simply point towards the lectern at the front of the room. Despite only experiencing
this interaction for 2 minutes, 88% of the subjects were able to successfully and
instinctively use the sound as an audible sense of orientation and point in the correct
direction. This effectively shows how sound information can be imbued with spatial
orientation information and that even with only 2 minutes of exposure to the system
the users were able to use their augmented senses of proprioception and hearing to
make confident spatial decisions about their orientation in the space.
Sonic Compass.Experiment 2
47 The speakers used for this experiment completely surrounded the user such that no sense of stereo-graphic orientation could be used
23
Fig. 15 - Reaching into Sound Compass.
The third experiment featured three
differenty pitched string instrument sounds
that were hidden within the room. The
sounds were modelled as spheres that
emitted a continuous note when occupied
by a hand. The volume of the sound
increased as the hand moved towards
the center of the sphere. The users had
to explore the space at different heights
until they made contact with one of the
sound volumes (Fig. 16). This interaction
relies solely on the connection between the
aural sense and the sense of the body’s
position through proprioception and sight.
The users were surprised to discover the
sounds at first but went on to discover the
others using various methods of scanning
the space (Fig. 17). 63% of the subjects
experienced the sounds they interacted with
as 3 dimensional forms in space, however
the perception of the actual shape of the
sounds in the air ranged from the accurate
description of spheres through to columns,
discs, walls and cubes. The experiment
was modelled such that the sounds were
not only different heights but varied in
radius between 30cm and 50cm, 67% of the
subjects perceived the change in physical
size of the sounds in the space. 83% of the
subjects were able to correctly relocate the
position of the invisible sounds in the room
and could even play them as chords using
both hands.
Sounds in Space.Experiment 3
Fig. 16 - Searching and Finding Sounds (Film Stills)
This ability to map the experience of a sound into a 3 dimensional position through
bodily interaction illustrates the potential of sensory chiasm when perceiving spatial
objects and sounds. The aural connections between sound and shape discussed in
the first chapter imply that the perceived shape of the sound would be affected by
the timbre and texture as well as just the interactive spatial experience. In the future,
this experiment could be expanded by using different types of sound as well as just
different positions, pitches and sizes to test if perceptions of shape and form were
influenced by audible timbre.
Fig. 17 - Searching and Finding Sounds (Sounds visualised)
24
24
Fig. 19 - Correct Zone Diagram
Fig. 18 - Visualisation of 3 Continuous Sound Zones
Fig. 20 - Incorrect Zone Diagram
Photoshop image here!
Experiment 4 tests how the presence of a modulating sound field can change a
user’s perception of a room as they move through it. The room was divided into 3
overlapping zones of music, such that as the inhabitants moved around the room,
the audible characteristics of the space changed with their position (Fig. 18). The
3 pieces of music had their tempo synced with each other such that there was a
smooth transition between each piece when the users moved between zones. The
users explored the room to discover the new audible significance of the different
spaces, most of them developed new preferences for particular positions in the
room. 96% of the users found that the music significantly changed the character of
the spaces, and 96% were able to perceive the spatial presence of a modulating
sound field. The experiment also caused some users to jump from one zone to
another at the end of a bar to further control the music as well as passively observe
the changes. The questionnaire required the users to draw a map of the sound
spaces on a floor plan of the room. 67% of the users accurately illustrated the
spatial layout of the music zones (Fig. 19). The remaining participants produced
less accurate diagrams of how the room was divided, yet still picked up some
attributes of the zones such as the fact that the zones overlapped (Fig. 20). Only
54% of all architecture students we able to successfully illustrate the zone divisions,
whereas the other 46% had close representations but generally overcomplicated
the spatial arrangement; this suggests that the architect’s refined skill for thinking in
plan had not necessarily expanded their ability to map the sounds into space but
perhaps added extra complexity from other factors in the music. However, 70% of
the musicians were able to draw the zones in plan with near perfect accuracy. This
is evidence that instrumental experience builds and reinforces strong associations
between corporeal action in space and audible reaction.
Spaces of Sound.Experiment 4
25
Fig. 21 - Walking through music, forwards and backwards.
The 5th experiment aimed to bring the temporal dimension of music into a physical 3
dimensional representation that the participants could interact with. The experiment
was developed by again looking to metaphor as a source of crossmodal chiasm,
particularly the words we use to talk about direction and time. We use the terms
such as “looking forward to something”, “backwards”, “behind” and “thinking
ahead” to talk about linear directions in time. Tversky describes the embodied
meaning of the word “forward” as the space in front of our bodies, or our primary
direction of movement.48 Using the principle that this linguistic metaphor has both
temporal and spatial associations, the experiment was developed such that any
forward motion of the user (such as walking) would cause the music to play; if
they stopped, the music stopped, and if the user walked backwards, the music
would also play backwards (Fig. 21). Through interacting with the space the users
were allowed to feel the alignment of their motion with the dimension of time and
therefore have the ability to move through or control it at will. 96% of the participants
agreed that the music moved in the same direction as their body with the strong
average agreement of 0.88 (MusicAndMotion.mp4). A common response to this
experiment was that the subjects tried to control the tempo of the music with how
fast they moved, this feature was omitted due to the technical challenges and time
constraints, however future versions of this experiment could be greatly enhanced
by allowing the subject to travel along the temporal dimension of music at their own
speed within the space as well as simply direction. These comments and statistics
of engagement clearly illustrate that we perceive both music and spatial motion as
dynamic temporal structures.
Music and Motion.Experiment 5
26
48 Tversky, B. (2008) p.201
Music, sound and noise are commonly experienced
as ubiquitous qualities of space. The sixth experiment
aimed to generate the perception that the music
playing in the space could be physically held and
manipulated with the hands and body. To generate
this perception of a physical and interactive sound
field, the actions that altered the sound were
designed to harness common sensory experiences
and associations to create an immediately intuitive
interface. The experiment focussed on the character
of muffling and dampening sounds and the spatial
factors that create this audible change. The sensory
ubiquity of sound and noise in the everyday
spaces we inhabit has built a comprehensive
perceptual understanding of how sounds act
within and permeate different environments. For
example the acoustic responses of sound in large
and small spaces have immediately recognisable
characteristics of reverb. This relationship also
allows artificial or reproduced sounds with greater
reverberation times to strongly elicit senses of
spatial scale without visual cues (such as the
aforementioned echo gesture in Aphasia). A similar
common experience of sound behaviour is the
natural muffling of sounds in enclosed spaces.
Obstacles between a listener and a sound source
will absorb and reflect different frequencies of the
audio depending on the form, depth and materiality
of the obstacle. For example, building walls tend to
transmit lower audio frequencies as their wavelength
exceeds the depth of the obstacle; as such, music or
noise from a neighbouring room with poor acoustic
Haptic Music.Experiment 6
Fig. 22 - Screenshots of “Haptic Music” program.
insulation will appear muffled and
will lack the definition of the higher
frequencies. From this common
experience, muffled sounds carry
with them associations of spatial
conditions such as being enclosed
in a box, underwater or hidden
behind a wall. The experiment
aimed to harness this spatial
association of muffled sounds
to generate a perception of a 3
dimensional source of pure music
that could be held in the hands,
gathered, enclosed, squeezed,
compressed and even released
back into the room. The interaction
was devised such that as the hands
of the user pressed together a low
pass frequency filter would act on
the music playing in the room, the
cutoff frequency of the filter was
directly proportional to the proximity
of the hands providing the illusion
that the hands were enclosing the
source of the sound. At the point at
which the hands came completely
together, the music had been
reduced to a deep, compressed
throb. If the hands were unclasped
rapidly the music appeared to have
“escaped” and all frequencies were
returned to normal. To capture the
music again the subject had to
extend their arms out to “gather”
the audio and then the music
was able to be compressed and
released at will.
Fig. 23 - Collecting and Releasing Music (augmented video stills - HapticMusic1.mp4).
27
In this experiment the subject was asked to stand in the middle of the space whilst
the music began playing through the speakers. The subject was then simply asked
to “compress the music” but was given no visual or gestural cues of how this
might be done. The verbal cue to “compress” the sound elicited various different
responses from the subjects, some extended all their limbs before crouching
and curling into a ball (Fig. 24), others raised their hands up then pressed them
together towards the floor and most of the subjects brought their hands together as
if squeezing a large ball (Fig. 25-26). These initial responses indicate the success
of the application of a spatial metaphor to the manipulation of the music. As the
subjects realised that their initial motions affected the music in an expected manner
they almost immediately began to interact with the music as an object or field in
space. 96% of the subjects agreed that the interaction between body and music
felt natural with a strong average agreement of 0.89. The experiment was not only
devised to test whether a spatial sense of music could be achieved through intuitive
interaction but also to test how the temporal structure of the music would influence
the subject’s choice of movement once the perceptual coupling between action and
effect had been made.
Haptic Music (Continued).
Fig. 24 - Crouching to hold the music (Video Stills).
Fig. 25 - Compressing to hold the music (Video Stills).
The music the subjects manipulated was chosen specifically to encourage
participation; the music featured a strong underlying rhythm with a catchy repetitive
chord progression and melody.49 The contemporary music used had very clear
dynamics, build ups and releases at the end of regular 16 bar phrases. The
repetitive nature of the beat and chord progression was chosen in order to introduce
the subjects to the temporal framework of the music quickly such that they were able
to comprehend and act on their expectations of change by reacting with appropriate
body movements. By compressing the sound and reducing the music to the
bassline and underlying beat, the subjects were able to contribute to the dynamic
changes of the music, generating a literal tension and anticipation of resolution that
is felt both musically and physically. This is a similar effect to the stretching action
and audio used in Applebaum’s Aphasia. 83% of the musician subjects agreed that
the structure of the music influenced their choice of motion, yet only 40% of the non-
musician non-architect group found this to be the case. The musicians had a clearly
refined awareness of corporeal action with regards to musical structures. Whereas
the primary aim of the non-musicians’ actions was simply to examine the effect of
the resulting tone of the sound. 100% of the Architect musician group agreed that
the structure of the music influenced their choice of movement with an average
agreement of 0.93. This clearly illustrates a relationship between the subject’s
musical and spatial experience and their ability to think in 4 dimensions.
49 The music was of a popular genre to the subject demographic yet not well known enough to distract the subject from the spatial interaction
Fig. 26 - Experiment photograph with overlaid illustration of sound.
28
28
Table 1. - Experiments Overview.
• 92% of participants placed higher pitches above lower pitches.
• Some musician’s instrumental experience had reversed this placement.
• Spatial associations to pitch are built through embodied actions.
Experiment 1:
Pitch and Height
• 88% of participants orientated themselves using proprioception and hearing.
• Participants could functionally use spatial arrangement of sounds to make decisions.
• Sounds are be imbued with spatial meaning over time and experience.
Experiment 2:
Sonic Compass
• Spatially interacting with sounds can perceptually give them a 3 dimensional form.
Experiment 3:
Sounds in Space
• Audible attributes of a space significantly change our experience.
• Sound fields can be perceived spatially (and accurately described).
Experiment 4:
Spaces of Sound
• We feel time as moving forwards.• Music can become a navigable string of
movement in space.
Experiment 5:
Music and Motion
• We can feel the spatial qualities of music and sound with our ears.
• We can use previous understanding of how sound works in space to manipulate sound and music as physical objects.
Experiment 6:
Haptic Music
29
Expe
rimen
t >1
25
Que
stio
n N
umbe
r >1
23
45
67
89
1011
1213
M/F
Mus
icia
nSu
bjec
t1
23
45
6Su
bjec
t Num
ber v
160
5960
4560
6060
585
5958
5760
F1
Arch
itect
ure
43
26
51
253
5430
1126
4657
475
4456
5455
M1
Phys
ics
45
62
31
360
6030
1545
1560
4317
5815
3050
M1
Engi
neer
ing
61
32
54
450
3760
6032
5757
600
5160
5951
M0
Arch
itect
ure
56
32
41
546
030
3047
4346
460
4430
6042
M1
Hist
ory
65
12
43
657
5730
5240
4030
5042
1554
5052
F1
Chem
istry
53
24
61
759
5959
5959
4242
592
3347
4546
M0
Arch
itect
ure
56
42
31
860
510
4660
5960
600
6030
5916
F0
Arch
itect
ure
62
34
51
934
1050
6058
6060
600
6011
6060
F0
Hist
ory
of A
rt4
53
26
110
059
3044
6060
6060
060
6060
60M
1Ar
chite
ctur
e5
64
23
111
5047
3426
3456
3347
257
5757
58M
0Ar
chite
ctur
e4
36
52
112
6060
6060
4360
6060
060
6060
60F
1Ar
chite
ctur
e2
56
34
113
1537
4517
3747
3640
1545
4050
54F
1Ar
chite
ctur
e2
51
46
314
5858
5858
2155
5530
056
5656
56M
0Ar
chite
ctur
e2
45
36
115
608
5052
3050
5050
650
6060
60F
0Ar
chite
ctur
e5
64
32
116
5460
6024
4060
6060
060
3060
60F
0Ar
chite
ctur
e5
64
13
217
6045
6060
6060
4560
752
6060
60M
1Ar
chite
ctur
e2
64
53
118
3760
5459
5959
5959
1242
5960
60F
1Th
eolo
gy5
61
23
419
6060
6060
4057
5960
060
4660
59F
0Ar
chite
ctur
e5
36
24
120
5745
442
4760
5755
457
1154
50F
0PP
S5
46
13
221
5545
4712
4060
6058
555
1352
30F
0Hi
stor
y5
64
13
222
5545
5554
5444
4440
1245
5356
45M
1En
gine
erin
g5
63
24
123
5052
1330
5052
5360
560
5853
53F
0PP
S3
46
52
124
5540
340
3060
4949
1048
5253
53F
0Bi
olog
ical
Sci
ence
s5
46
32
1
Aver
age
rank
of p
refe
renc
eAg
reem
ent C
oeffi
cien
t0.
840.
770.
680.
710.
740.
880.
870.
880.
100.
850.
750.
920.
876
45
31
2Sa
mpl
e siz
eAr
chite
cts
0.94
0.79
0.79
0.80
0.66
0.91
0.87
0.89
0.02
0.89
0.80
0.95
0.85
86
45
32
1M
usic
ians
0.86
0.77
0.64
0.61
0.75
0.69
0.82
0.79
0.24
0.69
0.74
0.86
0.84
64
63
25
1Ar
chite
ct M
usic
ians
0.65
0.87
0.85
0.75
0.87
0.96
0.87
0.93
0.09
0.92
0.93
0.96
0.98
56
31
45
2N
on E
ither
0.84
0.64
0.39
0.61
0.75
0.97
0.93
0.94
0.08
0.93
0.48
0.91
0.82
56
45
12
5
% A
gree
men
t92
8863
6783
9696
964
9675
9692
Succ
essf
ully
orie
ntat
ed
Arch
itect
s10
088
8875
7510
010
088
010
075
100
8863
Mus
icia
ns10
083
3350
8383
8310
017
8383
8310
083
Arch
itect
Mus
icia
ns60
100
8080
100
100
100
100
010
010
010
010
040
Non
Eith
er10
080
4060
8010
010
010
00
100
4010
080
60
34
6Ge
nera
l
Table 2. - Experiment Data.
The results of the experiments clearly show that we have the ability to perceive
both music and sound as spatial forms. We naturally use metaphors to mediate
meaning between musical and spatial experiences yet rarely actively engage our
experiential knowledge of one with the other. This ability is not reserved to those with
neurological synaesthesia, it is built through our everyday experience. It is clear that
we have moved beyond the question “Are music and space cognitively connected?”
to “How can we apply this unified understanding of embodied perception to
design?” and particularly “How can the diciplines of music and architecture
augment and inform each other?”
I have demonstrated that by using emerging technologies and systems, both
spatial and audible presence can be combined to augment a user’s navigation and
experience of architecture. The results of the experiments speculate that these forms
of navigable music could become an architecture unto themselves. In experiments 3
and 4 the subjects used words such as “columns”, “corridors”, “walls” and “rooms”
to describe the sounds they interacted with. This shows the potential to create
an immaterial architecture made of music that is still spatially navigated. I have
illustrated that the instrumental experience of the participants heavily influenced
their audio-spatial ability; however the architecture itself now has the potential to
become the instrument that musicians and non-musicians alike will inadvertently
play (and thus practice) with their bodies as well as their eyes. This implies that even
non musicians could become musicians of space as they are exposed to this type of
architecture.
Discussion.
30
30
The experiments have not only shown that we perceive spatial inferences in sounds
but also that music can have an affective role in our spatial interaction when the two
are coupled. This affective quality of music could extend the idea of an architecture
made of sound to become active navigational sound architecture for the blind,
where sounds are naturally imbued with spatial characteristics. Ascending scales
of notes could have clear spatial connotations to obstacles such as stairs, or
volumes of sounds with different textures could be used to generate perceptions
of proximity to various objects in a room. The refined musical ability of temporal
pattern recognition could be attuned to the motions of the body as it moves through
space. Phrases of music can now actually become “musical places” which develop
and change as we move from one to another. This combines both uses of the
hippocampus discussed and could be used as a framework to distinguish different
spaces and their relationships to each other. In the same way as Minsky described
musical learning, this system of navigable music would perceptually build into
intuitive wayfinding devices that cue memories of what lies ahead as inhabitants
move through them. These are just a few examples of embodied music concepts
that could contribute to a much larger vocabulary and temporal grammar of intuitive
architectural sound elements.
The implementation and curation of these elements need not be functional and
could become new kinds of architectural ‘detail’. Experiment 4 illustrated how our
visual perceptions of space are overridden by the audible characteristics, in the
same way that film soundtracks significantly enhance or alter the visual narrative.
Pockets of music can now be placed in buildings that change and augment the
qualities of that particular space. The study of how and what music makes us feel is
a larger avenue of research but the architect has the opportunity to change the way
their buildings are listened to as well as seen. The music and the architecture have
the potential to be composed together: Michael Gondry’s film for “Star Guitar” by
the Chemical Brothers is a fantastic example of how music changes our perception
of images, and also how features of space could be heard and augmented in
this way (StarGuitarExperiment.mp4). Every element in the landscape of the train
journey is highlighted by a different sound; attention is drawn to every element in
the scene as it is revealed and audibly signified in the music. Architecture could
use similar technology to literally become audible music in this way, or alternatively
(and perhaps more interestingly) sequences of spaces could be designed with
these principles of Music Aided Design to actively stimulate the same temporal
continuities, contrasts and empathic motions that move us so deeply in music.
31
There is the opportunity to apply this musical intelligence during the appraisal
and brief stages of architectural design. During the contextual analysis of a
site, particular attention could be given to the details, rhythms and forms of the
surrounding spaces and buildings, analysing the developments and changes of
a dynamic perspective through the area. Musical composition could be used to
understand how these rhythmic and dynamic textural changes lead to a dynamic
architectural language of the area as well as just a static, stylistic or materialistic
language. The gap (or silence) of the undeveloped site in the routes through the
area could be conceptually developed based on the expectations, contrasts and
rhythms (to name a few) of the temporal moments either side. This could even be
achieved by composing the music of the context and experimenting with potential
surprises or continuities in the currently empty site. The approaches and departures
from the building could therefore be actively engaging the visual and spatial aspects
of the site with a larger musical narrative. This would continue inside the building
where languages of ‘exposition’, ‘development’ and ‘recapitulation’ (taking sonata
form for example) could be used to create gradual changes, moments of surprise,
steady continuities or moments to reconfirm or change our perceptions of the whole.
The results of this discussion also have implications in other disciplines. Similar
technologies and bespoke software could allow sounds to be actively sculpted,
stretched, smoothed, textured and manipulated as spatial objects. The sounds
could even be passed between multiple collaborators or moulded and developed
together. This could have huge potential in creating music through dance, and also
in electronic sound design and musical composition.
The precedent studies, neurological studies and experimental studies have all
revealed that they may be explored in far greater depth than I have done here.
The experiments deserve a greater sample size to critically examine the changes
between refined spatial awareness and musical ability. The study would also benefit
from exploring the spatial responses to many different types of music and sound
in each experiment. To extend this research, empathy and empathic involvement
with space, motion and music will also need more comprehensive analysis. This
could even extend into how mirror neurons activate the sensorimotor system just by
perceiving the sounds and motions of other objects and people. This dissertation
has been a proof of concept in what clearly has the potential to be a much larger
practical and theoretical pursuit in the disciplines of both architecture and music.
32
32 33
Illustrations & Tables.
Fig. 1: Excerpt from Xenakis’ Metastasis graphical score.
Fig. 2: Excerpt from Xenakis’ Mycenes Alpha graphical score.
Fig. 3: Hyperbolic Paraboloid, Straight lines creating curved forms, diagram.
Fig. 4: Xenakis and Le Corbusier’s Philips Pavilion, photograph.
Fig. 5: Excerpt from Mark Applebaums Metaphysics of Notation graphical score.
Fig. 6: Kiki and Bouba diagrams.
Fig. 7: Superimposed frames of Oskar Fischinger’s Studie Nr. 7 showing expectation and resolution.
Fig. 8: Mark Applebaum playing his bespoke Mouseketeer instrument, photograph.
Fig. 9: Aphasia Echo gesture, film stills with illustrations.
Fig. 10: Aphasia Stretch gesture, film stills with illustrations.
Fig. 11: Guide Psychogeographique de Paris book cover, illustrates relative space mapping, photograph.
Fig. 12: Experiment Setup. Photograph with labels.
Fig. 13: Sketch plan of room and speaker layout, drawing.
Fig. 14: “High” and “Low” frequency sound plane, Illustrated film stills.
Fig. 15: Reaching into Sound Compass, Illustrated film frame - SonicCompass.mp4.
Fig. 16: Searching and finding sounds in space, film stills - SoundsInSpace.mp4.
Fig. 17: Searching and finding sounds in space, Illustrated film stills
Fig. 18: Visualisation of different audible music zones within the space, illustrated photograph.
Fig. 19: Participants correct zone diagram, drawing.
Fig. 20: Participants incorrect zone diagram, drawing.
Fig. 21: Walking backwards and forwards through the music, illustrated film stills.
Fig. 22: Screenshot of motion camera and “Haptic Music” program, screenshot.
Fig. 23: Collecting and releasing music, film stills - HapticMusic1.mp4.
Fig. 24: Crouching to collect the music, film stills.
Fig. 25: Compressing to hold the music, film stills.
Fig. 26: Tangible music, Illustrated photograph.
Table 1: Experiments overview.
Table 2: Experiment numerical data.
Arnold, R (2012). “Theres No Sound In My Head”, Online documentary about the
performances of Mark Applebaum’s “Metaphysics of Notation” https://vimeo.com/14469188
Bengtsson, I & Eggebrecht, H. (1973). Verstehen: Prolegomena zu einem semiotisch-
hermeneutischen Ansatz. In P. Faltin & H.-P. Reinecke (Eds.), Music und Verstehen:
Aufsatze zur semiotischen Theorie, Asthetik und Soziologie der musikalischen Rezeption.
Cologne: Volk.
Borries, F (2007). Space Time Play: Computer games architecture and urbanism, Germany:
Birkhauser
Brown S (1999). The “Musilanguage” Model of Music Evolution. In Wallin NL, Merker B, and
Brown S, eds. The Origins of Music. Cambridge MA: MIT Press.
Clark, A (2008). Supersizing the mind, embodiment, action and cognitive extension, Oxford:
oxford university press
Cupchik, G. C., Phillips, K., & Hill, D. S. (2001). Shared processes in spatial rotation and
musical permutation. Brain and cognition, 46(3), 373-382.
Debord, G. (1957). Guide psychogéographique de Paris. Édité par le Bauhaus imaginiste.
Douglas, K.M., and Bilkey, D.K. (2007). Amusia is associated with deficits in spatial
processing. Nat. Neurosci. 10, 915–921
Dourish, P (2004). Where the Action Is, the foundations of embodied interaction, Cambridge
MA: MIT Press.
Bibliography.
34
Ekstrom A.D, Kahana M.J, Caplan J.B, Fields T.A, Isham E.A, Newman E.L, Fried I. (2003).
Cellular networks underlying human spatial navigation. Nature 425:184–188.
Erneling, C. E., & Johnson, D. M. (Eds.). (2005). The mind as a scientific object: between
brain and culture. Oxford University Press: USA.
Fishwick, P. A, (2006). Aesthetic Computing, Cambridge MA: MIT Press
Gaser, C., & Schlaug, G. (2003). Brain structures differ between musicians and non-
musicians. The Journal of Neuroscience, 23(27), 9240-9245.
Gray J.A, Rawlins J.N.P. (1986). The hippocampus, Comparator and buffer memory: an
attempt to integrate two models of hippocampal function. Plenum: New York, p159–201.
Hatten, R. S. (1994). Musical meaning in Beethoven: Markedness, Correlation, and
Interpretation. Bloomington: Indiana University Press
Hebb, D. O. (1949). The Organisation of behaviour: A Neuropsychological theory. New York:
John Wiley and sons.
Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B., ... & Cattapan-
Ludewig, K. (2010). Musical training induces functional plasticity in human hippocampus.
The Journal of Neuroscience, 30(4), 1377-1384.
Hofmann, B (2005). Spatial Aspects in Xenakis’ Instrumental Works. In Makis Solomos,
Anastasia Georgaki, Giorgos Zervos (ed.), Definitive Proceedings of the “International
Symposium Iannis Xenakis” (Athens, May 2005), www.iannis-xenakis.org, October 2006
Howard, D (2009). Sound and Space in renaissance venice, China: Yale University Press
Huron, D. (2006). Sweet Anticipation: Music and the Psychology of Expectation. Cambridge,
MA: MIT Press.
Husserl, E, (1928). Lectures on the Phenomenology of Internal Time Consciousness,
Bloomington : Indiana University Press
Köhler, W (1929). Gestalt Psychology. New York: Liveright.
34
Knight, R (1996). Contribution of human hippocampal region to novelty detection. Nature
383:256–259.
Knights S, (2012). Visual Music: the films of Oskar Fischinger. Cambridge: History of Art
dept. Dissertation
Kumaran D, Maguire EA (2007). Which computational mechanisms operate in the
hippocampus during novelty detection? Hippocampus 17:735–748.
Lakoff, G., & Johnson, M. (1980). Metaphors we live by (Vol. 111). London: Chicago.
Lakoff, G. J., & Johnson, M. M.(1999). Philosophy in the Flesh. The Embodied Mind and Its
Challenge to Western Thought. New York: Basic Books.
Leman, M, (2008). Embodied Music Cognition and Mediation Technology. Cambridge MA:
MIT Press
Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007). Spatial associations for musical
stimuli: A piano in the head?. Journal of Experimental Psychology: Human Perception and
Performance, 33(5), 1189.
Maguire, E.A. (2001). Neuroimaging, memory and the human hippocampus. Rev Neurol
Paris 157:791–794.
Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D.
(2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc Natl
Acad Sci USA 97: 4398-4403.
Mallgrave, H. F, (2010). The Architects Brain. UK: Wiley-Blackwell
Maurer D, Pathman T & Mondloch CJ (2006). The shape of boubas: Sound-shape
correspondences in toddlers and adults. Developmental Science 9 (3): 316–322
Minsky, M (1981). Music, Mind and Meaning. In Clynes M, ed. Music, Mind and the Brain:
The Neuropsychology of Music. Plenum: New York
Piaget, J. (1970). Piaget’s theory.
35
Pratt, C.C (1930). The spatial character of high and low tones Journal of Experimental
Psychology. 13 (1930), pp. 278–285
Merleau-Ponty, M. (1996). Phenomenology of perception. Motilal Banarsidass Publishe.
Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia - a window into perception,
thought and language. Journal of Consciousness Studies, 8(12), 3-34.
Rasmussen, S. E. (1964). Experiencing architecture (Vol. 2). Mit Press.
Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010). Musical training, neuroplasticity and
cognition. Dementia & Neuropsychologia, 4(4), 277-286.
Rusconi E, Kwan B, Giordano B.L, Umilta C, Butterworth B (2006). Spatial representation of
pitch height: The SMARC effect. Cognition, 99 (2), pp. 113–129
Sacks, O, (2008). Musicophilia; Tales of music and the Brain, New York: Knopf
Shapiro, L, (2010). Embodied Cognition. USA: Routledge
Stewart, L., & Walsh, V. (2007). Music perception: sounds lost in space. Current Biology,
17(20), R892-R893.
Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999). Segregating the
functions of human hippocampus. Proc Natl Acad Sci U S A 96:4034–4039.
Strange B.A, Dolan R.J. (2001). Adaptive anterior hippocampal responses to oddball stimuli.
Hippocampus 11:690–698.
Tversky, B. (2008). Spatial cognition: Embodied and situated. The Cambridge handbook of
situated cognition, 201-217. USA: Cambridge University Press.
Van Gelder, T, (1998). The dynamical hypothesis in cognitive science, Behav. Brain Sci. 21,
615–628. doi: 10.1017/S0140525X98521735.
Vinogradova, O.S. (2001). Hippocampus as comparator: role of the two input and two output
systems of the hippocampus in selection and registration of information. Hippocampus
11:578–598.