Since 1959, when electronic music was estab-lished as a new way of music composition, the

rules of traditional music performance and enjoy-ment have changed to include space, motion,and gesture as musical parameters. For example,musicians are often located somewhere otherthan the stage—sometimes even in the audi-ence—and where the music will be performedoften influences compositional thinking. Loud-speakers move sound through the space at vary-ing speeds (based on other musical parameters).In addition, the development of live electronics—that is, computers applied to real-time processingof instrumental sounds—has allowed space as amusical instrumental practice to flourish.

Electro-acoustic technologies let composersexplore new listening dimensions and considerthe sounds coming from loudspeakers as pos-sessing different logical meanings from thesounds produced by traditional instruments.

Medea, Adriano Guarnieri’s “video opera,” isan innovative work stemming from research inmultimedia that demonstrates the importanceand amount of research dedicated to soundmovement in space. Medea is part of the Multi-sensory Expressive Gesture Application project(http://www.megaproject.org). MEGA aims todevelop mediated experiences and supportingtechnologies through cross-fertilization of expres-sive communication theories and expressive andemotional content by modeling nonverbal inter-action with multisensory interfaces in sharedinteractive mixed-reality environments. As thisarticle shows, real-world artwork and perfor-mance enhance the research’s scientific aspects.

Among Medea’s intentions, derived from artis-tic and musical suggestions and needs, is a desireto establish an explicit connection betweensound movement and expressiveness and to

show how engagement can be enhanced acousti-cally in multimodality environments—for exam-ple, through the motion of sound throughvirtual spaces. Whereas sound positioning andmovement have seldom been used in concert set-tings, the ear has great detection capabilities con-nected to its primary role (a signaling device forinvisible or unseen cues); music is now trying toput these capabilities to creative use.

Space as a musical parameterComposers have long considered space in

music; however, aside from some work by Gio-vanni and Andrea Gabrieli (the antiphonal choirsin Venice’s churches) and Wolfgang AmadeusMozart (the three orchestras playing simultane-ously in Don Giovanni), current trends in con-temporary music are developing space as afull-blown musical parameter. For example, Karl-heinz Stockhausen’s Gruppen (1955–1957) fea-tures three orchestras scattered throughout theaudience: “All the processes of this music havebeen determined by the spatial disposition of thesound, by its direction and its movement.”1

New electro-acoustic technology widely usesthe spatial movement of sound. A number ofworks may have contributed to the ideas behindMedea. For example, Pierre Boulez wrote Répons(1984) for ensemble and live electronics. Theaudience surrounds the stage, and six soloists sur-round the audience. The soloists are amplified,causing the live-electronics setup to perform dif-ferent kinds of sound movement.

Another work, Luciano Berio’s Ofanim (1988),uses sound movement in space to build unusualorchestral and counterpoint functions. Theinstrumentation is peculiar (two 10-elementwoodwind–brass ensembles, two percussionsgroups, two 20-element childrens choirs, and one

4 1070-986X/04/$20.00 © 2004 IEEE Published by the IEEE Computer Society

Artful Media Doree Duncan SeligmannAvaya Labs

Amalia de GötzenUniversità di

Verona

Enhancing Engagement inMultimodality Environments bySound Movements in a Virtual Space

singer), so the spatialization helps avoid sound-masking phenomena in the dense passages andclarify the counterpoint among parts.

As a last example, Adriano Guarnieri’s Passionesecondo Matteo (2000) features the composer’sconcept of sound movement. Rather than usingphysical gesture, live-electronic processingenhances and clarifies the gesture in this piece(for instance, the trumpets’ amplitude envelopecontrols some space location parameters toenhance the musical meanings of crescendo anddiminuendo).

Sound motion through space is an establishedtradition in much of contemporary music, muchof which exploits multimodality to enhance per-formance. Music—specifically sound motion inspace—conveys expressive content related to per-formance gestures. Although composers haveinvestigated the connection between music andemotion for traditional parameters, such asintensity, timbre, and pitch, spatialization is stilla new research path. The use of space as a musi-cal parameter in an expressive dimensionrequires new paradigms for interaction, mappingstrategies, and multimedia interfaces based onreal-time analysis and synthesis of expressivecontent in music and gesture.

Researchers have developed and applied mod-els and algorithms for extracting high-level, qual-itative information about expressive content toreal-time music and multimedia applications.Analysis of expressive gestures has sought toextract expressive information from humanmovements and gestures and to control the gen-eration of audio content depending on the analy-sis. Figure 1 diagrams the link between physicaland spatial movement. Medea offers real-worldexamples of such multimodal extensions.

Medea: Exploring sound-movementparameters

Medea is a large musical work for soli, orches-tra, choir, and live electronics that Guarnieridescribes as a video opera. Video opera refers tothe full scope of the underlying musical vision,which consists of metaphorical references tovideo devices such as zooming and editedsequences. Medea features Guarnieri’s intuition

about the expressive matching of instrumentaland sound movement gestures, each reinforcingthe other to produce a rich and powerful mes-sage. In scoring Medea, the composer explicitlydescribes the expressive mapping between per-former and sound movement gestures.

Gesture generally refers to dance movementsand sometimes to specific body expressions. How-ever, gesture can also be considered a structurewith definite semantics defined into an abstractspace, as it is here. For example, a musical phraseis a gesture that expresses an emotion using onlymusical parameters, where music is the abstractspace. In dance, the connection between musicand body movement is explicit. Choreographycan better express a given musical work in a bal-let and vice versa: the music carries emotionalstates that the body movement expresses.

Many psychological studies about music andemotion maintain that music can represent thedynamic properties of emotions, such as speed,strength, and intensity variations. In particular,music can express these emotions by associatingaffinities between musical characteristics andbody movements.2 Imberty states that the bodyand the emotions share some kinetic tension andrelease schemes, so movements and emotionalstates form a coherent set and gesture becomes acommunication channel.3 Sound movement inspace may be a link through which to explorethis connection between gestures and emotions.

Psychoacoustic test modelAs a researcher with the MEGA project, I per-

formed a psychoacoustic test to evaluate soundmovement as a musical parameter and determinewhether it has potential expressive content forthe listener. The test sought to establish a per-ceptive paradigm with which to construct amodel for use in a musical context. This prelimi-nary study focused on three parameters that areconsidered sound movement’s basic compo-nents: speed, articulation, and path. I designedthe parameters to have only two semanticallyopposed values in each example:

❚ Speed—two opposed time values represent thesound persistence on each loudspeaker

5

Ap

ril–June 2004

Movement Gestureanalysis

Expressivecontent

description

Expressivespatialization

model

Spatialmovement

Figure 1. Connection

between physical and

spatial movements.

❚ Path—continuous circular/discontinuousrandom

❚ Articulation—the degree of crossed overlapbetween the loudspeakers’ amplitudeenvelopes performs a staccato or legato sound

Eight of the audio stimuli produced featuredwhite noise while another eight examples fea-tured a harmonic continuous sound (a loopedtrombone sample). I intended to use these stim-uli to evaluate the influence of timbre on the per-ception of movement. To collect data, I askedlisteners (both musicians and nonmusicians) touse a dimensional approach. A dimensionalapproach uses representations of emotional statesdrawn from the psychological tradition, which isan alternative to the categorical tradition, inwhich listeners choose from a list of adjectivesdescribing emotional states.

I organized the bidimensional spaces to estab-lish two dimensions: valence and arousal. Theclearest common element of emotional states is

that the subject is materially influenced by feel-ings that possess a valence: they’re centrally con-cerned with positive or negative evaluations ofpeople, things, or events. Furthermore, emotion-al states involve a disposition to react in certainways. I rated arousal states simply in terms of thesubject’s disposition to take some action ratherthan no action at all.4

Figure 2 shows the valence-arousal space withthe adjectives composing the Russel circumplexmodel used to evaluate emotions.5 Using the col-lected data and its statistical analysis,6 I draw sev-eral conclusions:

❚ The values obtained for each timbre indicatethat the same movements occupy a similarspace location in the valence/arousal space,revealing that listeners identify differentexpressive intentions with coherence. How-ever, the valence component often alsodepended on the sound’s pleasantness.

❚ Articulation and speed are stronger parametersthan path, even though the Russel model fea-tures a specific emotion related to path.

❚ Speed is strongly related to the arousal axis,and it’s the most coherent parameter in sub-jective appreciation involving examples withboth timbres.

❚ Path type appears to be a weak parameter,with different path types (circular or random)evoking the same emotion in many listeners.Speed and legato/staccato, on the other hand,appear to be strong parameters, directlyrelated to emotions.7

A relationship between movements and per-ceived emotions is evident. Table 1 shows thedesign of a model to use this data in musicalapplications.

Artistic performanceMedea’s score cites sound spatialization as a

fundamental feature of the opera. The musiciansin the hall should be considered a sonic body liv-ing among the audience to create a sort of gravi-tational center for the trumpets located on eitherside of the audience. The presence of tromboneswith their gestural posture becomes a centralexpressive feature.

Medea’s score begins with articulated instruc-tions related to space. These instructions encom-

6

IEEE

Mul

tiM

edia

Artful Media

ExcitedAstonished

Delighted

GladHappy

Pleased

SatisfiedContent

At EaseRelaxed

Sleepy

MiserableDepressed

FrustratedAnnoyed

Distressed

TenseAngry

Afraid

AlarmedAroused

SadGloomy

BoredDroopy

Tired

CalmSerene

AROUSAL

VALENCE

Figure 2. The Russel

circumplex model in

the valence/arousal

space.

Table 1. Emotions from the Russel circumplex model

associated with each movement.

Emotion Speed Articulation PathFrustrated, distressed Slow Staccato Discontinuous

Depressed, unhappy Slow Staccato Continuous

Tired, sleepy Slow Legato

Serene, relaxed Fast Staccato Continuous

Happy, glad Fast Legato Continuous

Excited, delighted Fast Legato Discontinuous

pass four sound-reinforcement modes:

❚ Transparent reinforcement, in which delays areused to keep a natural perspective of soundpositioning;

❚ Celluloid, which is composed by randommovement among the four stereo front speak-ers and the front central cluster;

❚ Rain, which is created by fast random move-ment on specific speakers above the public;and

❚ Olophonic, in which controlling volumethrough low-pass filtering producesmovement simulation.

The live-electronics performers executed allmovements and transformations following theconductor and the score (each instrument has itsown spatialization modes, and the score markseach transformation and movement precisely),with all sound movements except for those com-ing from the trombones having been predeter-mined.

A real-time application developed on theEyesWeb (http://www.eyesweb.org) platformprocessed the sound movements. Camurri et al.developed the application by interconnectingseveral software modules for real-time expressivegesture analysis and synthesis.8 Guarnieri defined11 modalities for the trombone movements. AnEyesWeb patch controlled the trombones’ ran-dom space movements, a Webcam capturedmovements derived from trombone players’ ges-tures, and the EyesWeb program digitallyprocessed them to provide each movement’sspeed parameter using a gesture-speed mapping.This method’s functionality derives from a trans-lation of the image bitmap in terms of speed:intense instrumental gestural activity (rockingoff) leads to a large bitmap variation and there-fore to high speed, while reduced gestural activi-ty corresponds to a moderate movement speed.

Figure 3 shows one of the four trombone play-ers during Medea’s premiere performance. In thecontext of Medea as a video opera, the expressivematching between physical movement (by theinstrumentalist) and sound movement throughspace clearly plays the metaphorical role of a“camera car,” where the public enters the physi-cal movement through the movement of sounditself. It should be noted that all of these consid-

erations are subtle and subliminal as is most of theexperience of listening to contemporary music.

ConclusionAs this experience shows, instrumental ges-

tures can be translated into spatial movementseven though the translation is fairly complicat-ed in its conception, from both technologicaland musical viewpoints. This type of indirectmapping of sound and gestural movement canmore powerfully engage the public, who mightbe able to distinguish between different kinds ofmovements performing different kinds of expres-sive intentions.

Medea is simply one prominent example ofhow contemporary music can use multimediatechnologies to convey languages and experi-ences that continue to grow in complexity andscope. To attract the public, concert events mustsharply differentiate themselves from sophisti-cated home theatre experiences. In this context,all multimodal matching between senses will play

7

Ap

ril–June 2004

Medea: A Video OperaIn Adriano Guarnieri’s Medea (2002), the public is completely enveloped

by a multitude of independent sound sources. Some are generated by actu-al players (trumpets, bass flute, and trombones), while others come fromthe live-electronic processing of the orchestra, choir, and singers project-ed over a 16-loudspeaker system that is distributed around and above theaudience. Sound motion in space isn’t narrative in this music (that is, spacedoesn’t represent “where the sound comes from”) but it’s intended ratherin terms of musical trajectories and paths.

Figure 3. Trombone

during the premiere

performance of Adriano

Guarnieri’s Medea.

a stronger role in future musical works, and play-ing with the public’s senses will indeed becomean integral part of musical composition. MM

References1. K. Stockhausen, Musik in Raum, Dumont-Verlag,

1958.

2. M. Budd, Music and the Emotions: The Philosophical

Theories, Routledge, 1992.

3. M. Imberty, “Entendre la musique,” Semantique

Psychologique de la Musique, Dunod, 1979.

4. R. Cowie et al., “Emotion Recognition in Human-

Computer Interaction,” IEEE Signal Processing, vol.

18, no. 1, Jan. 2001, pp, 32-80.

5. P. Juslin and J. Sloboda, Psychological Perspectives

on Music and Emotion, Music and Emotion—Theory

and Research, Oxford Univ. Press, 2001.

6. A. de Götzen, “Expressiveness Analysis of Virtual

Sound Movements and Its Musical Applications,”

Proc. XIV Colloquium on Musical Informatics (XIV

CIM 2003), Centro Tempo Reale, 2003, pp. 64-68.

7. S. Gabrielsson and E. Lindstrom, The Influence of

Musical Structure on Emotional Expression, Music,

and Emotion—Theory and Research, Oxford Univ.,

2001.

8. A. Camurri, B. Mazzarino, and G. Volpe, “Analysis

of Expressive Gesture: The EyesWeb Expressive Ges-

ture Processing Library, Gesture-Based Comm. in

Human-Computer Interaction, A. Camurri and G.

Volpe, eds., LNAI 2915, Springer-Verlag, 2004.

Amalia de Götzen is a PhD student in the Department

of Informatics at the Univerità di Verona. Contact her at

degotzen@sci.univr.it.

Contact Artful Media editor Doree Duncan Seligmann at

Avaya Labs, 666 Fifth Ave., 11th floor, New York, NY

10103; doree@avaya.com.

8

IEEE

Mul

tiM

edia

Artful Media

http://computer.org

IEEE

AND A PPLICAT IONS

EDIT

OR

IAL

CA

LEN

DA

R

January/February: Emerging Technologies* This issue covers the Siggraph 2003 Emerging Technologies exhibit,

where the graphics community demonstrates innovative approaches tointeractivity in robotics, graphics, music, audio, displays, haptics,sensors, gaming, the Web, artificial intelligence, visualization,collaborative environments, and entertainment. *Bonus CD-ROM ofinteractive demos included with this issue.

March/April: Haptic RenderingThis special issue will examine some of the latest advances on hap-

tic rendering and applications, while looking forward to exciting futureresearch in this area. Articles will present novel research results andinnovative applications of that take advantage of the haptic interactionsensory modality.

May/June: Computer Graphics InnovationsThis issue covers an array of computer graphics innovations. It pre-

sents new developments in virtual reality, novel approaches in visual-ization, and innovative computer graphics applications. The range oftopics highlights the usefulness of computer graphics for everyone.

July/August: Point-Based GraphicsPoint primitives can be efficiently rendered and displayed in high

quality, and novel algorithms for advanced shape modeling and digitalgeometry processing are entirely based on point samples. These effortshave spawned a new field called point-based computer graphics. Thisissue presents novel research results in point-based computer graphics.

September/October: Visual AnalyticsVisual analytics is the formation of abstract visual metaphors that

provide the basis for human interaction and discourse about complexmultidimensional data. Effective use of visual analytics can potentiallyalleviate the ongoing information overload problems. This issue pre-sents articles that describe innovative results in this area. Particularattention is paid to technology transfers and applications in various dis-ciplines and communities.

November/December: Smart Graphics— Touch-Enabled Interfaces

People use tactile and force feedback to identify objects and to explorethe environment around them. This sensory cue is also used in manipu-lating objects. Therefore, an ideal man–machine interface for under-standing virtual environments and visualizing complex information shouldenable the user to feel and orient objects and to manipulate data usingtouch-enabled interfaces. This issue examines the senses of touch as a com-munication medium, in addition to graphical display. Specifically, it pre-sents different approaches to designing touch-enabled interfaces forvarious applications including scientific visualization, model design andediting, virtual prototyping, 3D painting, and medical training.

2004