Simple mappings, expressive movement: a qualitative investigationinto the end-user mapping design of experienced mid-air musiciansDom Brown, Chris Nash and Tom Mitchell

Creative Technologies Laboratory, University of the West of England, Bristol UK

ABSTRACTIn a New Interface for Musical Expression (NIME), the design of the relationshipbetween a musician’s actions and the instrument’s sound response is critical increating instruments that facilitate expressive music performance. A growingbody of NIMEs expose this design task to the end performer themselves,leading to the possibility of new insights into NIME mapping design: whatcan be learned from the mapping design strategies of practicing musicians?This research contributes a qualitative study of four highly experienced usersof an end-user mapping instrument to examine their mapping practice. Thestudy reveals that the musicians focus on designing simple, robust mappingsthat minimize errors, embellishing these control gestures with theatricalancillary gestures that express metaphors. However, musical expression ishindered by the unintentional triggering of musical events. From thesefindings, a series of heuristics are presented that can be applied in the futuredevelopment of NIMEs.

KEYWORDSMid-air interaction; action–sound mapping; gesturalinteraction; experiencedusers; new interfaces formusical expression;embodied cognition; musicalmetaphors

Introduction

The constant improvement of computationalsystems and sensors used in music technology(McPherson, Jack, and Moro 2016) is enablingthe rapid and cheap development of New Inter-faces for Musical Expression (NIMEs). A popu-lar interaction technology used in instrumentdesign is gestural mid-air interfaces that usethe direct bodily movements of the performerfor musical control (Mitchell, Madgwick, andHeap 2012; Mainsbridge 2018; Otondo 2018),and it is becoming increasingly important tounderstand how these types of instrumentscan be designed to effectively facilitate musicalexpression.

In the NIME literature, one of the mostimportant expression facilitating facets of a

NIME is the relationship between the controlinterface and the sound response, or mapping(Rovan et al. 1997; Hunt, Wanderley, and Para-dis 2003), and it is argued that an instrument’smapping can make the difference between amusical toy and an instrument that supportsvirtuosic performance (Hunt, Wanderley, andParadis 2003). In most NIMEs, the mapping iscreated by its designer and remains a constantfeature of the instrument; however, there is agrowing body of NIMEs that expose the intrica-cies of mapping design to the instrument’splayers, referred to as ‘end-user mapping’ (Mal-loch, Sinclair, and Wanderley 2007; Fiebrink,Trueman, and Cook 2009; Brown, Nash, andMitchell 2018). As well as providing new oppor-tunities for creativity that are unavailable to

© 2018 Informa UK Limited, trading as Taylor & Francis Group

CONTACT Dom Brown dom.brown@uwe.ac.uk Creative Technologies Laboratory, University of the West of England, Bristol BS161QY, UK

DIGITAL CREATIVITYhttps://doi.org/10.1080/14626268.2018.1510841

traditional instrumentalists, this approach leadsto new questions about mapping design,namely: what factors influence the designchoices of a musician and what can be learntfrom their practice to inform the design ofnew instruments? To explore this question,this research examines the mapping practiceof four experienced NIME musicians, who, forseveral years, have made a mid-air, end-usermapping instrument their primary musicalinstrument. Studying this group providesinsights into the characteristics and processesof expressive mapping design in experiencedNIME musicianship.

Previous literature argues that simple map-pings do not provide engaging interactions,and that complex mappings, which use combi-nations of one-to-many, many-to-one, andmany-to-many mapping relationships, areneeded to facilitate expressive musical inter-action (Rovan et al. 1997; Hunt and Kirk2000; Dobrian and Koppelman 2006; Momeniand Henry 2006). This study explores this argu-ment: the four experienced NIME musiciansunanimously select simple rather than complexmapping strategies, enabling them to incorpor-ate theatrical ancillary movements into theirperformances. The musicians’mapping practiceprioritises the expression of personal aesthetics,using visual metaphors to enhance and commu-nicate musical and lyrical meaning. Ourfindings show that simple mapping strategiescan facilitate expressive music performance,and that expressive mappings are not relianton complexity. These findings inform a seriesof design heuristics that are presented, whichcan be applied to the development of futureNIMEs.

Background

Mapping design

Unlike acoustic instruments, NIMEs do not relyon a physical connection between a musician’sactions and an auditory response. As such, the

permutations of how a player’s actions generatea musical response are limited only by the affor-dances of a system’s interface and the con-straints of the auditory synthesis to which theinterface is connected. This highly-configurablearrangement presents a large sandbox withinwhich instrument designers can work, and thedesign of mappings remains a focal point ofNIME research (Hofmann et al. 2017; Scurto,Bevilacqua, and Françoise 2017; Visi et al.2017).

Traditionally, mapping design has been con-sidered a series of relationships between actionand sound parameters: one-to-one, one-to-many, many-to-one, and combinations of thelatter two: many-to-many (Hunt, Wanderley,and Paradis 2003). This conceptualisation hasbeen used to design several NIMEs (Goude-seune 2002; Momeni and Henry 2006; Paine,Stevenson, and Pearce 2007), and it has beenargued that more complex mappings lead tomore expressive instruments (Rovan et al.1997; Hunt, Wanderley, and Paradis 2003).Mapping design literature has also focussed onhow designers implement mappings: Explicitmapping requires the designer to explicitlydefine the relationship between action andsound parameters (Hunt and Kirk 2000),while implicit mapping employs machine learn-ing to perform the actual mapping, while theinstrument designer provides the algorithmwith training examples (Fiebrink, Trueman,and Cook 2009; Francoise 2015). Much of therecent mapping literature has focussed on theuse of implicit techniques (Caramiaux et al.2014; Scurto, Bevilacqua, and Françoise 2017),and it is argued that implicit mapping affordsdesigners more complexity and therefore moreexpressivity as the specifics of mapping designare abstracted. Fiebrink et al. (2010) argue thatimplicit systems are better suited to experimen-tal exploration of musical control than design-ing mappings for a specific purpose, wherepredictable and reliable responses to a user’sactions are favoured over any serendipity ofmachine learning misclassifications.

2 D. BROWN ET AL.

While some mid-air performance systemshave drawn from existing gestural disciplinessuch as Soundpainting (Van Nort 2018),there is a growing body of NIME mapping sys-tems that enable the end-users themselves todefine the connections between actions andsound. In such instruments a software interfaceallows the musician to implement connectionsbetween gestural parameters, from humaninput devices such as cameras, to auditory par-ameters, typically MIDI or Open Sound Con-trol (OSC). These instruments use eitherimplicit machine learning methods such asthe Wekinator (Fiebrink, Trueman, and Cook2009), or explicit methods such as MyoMapper(Di Donato 2017) or Glover (Brown, Nash, andMitchell 2018). These systems give musiciansthe ability to express their personal aestheticsin action-sound relationships (Fischer and Gir-gensohn 1990).

Musical expression in NIMEs

The text/act paradigm of musical expressionholds that expressive musical performance isachieved through the process of deviatingfrom the given musical ‘text’ in the performanceof the musical ‘act’ (Taruskin 1995). Here, thelevel of control intimacy available plays animportant role in determining the expressivepotential of a musical instrument, with expres-sive performance only achievable through virtu-osity, which in turn is facilitated by complexmappings (Dobrian and Koppelman 2006).

However, expression cannot be consideredonly as the level of control intimacy availablein an instrument. Musical expression is a pro-cess of communication between the perfor-mer, their audience, and the composer,situated within a wider cultural and socialcontext (Gurevich and Treviño 2007). Assuch, an instrument’s expressive potentialcannot be considered in and of itself; the con-text in which the instrument is being playedmust be considered.

Metaphor in NIME mapping

The use of metaphor in interaction design is anapproach that exploits a user’s existing knowl-edge of the world to help them learn new tech-nologies. Much research has argued thatmetaphor should be used in the design ofNIME mappings to provide more intuitive,transparent interactions for both audiences andperformers (Fels, Gadd, and Mulder 2002; Wes-sel and Wright 2002).

A typical understanding of metaphor is whenone domain, the target, is described through theterminology of another, the source (Lakoff andJohnson 1980; Blackwell 2006), for example,the metaphor TIME IS A RESOURCE gives usthe phrase ‘time is running out’.

The use of metaphor in HCI is widely basedin Lakoff and Johnson’s Conceptual MetaphorTheory (Blackwell 2006). Conceptual MetaphorTheory (CMT) argues that our bodily experi-ences of the world are used to understand allabstract concepts (Lakoff and Johnson 1980).This is evidenced in language, for example, inthe metaphor ARGUMENT IS WAR, theabstract concept of arguing is described in thebodily experienced domain of war and conflict:‘His claims are indefensible’, ‘Her position isweak’. Using the phenomenon that languagereveals our conceptual metaphors, musicalmetaphor has been examined through thestudy of musicians’ language (Wilkie, Holland,and Mulholland 2010), finding metaphoricalexamples such as HARMONIC PRO-GRESSION IS MOVEMENT ALONG APATH, A KEY/CHORD IS A CONTAINERFOR NOTES and MUSICAL SILENCE IS ABLOCKAGE TO MOVEMENT.

Metaphor is also present in gestures, whichprovide depictions of spatial elements of a sourcedomain. For example, the metaphor FOR-WARDS AND BACKWARDS IN TIME ISFORWARDS AND BACKWARDS IN SPACE,is present in a forward pushing gesture indicat-ing the postponing of an event: ‘we can put itoff until next week’ (Cienki and Müller 2008).

DIGITAL CREATIVITY 3

Embodied cognition

ConceptualMetaphorTheory builds on the phil-osophy of embodied cognition, which holds thatour cognition and understanding of the worldare shaped by our bodily interactions within it(Dourish 2004; Leman 2008; Cox 2016), withLakoff and Johnson arguing that all metaphorsare based on source domains related to ourbodies. Embodied cognition rejects theCartesianduality of the mind controlling the body, akin toan automata and its operator, and holds that ourbodies, particularly our hands (Wilson 1998;Pallasmaa 2017), are a part of our cognitive pro-cesses (Merleau-Ponty 2013).

This cognitive model has been applied to ourunderstanding of music, and how we conceptual-ise music in terms of movement and our bodies(Johnson and Larson 2003; Leman 2008; Cox2016). Cox (2016) argues that when we listen tomusic, we experience an imagined movement,either imagining ‘what is it like to do the music’,where we imagine the actions are necessary to cre-ate the music, drawing on our experiences withmusical instruments; or what ‘is it like to be themusic’, imagining the changes in the music(changes in pitch, timbre, rhythm etc.) to be ourown movements.

The theory of embodied cognition is used inthe HCI field of Embodied Interaction. Embo-died Interaction is when embodied cognitionconcepts such as ‘being-in-the-world’ are usedto inform interface design and analysis (Jaasmaet al. 2017; Klipfel 2017; van Dijk and Hummels2017), and has become influential due to itsapplication to tangible interfaces and social com-puting (Dourish 2004). Embodied Interactionhas become particularly pertinent for gesturaland wearable performance systems (Birringerand Danjoux 2009; Klipfel 2017; Mainsbridge2018; Otondo 2018) as the performer’s bodybecomes the control interface for the system.

Examining NIME performance

There have been many studies into NIME per-formances from audience (Bin, Bryan-Kinns,

andMcPherson 2016; Otondo 2018) and perfor-mer perspectives (Tahiroglu, Vasquez, and Kil-dal 2016; Jack, Stockman, and McPherson2017). However, it is often the case in perfor-mer-perspective studies that the performerssolicited are new to the NIME in question.This is due of a lack of experienced users formost NIMEs, due to the novelty of the interfacein question: it is difficult to build a communityof experienced practitioners around an instru-ment that has only existed for a few years(McPherson and Kim 2012). This lack of experi-ence can bemitigated bymaking use of the exist-ing expertise in traditional instruments that isabundant in society, and is useful in cases ofaugmented instruments, which offer extendedperformance techniques on top of existinginstruments such as pianos or cellos (McPher-son, Gierakowski, and Stark 2013; Eldridgeand Kiefer 2017; Harrison and McPherson2017). However, this is not the case for instru-ments whose intention is to move away fromtraditional or pre-existing musical interfaces,as is the case for many mid-air instruments.An existing body of experienced musicians can-not be called upon for the study of instrumentsthat are not based on the augmentations of otherinstruments. Instead, these instruments requireentirely new musical expertise to be developed(Brown, Nash, and Mitchell 2018). While therehas been much work on the design and develop-ment of new gestural performance systems(Klipfel 2017; Otondo 2018), there are veryfew studies that explore and understand thedevelopment of mid-air musical ability (Mains-bridge 2018).

Examining a new user’s experience of tech-nology certainly has its benefits; however, thepotential scope of findings becomes limited. Asmall number of experienced users can beused to effectively study new technology, astheir intimate knowledge of the problemdomain and current software solutions givesthem a knowledge and appreciation of the tech-nology that would elude most novices (Nielsen1994). Similarly, the skill and expertise of a

4 D. BROWN ET AL.

small number of experienced musicians can beelicited to provide meaningful insights intomusical creativity (Johnston 2009; Gelineckand Serafin 2012; Mainsbridge 2018).

NIME research has traditionally drawn fromHCI methods (Wanderley and Orio 2002; Kie-fer, Collins, and Fitzpatrick 2008), using dis-crete musical tasks to study musicalinteraction. Recent work highlights the benefitsof using qualitative and observational methodsto gain insights into the subjective experiencesof NIME musicians (Morreale, De Angeli, andO’Modhrain 2014; Brown, Nash, and Mitchell2017). These approaches focus on contextuallyrelevant data over quantitative precision (Mack-enzie 2013), and allows researchers to exploreinteraction in the context of the user’s ownpractice. These methods have been used suc-cessfully in the research of compositional andcreative processes (Collins 2007; Fiebrink et al.2010; Gelineck and Serafin 2012), and a movetowards context-based, observational researchis also being championed in the general HCIdiscipline (Kaye 2009; Kuutti and Bannon2014; Bødker 2015).

The Gloves

The NIME used in this study was a pair of datagloves (Figure 1) and their dedicated action–sound mapping software (Figure 2) (Mitchelland Heap 2011; Mitchell, Madgwick, andHeap 2012). The gloves incorporate an arrayof sensors to detect the orientation (Inertial

Measurement Unit on the wrist), flex, and pos-ture (flex sensors down each finger) of a wear-er’s hands, which are presented as gesturalinput parameters in the mapping software.The gloves include a button, placed on theside of the index finger and operated by thethumb, a vibration motor on the wrist and aprogrammable LED for haptic and visual feed-back respectively.

The input parameters from the gloves belongto one of three categories:

. Movements. Continuously streaming par-ameters such as the pitch, yaw and roll ofthe hand or the joint angles of the fingers.

. Qualifiers. Parameters that have a binarystate behaviour: they can either be occurringor not, such as hand postures (it is either afist or not-a-fist) or the button press.

. Events. Parameters that represent one-shottrigger controls, usually derived from peaksin sensor readings, such as ‘drum hit’ or‘wrist flick’.

Gestural parameters for the gloves are pre-sented in the mapping software organised inthe following categories.

. Orientation (Movements). The pitch, yawand roll of the hand.

. Flex (Movements). The individual flex valuesof the finger bend sensors.

. Directions (Qualifiers). Directions in whichthe hand can be pointed (directional lobes):up, down, left, right, forwards, backwards.

. Postures (Qualifiers). Classified from thefinger flex sensors using a machine-learningalgorithm, trained by the user.

. Button (Qualifiers). Indicating when the but-ton is pressed or released.

. Drum Hits (Events). Rotational peaks on thehand’s three axes: slap, wrist flick and drumhit.

The software employs a machine learningclassification algorithm to identify patterns theFigure 1. The Gloves.

DIGITAL CREATIVITY 5

finger flex sensor data as hand postures, withusers able to train the software their own cus-tom posture classes.

The mapping software allows users to con-nect the gestural parameters to musical MIDIdata. The interface uses a patch-cord metaphor(Figure 3) similar to visual programminglanguages like Max/MSP or PureData. It alsoincorporates two mapping ‘instruments’,designed to facilitate and simplify commonmapping tasks:

. Chord Machine. This allows users to easilymap gestural qualifiers to multiple notes ona piano keyboard interface (Figure 4(a)).

. Note Matrix. This splits a given movementparameter into a series of thresholds, which,when crossed, trigger notes (Figure 4(b)).

The gloves project began with a single user(and co-creator) Imogen Heap in 2011. Sincethen, an international community of morethan 30 practitioners has grown around the pro-ject. Although there is no official pedagogy,established techniques and practices haveemerged organically through collaboration(Figure 5) and the process of introducing anddemonstrating the technology to new prac-titioners. These techniques have their foun-dations in the early practice of Heap, such as

Figure 2. The Gloves’ dedicated action—sound mapping software.

Figure 3. Examples of the patch-cord mapping metaphor.

6 D. BROWN ET AL.

the four postures that are considered as defaultsin the posture recognition feature: fist, openhand, puppet hand and one finger point(Figure 6).

Glove practitioners have used the gloves toperform electronic music in national and inter-national tours, as well as at events such as TED,Ableton Loop and Sonar (Figures 7 and 8). Thiscommunity, although small, represents a rare

resource in the NIME field, and one apt forstudying to gain insights into the musical prac-tice of experienced NIME practitioners.

Figure 4. (a) Example Chord Machine mapping: ‘OpenHand’ posture to C major, ‘Fist’ posture to G major. (b)Example Note Matrix mapping: Pitch parametermapped to a C major chord.

Figure 5. Kris Halpin and Imogen Heap collaborating. Photograph by Lee Cogswell, courtesy of Kris Halpin.

Figure 6. The default postures (clockwise from topleft): Fist, Puppet Hand, One Finger Point, Open Hand.

DIGITAL CREATIVITY 7

Figure 7. Chagall performing with the gloves. Photograph by Ben Houdijk, courtesy of Chagall Van Den Berg.

Figure 8. Imogen Heap performing with the gloves. Photograph by Tadej Vindis, courtesy of Imogen Heap.

8 D. BROWN ET AL.

Method

The method used to examine the musical prac-tice of glove musicians was Grounded Theory(Glaser and Anselm 1967). Four experiencedglove musicians took part in an unstructuredinterview focussing on the factors that affecttheir mapping design and the use of the glovesin their musical practice. The interview datawas concurrently collected and analysed tofacilitate theoretical sampling. The four musi-cians approached were those whose professionalpractice with the gloves includes live perform-ances for large audiences.

Qualitative research is highly dependent onthe interpretations and perspective of theresearcher (Elliott, Fischer, and Rennie 1999).The lead researcher is not a practising Glovemusician, but does possess intimate knowledgeof the associated systems and software, havingconducted research on the project for twoyears. The analysis builds on an understandingof embodied interaction (Dourish 2004),where knowledge is considered to be builtfrom an individuals’ personal bodily experi-ences in the world. This position has beeninfluenced by HCI and NIME literature, inwhich embodiment is a common epistemologi-cal position (Dourish 2004; Leman 2008; Kaye2009; Cox 2016; van Dijk and Hummels 2017).

Participants

The four experienced participants have all beenusing the gloves in their professional perform-ance practice for several years, each performingat national and international tours and events.The group are a strong community, and eachmusician is well known to the others. Theyalso meet regularly (every six to twelvemonths) to share their work and provideeach other with support and feedback.Throughout the interviews, the musiciansreferred to each other’s work, so each musicianhas been assigned a letter: A, B, C and D. Inaddition to occasional performances with the

gloves, Musician C works extensively as a facil-itator, designing and developing mappings forothers’ performances.

All four musicians have been tied to thedevelopment of the gloves and its software tovarying degrees, providing at times significantdesign input and feedback, as well as suggestingand designing mapping features such as thechord machine and note matrix instruments.

Results

Simple mappings

The musicians were found to use simple, one-to-one and few-to-one mappings that minimizethe potential for performer-related errors.Often referred to by the musicians as ‘practical’mappings, they provide the musicians with con-trol over their musical content that can be mas-tered with little effort. For instance, Musician Aroutinely uses combinations of open hand andfist postures, coupled with the directionallobes of movement to quickly facilitate auditoryfeedback.

… practical, make it work quickly, use thedifferent directions and opposite postures sofist and open hand or something … that’slike the quickest way to do a lot of differentthings. – Musician A.

Simple mappings are also used due to thepressures of performing in front of large audi-ences, where they are used to minimise perfor-mer error.

I had to figure out a way of mapping every-thing that I was going to teach them, thatwas, interesting for people to watch, andthen interesting for them to play, but deadsimple… it’s not that [name] couldn’t dothat if they had the time, they just did nothave the time. I literally had 45 minutes toteach them the song and then they performedit in front of 6000 people. – Musician C.

Simple mappings are also used to make con-trol relationships obvious to audiences. Forinstance, Musician B described how they have

DIGITAL CREATIVITY 9

simplified their mappings as audience membersare unable to accurately perceive more complexcontrol relationships. They found there was lit-tle point in making mappings overly complexwhen they could communicate their musicalintentions using mappings that are easier toperform.

That’s another thing, it’s interesting when techpeople think they know how it works, and I’vehad quite a few people think that I’m launch-ing lots of clips, so when I’m doing the violinthing I’m just triggering a sample and thenmiming to it… there’s times for having abackup clip that I could launch if I needed tofall back, because sometimes it doesn’t comeacross, and the audience isn’t part of that con-versation, no matter how hard I make it formyself. – Musician B.

Expressive movement

While the musicians use simple excitation map-pings, they embellish their excitation move-ments with theatrical ancillary movement.This ancillary movement has no effect on thesound parameters being controlled, but is usedby the musicians to express aesthetic intentions,and to make performances more engaging fortheir audiences. The incorporation of ancillarymovement comes from the performance con-text of the musicians’ practice, and was oftenreferred to as ‘performance theatrics’. Makingtheir performance movements more visuallyengaging was a priority for both Musicians Aand B, who also discussed their work or desireto work with choreographers.

… but also exaggerating certain movements… first of all you’re on a stage so people arelooking at you, and before I did anything ofthe choreography it made me really aware ofhow I moved on stage and felt not that supercomfortable about it, because I’m not a traineddancer and suddenly I had to do movementthat I didn’t necessarily like. So thinkingmore about the choreography and making itmore exaggerated and theatrical made meway more confident performing because nowat least I knew what I was doing and I was

sure it would look cool because we thoughtabout it, you know. – Musician A.

This aesthetic consideration in the design ofmappings and consequently movement is some-thing that has become more prominent in themusicians’ practice as their Glove performancehas developed. Both musicians A and Bremarked that in their early mapping practicethey would use the ‘next available’ control, andtheir focus was on creating performable sol-utions. As their practice with the gloves hasdeveloped, they have moved towards consideringthe visual aesthetics of their performances.

There’s a video of me on the day that I cameup with it, and I just have one glove and Ijust go two finger point, fist, two fingerpoint, fist, and just pitch, that’s all, and thatworks exactly the same way as I do it nowbut with this and the turning around and put-ting my hands to the side for no reason –Musician A.

[T]hings would be quite small, I used to thinkin terms of the next available thing, like if I’mpointing up then I could point forwards. –Musician B.

This development reflects the advancementof the musicians’ creative practice, and a movefrom mapping for functional control to amore abstracted, aesthetically driven approach.Musician A discussed their use of ancillarymovements to develop more aesthetically enga-ging performances. They discuss how in onemusical phrase the last gesture has no musicaleffect, but is performed due to the perceivedmovement in the music.

… there’s one thing that I do that doesn’t trig-ger a different chord, I do [gestures] this thenthis, and the two finger point actually doesn’ttrigger anything. I don’t know why I do it,but it’s just because in the music it feels likesomething changes so I feel stupid if I don’tchange my posture, but it’s not triggering any-thing – Musician A.

While the musicians’ mappings are simple,the use of theatrical ancillary movement allows

10 D. BROWN ET AL.

them to express their aesthetic intentions andprovided engaging performances. Musician C,who believes that something was ‘aestheticallylost’ when mappings were simplified, remarkedthat Musician A’s use of ancillary gesture addedto their performances.

Something is aesthetically lost, for sure. That’swhy I think it’s so cool what [Musician A] isdoing as they’re actually probably doingsimple things, but they’re incorporating theminto a choreography that makes them seemand look and feel more subtle. – Musician C.

Similarly, Musician D remarks how it wasthe simplicity of Musician B’s performancesthat provided an engaging performance.

When I saw [Musician B] for the first time,they were very specific in the things thatthey did, and one of the things that reallycaught me was that I was trying to do toomany things, why am I doing so much? Icould really par down the pallet and be justas impactful. – Musician D.

Metaphors in mapping

In their expressive movements, the musiciansoften used visual metaphors of lyrical or musicalmaterial. The metaphors would often correlatewith established metaphors relating music tospatial properties, such as space being used toconceptualize musical pitch: UP AND DOWNIN PITCH IS UP AND DOWN IN SPACE(Lerdahl 1988; Wilkie, Holland, and Mulhol-land 2010).

This feels to me, terms of pitch, the audiencewill perceive that the chord does indeedchange, it goes down, but it’s not a big change,so you need a bigger gesture to get to the nextchord. So, if the interval’s further away youneed to make the posture bigger. –Musician B.

Other times, gestural metaphors representinglyrical content were used. Musician A designedmappings that reflected the meaning of theirlyrics, for example, Musician A used a metaphorof OPENING HANDS IS OPENING EYES.

… it’s about my friend waking up from acoma, which actually happened like a yearago, so the chords that I trigger, the firsttime I trigger the chords I go like this [openhand gesture in front of eyes] because itseye-opening stuff. – Musician A

These metaphors aid the musicians in creat-ing visually engaging performances throughtheir simple mapping strategies. Musician Dreflected on a performance of Musician B,who incorporated a visual metaphor ofRELEASING A FEATHER IS RELEASING ANOTE into a simple mapping.

I saw them have five or six chords in a spaceand it was just a piano sound, and whenthey opened their hand in a zone it would letout a chord, and it looked like they were let-ting them off like releasing a feather or some-thing, into the air, and it was so beautiful. Iknew that it was just forward, up, left, rightbut it looked like “I’m going to put it intothat wind bit over there, and I’m going to letthe feather go over there” and then theywere singing with it and it was just really gor-geous. – Musician D.

The musicians’ use of metaphor has alsodeveloped from being based on the interactionsof traditional instruments to more abstractmetaphors reflecting the relationship betweenmovement and music (Johnson and Larson2003; Cox 2016). For instance, Musician Bused mappings that used interaction metaphorsof existing instruments in their early perform-ances (such as guitars and violins), whichafforded both them and their audience trans-parency (Fels, Gadd, and Mulder 2002). How-ever, they now find this approach to belimiting creatively, and they have moved tothinking more abstractly about how their move-ments relate to the music itself.

I think it would be good to work with a choreo-grapher at some point. For me the mileage hadrun out [with their previous show]. But it wasgreat for what it was at the time, to go thereand play those shows and be like here’s myinvisible guitar, here’s my invisible drum kitand whatever, that sets it up as a gestural

DIGITAL CREATIVITY 11

thing that people can understand. If I startedfrom day one with all this abstract stuff thenwhat is it? Nobody knows, there’s no way in.But that has to evolve. I’m thinking aboutwhat is the movement of the music. It’s thefirst time I’ve gone the other way around andthought “what would it be?”. – Musician B.

Accidental triggering

A hindrance to the musicians use of aestheticmappings is the prevalence of accidental trig-gering. For instance, due to the snap-to-near-est-class behaviour of the machine-learning-based posture recognizer, it often mistakesthe relaxed hands of the wearer as an ‘openhand’ posture. If the musicians map any con-trols to an open hand, they can often be trig-gered unintentionally. The musicians havedeveloped several strategies for dealing withaccidental triggering, one being the avoid-ance of certain controls. For instance, Musi-cian A avoids using ‘open hand’ postures intheir mappings, and instead uses anotherposture they call ‘secret finger’ (Figure 9), aposture similar but subtly different to anopen hand.

I do [secret finger] when I don’t want peopleto pay attention to my posture because it’salmost no posture, and I usually do this onewhen I kind of want to do it with an openhand but I just need the control. –Musician A.

One strategy that the musicians employwhen they wished to use the open hand posturefor metaphoric or aesthetic purposes is to use‘hidden’ controls alongside the open hand pos-ture (with a Boolean AND relationship) toensure intended control. For instance, MusicianB described an instrument using a metaphor ofpushing a note between directional lobes. Tostop accidental triggering, they use anadditional qualifier for each control: the downdirection of their other hand. This extra qua-lifier is considered ‘hidden’ as it does not formpart of the performance movement, and theintention is that it goes unnoticed by theaudience.

When I did the pushing the synth thing there’sa qualifier on the other hand so it’s only activewhen I’m pointing down. – Musician B.

So I want to play with open hand, right openhand forwards, which I will map, but that’sgoing to trigger all the time, so I’m going tohave to have a sneaky other thing, like onlywhen my left hand is down or something. –Musician A.

Similarly, the name ‘secret finger’, a postureshared by Musician’s A and D, suggests that itis the musicians’ intention that the exact natureof the control is not perceived by the audience.Musician A does use open hand postures whenthey wish to express a specific aesthetic intent,such as the mapping used to represent a lyricabout opening eyes. However, Musician A wasacutely aware that this mapping choice is vul-nerable to accidental triggering and describedhow they immediately return to a ‘fist’ postureonce the phrase has been performed to mini-mise risk.

I definitely don’t make too much dependenton open hand. In that one with the openingFigure 9. The ‘Secret Finger’ posture.

12 D. BROWN ET AL.

eyes thing I do trigger stuffwith open hand butonce I’ve done it I immediately go back to fistjust to make sure I don’t trigger it again. –Musician A.

Reliability in gestural controls

The issues with accidental triggering leads to themusicians needing to balance their aestheticintentions with reliable controls, and influencesthe way musicians create simple, ‘practical’mapping strategies. For instance, when Musi-cian A designed mappings that are used toexpress their aesthetic intentions through meta-phors, it is important that the controls not onlydid this, but could be reliably triggered.

I sing “rewired” [gestures bringing handstogether] because this is the things that comeback together. It works and its practical. –Musician A.

This leads to the musicians considering the‘robustness’ of their mapping choices, whichis mostly done when the musicians considerposture controls. The behaviour of the soft-ware’s Posture Recognition algorithm causesit to recognise and trigger ‘pass through’ pos-tures: postures that the hand unavoidably‘passes through’ as it transitions from one pos-ture to another. For example, if a musician ismoving from a fist to an open hand posture,if they move in such a way that the index

finger starts to extend before the otherfingers, the posture recogniser may briefly reg-ister a first finger point posture Figure 10). Thisvulnerability causes the musicians to considerthe kinematics of their hands and their choiceof postures carefully; what they frequentlyreferred to as the ‘robustness’ of their posturechoices.

Puppet hand isn’t a very stable posture to bedoing that with either. It’s not very robust interms of the likelihood of it happening duringother things, you know, when you’re gesticu-lating, there’s normally a puppet hand inthere. You think about hands in a really differ-ent way. – Musician B.

While most of the musicians reported thatthey often change their posture choices tomore robust, practical set that minimises thesetypes of errors, Musician C reported that theypractise with the posture recogniser to developthe proprioception necessary to master theirposture choices. Furthermore, Musician C alsoadded how they can gain reliable control bypractising their movements rather changingtheir posture choices.

… and I’ve spent enough time with the soft-ware, a lot of time with the software, so I feellike I’ve had a chance and I continually havea chance to build for myself quite subtle androbust posture changes. – Musician C.

Figure 10. Example of a ‘pass-through’ posture: as the hand moves from ‘Open Hand’ to ‘Fist’, a ‘One Finger Point’posture is recognised.

DIGITAL CREATIVITY 13

If it’s just me I’ll persevere and I’ll practise andI’ll practise and I’ll practise. – Musician C.

This motivation could be due to the natureof Musician C’s personal Glove practice, asthey do not perform in front of audiences tothe extent that the other musicians do, andtherefore they may not have the same motiv-ation to develop mappings that mitigate per-formance errors.

Personal aesthetics and gestural identity

Mapping design has become a very personalexpression of aesthetics for the glove musicians.Enabling musicians to design their own map-pings means that choices vary wildly betweenpractitioners.

What we’re doing is different enough fromeach other that we’ve all invented our ownstandard way of doing things. I inherited a lit-tle bit from what we were developing togetherwith [Musician D], but I know [Musician A]does things their way, [Musician B] doesthings their way, they also inherited thingsfrom us, because we taught them initiallyhow to use them, but I’m sure they’ve devel-oped their own workarounds. – Musician C.

This lack of shared practice is interestingconsidering the closeness of the glove commu-nity and given that new users are often intro-duced to the gloves by experienced usersdemonstrating and sharing elements of theirown practice. This suggests that mapping is avery personal creative endeavour for the musi-cians. For instance, Musician A expressed areluctance to perform using mappings designedby others, as they saw the design of their map-pings as a dimension of their musicianship.Musician C also remarked on the importanceof designing one’s own mappings to provideengaging and distinct performances.

I feel like playing with the gloves is such anexpression of how I see and feel music? Sothere’s almost no point in copying someoneelse’s movements or sound–gesture relation-ships because playing with them is part of

the expression, totally, in how you use them.– Musician A.

I feel like that the ability to spend time withyour own mappings, and create your ownmappings, is really important for makingsomething that is really engaging visually –Musician C.

One exception to the lack of shared map-pings is Musician C’s work with a collaborator.The collaborator’s glove musicianship is purelyperformative, and they do not have the samecreative investment in mapping design. The col-laborator wanted to perform a cover of Musi-cian D’s material, and expressed a desire toperform it using mappings designed by Musi-cian D.

I had kind of figured out how she could startthe song, and then [collaborator] decidedthey wanted to have [Musician D] do it forthem – Musician C.

While aesthetic mapping practice hasbecome very individual, there has been adevelopment of standard practice aroundtechnical aspects of glove mappings, such asusing the buttons on the glove to initialisethe glove’s orientation parameters, asobserved in previous research (Brown, Nash,and Mitchell 2018). As these mappings arerelated to solving system related issues (forinstance, Musician C advocates for a ‘kill allnotes’ control on the left-hand button),there is no personal aesthetic investment inthe controls, and they are freely shared andcopied between musicians.

The desire for personal customisationextends to the hardware interface and the low-level workings of the mapping software. Forexample, Musician C desires detailed controlover the posture training process, such as theability to remove sensors from the algorithm.In their current practice, Musician C ‘frees’ sen-sors from the algorithm by providing it withenough varied training examples so that classifi-cation result becomes unaffected by the positioncertain fingers (in this case the thumb).

14 D. BROWN ET AL.

Almost any posture, if I program the posturesin a way that the thumb is independent anddoesn’t add to the posture, I can move thethumb around. – Musician C.

I want more degrees of freedom. I want to beable to choose for myself which sensors are theones that are contributing the postures thatcan be used as triggers… if you could say Idon’t want that to be in the posture recog-nition algorithm, because if I move that par-ticular sensor, usually it’s my thumbs, if Imove that sensor I don’t want that to messup my postures, I want to be able to reallyuse my fingers a lot more, in a more nuancedway. – Musician C.

Discussion

Traditional mapping literature argues thatcomplex action–sound mapping relationshipsare needed to facilitate musical expression(Rovan et al. 1997; Hunt, Wanderley, andParadis 2003; Dobrian and Koppelman2006). Our findings suggest that glove perfor-mers achieve expressive performances usingsimple one-to-one and few-to-one mappingsto minimise the risk of performance error,while embellishing these simple mappingswith theatrical ancillary gestures. The musi-cians use of simple mappings suggests thatit is not the complexity of their mappingsthat facilitates their musical expression,going against the argument that simple map-pings lead to musical toys that musiciansquickly grow tired of (Hunt, Wanderley,and Paradis 2003). For these musicians,expressive performance can be achievedthrough simple action–sound mappings thatfacilitate theatrical movement and their per-sonal ideas and aesthetics of action—soundrelationships.

However, the musicians’ ability to expresstheir personal aesthetics is hindered by issueswith accidental triggering caused by the snap-to-nearest behaviour of the posture recogniser,with the musicians’ mapping decisions beinginfluenced by the need to consider the robust-ness of their mapping choices.

An interesting finding in this research is theimportance of a musicians’ personal mappingstrategies. While it might be expected that map-ping practice would be similar between musi-cians due to their frequent collaborations andsharing of ideas, this research has found thatmapping practice is an incredibly personalendeavour, with glove musicians consideringmapping design to be an important aspect oftheir creative practice. Providing musicianswith the ability to define their own ideas aroundaction—sound mapping forms an importantpart of musical expression with the gloves.

Another point of interest is the desire ofsome of the musicians to work with choreogra-phers to aid them in developing visually sophis-ticated and expressive movements. This focuson expressive movements was influenced bythe musicians’ aim to provide engaging per-formances for their audiences, and highlightsthe importance of the performer-audiencerelationship in mapping design; both the mini-misation of performer errors and the use ofexpressive ancillary gesture come from thedesire to provide a good performance, andsuch factors are more important for musiciansin the context of live performance than inother domains, such as composition (Fiebrinket al. 2010). This is particularly highlighted bythe personal glove practice of Musician C,who does not have the same error minimisationpriority of the other musicians, being more will-ing to spend time mastering difficult mappingsin their personal practice, while in their map-ping design for other performers, simplicityand audience engagement remain importantfactors.

NIME mapping design literature advocatesfor the use of metaphors in mapping design tofacilitate both musician and audience engage-ment (Fels, Gadd, and Mulder 2002; Hunt,Wanderley, and Paradis 2003). The glove musi-cians used metaphors in their mapping design,mainly using metaphors in their ancillary ges-tures to communicate meaning, for examplemusician A’s ‘opening eyes’ metaphor. This

DIGITAL CREATIVITY 15

again was mainly influenced by wanting to pro-vide engaging performances.

The focus on end-user mapping in the con-text of professional performance reveals map-ping design influences that might not beapparent in laboratory-based studies (Françoise2013; Caramiaux et al. 2014), which find theimportance of designing mappings with embo-died metaphors, but do not touch upon factorsraised in this study around reliability, an impor-tant aspect of mapping design for these musi-cians as they are aware that their mappingsare being designed for a live performance.

Conclusion

Through investigating the mapping practice offour experienced musicians who employ end-user mapping in their performance practice,this research has discovered a series of factorsthat influence their mapping design for musicperformance. Primarily, the musicians focuson creating simple mappings that reduce thepossibility of performer error, focussing ondeveloping expressive, performative ancillarymovement, with the underlying aim of thesefactors being the desire to provide engaging per-formances for their audiences.

By studying the creative mapping practice ofexperienced mid-air musicians, an understu-died group of NIME practitioners (McPhersonand Kim 2012), this research contributes novelinsights into expressive NIME mapping design.The findings from this research have informedthe following heuristics that can be applied inthe design of future NIMEs.

(1) NIMEs can use simple action–sound map-pings to create engaging performances.This research has found that experiencedmusicians can use simple mapping sol-utions while providing engaging perform-ances through expressive movement.

(2) NIMEs should allow end-users to expresstheir personal action–sound aesthetics.This research has found that it is important

for NIME mappings to reflect a musician’spersonal interpretations and aesthetics ofmusic andmovement, and that themappingdesign process is an important part of crea-tivity with the gloves. Therefore, NIMEsinstruments that permit end-user customi-zation and mapping personalisationempower musicians to express their ownpersonalized action–sound relationships.

(3) NIME mapping should use metaphors ofmusic to provide engaging instruments.This research has found that experiencedmusicians use metaphors in their mappingdesign to communicate aesthetic intent andto provide engaging performances. The useof metaphor has been advocated in thedesign of music interaction (Fels, Gadd,and Mulder 2002; Wessel and Wright2002), which this research supports.

(4) NIMEmappings should mitigate the poten-tial for accidental triggering. This researchhas found that a major barrier in the musi-cians’ practice is accidental triggering. Thebehaviour of the posture recogniser andoccurrence of ‘pass-through’ postures forcethe musicians to move away from expres-sing their personal aesthetics and focus onthe ‘robustness’ of their posture choices.NIMEs could minimise accidental trigger-ing through excitation controls that avoidthe need for performers to pass throughother body states that trigger excitation con-trols, and through avoiding controls thatrely on gestures that are similar to a musi-cian’s relaxed body state.

Themes emerging from this study suggestdirections for future research. The musiciansspoke of how their mapping practice has devel-oped over the time with the gloves, such as theirmove away from functional mappings to thea-trical, aesthetic mappings. Tracking the devel-opment of mapping practice over alongitudinal study could provide insights intothis process. As well as this, future work couldexamine the extent to which accidental

16 D. BROWN ET AL.

triggering or system related issues affect a per-former’s experience with the gloves.

Acknowledgements

This work is supported by the University of the Westof England, The Higher Education Innovation Fund(Research England), Innovate UK and EuropeanCommission. We thank our anonymous reviewersand study participants for their insightful feedbackalong with the those involved in the Mimu glovesproject: Imogen Heap, Kelly Snook, Seb Madgwick,Hannah Perner-Wilson, Adam Stark, Rachel Freire,Chagall Van Den Berg, Kris Halpin, the gloves col-laborators and many others who have helped alongthe way.

Disclosure statement

No potential conflict of interest was reported by theauthors.

Funding

This work is supported by the University of the Westof England, The Higher Education Innovation Fund(Research England), Innovate UK and EuropeanCommission.

Notes on contributors

Dom Brown is currently a PhD candidate at the Uni-versity of the West of England, looking at end-usermapping design in musical performance with mid-air gestural technology.

Dr Chris Nash is a Senior Lecturer at the Universityof the West of England, with research interests inmusic technology and related fields, with a specificgoal of improving the user experiences of compu-ter-based composers and musicians.

Dr Tom Mitchell is an Associate Professor of Crea-tive Technologies at the University of the West ofEngland and head of the Creative Technologies Lab-oratory, with interests in artificial intelligence, audioand music interaction.

References

Bin, S. Astrid, Nick Bryan-Kinns, and Andrew P.McPherson. 2016. “Skip the Pre-concert Demo:

How Technical Familiarity and Musical StyleAffect Audience Response.” Proceedings of NewInterfaces for Musical Expression (NIME), 200–205, Brisbane, Australia.

Birringer, Johannes, and Michãšle Danjoux. 2009.“Wearable Performance.” Digital Creativity 20(1–2): 95–113. doi:10.1080/14626260902868095.

Blackwell, Alan F. 2006. “The Reification ofMetaphor as a Design Tool.” ACM Transactionson Computer-Human Interaction (TOCHI) 13(4): 490–530.

Bødker, Susanne. 2015. “Third-Wave HCI, 10 YearsLater—Participation and Sharing.” Interactions 22(5): 24–31. doi:10.1145/2804405.

Brown, Dom, Chris Nash, and Tom Mitchell. 2017.“A User Experience Review of Music InteractionEvaluations.” Proceedings of New Interfaces forMusical Expression (NIME), Copenhagen,Denmark.

Brown, Dom, Chris Nash, and Tom Mitchell. 2018.“Understanding User-defined Mapping Designfor Mid-air Musical Performance.” Proceedingsof the 5th international conference on movementand computing (MOCO), ACM, Genoa, Italy.

Caramiaux, Baptiste, Jules Françoise, NorbertSchnell, and Frédéric Bevilacqua. 2014.“Mapping Through Listening.” Computer MusicJournal 38 (3): 34–48.

Cienki, Alan, and Cornelia Müller. 2008. “Metaphor,Gesture, and Thought.” In The CambridgeHandbook of Metaphor and Thought, Chapter 5,edited by Raymond W. Gibbs Jr., 483–501. NewYork, NY: Cambridge University Press.

Collins, David. 2007. “Real-time Tracking of theCreative Music Composition Process.” DigitalCreativity 18 (4): 239–256.

Cox, Arnie. 2016. Music and Embodied Cognition.Bloomington, IN, USA: Indiana University Press.

Di Donato, Baladino. 2017. “MyoMapper.” http://www.balandinodidonato.com/myomapper/.

Dobrian, and Daniel Koppelman. 2006. “The ‘E’ inNIME: Musical Expression with New ComputerInterfaces.” Proceedings of New Interfaces forMusical Expression (NIME), Paris, France.

Dourish, Paul. 2004. Where the Action is: TheFoundations of Embodied Interaction.Cambridge, MA: MIT Press.

Eldridge, Alice, and Chris Kiefer. 2017. “The Self-Resonating Feedback Cello: Interfacing Gesturaland Generative Processes in ImprovisedPerformance.” Proceedings of New Interfaces forMusical Expression (NIME), 25–29, AalborgUniversity Copenhagen, Copenhagen, Denmark.http://sro.sussex.ac.uk/68252/.

DIGITAL CREATIVITY 17

Elliott, Robert, Constance T. Fischer, and David L.Rennie. 1999. “Evolving Guidelines forPublication of Qualitative Research Studies inPsychology and Related Fields.” British Journalof Clinical Psychology 38 (3): 215–229.

Fels, Sidney, Ashley Gadd, and Axel Mulder. 2002.“Mapping Transparency Through Metaphor:Towards More Expressive Musical Instruments.”Organised Sound 7 (2): 109–126.

Fiebrink, Rebecca, Daniel Trueman, N. Cameron Britt,Michelle Nagai, Konrad Kaczmarek, Michael Early,M. R. Daniel, Anne Hege, and Perry R. Cook. 2010.“Toward Understanding Human-ComputerInteraction in Composing the Instrument.”Proceedings of the International Computer MusicConference (ICMC), New York, NY, USA.

Fiebrink, Rebecca, Dan Trueman, and Perry R.Cook. 2009. “A Meta-Instrument for Interactive,On-the-Fly Machine Learning.” Proceedings ofNew Interfaces for Musical Expression (NIME),280–285, Pittsburgh, PA, USA.

Fischer, Gerhard, and Andreas Girgensohn. 1990.“End-user Modifiability in DesignEnvironments.” Proceedings of the SIGCHIConference on Human Factors in ComputingSystems, 183–192, ACM, Seattle, WA, USA.

Françoise, Jules. 2013. “Gesture–sound Mapping byDemonstration in Interactive Music Systems.”Proceedings of the 21st ACM InternationalConference on Multimedia, 1051–1054, ACM,Barcelona, Spain.

Francoise, Jules. 2015. “Motion-Sound Mapping ByDemonstration.” PhD, McGill University.

Gelineck, Steven, and Stefania Serafin. 2012.“Longitudinal Evaluation of the Integration ofDigital Musical Instruments into ExistingCompositional Work Processes.” Journal of NewMusic Research 41 (3): 259–276.

Glaser, Barney G., and L. Strauss Anselm. 1967. TheDiscovery of Grounded Theory: Strategies forQualitative Research. Hawthorne, NY, USA:Aldine Publishing Company.

Goudeseune, Camille. 2002. “Interpolated Mappingsfor Musical Instruments.” Organised Sound 7 (2):85–96.

Gurevich, Michael, and Jeffrey Treviño. 2007.“Expression and its Discontents: Toward anEcology of Musical Creation.” Proceedings ofNew Interfaces for Musical Expression (NIME),106–111, ACM, New York, NY, USA.

Harrison, Jacob, and Andrew McPherson. 2017. “AnAdapted Bass Guitar for One-handed Playing.”Proceedings of New Interfaces for MusicalExpression (NIME), 507–508, Aalborg University

Copenhagen, Copenhagen, Denmark. http://www.nime.org/proceedings/2017/nime2017_paper0102.pdf.

Hofmann, Alex, Bernt Isak Waerstad, SaranyaBalasubramanian, and Kristoffer E. Koch. 2017.“From Interface Design to the Software Instrument- Mapping as an Approach to FX-InstrumentBuilding.” Proceedings of New Interfaces forMusical Expression (NIME), 133–138, AalborgUniversity Copenhagen, Copenhagen, Denmark.

Hunt, Andy, and Ross Kirk. 2000. “MappingStrategies for Musical Performance.” Trends inGestural Control of Music 21: 231–258.

Hunt, Andy, Marcelo M Wanderley, and MatthewParadis. 2003. “The Importance of ParameterMapping in Electronic Instrument Design.”Journal of New Music Research 32 (4): 429–440.

Jaasma, Philemonne, Dorothé Smit, Jelle van Dijk,Thomas Latcham, Ambra Trotto, and CarolineHummels. 2017. “The Blue Studio: Designing anInteractive Environment for Embodied Multi-sta-keholder Ideation Processes.” Proceedings of theeleventh international conference on tangible,embedded, and embodied interaction, 1–10, TEI‘17, ACM, New York, NY, USA. doi:10.1145/3024969.3025002.

Jack, Robert H., Tony Stockman, and AndrewMcPherson. 2017. “Rich Gesture, ReducedControl: The Influence of Constrained Mappingson Performance Technique.” Proceedings of the4th international conference on movement com-puting, ACM, London, UK.

Johnson, Mark, and Steve Larson. 2003. “‘Somethingin the Way She Moves’ – Metaphors of MusicalMotion.” Metaphor and Symbol 18 (2): 63–84.

Johnston, Andrew. 2009. “Interfaces for MusicalExpression Based on Simulated Physical Models.”PhD Thesis, University of Technology, Sydney.

Kaye, Joseph Nathaniel. 2009. “The Epistemology &Evaluation of Experience-Focused Hci.” PhDThesis, Cornell University.

Kiefer, Chris, Nick Collins, and GeraldineFitzpatrick. 2008. “HCI Methodology ForEvaluating Musical Controllers: A Case Study.”Proceedings of New Interfaces for MusicalExpression (NIME), 87–90, Genova, Italy.

Klipfel, Kristofer. 2017. “MIDI Motion: InteractiveMusic Composition Gloves.” Proceedings of theeleventh international conference on tangible,embedded, and embodied interaction, 757–760,TEI ‘17, ACM, New York, NY, USA. doi:10.1145/3024969.3035535.

Kuutti, Kari, and Liam J Bannon. 2014. “The Turn toPractice in HCI: Towards a Research Agenda.”

18 D. BROWN ET AL.

Proceedings of the 32nd annual ACM conferenceon human factors in computing systems, 3543–3552, ACM, Toronto, ON, Canada.

Lakoff, George, and Mark Johnson. 1980. MetaphorsWeLiveBy. Chicago, IL:University ofChicagoPress.

Leman, Marc. 2008. Embodied Music Cognition andMediation Technology. Cambridge, MA, USA:MIT Press.

Lerdahl, Fred. 1988. “Tonal Pitch Space.” MusicPerception: An Interdisciplinary Journal 5 (3):315–349.

Mackenzie, I. Scott. 2013. Human ComputerInteraction: An Empirical Research Perspective.Edited by Meg Dunkerley and Heather Scherer.Waltham, MA, USA: Morgan Kaufmann.

Mainsbridge, Mary. 2018. “Gesture-ControlledMusical Performance: From MovementAwareness to Mastery.” International Journal ofPerformance Arts and Digital Media 14 (1): 1–18.

Malloch, Joseph, Stephen Sinclair, and Marcelo M.Wanderley. 2007. “From Controller to Sound:Tools for Collaborative Development of DigitalMusical Instruments.” Proceedings of theInternational Computer Music Conference(ICMC). Copenhagen, Denmark.

McPherson, Andrew P., Adrian Gierakowski, andAdam M. Stark. 2013. “The Space between theNotes: Adding Expressive Pitch Control to thePiano Keyboard.” Proceedings of the SIGCHIconference on human factors in computing sys-tems, 2195–2204, ACM, Paris, France.

McPherson, Andrew P., Robert H. Jack, and GiulioMoro. 2016. “Action-Sound Latency: Are OurTools Fast Enough?” Proceedings of NewInterfaces for Musical Expression (NIME),Griffith University, Brisbane, Australia.

McPherson, Andrew P., and Youngmoo E. Kim.2012. “The Problem of the Second Performer:Building a Community Around an AugmentedPiano.” Computer Music Journal 36 (4): 10–27.

Merleau-Ponty, Maurice. 2013. Phenomenology ofPerception. Abingdon: Routledge.

Mitchell, Thomas, and Imogen Heap. 2011.“SoundGrasp: A Gestural Interface for thePerformance of Live Music.” Proceedings ofNew Interfaces for Musical Expression (NIME),Oslo, Norway.

Mitchell, T. J., S. Madgwick, and I. Heap. 2012.“Musical Interaction with Hand Posture andOrientation: A Toolbox of Gestural ControlMechanisms.” Proceedings of New Interfaces forMusical Expression (NIME), Ann Arbor, MI, USA.

Momeni, Ali, and Cyrille Henry. 2006. “DynamicIndependent Mapping Layers for Concurrent

Control of Audio and Video Synthesis.”Computer Music Journal 30 (1): 49–66.

Morreale, Fabio, Antonella De Angeli, and SileO’Modhrain. 2014. “Musical Interface Design: AnExperience-Oriented Framework.” Proceedings ofNew Interfaces for Musical Expression (NIME),467–472, London, UK.

Nielsen, Jakob. 1994. “Heuristic Evaluation.”Usability Inspection Methods 17 (1): 25–62.

Otondo, Felipe. 2018. “Using Mobile Sound toExplore Spatial Relationships between Danceand Music Performance.” Digital Creativity 1: 1–14. doi:10.1080/14626268.2018.1459731.

Paine, Garth, Ian Stevenson, and Angela Pearce.2007. “The Thummer Mapping Project(ThuMP).” Proceedings of New Interfaces forMusical Expression (NIME), 70–77. ACM,New York, NY, USA.

Pallasmaa, Juhani. 2017. “Embodied and ExistentialWisdom in Architecture: The Thinking Hand.”Body & Society 23 (1): 96–111.

Rovan, Joseph Butch, Marcelo M Wanderley,Shlomo Dubnov, and Philippe Depalle. 1997.“Instrumental Gestural Mapping Strategies asExpressivity Determinants in Computer MusicPerformance.” Proceedings of Kansei - the tech-nology of emotion workshop, 3–4, Genoa, Italy.

Scurto,Hugo, FrédéricBevilacqua, and Jules Françoise.2017. “Shaping and Exploring Interactive Motion-Sound Mappings Using Online ClusteringTechniques.” Proceedings of New Interfaces forMusical Expression (NIME), 410–415, AalborgUniversity Copenhagen, Copenhagen, Denmark.

Tahiroglu, Koray, Juan Carlos Vasquez, and JohanKildal. 2016. “Non-Intrusive Counter-Actions:Maintaining Progressively Engaging Interactionsfor Music Performance.” Proceedings of NewInterfaces for Musical Expression (NIME), 444–449, Brisbane, Australia.

Taruskin, Richard. 1995. Text and Act: Essays onMusic and Performance. New York, NY: OxfordUniversity Press.

van Dijk, Jelle, and Caroline Hummels. 2017.“Designing for Embodied Being-in-the-World:Two Cases, Seven Principles and One Framework.”Proceedings of the eleventh international conferenceon tangible, embedded, and embodied interaction,47–56, TEI ‘17, ACM, New York, NY, USA. doi:10.1145/3024969.3025007.

Van Nort, Doug. 2018. “Conducting the in-between:Improvisation and Intersubjective Engagement inSoundpainted Electro-Acoustic EnsemblePerformance.” Digital Creativity 29 (1): 68–81.doi:10.1080/14626268.2018.1423997.

DIGITAL CREATIVITY 19

Visi, Federico, Baptiste Caramiaux, MichaelMcloughlin, and Eduardo Miranda. 2017. “AKnowledge-based, Data-driven Method forAction-sound Mapping.” Proceedings of NewInterfaces for Musical Expression (NIME),231–236, Aalborg University Copenhagen,Copenhagen, Denmark.

Wanderley, Marcelo Mortensen, and Nicola Orio.2002. “Evaluation of Input Devices for MusicalExpression: Borrowing Tools from HCI.”Computer Music Journal 26 (3): 62–76.

Wessel, David, and Matthew Wright. 2002.“Problems and Prospects for Intimate MusicalControl of Computers.” Computer Music Journal26 (3): 11–22.

Wilkie, Katie, Simon Holland, and Paul Mulholland.2010. “What Can the Language of Musicians TellUs about Music Interaction Design?” ComputerMusic Journal 34 (4): 34–48.

Wilson, Frank R. 1998. The Hand: How Its UseShapes the Brain, Language, and HumanCulture. New York: Pantheon Books.

20 D. BROWN ET AL.