Playful Technology-Mediated AudienceParticipation in a Live Music Event

Fares KayaliOliver HödlGeraldine FitzpatrickPeter PurgathoferAlexander FilippInstitute for Design andAssessment of Technology, TUWien1040 Vienna, Austriafares@igw.tuwien.ac.at

Johannes KretzDepartment for Composition,Electroacoustics and TonmeisterEducation, University of Musicand Performing Arts Vienna1030 Vienna, Austriakretz@mdw.ac.at

Ruth Mateus-BerrUlrich KühnThomas WagensommererInstitute for Art Sciences and ArtEducation, University of AppliedArts Vienna1010 Vienna, Austriaruth.mateus-berr@uni-ak.ac.at

Susanne KirchmayrElectric Indigo1030 Vienna, Austriaindigo@indigo-inc.at

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s). Copyright is held by theauthor/owner(s).CHI PLAY’17 Extended Abstracts, October 15–18, 2017,Amsterdam, The Netherlands.ACM ISBN 978-1-4503-5111-9/17/10.http://dx.doi.org/10.1145/3130859.3131293

AbstractThis paper presents the evaluation of playful technology-mediated audience participation (TMAP) during three musicperformances in a recent music event. It captures prelim-inary impressions from a wide range of perspectives andincludes critical reflections of music artists, video analysisand qualitative interviews with audience members to coverhypotheses designed to capture both the artists’ and theaudience’s point of view. Results indicate a willingness fromboth sides to engage in playful TMAP, and a high potentialfor exploration and playful collaboration within the audience,but the experience is restricted by the need to retain con-trol on the side of artists and the need for clear instructions,feedback and reliable technical systems on the side of theaudience.

Author KeywordsTechnology-Mediated Audience Participation; Music Perfor-mance; Playful Interaction

ACM Classification KeywordsH.5.m. [Information Interfaces and Presentation (e.g. HCI)]:Miscellaneous; K.8.0 [Personal Computing]: Games; H.5.5[Sound and Music Computing]: .

IntroductionThis paper reflects playful technology-mediated audienceparticipation (TMAP) in live music during a recent musicevent. Across three music performances and a supportingartistic performance technology was used to facilitate play-ful interaction between artists and audience. We focus onfirst impressions of the performance from the perspectivesof participating artists and audience. We first describe thefour performances along with giving a critical reflection fromthe perspective of the participating artists based on informalpersonal and email de-briefs. These perspectives are thencontrasted by presenting preliminary results of a qualitativeevaluation with audience members. To the left we present aseries of hypotheses that define the research context of thiswork. Given that this is a work-in-progress paper, we willnot validate these hypotheses but present first directions inwhich results are potentially leading.

Hypotheses

(H1): Music artists welcomeinteractive systems for audi-ence participation because itenables new ways of artisticexpression.

(H2): Interactive systems foraudience participation posi-tively influence an audience’sexperience of a performance.

(H3): Artists perceive inter-active systems for audienceparticipation as a way ofcommunicating with theaudience.

(H4): Interactive systems foraudience participation makethe spectators feel includedin the performance.

Related WorkAn example for audience participation goes back to Mozart(1756-1791), who allegedly composed the parts of the“Musikalisches Würfelspiel” [7] (musical dice game min-uet). He made a quite conscious game design decision. Herecognized chamber music as a participatory musical formin the need for an interactive diversion for the audience.Thus he introduced two dice, thrown to determine one ofmany possible combinations of musical segments of waltzmusic played afterwards. This playful approach seemed torepresent the very antithesis of compositional strategies[9]. For this purpose Mozart abstracted waltz music fromcontinuous pieces of music to smaller segments, which canbe rearranged freely. The common denominator of manyworks in the field of sound art and music-based games [8],is that they make aspects of playing music and compositionaccessible to the audience by abstracting from its originalcomplexity. The same is needed for playful TMAP. Previ-

ous work in this field has resulted in the creation of designcards to support creative processes in TMAP [4], metricsto describe and evaluate the characteristics of participatoryperformances [6], and design implications such as expres-siveness, communication and appropriateness that are ofconcern to both audience and musicians in TMAP appli-cations [3]. The research presented here will extend thesefoundations by evaluating three distinct music performancesusing TMAP.

Revisiting the Performances from an Artist Per-spectiveIn this section we discuss the three performances and thesupporting performance from the perspective of the par-ticipating artists. Each performance is first described andthen critically reflected. The critical reflections are based oninformal personal and email discussions with the artists.

Johannes Kretz - RobotsIn this 20 minute-long opening piece, created by a com-poser and computer musician, the audience is introducedto the overall setting and general atmosphere of the event.An environment at the intersection of performance and in-stallation offers the audience a space of exploration anddiscovery. The performance iStressTest for three robotsand audience provided three autonomous interactive mu-sical agents, each with its own musical personality, react-ing to collisions with the wall or with the feet of the visitors,both visually and acoustically. The more the robots werehit, the more they were stressed, emitting “stress sounds”and also visually reflecting their stress level with differentcolours on the display. In order to achieve a common globalsoundscape, the robots transmitted their states to a laptopcomputer through WLAN, which allowed on the one handto create a common acoustical atmosphere, and on theother hand allowed to synchronise the rhythmical patterns

of the robots. Synchronization could only happen, when therobots were in low stress level and “undisturbed” from theaudience. The robots’ movement information is also usedto (relatively) operate a first-person character in a virtual,photogrammetric replica of the event’s location displayed ontwo screens on the floor (figure 1). The position of the char-acter adds auditive responses to the soundscape. So, theway the audience interacts with the robots dynamically af-fects the overall alternation of sound in a playful way, whilethe visual representation acts as a game-like interface in-tended for a multitude of users.

Critical ReflectionFrom a composer perspective – who is used to creating awork and to being able to accurately control the soundingresult of a work – it was a challenge to create an environ-ment, where the audience is capable of modifying the mu-sical outcome interactively. In general the playful setup ofthe performance worked nicely. Groups of audience mem-bers spontaneously decided to surround / trap one of therobots and to play with it together within their circle. (Thisbehaviour was not expected by the composer but neverthe-less was an interesting and welcome result of the exper-iment.) It seemed not so clear for the audience, whetheriStressTest was a “composition” or an interactive “funny sit-uation” during the period of the audience entering the hall.This relaxed situation seemed to be welcome for the audi-ence and they were able to decide individually, whether theywanted to chat with other visitors or whether they wouldinvolve themselves in the sound game. In any case the in-stallation seemed to trigger interest and curiosity.

Breaking The Wall Event

The event was held as partof an art-based researchproject to put created sys-tems for TMAP to a practicaltest and to evaluate theiruse with audience membersand music artists. The eventwas held in Vienna on June2nd, 2017. There were 288attendants who took part inthree music performancesand a supporting artisticperformance. All three per-formances used the samelayout of the space (seefigure 1). The stage (usedin the Smartphones andLasers performances) waspositioned in the centre al-lowing the crowd to gatheraround it from all sides. Twovideo screens were placedon the floor and used in allthree performances. Fourpedestals with speakerswere positioned close to thecorners of the room and al-lowed attendants to interactduring the Smartphones per-formance. The four movinghead spotlights stood on thefloor and were used in theLasers performance.

Electric Indigo - SmartphonesBy using an app developed for this performance the smart-phones of the attendees are transformed into mobile soundsources. On the one hand these can be manipulated in

real-time by the composer and musician, on the other theyare freed of a static location-dependance by the behaviourand movement of the audience. This way the audience (asa swarm) is able to independently affect the distribution ofsound thereby acting as a dynamic mediator between per-former and performance space. This bridge is realised tech-nologically through the use of ultra sound (high-frequencysound IDs) [5], which connects the audience’s smartphonesto the performer without the need to join a WiFi or mobilephone network, thus lowering the technological thresholdfor participation. The audience uses the app on their ownsmartphones. Based on inputs from the artist on stage thephones create sounds and blinking display lights. Audiencemembers can also use four pedestals in the corners of theroom (figure 1) to augment the sounds from their phones.The audience interacts by positioning their devices (andthus sound sources) in the room to co-create the resultingelectro-acoustic soundscape.

Critical ReflectionSome of the files running off the phones were less audi-ble than others, and the audience could be observed intheir quest to understand the role of the equipment on thefour pedestals. However, this created an atmosphere ofwonder and excitement for many, and they continued tosearch for the "right" way to contribute the things happen-ing on the phones to the overall performance. The artist, onthe other hand, tried her best to make the inclusion of thesounds from the smart phones as noticeable as possible,with room to improve. Despite testing the app beforehand(under different conditions) some phones did not alwaysreact to the high-frequency audio triggers. This resulted infewer phones playing back the audio files than desired. Still,as the modes of interaction were not perfectly transparent,many people seemed to think that this was a deliberate dif-ferentiation among the large number of smart phones. This

made them enjoy the moments when their own phones re-acted. An additional aspect of audience participation wasdescribed by participants, as the music itself was modu-lated from moving around, so that through variation of theirlocation and motion within the performance space madethem interact with the sound in unexpected ways. Weav-ing through people who had their smart phone blinking andmake sounds varied the experience from moment to mo-ment, and it created a bond between participants.

The original plan was to add amplification possibilities forthe phones at the four pedestals as an additional interactionfeature for the audience. After the dress rehearsal on theday before the premiere, when more people were presentthan during the rehearsals before that, two problems withthe amplification became visible: One being the quite natu-ral tendency of people to try out the amplifying coils repeat-edly which would result in harsh (and from the artist’s sideunintended), more or less rhythmical volume alternations ofvery loud and quiet. This became annoying and stressful forthe artist as she had no adequate option to react to it in away that would make sense musically. Second the opera-tion of the amplification coils has a huge flaw: They produceamplified interference noises every time the coil is close tothe phone but does not touch it. Therefore, in the morningof the day of the premiere, the artist decided not to use thecoils for amplification of the phones but play the concert atlower volume instead.

Figure 1: The centre stage layoutof the The Breaking The WallEvent.

Figure 2: Audience members,illuminated by the screens of theirmobile phones, gather around thecentral stage and Electric Indigo.

null.head - LasersThe collaboration with the performance artists null.head(Didi Bruckmayr and Chris Bruckmayr) focuses on the tech-nological and dramaturgical connection of body, sound,light and room. Through this multi-sensory experience, theaudience should be able to reflect on and question digitalsurveillance and technological authority as it may be part

of technology-mediated audience participation. This kind ofembodied and technological intervention creates an experi-mental situation questioning accepted customs and habits.Technically, the audience will be located in the room withthe use of laser traps scattered in the whole area. Thus theposition of audience members in those traps becomes acharacteristic parameter of the performance. Every timesomeone crosses an ankle-high laser barrier, a certainchange in sound is initiated, while the moving head spot-lights (figure 1) focus on this certain person.

Critical ReflectionDue to the unexpectedly large number of initial partici-pants, too many laser tripwires were activated too often.So the interaction between audience and technology wasa bit harder to follow than expected. Thanks to the perfor-mance and voice of the music artists the interaction some-times switched to a very direct human-human interaction.The next time this performance will take place, more lasertraps and better positioning of those traps can be used tomake the interaction more rewarding and easier to follow.A lot of visitors still seemed very impressed with the perfor-mance, in that they articulated their desire for higher vol-umes to experience the music. Many visitors felt great plea-sure to move around in an uncomplicated and professionalstage lightning and felt truly invited to "play around" with thelasers and moving lights. Visitors gave very good feedbackabout the overall design of the stage, the lighting situationand the ambience of the whole performance space.

Artistic PerformanceThe artistic "counter-performance" Treat On Toes | Rules OfInteraction was used to provoke interaction within the con-cert. Rules and possibilities for the interaction at the con-cert were not declared or written and had to be observedor provoked by the artists counter-performance (e.g., step-

ping on toes, blocking the way, stalking). The performancestarted with a temporary barrier in the staircase, which re-minds of the often senseless labyrinth-like barrier lines inairports. Just one person dared to ignore or even destroythe senseless hindrance, which was immediately reinstalledand never touched again until the end of the event. Theartist provokes by coming too close or blocking the way.Only people not concentrating on the music (talking dur-ing the concert, posting things on facebook, sitting on abench etc.) were targeted. The artist was wearing darksunglasses with an integrated HD-camera. The objectivewas to observe reactions. She aimed to get very close tovisitors in order to provoke them to move towards possibleinteractions with the lasers or the smart phones installed onpedestals. Audience members were touched with one handand given an event-branded business card.

Critical ReflectionObserved reactions to the counter-performance interventioninclude feelings of embarrassment, when people felt theywere transgressing orders. Women mostly rapidly movedaway, some of them giggled. Men reacted with flash-lightingher face with their smart phone LED or by asking: "what doyou want with these glasses?", or even by acting aggres-sively. One man grasped his bag asking: "what do you wantfrom me?". Only some men moved away. Most seemed tobe provoked and did not move an inch. Just some of theparticipants had the courage to face her and started to un-derstand that the glasses are blinking and that this mighthave a meaning related to the performance. E.g., someonesaid: "now she is looking at us, this might have an influenceon the performance". Some people also initiated playful in-teractions with the artist. While not all of the reactions couldbe documented by the glasses’ built-in camera, the surveil-lance characteristic remained intact.

Audience EvaluationIn his article "Designing for Audience Engagement" Bilda[1] comes to the conclusion, that "an evaluation viewpoint isone that tries to gain knowledge and understanding of theaudience that is as reliable as possible." In her article "Theart of research in live music performance" Mine Dogantan-Dack [2] describes options including questionnaires, inter-views, audio- and video-analysis, and observational casestudies. To evaluate (H2) and (H4) we settled on a mixed-method approach of video analysis and interaction data log-ging during the performance, as well as interviews beforeand after the event.

For the video analysis, we used four cameras to record theperformance and the audience. Overall we could observea lot of interaction during the first two performances, mostnotably people building groups and collaborating aroundthe robots, the four pedestals and the screens. In the thirdperformance we saw people experimenting with lasers andlights, but a lot less interaction due to a lack of recognisablefeedback.

We recorded 25 interviews before the event. The first inter-view consisted of 16 questions and was estimated to takearound four minutes. In the interviews, we asked about theirexpectation of the opportunity to interact during a perfor-mance with robots, smartphones and lasers and they hadthe chance to rate their notion. Most of the surveyed peopleseemed to like the idea of interacting during a performancewith lasers. On the other hand, the interviewees often dis-liked the opportunity to interact with smartphones during amusic event. After the event, we recorded 22 interviews (11of them also did the interview before the event). The sec-ond interview was designed to take about 8 minutes and in-cluded 31 questions. In general, most of the people enjoyedthe performance. One person told us that it was a unique

experience for him and he was all lost in thought during theevent. Another person said: "It felt like a conspiracy theory- it was a little bit creepy at the beginning, but not negativelyso - it was fascinating and scary". Some people thoughtthat the event was too challenging and that it was hard tounderstand the individual performances. Some of the in-terviewed people would have preferred a short instructionbefore each performance to better understand possible in-teractions. Then again, one of the interviewees stated thatit was good to have no briefing before: "It inspired me tothink!"

Acknowledgments

Funded by FWF AustrianScience Fund (AR 322-G21).Thanks to: Christoph Bart-mann, Chris Bruckmayr,Dietmar Bruckmayr, JuliaSoto Delgado, Simon Hol-land, Anna Lerchbaumer,Hande Saglam.

ConclusionsThe presented critical reflection by artists and preliminaryinsights from the audience study lead to the following ten-dencies regarding the four initially presented hypotheses.

We expect mixed results regarding (H1). While all of theparticipating artists were excited to create systems forTMAP to be used in their performances, the actual finalresults varied. The more media installation-like first per-formance with robots fully relied on audience interactionto a degree where the music artist, though present, is notreally visible to the audience. Conversely Electric Indigo de-cided not to include one of the primary modes of audienceinteraction in the morning of the performance because re-hearsals with more people present made the musical resulthard to control. Overall (H1) will probably be neither vali-dated nor invalidated. Instead a balance between handingover aspects of the performance to the audience while re-taining control as an artist can be be described.

Regarding (H2), overall people were open-minded to theidea of interacting within a live music performance. Recordedexperiences were mixed, with some describing intrigue, cu-riosity and immersion, while others were confused by ambi-

guities in the presented interaction and feedback. We canconclude here that the clearer instructions and feedbackare, and the better and more reliable the used technologyworks, the better the audience’s experience is in TMAP.Also providing an environment and ambience that facilitatesplayful interactivity and exploration is beneficial to the expe-rience.

(H3) discusses artists using TMAP to communicate withthe audience. In the first performance the artist discussedemergent audience behaviour (crowding of the robots)that was unexpected and prompted different interactionsthan what we planned for. The second performance con-tained the strongest dialogue with the audience as thesoundscape was very actively defined by the audience andtheir phones prompting the artist to both act (by triggeringsounds) and react (by adjusting the volume balance of thePA and surround speakers with what was played back onthe phones). In the third performance artist-audience com-munication was used to compensate for technical problems.

Regarding (H4) we saw that people feel more included themore feedback they get. The robots gave very immediatefeedback and the smart phone interaction was on the onehand very palpable and on a larger scale mysterious andintriguing enough to trigger curiosity. The third performanceshowed that when feedback is missing the audience startsto retreat to normal, more passive roles. We also were ableto observe a lot of collaboration within the audience duringthe first two performances, which was only partially plannedfor. Playful collaboration should be considered to have highpotential in future systems for TMAP as a means of includ-ing the audience more tightly.

Future work will present a detailed evaluation of the four hy-potheses. Building on this data we will also present detaileddesign considerations for playful TMAP in live music.

REFERENCES1. Zafer Bilda. 2011. Designing for audience engagement.

Interacting: art, research and the creative practitioner.Libri Publishing Ltd., Faringdon (2011), 163–181.

2. Mine Dogantan-Dack. 2012. The art of research in livemusic performance. Music Performance Research 5(2012), 34–48.

3. Oliver Hödl, Geraldine Fitzpatrick, Fares Kayali, andSimon Holland. 2017. Design Implications forTechnology-Mediated Audience Participationin LiveMusic. In Sound and Music Computing 2017, Helsinki,Finland, July 5–8, 2017.

4. Oliver Hödl, Fares Kayali, Geraldine Fitzpatrick, andSimon Holland. 2016. TMAP Design Cards forTechnology- Mediated Audience Participation in LiveMusic. In 2016 Conference on Human Factors inComputing Systems, San Jose, US, may 7–12, 2016,Workshop on Music and HCI. ACM, 1–4.

5. Fares Kayali, Christoph Bartmann, Oliver Hödl, RuthMateus-Berr, and Martin Pichlmair. 2017. PoèmeNumérique: Technology-Mediated AudienceParticipation (TMAP) Using Smartphones andHigh-Frequency Sound IDs. In Intelligent Technologies

for Interactive Entertainment: 8th InternationalConference, INTETAIN 2016, Utrecht, TheNetherlands, June 28–30, 2016, Revised SelectedPapers. Springer, 254–258.

6. Dario Mazzanti, Victor Zappi, Darwin G Caldwell, andAndrea Brogni. 2014. Augmented Stage forParticipatory Performances.. In Proceedings of theInternational Conference on New Interfaces for MusicalExpression 2014. 29–34.

7. Johannes Chrysostomus Wolfgang Gottlieb Mozart.1793. Anleitung so viel Walzer oder Schleifer, mit zweiWürfeln zu componieren so viel man will, ohnemusikalisch zu seijn, noch etwas von der Compositionzu verstehen. J.J. Hummel, Berlin-Amsterdam.

8. Martin Pichlmair and Fares Kayali. 2007. Levels ofSound: On the Principles of Interactivity in Music VideoGames.. In Proceedings of the Digital GamesResearch Association 2007 Conference: Situated play.

9. Lawrence M Zbikowski. 2002. Conceptualizing Music:Cognitive Structure, Theory, and Analysis. OxfordUniversity Press.