dance jockey: playing music by dancing at tedxluanda

7/31/2019 Dance Jockey: Playing Music by Dancing at TEDxLuanda

1/10

1

DanceJockey:PlayingMusicby

DancingatTEDxLuanda

YagodeQuay

Aug us t 25 , 20 12

AbstractThispaperreportsonDanceJockeysperformanceattheTEDxLuandaconference.It

examinesthemethodologybehindthesoftware,hardware,musicand

choreography.Describesthesocialcontextthatwascrucialtothedevelopmentof

thehardwareandsoftwareininteractivemusicperformances.Concludeswith

lessonslearnedfromtheperformance.

Keywords:userinnovation;interactivemusic,interactivedance,interactive

projection,Kinectcamera,motioncapture

1IntroductionResearchersandhobbyistsalikeareinnovatinginnewfieldswhichtheymightnotbe

consideredexperts.Theavailabilityofaffordablesensorstogetherwithwillingness

tosharesoftware,ideasandmethodsarefuelingthedevelopmentofhome-brewed

entertainmentsystemsthatempoweruswithnewwaysofinteractwithpeopleand

things.DanceJockeyisanexampleofasystemthatenablesmusicandvisualstobe

generatedwithbodymovements.Usingcamerasandsensorsthebodyofadanceris

turnedintoalivingsynthesizer.

ThispaperstartsbydescribingthecontextinwhichDanceJockeywascreated.The

workscitedexplainwhatfactorscontributedtothedevelopmentofitssoftwareand

hardwareandwellasthemotivationsbehindinnovativeworks.Insection3wewill

seeexamplesofartistsandresearcherswhohavetakenituponthemselvesto

innovatethefieldofmusiccompositionandperformance.Section4willprovide

insightintohowmovementwascoupletointeractivemedia,andsounddesign

techniques.Thelastsectionwillreviewtheexperiencesgatheredfromtherehearsalsandperformance.Inthispaper,DanceJockeyreferstoaninteractive

music/danceperformancepresentedatTEDxLuandaon26ofMay,2012.Avideoof

theperformancecanbefoundhere:http://youtu.be/uYjdLVAEQzY

2UserInnovationEndusersmaychoosetoinnovatewhenaspecifictechnologydoesnotfittheir

needs.(Hippel2005)citesastudyillustratingthefrequencywithwhichuserfirms

andindividualconsumersdevelopormodifyproducts.Findingsshowthatinoutdoor

consumerproducts,mountainbikingequipmentandextremesportingequipment,thepercentagesofusersdevelopingtheproductforownuseare9.8%,19.2%,and


2/10

2

37.8%respectively.Heclaimsthatinnovationisbeingdemocratizedandthosewho

dontinnovatecanbenefitfrominnovationsfreelysharedbyothers.

Withintheseendusers,(Franke,Hippeletal.2006)identifiedagroupthatthey

calledleadusers,whocanhelpdevelopbreakthroughproducts.Leadusersare

membersofauserpopulationthat1)expectobtaininghighbenefitsfromthedevelopmentofaproduct,2)areaheadofthemarketplacetrendandthus

anticipatingtheneedsofmanyusers.Thelattercharacteristicalsopredictsbothuser

innovationandlikelihoodofinnovationattractiveness.

(ShahandTripsas2007)arguethatthecommercializationofinnovativeproductsby

individualusersismorelikelytohappeninindustrieswhereuseprovidesenjoyment

insteadofpurelyeconomicbenefits,haverelativelylowopportunitycostsandhigh

varietyofdemand.

Theauthorsclaimthatprioremploymentoruniversitytiescanprovideinsightinto

theapplicationofemergingtechnologies.Furthermore,usersfromoutsideagiven

fieldareofteninabetterpositiontofindinnovativesolutionsbecausetheyframe

theproblemdifferently.

(JeppesenandFrederiksen2006)confirmthepreviousfindingsinanonline

communityrevolvingaroundPropellerheadsdigitalaudioproductionsoftware.In

thisforum,userscouldaskandanswerquestions,testsandredesignmanyofthe

softwaredevelopedbythecompany.Thekeyfindingisthatinnovativeusersare

likelytobehobbyistsinthefieldinwhichtheyinnovate.

Hobbyistsinthiscontextarealsomorelikelytoshareinnovationbecausetheyarenotinacompetitionwithotherusers.Infact,56%ofthesoftwaremodifications

weresharedintheonlinecommunityand33%ofusersreportedbuildinguponthe

workofotherusers.Theauthorsconcludethatthesefactorsincreasethequalityof

innovationtheymaketheproductmoreattractiveandaheadofmarkettrend.

Inthefieldofelectronicmusic,(Rowe1996)exploresthemotivationsbehindthe

efforttotransfermusicalconceptstocomputerprograms.Heclaimsthatcomputers

openthepossibilitiesforexploringthecompositionoftimbresandrhythms,andthe

executionofmeta-compositionalmethodsforgeneratingmusicaltexturesand

structures.Hisinterestsinthisfieldaremakingprogramsmoreresponsivetomusicalinputandbetterperformers.

(Winkler1999)addsthattheabilitytohavecompletecontroloverallaspectof

musicalcompositionwaspartofwhatattractedcomposerstowardscomputer

music.Healsopointsouthowmostoftheearlycomputermusicinnovation

happenedinresearchfacilitieslikeColumbiaUniversity,StanfordUniversity,andBell

Labs.Thissuggeststhatclosetiestoresearchanddevelopmentfacilitiesmay

increasethelikelihoodofinnovationinthisfield.


3/10

3

3PreviousWorksResearcherandperformerToddWinklerusedDavidRokebysVeryNervousSystem

thatdetectsspeedandlocationofdancerstomappositionaldatatoinstruments

andlargemusicalprocesses(Winkler1997).Sectionsintheperformancehadoneof

fourdifferenttypesofmovement:mechanicalandrepetitiveforpercussiveinstruments;slowandfluidforevolvingsounds;practicalandtechnicalforlive

mixing;andtheatricalfortriggeringcomicalsamples.Mappingsrangedfromnatural

correlationsbetweenlowenergymovementsandsound,tocontradictoryand

unpredictableresponse.Theauthordescribedthesoundsasmainlypercussive,vocal

andmechanical.

ThePalindromedancecompanyhasbeensonicallyandvisuallyaugmentingtheir

performancesusingtheirownEyeConmotiontrackingsoftware(Wechsler2011).

Therearetwoapproachesformotiontracking:Usingexternaldeviceslikecameras

orbody-wornsensorslikeelectrodes.Hearguesthatalthoughmotiontracking

technologyhasthepotentialtoincreaseinteractivityinperformingarts,itsthe

techniquesusedforcouplingcaptureddatatomediathatwillultimatelyvalidatethe

interactivity.

TheartistImogenHeaptogetherwithresearcherTomMitchelldevelopDataGloves,

apairofgloveswornbytheartistasaninterfacetoalivemusicalcomposition

systemwhichisentirelycontrolledbygestures.Theglovestrackorientationand

fingerandupperbodymotion.Examplesofinteractionincludehigh-passfilters

mappedtohanddistance,playingairdrum,manipulatingvolume,reverb,and

panning(Shepherd2011).

Inanotherexampleusinghand-heldcontrollers,OnyxAshantiimprovisesjazzmusic

withasensorthatconvertshisfinger,handandbreathmovementsintocontroldata

(Ashanti2012).Themusicalinteractionwiththesensorissimilartothatwitha

saxophone.Duringaperformance,alightshowillustratestotheaudiencewhichpart

ofthearrangementisbeingplayed.

(Guedes2007)exploreshowsimilaritiesbetweenmusicalrhythmandrhythmin

dancecanbringnewpossibilitiestocomputer-mediatedinteractionsbetween

dancersandmusicians.UsingasetofMAX/MSPalgorithmscalledm-objects,his

variousinstallationsandperformancesusuallyfeatureadancercontrollingthe

tempoofthemusicwithhis/hermovements.

ChrisVikdevelopedandperformswithasoftwarecalledKinectarthatusesdata

fromMicrosoftsKinectcamera(Microsoft2011)tomapbodylimbstoMIDIevents

(Vik2012).Thesoftwareisdesignedtofacilitatemelodicandharmonicinteraction

withdigitalaudioworkstations.

Inanexampleusingbiofeedback,PamelaZdevelopedasystemallowinginterplay

betweenhervoiceandagesture-basedMIDIcontrollercalledBodySynththat

translateselectricalsignalsgeneratedbymusclecontractionsintoMIDIcommands

(Zone2011).TheNewYorkdancecompanyTroikaRanchhasbeenknowntousea


4/10

4

wirelesssuitofuptoeightbendsensorstomanipulatevisualandsonicelements

(Ranch2011).

4Methodology

Usingcheapvideogamecontrollers,DanceJockeyextractsdancefeatureslikeposesandgesturesareinterpretedandlinkedtosoundgeneratingsoftware.Contraryto

mostconcertsfeaturingelectronicmusic,inthereisnooneworkingbehindthe

computer.Allthecommunicationdonebetweentheperformersandthecomputer

arethroughgestures,andmostofthetimestheonlyfeedbackissound.

4.1MotionCapture

Originally,DanceJockeywasperformedwiththeexpensiveandadvancedXsens

MVNmotioncapturesuitnormallyusedforanimatingcomputergenerated

charactersinfilms.Howeverduetothegrowthoffree,opensourcesoftwareand

theavailabilityofcommercialmotioncapture,simplerandmoreflexiblesolutionswerefounds.Becausecheapmotioncapturesoftwareandhardwarecanbehacked

easilyusershavethefreedomtocustomize.

Twomotioncapturetechnologieswereusedinthisperformance,MicrosoftsKinect

cameraandNitendosWiiRemote.Thefirstisaspecialcamerawithdepth

perceptionandlimbtrackingthatcomeswithvideogameconsoleXbox360.

SoftwareforconnectingtheKinecttocomputersissharedfreelybyvarious

developers,whichcontributesitspopularityamongmediaartists.Recognizingthese

unexpectedimplementations,Microsoftreleasedavideothatbecamethemost

viewedvideoaboutKinectonyoutube.com.Amongthetophitsisalsoavideoshowingthe12bestKinecthacks.Thesecondmotioncapturetechnologyisa

handheldsensorremotecontrollerfortheWiivideogameconsole.Itcansense

accelerationamongthreeaxesandinfraredlight.Again,oneofthetop10most

viewedvideosoftheTechnology,EntertainmentandDesign(TED)conferences

youtube.comchannelisoneaboutWiiRemotehacks.

EventhoughtheKinectandWiiRemotewereoriginallydesignedascontrollersfor

videogameconsoles,softwaredevelopedbyhobbyistsenableonetoconnectthese

controllerstocomputers.TheKinectplugsintothecomputerthroughtheUSBport.

TheapplicationSynapse1thengrabsthedatacapturedbytheKinectandsendsitto

anyapplicationthatcanreceiveOpenSoundControlprotocol.TheWiiRemotelinks

tothecomputerbytheBluetoothwirelessnetwork.TheapplicationOsculator2

convertsdatareceivedfromthecontrollerandconvertsittovariousformatsthat

canbeusedbyotherapplications.

TheKinectwasplacedatthecenterofthestage,3metersawayfromtheperformer.

Thisgaveanareaofupto2x2metersfortheperformertomoveaboutwithout

1http://synapsekinect.tumblr.com/developedbyRyanChallinor2http://www.osculator.net/


5/10

5

gettingoutofthecamerasfieldofvision.TheWiiRemotewasheldbyoneofthe

performersduringthetalkandinthebeginningoftheconcert.

4.2ControlUnit

Thecontrolunitwasahubwherealltheinformationfromsensorswasaggregated,

transformedtocommands,andthentransmittedtodifferentapplications.Itwastheonlysoftwaredevelopedfromscratch,usingthevisualprogramminglanguage

Max/MSP.Itsarchitecturewasbasedonthefinite-statemachinemodelthatis

comprisedofanumberofstates,eachstatecontainsinstructionsonhowtomap

incomingmotioncapturedatatoaudiocommands,andconditionstotransfertothe

nextstate.Duringtheperformancethecontrolunitpassessequentiallythroughalist

ofdifferentstates,oneatatime.Theorderandmappingofthestatesareessentially

themusiccomposition.

DanceJockeyisinterestedinexploringthepossible

connectionsbetweenmovementandsound.Thereforethegoalofthecontrolunitwasnottoperfectone

specificmappingbuttoquicklyexperimentwithdifferent

mappings.Toachievethis,eachstate,togetherwithits

mappings,wasencapsulatedinamodule.Modulecould

becopy/pastedfromoneperformancetoanother

withoutloosinganyinformation.Sincecompatibilitywith

differentmotioncapturedevicesandsoftwarewas

importantfortheinteraction,datawastransferredin

OpenSoundControlformatthroughtheUserDatagram

Protocol(UDP).

Figure1showsthemappinginsideLuzmodule.Kinect

dataisfilteredtogetthelefthandheightposition

relativetothewaist.Minimum/maximumboundsscale

theinputtoa0.0-1.0range.Whenthehandgetsbelow

0.9,aninstructiontoplayasoundclipofalightbulb

beingswitchedonissenttothemusicsoftware.

Immediatelythecontrolunitgoestothenextscene

calledpitchglitchascanbeseenonError!Reference

ourcenotfound..

4.3Music

Theperformanceconsistedofatalkandaconcert.Duringthetalk,soundeffects

weretriggeredandmanipulatedwithgesturescapturesbytheWiiRemoteorKinect.

Thesoundsreinforcedtheactionshappeningonstagesotheaudiencewould

understandthatthelecturersmovementswerecausingthesound.

Theconcertplayedtwosongs.Thisfirstsongstartedwiththeperformerspassinga

sonicobjecttoeachother.Thentheobjectwasthrowntothecrowdandstarteda

beat.Anyregisteredupperbodymovementperformedbythedancerwouldunmute

thebeatandthevisuals.Allthemappingsbetweensound,visualsandmovement

Figure1-Insideoneofthe

scenes


6/10

6

wereobvioustotheobserver.Afterthedancerperformedahandstandthenext

songwasintroduced.

Thesecondsongmixedlivebeatboxingbythemusicianwiththesynthesizersounds

producedbythedancer.Themusiciancreatedaloopingbeatbystackinglayersof

vocalpercussionandmelodies.IntheAsectionofthesongthedancerlockedintooneofthefivepositionsthattriggeredauniquesustainingchord.Thechoreography

andchordprogressionwerecomposedsimultaneously.Positionswererecognized

usingtherelativeheight,width,anddepthdistancesfromeachhandtothechest.

Chordswerequantizedtothebeattokeepthemontime.Furthermore,relative

hand-to-chestheightopenedorclosedafilterappliedtothechords.Thelefthand

affectedtheleftchannelwhiletherighthandaffectedtherightchannel.The

musicianandthedancerplayedsimultaneously.

Figure2-Allthestatesarrangedchronologically

IntheBsectionthemusicianmutedorunmutedthepreviouslyrecodedloopswhilethedancerplayedasynthesizerwithherhands.Lefthandheightraisedthevolume,

widthreducedawobbleeffect;righthandheightclosedahighpassfilter,depth

reducedthereverb.Theendingsectionfeaturedaloopthatsloweddownasher

headwaslowered.

AllaudiowasproducedindigitalaudioworkstationcalledAbletonLive8.This

softwareisrobustforliveperformancesandcanincorporateMax/MSPpatchesused

inthecontrolunit.AseriesofmodelsweredevelopedforAbletonLivethatcould

receiveawidevarietyofinstructionsfromtheControlUnitthruOpenSound

Control.TheseinstructionscouldbemappedtoanymusicalparameterinAbletonLive.ThemusiciancontrolledloopsandmanagedsectionswithMIDIkeyboard.


7/10

7

4.4Choreography

ThefinalchoreographywasaconsensusbetweentheartisticdirectorLauraFerro3,

dancerAnaisaLopes,andI,themusician.Decisionsaboutwhatmovementstouse

hadnotonlytobevisuallyattractivebutalsosonicallypleasing.Moreover,the

relationshipbetweenagestureanditssonicconsequencehastobedemonstratedto

theaudiencesotheywouldunderstandtheinteraction.

EverydayIwouldbringasoundtorehearsals,maptheinteractiveparametersto

Anaisaslimbs,andthenLaurawouldcreateachoreographywhileIlistenedtothe

result.Wedtakenotesonwhatsounds/movementsworkedandIdgohometo

improvethesystemoraddnewsounds.Slowlybuildinguponpreviouslyaccepted

soundsandmovementsweetchedoutthewholeconcert.

Figure3-MIDIKeyboardusedbythemusicianwithwrittencommands

4.5Projection

Otherthantheslidesandanimationsusedtosupportthetalk,theconcertfeatured

videoclipsthatrespondedtothemovementofthedancer.Insteadofcomputer-

generatedimages,weoptedformanipulationofpre-recordedvideosequences.The

footagewaschosenfromapoolofhighquality,artisticclipsfilmedbyPatriciaVidal

Delgado4.Theyhaveanorganicfeelthatisnoteasilyfoundincomputer-generated

images.Itwasalsofasterthanhavingtocreatenewgraphicssomoretimecouldbe

spentdecidinghowtheywouldbemanipulated.Thefirstcomputerrunningthe

controlunitwouldsendinstructionsandinformationthrutheEthernetandOpen

3www.lauraferro.com4www.pvdelgado.com


8/10

8

SoundControltoasecondcomputerprocessingthevisuals.RodrigoGuedes5

designedhowthevideowasmanipulated.

Theconcertfeaturedadifferentvideoclipineachsectionandmotioncapturedata

usedtomanipulateaudiowasalsousedtomanipulatethevideo.Inthefirstpartof

theconcerttheprojectiondepictedaballthattransformedintoabeamoflightwhenmovementwasdetectedwiththeKinectorWiiRemote.Theoriginalfootage

wasofthemoonatnightafteraddingsomefilters.Thenaseriesofclipswereshown

whenmovementwasdetected.Theseclipscapturedtheprocessofdisintegratinga

photographbysubmergingitinaspecialchemical.IntheAsectionwiththebeat

boxingandthedancerperformingatthesametime,thehandheightofthedancer

madethevideopanlefttoright.InBfollowingsectionwiththedancershandheight

stretchedthevideo.Inthelastsection,headheightdecreasedtheopacityofthe

video.Thusintheendherheadwasdownandthescreenwasblack.

5ResultsDifferentmotioncapturetechniqueswillresultindifferenttypesofmovementdata.

Intheenditsaquestionoffindingouthowtoworkwithaspecifictechnology.In

thiscasetheKinectwasabletoprovidewithlowlatency,detailedinformationabout

thepositionofallthelimbsonstage.HowevertheKinectisacamerasothemotion

captureprocessisinvisibletotheaudience.Thetalkpriortotheperformancewas

crucialinmakingcleartherelationshipbetweenmovementandsound.Thisalso

meantthatoftenthemovementshadtobeclearanddidactic.Also,theKinecthasa

smallcaptureareasothedancercouldntmoveaboutwithhertypicalfreedom.The

dancersawthisasalimitation.

ThecontrolunitofferedaflexiblewaytohandleanddistributedatafromtheKinect.

Ontheflymodificationstothecontrolunitcouldbedonewithoutworryingabout

side-effects,whichresultedinasmallgapofexecution.Thefinite-statemachine

modelforschedulingthemappingsprovedtobearobustperformancetechniquefor

linearcomposition.However,insomeoccasions,theperformerwouldtriggerthe

nextstateaccidentallyandtherewouldbenowaytogobacktothepreviousstage

withoutgoingtothecomputer.

Nothavingsomeonebehindacomputeronstagehelpedgivetheperformancea

moremagicalfeel.Italsomadetheperformersfocustheirattentionontheirown

movementsnotthescreen.DanceJockeygoesbeyondthetraditionalprocessesof

producingsoundsthroughinstrumentslikeanacousticinstrumentoraDJdeck.To

theaudiencetheinteractionbetweenmovementandsoundisunquestionable;its

innovativesynergybetweendanceandmusicthattheyappreciate.

Havingtheprojectionpointedonthedancerprovidedsomeunexpectedresults.

Initiallywethoughttheshadowofthedancerwasaproblemandconsideredusing

rear-projectionbutthespacedidntallowit.However,manyoftheaudience

5www.vimeo.com/visiophone


9/10

9

commentedthatthatlikedtheshadowonthescreen.Furthermoretheprojection

addedtexturestothedancersskin.

Figure4-Dancerwiththeprojectiononthebackground

6ConclusionThispaperexplainedtheinnerworkingsofaperformancecalledDanceJockeywhere

electronicsoundsarecontrolledbythemovementsofadance.Itarguesthatthe

underlyingcontextofthesetypesofnon-professionalmediaworkisuserinnovation.

Thegoaloftheperformancewasnottosonifymovement,butdancefymusic.

Thereforemusicalcoherenceanddiversitywasthetoppriority.Usingopen-source

software,proprietarysoftwareandvideogamecontrollers,DanceJockeyharnessed

motioncapturedatafromdifferentsourcesanddynamicallymappedittointeractive

media.Theshort8minuteconcertatTEDxLuandarepresentedthefirsttimeDance

Jockeycombinedlivesoundswithelectronicsoundsatthesametimeonstage.


10/10

10

BibliographyAshanti,O.(2012)."OnyxAshanti."RetrievedJune,2012,from

http://www.onyx-ashanti.com.

Franke,N.,E.v.Hippel,etal.(2006)."Findingcommerciallyattractiveuserinnovations:Atestofleadusertheory."JournalofProductInnovation

Management23:301-315.Guedes,C.(2007)."TranslatingDanceMovementIntoMusicalRhythmInReal

Time:NewPossibilitiesForComputer-MediatedCollaborationIn

InteractiveDancePerformance."TheInternationalComputerMusicConference,Copenhagen,Denmark.

Hippel,E.v.(2005)."DemocratizingInnovation."

Jeppesen,L.B.andL.Frederiksen(2006)."WhyDoUsersContributetoFirm-HostedUserCommunities?TheCaseofComputer-ControlledMusic

Instruments."OrganizationScience17(1):45-63.Microsoft.(2011)."KinectGames."RetrievedJan9,2011,from

http://www.kgames.org/.Ranch,T.(2011)."TroikaRanch"RetrievedApril,2011,from

http://www.troikaranch.org.

Rowe,R.(1996)."Incrementallyimprovinginteractivemusicsystems."

ContemporaryMusicReview13(2):47-62.Shah,S.K.andM.Tripsas(2007)."Theaccidentalentrepreneur:Theemergent

andcollectiveprocessofuserentrepreneurship."StrategicEntrepreneurshipJournal1:123140.

Shepherd,I.(2011)."ImogenHeapDataGlovesvideo@WiredFutureOfMusic

20July2011."RetrievedJune,2012,fromhttp://productionadvice.co.uk/imogen-heap-data-gloves-video/.

Vik,C.(2012,2012)."Kinectar."RetrievedJune,2012,fromhttp://www.kinectar.org/.

Wechsler,R.(2011)."PalindromeIntermediaPerformanceGroup."Retrieved

March10,2011,fromhttp://www.palindrome.de/.Winkler,T.(1997)."CreatingInteactiveDancewiththeVeryNervousSystem."

ConnecticutCollegeSymposiumonArtsandTechnology.

Winkler,T.(1999)."Composinginteractivemusic:techniquesandideasusingMax."Cambridge,Mass.,MITPress.

Zone,S.(2011)."BodySynth."RetrievedApril,2011,fromhttp://www.synthzone.com/bsynth.html.

dance jockey: playing music by dancing at tedxluanda

Documents