mallo: a distributed, synchronized instrument for internet...

MalLo: A Distributed, Synchronized Instrument

for Internet Music Performance

Zeyu Jin⇤Department of Computer

SciencePrinceton University

[email protected]

Reid Oda⇤Department of Computer

SciencePrinceton University

[email protected]

Adam FinkelsteinDepartment of Computer

SciencePrinceton [email protected]

Rebecca FiebrinkDepartment of ComputingGoldsmiths, University of

[email protected]

ABSTRACTThe Internet holds a lot of potential as a music listening,collaboration, and performance space. It has become com-monplace to stream music and video of musical performanceover the web. However, the goal of playing rhythmicallysynchronized music over long distances has remained elu-sive due to the latency inherent in networked communica-tion. The farther apart two artists are from one another, thegreater the delay. Furthermore, latency times can changeabruptly with no warning.In this paper, we demonstrate that it is possible to cre-

ate a distributed, synchronized musical instrument that al-lows performers to play together over long distances, despitelatency. We describe one such instrument, MalLo, whichcombats latency by predicting a musician’s action before itis completed. MalLo sends information about a predictedmusical note over the Internet before it is played, and syn-thesizes this note at a collaborator’s location at nearly thesame moment it is played by the performer. MalLo also pro-tects against latency spikes by sending the prediction dataacross multiple network paths, with the intention of routingaround latency.

Author KeywordsNetworked Performance, Novel Musical Interface, ObjectTracking, Network Latency Reduction

1. INTRODUCTIONJust as the Internet has come to mediate so many otheraspects of our culture, it also o↵ers huge potential as ashared musical performance space. It a↵ords new oppor-tunities like discovering collaborators among a worldwidecommunity of musicians, holding impromptu improvisationsessions, and playing with others who share similar artis-tic goals and vision. Unfortunately, this potential remainslargely unrealized because communication latency inhibits

⇤These authors contributed equally.

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that copies

bear this notice and the full citation on the first page. To copy otherwise, to

republish, to post on servers or to redistribute to lists, requires prior specific

permission and/or a fee.

NIME’15, May 31-June 3, 2015, Louisiana State Univ., Baton Rouge, LA.Copyright remains with the author(s).

rhythmic synchronization among remote performers usingtraditional instruments.Depending on musical context, latency greater than some

threshold causes tempo drag : each musician repeatedly slowsdown in response to apparent delay from their remote part-ner [6, 7]. As latency increases it becomes increasingly di�-cult to stay on beat, until eventually the performance breaksdown. In practice, this means that artists have a limitedradius of collaboration, meaning they can only play withartists who are within a certain distance (because networklatency roughly correlates with physical distance). Playerscan adjust to a certain amount of lag, so if we assume thatpeople can tolerate 45 ms of latency [6, 7], it is possiblefor artists in New York and San Francisco to play with oneanother, but they will have to train themselves to ignorethe fact that the music is audibly out of sync, and be con-scious of not slowing down. Furthermore, they will be atthe mercy of network tra�c, where bursts of congestion arethe norm and cause unpredictable delays. This approachhas been the basis of many telematic performances playedin spite of lag [14]. Another approach to working with la-tency has been to allow one side of the connection to leadwhile the other follows [11]. However, these approaches donot attempt to remove latency, and so they cannot trulyallow for distributed ensembles in which the audio is syn-chronized among multiple locations. We envision a newcategory of musical instrument that is specifically designedto enable natural, latency-free collaborations between mu-sicians. These instruments are distributed, in the sense thatthey exist and generate audio in many locations, as well assynchronized, in the sense that audio is generated and heardin nearly the same moment in all locations. We demonstratein this paper that building such distributed, synchronizedmusical instruments (DSMIs) is achievable, and we presentthe implementation details of our approach alongside anevaluation of its e↵ectiveness.The synchronization in a DSMI is accomplished by pre-

dicting a note before it is played. These note predictionscan then be transmitted over the network early enough thatthey can be played remotely at the same time as they oc-cur locally (Figure 1). DSMIs capitalize on the fact thatthe musician no longer needs to manually excite a physicalinstrument in order to generate sound. Instead, we can usesynthesis algorithms along with a sound system to generateaudio. This decoupling is essential, because it allows us tobegin the transmission process without waiting for soundto be generated locally. Thus, when constructing a DSMI,we seek to design an instrument with predictable properties

293

Proceedings of the International Conference on New Interfaces for Musical Expression, Baton Rouge, LA, USA, May 31-June 3, 2015

rather than particular acoustic properties. Two crucial as-pects of predictability are the anticipation time (i.e., howfar in advance can it be made) and the prediction accuracy(i.e., how di↵erent is the predicted time from the time of theactual note event). Making predictions earlier in advancecan mask longer network delays (which generally correspondto longer distances), and making more accurate predictionsallows more authentic collaboration.In this paper, we present our DSMI named“MalLo”(short

for Mal let Locator), which combines prediction of a mu-sician’s gesture with a network forwarding strategy in or-der to support synchronization between distant performers.We show that MalLo is capable of playing a note that isnearly synchronized (within 30 ms) with an actual malletstrike thousands of miles away. MalLo builds on the work ofDahl [8] and Oda [13], who showed that the path of a per-cussion mallet head is su�ciently predictable that a DSMIcould be built around it. The swing gesture of a mallet islong (on the scale of 40-80 ms) and smooth. This smooth-ness stems from the purpose of the gesture: to e�cientlybuild kinetic energy to convert into audio, via a vibratingsurface. We describe MalLo’s sensing and prediction mech-anisms in Section 3.MalLo sends predictions from one location to another us-

ing a multipath forwarding approach, similar to an over-lay network [1]. This aspect of the instrument addressesthe problem that message latency on the Internet variesbased on network tra�c. By sending duplicate messagesover many di↵erent network paths at once, MalLo can avoidmoments of high latency that are associated with burstsof Internet congestion. Figure 2 illustrates these servers,placed in multiple strategic geographic location so as to cre-ate diverse routes through the Internet. We show that ourmethod reduces peak latencies, thereby increasing overallprediction accuracy and robustness.

2. RELATED WORKThere have been a number of musical instruments builtspecifically for the Internet. In particular, Sarkar and Ver-coe [16] built a predictive percussion instrument, which de-termines which pattern a sending performer is playing, andsynthesizes it for the Receiver. Our system is inspired bytheirs, but ours plays the same musical content on each endof the connection, while theirs might di↵er if the senderplays any material not in the learned repertoire. Somewhatsimilarly, Derbinsky uses reinforcement learning to allow thecomputer to learn drum patterns from a human performer,in order to act as a stand-in [9].Barbosa provides an in-depth survey of Internet music

collaborative systems that outlines the challenges of play-ing over the Internet, and it discusses previous uses of notemessages as a means of communicating between distributedensemble members [2] . Other works explore using the net-

Gesture Sense Predict Send Schedule Synthesize

Figure 1: An example of a distributed, synchronizedmusical instrument (DSMI). The instrument is com-posed of various parts that exist in di↵erent loca-tions. It uses prediction in order to synthesize thesame audio in each location at the same moment. Itcapitalizes on the predictable path of a percussionmallet in order to accomplish this goal.

Figure 2: To reduce latency peaks, messages aresent along multiple simultaneous paths by sendingboth to the Receiver and to forwarding servers, inan attempt to route around local network conges-tion.

work itself as a resonant instrument [5, 4], while Tanakacreated a hybrid instrument that uses both the Internet andphysical space [18].

3. MALLO CONSTRUCTIONThe key components of MalLo are (1) the sensing mecha-nism, (2) the prediction algorithm, which determines howthe mallet path is interpreted by both the Sender and Re-ceiver, and (3) the multipath forwarding network. The im-plementation of each component is described here, and thecomponents are evaluated in Section 4.We have built two versions of the MalLo sensing mech-

anism, one using a high-speed RGB camera and one usingthe Leap Motion sensor (Figure 3). The high-speed RGBversion is more accurate and was used for all tests in Sec-tion 4. The Leap Motion version was built as a proof ofconcept, to show that it is possible to construct MalLo withinexpensive components.

3.1 High-Speed RGB Camera VersionThe RGB camera version of MalLo has low latency andtight timing synchronization. The host for this system isa Linux host running Ubuntu 12.04, with a dual-core 2.40GHz Core 2 Duo processor. The camera is a Point GreyGRAS-03K2C-C FireWire 800 camera. It was chosen be-cause it can isochronously transfer images with low latency(6 ms) from the camera to computer. Furthermore, the 1394bus clock allows us to estimate shutter times with high ac-curacy (< 1 ms). It has a frame rate of 200 fps, a resolutionof 640⇥ 480 pixels (turned sideways), and a global shutterto reduce motion blur. These specifications roughly meet

High-speed RGB Camera Leap Motion Sensor

Figure 3: Two versions of MalLo. The high-speedRBG version uses a 200 fps camera, OpenCV totrack the mallet head, and a MIDI drum pad tocollect timing ground truth. The Leap Motion ver-sion version has no striking surface, and uses a yel-low highlighter pen as a mallet. It is a↵ordable andportable, but it has lower accuracy and higher la-tency.

294


Time (s)32.95 33 33.05 33.1 33.15 33.2 33.25 33.3 33.35 33.4 33.45

Malle

t H

eig

ht (%

)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Current time

Predicted timeFuture mallet samples

Past mallet samples

Samples used in prediction

Predicted trajectory

Current time

Predicted strike time

Actual strike time

Figure 4: To predict the strike time of a mallet,we fit a second-degree polynomial to the previous 7mallet locations. We compute the zero crossing toobtain the predicted time pi of the strike, and thederivative of the polynomial at time pi to obtain thevelocity vi.

the minimum requirements for tracking percussion malletsshown in [13]. Our striking surface is a Roland SPD11MIDI percussion pad, which is used to collect timing groundtruth data for the experiments presented in Section 4.The mallet head is tracked using OpenCV 2 libraries [3],

via the template matching algorithm [17]. This search al-gorithm is the most computationally expensive operation inour pipeline. A typical search time is 5–15 ms, bringing thetotal processing time to 11–21 ms per frame.

3.2 Leap Motion VersionWe built an inexpensive version that uses the Leap Motionas input. The Leap Motion is a small, portable commer-cial 3D sensor. It collects images using IR illumination andmultiple cameras at a rate of 200 fps, and it has a transferlatency of 20 ms. The device ships with the ability to trackthe tip of a stick, but the algorithm is proprietary. In gen-eral, the Leap Motion delivered noisier data than the RGBcamera. A companion video1 submitted with this paperdemonstrates the Leap Motion and RGB camera versionsof MalLo in action. From this point on in the paper, allreferences to MalLo refer to the high-speed RGB version.

3.3 Prediction, Transmission, and PlaybackWe perform a new prediction for each new mallet heighthi (one per frame). If the mallet is descending, the Senderfits a second degree polynomial (quadratic regression) tothe last k heights (7 in our tests) and solves for the zerocrossing, which is the predicted impact time, pi. The veloc-ity vi at impact time is computed as the derivative of thepolynomial at pi, in pixels per second. This curve-fittingapproach is shown in Figure 4. Finally, pi and vi are sentin a message to the Receiver via the multipath forwardingnetwork described in Section 3.4.For each new message received, the Receiver determines

if the message is newer than the last message received, anda user-specified inter-note wait time Tmin has passed; if so,the message is processed. If it is an “unschedule” message,then all events from that Sender are removed from the notescheduling queue. This is a standard scheduling method forrealtime systems. If it is a prediction message, then a noteevent is scheduled at time pi with velocity vi. When thenote is played, t0 is updated to pi. The Receiver continuesto replace older predictions with newer predictions until oneis finally played. In doing this the Receiver assumes thatprediction error monotonically decreases as the next strikeapproaches. In practice we found this to be true; each newprediction is almost always better than the previous.

1http://gfx.cs.princeton.edu/pubs/Jin 2015 MAD/

Latency (ms)21ms 40ms 48ms 61ms 78ms 90ms

Err

or

(s)

-0.04

-0.02

0

0.02

0.04MeanMedian

Figure 5: A pattern played at 120 bpm, with vary-ing average network latency, accomplished by send-ing note messages to various locations around theworld. Higher latencies created more variance inprediction accuracy. At 78 ms latency 99.6% of pre-dictions are within 30 ms of the ground truth.

3.4 Multipath Forwarding StrategyNaively sending the note prediction packet directly from theSender to the Receiver over the Internet leaves our packetat the mercy of temporary congestion that may lie in thepath chosen for it by the series of routers it encounters.To reduce the risk of sudden, severe congestion delayingreceipt of the prediction, we have constructed a multipathforwarding system that uses a series of servers throughoutthe Internet, hosted on Amazon EC2 and other commer-cial hosting companies. For each prediction to be sent, theSender sends one copy directly to the Receiver, as well asone copy to each of the forwarding servers. They, in turn,forward the message to the Receiver. In this manner, theredundancy helps to route around congested areas of theInternet. Ideally the servers should be located so that theirpaths are di↵erent than the direct route, in an arrangementsimilar to Figure 2.

4. EXPERIMENTS AND RESULTSWe have evaluated the e↵ectiveness of the prediction andmultipath forwarding components of MalLo using a series ofexperiments over the Internet. Our first experiment inves-tigates the timing accuracy of our prediction method underdi↵erent latency conditions. The second experiment inves-tigates the accuracy of our note velocity predictions. Ourthird examines the e↵ectiveness of the multipath forwardingstrategy in reducing latency spikes arising from congestion.Finally, we compare our current prediction system againsta previously proposed version [13].

4.1 Timing AccuracyOur main goal is to show that prediction can allow a per-former to play the same sound, synchronized, in multiplelocations, in spite of latency. Faster tempos and higherlatency make percussive strikes more di�cult to accuratelypredict, so we tested the system under di↵erent latency con-ditions and at di↵erent playing tempos.We recorded a musician playing a syncopated, repeating

pattern made up of quarter and eighth notes, at 60, 80, 100,120, and 140 bpm for 20 seconds. The recordings resultedin a sequence of tuples containing a mallet height and acorresponding timestamp. We also used the MIDI drumpad to collect the corresponding timing ground truth. Eachsequence has approximately 30 notes.To test the accuracy of our note timing predictions under

di↵erent latency conditions, we used the height/timestamptuples to generate timing predictions that were each sent

295


to di↵erent servers around the world. Each server immedi-ately redirected the message packets back to our lab, wherethey were received by a Receiver process running on thesame computer as the Sender. (This allowed perfect clocksynchronization between Sender and Receiver for our tests.)We consider the entire round trip time, to the server andback, as the tested latency. There were 6 servers in totalcorresponding with 20, 40, 47, 61, 73, and 88 ms of averagelatency. In general the latencies were stable, varying by +/-5 ms for each. However, at times we observed large momen-tary spikes in latency. (See Section 4.3 for more informationabout latency behavior.)We compare the predicted note onset time with the tim-

ing of the ground truth. Based on Chafe’s analysis of themusical consequences of di↵erent latencies between perform-ers [6], we consider notes played by the Receiver within 30ms of the ground truth as “ideal”, notes within 30–50 msof ground truth as “tolerable”, and those beyond 50 ms as“missed”. Table 1 shows the percentage of notes that were“ideal”. The table also includes false positives in the countof “missed” notes. (False positives occur when the musi-cian moves the mallet downwards but does not completethe motion with a strike and the Receiver does not receivean“unschedule”message in time.) Note that the system be-gins to break down with 88 ms of Internet latency, dropping6 out of every 100 notes at 120 bpm. However, at 73.0 ms,the system accurately predicts 99.6% of all notes. 73 ms isroughly the time it takes for a network packet to travel fromBerlin to Cairo, New York to Lima, Tel Aviv to Halifax, orTokyo to Los Angeles [15].For a closer look at the behavior of MalLo, Figure 5 shows

the distributions of predictions as average latency is var-ied (by using di↵erent servers), for a pattern played at 120bpm. Note that as latency increases, so does the varianceof the predictions. Also note that for each average latencylevel, the extreme predictions are never more than half theamount of latency. As latency increases, the mean predic-tion becomes later, and a noticeable protrusion begins toform at the top of the distribution. This is due to the eighthnotes in the pattern occurring too quickly to be predictedon time. However, although they are predicted late, theystill fall within the “tolerable” range.

LatencyBPM

60 80 100 120 140

20.0 ms 100.0 100.0 100.0 100.0 100.040.0 ms 100.0 100.0 100.0 100.0 100.047.0 ms 100.0 100.0 100.0 100.0 100.061.0 ms 100.0 100.0 100.0 100.0 100.073.0 ms 99.8 99.6 99.9 99.6 99.688.0 ms 100.0 100.0 97.9 94.7 94.7

Table 1: A pattern of eighth and sixteenth noteswas played back with various latencies and tempos.Shown are the percentage of notes predicted within30 ms of the ground truth. 30 ms is only barelyperceptible as non-synchronous.

Figure 6 shows the accuracy of predictions at a latencyof 77.5 ms at various tempos. We see the same behavior asFigure 5, with higher tempos showing higher variance. Thisis because higher tempos and higher latencies are roughlyequivalent as far as the algorithm is concerned. A fastertempo means there is less time between notes. Higher la-tency requires predictions to be made earlier in order tosend over the Internet in time. Both require that a pre-diction be made earlier in the stroke of the mallet, and if

Tempo (beats per minute)60 80 100 120 140

Err

or

(s)

-0.04

-0.02

0

0.02

0.04MeanMedian

Figure 6: A pattern played at varying tempos, over77.5 ms of average latency. Higher tempo causesmore variance, but in the worst case of 140 bpm,99.8% of predictions are within 30 ms of groundtruth.

Latency (ms)20 30 40 50 60 70 80

Velo

city

Err

or

(dB

)

-15-12

-9-6-30369

121518

MeanMedian

Figure 7: The di↵erence between ground truth andpredicted velocities (converted to probable soundpressure level when striking a rigid body). Perfor-mance is within an acceptable range up through 50ms, but begins to degrade. At 80 ms, 5% of esti-mates are at least 15 dB louder than ground truth.

the prediction is made too early (e.g. before the mallet hasdescended long enough) there will not be enough data topredict accurately.

4.2 Velocity AccuracyTo test velocity accuracy, we compare the predicted veloc-ity of the mallet with the measured velocity of the mal-let at the time of impact (as derived from camera trackingdata). Each measure of velocity is accomplished by fittinga quadratic to the mallet path. The di↵erence is that theprediction is made before the actual impact, and the groundtruth is measured at the exact time of impact. (We did notcompare the predicted velocity to the drum pad’s MIDI ve-locity because we found the velocity sensitivity of the drumpad to be quite noisy, producing widely di↵erent MIDI ve-locities for similar strike velocities.) We use the same veloc-ity error measure used by Oda et al. [13] which comparesthe relative sound pressure levels of the two strikes using anequation derived from Hertz’s original study of rigid bodyvibrations [10].Figure 7 displays the results. We found that velocity

prediction was quite precise up through 50 ms of averagelatency. 60 ms of latency brings an audible di↵erence invelocity, but not an extreme one. Performance then startsto degrade until 80 ms latency, where 5% of predictionsare at least 15dB louder than the ground truth. Using ourcurrent prediction algorithm, velocity prediction is not asrobust as timing prediction, but it is not as important forsynchronization.

4.3 Latency Peak ReductionTo test latency variation with and without the multipathnetwork, we sent a stream of packets from Sender to Re-ceiver via the network of 4 forwarding servers located inVirginia, Texas, Georgia, and London. The Sender machinewas located in New Jersey, USA, and the Receiver machine

296


was located in Oregon.On the Receiver’s end, we measured and compared the

one-way transmission time from Sender to Receiver for eachpath. The results of the latency measurements, with andwithout the multipath network, are shown in Figure 8. (Forthis experiment we synchronized the sending and receivingservers’ clocks using Network Time Protocol, which is onlyaccurate within tens of milliseconds [12]. But it is not cru-cial that the clocks be synchronized exactly, because we areinterested in the change of latency over time, not the abso-lute latency.)

200 400 600 800 1000 1200 1400 1600 1800

La

ten

cy (

ms)

0

50

100

150

200Direct Connection Latency Compared to Forwarders

Direct

Virginia

Texas

Georgia

London

200 400 600 800 1000 1200 1400 1600 1800

La

ten

cy (

ms)

0

20

40

60

80

100

120Latency Reduction

Original

Final

Diff

Time (s)200 400 600 800 1000 1200 1400 1600 1800

Fo

rwa

rde

rs

Direct

VA

TX

Fastest Route

Figure 8: The multipath network helped to reducelatency peaks. The top plot shows measured laten-cies for di↵erent paths. The middle plot shows theoriginal latency, the final latency, and the di↵erencebetween them. Note that two large spikes around1000 and 1500 seconds were reduced by roughlyhalf. The lower plot shows the shortest route ateach time point. The previously mentioned latencyspikes were reduced by routing through Texas.

This experiment shows the multipath system works to re-duce latency peaks. Latency peaks may be bursts (whichlast on the scale of seconds or milliseconds), or they may besustained (which can last several minutes, or even hours).Figure 8 shows that the multipath network is e↵ective in re-ducing both types. At times the route through Virginia wasfaster than the direct route, but when congestion caused thispath to slow down, the direct route became the preferredpath. When in use, the Virginia route reduced latency by 5ms, which can be enough time to allow an“unschedule”noteevent to arrive in time to cancel a false positive. The mul-tipath system was also e↵ective at preventing some burstlatency. At around 1000 and 1500 seconds there are twomajor bursts that are reduced by almost half by routingthrough Texas. These two reductions kept all peaks under80 ms, as opposed to greater than 100 ms. Finally, somebursts were not possible to alleviate. These occurred whenthe congestion was local to Sender or Receiver, or simplyincluded all the paths.

4.4 Comparison To Previous WorkOur current algorithm improves on the previous predictionmethod used by Oda et al. [13] in the way it deals withnetwork latency, even without considering the multipathforwarding strategy. In the original method, the Sendertransmits a single message per strike, sent once the Senderhas determined that a strike is imminent and the cannotwait any longer due to its estimate of the network latency.

60 80 100 120 140

Err

or

(ms)

0

5

10

Latency = 48.1ms

OldNew

60 80 100 120 1400

5

10

Latency = 61.0ms


Err

or

(ms)

0

5

10

Latency = 77.5ms


0

5

10

Latency = 89.8ms

Figure 9: Our new prediction algorithm has a lowermean error in nearly every case than the methodused in [13].

However, if the Sender estimates the network latency in-correctly, this can cause a note to be missed. Our currentmethod sends all predictions (on a per-frame basis) as longas the mallet is descending at a certain speed. This allowsthe Receiver to determine which of the predictions to acton. This, in essence, gives us a perfect knowledge of theper-packet latency.To compare our current algorithm to Oda et al.’s method,

we implemented a slightly-improved version of their algo-rithm that estimates the latency every 0.5 seconds, by send-ing a timestamped packet from Receiver to Sender and com-puting the one-way packet latency. We then model the la-tency as a Gaussian distribution with the packet latency asthe mean, and use the 95th percentile in the algorithm la-tency estimate. We use this conservative estimate in orderto avoid sending messages too late.We tested the relative e↵ectiveness of both methods by

running them simultaneously in the same manner as theexperiment in Section 4.1. The results of this comparisonare shown in Figure 9. At all levels of latency, the newalgorithm did as well as or better than the old approach.

5. DISCUSSION5.1 Widening the Radius of CollaborationOda et al. previously hypothesized that 40 ms of predic-tion at 120 beats per minute might be possible with an in-strument such as MalLo, but our experiments have shownthat nearly twice that is possible, in spite of using a rela-tively primitive prediction algorithm. This means a muchlarger radius of collaboration than expected. In our experi-ments we found that the round trip latency from New Yorkto San Francisco is roughly 80 ms. This means that thisDSMI can easily synchronize audio between artists in eachcity by making up for 40 ms of one-way latency on eachside. It also means that if there are latency spikes of upto 80 ms, MalLo can gracefully absorb them. Synchronizedperformances between major cities such as Amsterdam andDetroit (51 ms), London and Fez (47 ms), Hangzhou andHyderabad (41 ms), and Tel Aviv and Varna (57 ms) areall theoretically possible [15] without reaching the limits ofprediction. In our companion video we demonstrate a jamplayed over a latency of 90 ms, equivalent to the one-waylatency between New York and Buenos Aires.Our current prediction algorithm, which uses quadratic

regression, is fairly primitive. We chose this approach tohighlight that in this case it is relatively simple to predictthese notes before they are played. We anticipate that moreadvanced algorithms would add to the system’s range andstability.

5.2 Tradeoffs and ChallengesMalLo is a new type of instrument, and one that requires amusician to practice it to gain proficiency. The basic move-ment is easy to understand, because it uses the metaphor of

297


striking an object, but it is slightly di↵erent from a typicalpercussion instrument. This is likely to be the case with allDSMIs. In order to have su�cient time to predict reliably,it is useful to have longer, slightly slower movements thantraditional instruments.There are limits on how fast a musician can play using

MalLo. The fastest playable note sequence depends on theamount of latency. At a latency of 100 ms, a musician isroughly limited to playing eighth notes at 150 bpm beforeprediction accuracy fails.In some cases the multipath forwarding approach will not

protect against congestion. If the congestion is near theSender or Receiver then it may not be possible to routearound it.

5.3 Design PhilosophyInternet latency will not significantly improve without fun-damental changes to the architecture of the Internet. Ifwe want to take full advantage of the Internet as a per-formance space we must design instruments that can beplayed despite its latency characteristics. In this work, wedemonstrate a new approach to designing Internet-basedinstruments that preserves certain characteristics of mostexisting instruments—namely, the ability to be played innote-by-note synchrony by ensembles of musicians.What properties of an instrument make it well suited to

distributed, synchronous performance? What might be theshared design strategies for new DSMIs? These questionsare wide open to future exploration. So far, it seems clearthat an instrument can be made to be more predictable byslowing down the movements of a musician, or by restrictingpaths of movement in particular ways. One might be ableto predict the fingering of a flute-like controller by placingpressure sensors on the keys, while also making them requirea fair amount of force to press. This would give the systemtime to infer the final finger position. A kick drum pedalmight also be a good candidate for pressure sensing andtracking, because of its long travel and constant contactwith a foot. However, we do not yet know if this slowing ofinstruments would be enjoyable for musicians, or irritating,and we intend to explore this question in future work.

6. CONCLUSIONS AND FUTURE WORKIn this paper, we have shown how it is possible to use pre-diction in order to create a truly distributed, synchronized,musical instrument (DSMI) despite the latency inherent inthe Internet. We described the implementation of one suchinstrument, MalLo, and showed that its timing predictionis within the perceptual bounds of synchrony for networkone-way latencies up to 73 ms, and that its velocity predic-tion performs well for latencies below 50 ms. We createdstraightforward algorithms for both the sending and receiv-ing parts of the DSMI. Finally, we presented a method forlatency peak reduction, using a network of servers to achievemultipath transmission. These proved e↵ective at reducingthe e↵ects long term network fluctuations on the order of5ms, and some burst latency on the order of 40 ms.There are a number of new directions to explore using

MalLo and DSMIs. While synchronization can be achievedwith purely audio cues, it helps to have visual cues as well.In the future we would like to explore using the same pre-diction scheme to drive a visualization of the Sender’s move-ments. Furthermore, the prediction strategies describedhere could be used not only for combating latency, but forlocal purposes, such as time correcting a note that is playedlate. They could also be used to enable expensive synthesisalgorithms, such as simulating the sympathetically vibrat-ing strings of a piano. Finally, of course, there is ample

space for future work exploring how to construct new DSMIinterfaces that best enable accurate prediction of musicallyexpressive actions.

7. ACKNOWLEDGMENTSThis work was supported by the Project X fund at Prince-ton University. Much thanks to Huiwen Chang and ShuranSong for help in developing the MalLo application, and toKatie Wolf for valuable advice.

8. REFERENCES[1] D. Andersen, H. Balakrishnan, F. Kaashoek, and

R. Morris. Resilient overlay networks. In Proc. ACMSymposium on Operating Systems Principles, 2001.

[2] Á. Barbosa. Displaced soundscapes: A survey ofnetwork systems for music and sonic art creation.Leonardo Music Journal, 13, 2003.

[3] G. Bradski and A. Kaehler. Learning OpenCV:Computer Vision with the OpenCV library. O’Reilly,2008.

[4] J. P. Cáceres and A. B. Renaud. Playing the network:the use of time delays as musical devices. In Proc.International Computer Music Conference, 2008.

[5] C. Chafe. Distributed internet reverberation for audiocollaboration. In Proc. AES 24th InternationalConference. Audio Engineering Society, 2003.

[6] C. Chafe. Living with net lag. In Proc. AES 43rdInternational Conference, 2012.

[7] E. Chew, A. Sawchuk, C. Tanoue, andR. Zimmermann. Segmental tempo analysis ofperformances in user-centered experiments in thedistributed immersive performance project. In Proc.Sound and Music Computing Conference, 2005.

[8] S. Dahl. Striking movements: A survey of motionanalysis of percussionists. Acoustical Science andTechnology, 32, 2011.

[9] N. Derbinsky and G. Essl. Exploring reinforcementlearning for mobile percussive collaboration. Proc.NIME, 2012.

[10] H. Hertz. On the contact of elastic solids. Math, 92,1881.

[11] A. Kapur, G. Wang, P. Davidson, and P. Cook.Interactive Network Performance: A dream worthdreaming? Organised Sound, 10, 2005.

[12] D. L. Mills. Internet time synchronization: Thenetwork time protocol. Communications, IEEETransactions on, 39, 1991.

[13] R. Oda, A. Finkelstein, and R. Fiebrink. Towardsnote-level prediction for networked musicperformance. In Proc. NIME, 2013.

[14] P. Oliveros, S. Weaver, M. Dresser, J. Pitcher,J. Braasch, and C. Chafe. Telematic music: Sixperspectives. Leonardo Music Journal, 19, 2009.

[15] P. Reinheimer and R. Will. WonderNetwork: GlobalPing Statistics, Jan. 2015.https://wondernetwork.com/pings.

[16] M. Sarkar and B. Vercoe. Recognition and predictionin a network music performance system for Indianpercussion. Proc. NIME, 2007.

[17] R. Szeliski. Computer vision: Algorithms andapplications. Springer, 2010.

[18] A. Tanaka and B. Bongers. Global string: A musicalinstrument for hybrid space. In Proc. Cast01//Livingin Mixed Realities, 2001.

298


mallo: a distributed, synchronized instrument for internet...

Documents