papersynth: distributed orchestra using mobile devices

PaperSynth: Distributed Orchestra Using Mobile DevicesPramod Verma

Johns Hopkins UniversityBaltimore, MD, 21218, USA

pramod@cs.jhu.edu

Figure 1. A prototype of a system for creating distributed orchestra using mobile devices.

ABSTRACTToday we are living in the age of handheld computing de-vices, equipped with powerful processing power, networkingcapabilities, and rich input-output interfaces such as camera,microphone, speaker, touch screen etc. In OzCHI24 designchallenge, we propose a system for creating real-time dis-tributed orchestra using smart phones.

Author KeywordsMobile HCI, Cloud-Computing, Orchestra

DESIGN REQUIREMENTSWe set following requirements for the concepts.

1. Users (Artists) are located at various places across theglobe.

2. They need a system or interface to play a synchronized or-chestra.

3. System should not cost additional resources, means itshould be designed using personal computing devices orgadgets such as iPhone or iPad.

PROTOTYPEOur prototype consists a pair of hand-held devices equippedwith camera and internet. For example users may use oneiPhone and one iPad. Two cameras (in two devices) are usedto detect depth information. For the visual assistance, weneed a piano keyboard layout that can be affixed on a tableor any surface. Paper layout also provides a tangible interac-tion.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.OZCHI’12, November 26-30, 2012, Melbourne, Victoria, Australia.Copyright 2012 ACM 978-1-4503-1438-1/12/11...$10.00.

Figure 2. Multiple users across different locations can collaborativelycreate orchestra with the help of communication and sharing using cloudcomputing.

Software, running on devices, uses a computer vision basedalgorithm to tack location of fingers on the paper layout, andplays related musical notes. User hears sound as a feed-back as soon as keys are touched. Multiple systems can beconnected with the help of cloud and data (musical sound)is shared in real-time with multiple users performing at dis-tributed places.

System also has some additional features such as recordingof any particular performance, music or orchestra, communi-cation for the preparation and sharing. Because some devicehas front and backside camera, we can use front camera forfinger detection and screen to display notes or real-time viewof other artists.

CONCLUSIONIn this design concept, we used mobile devices, cloud-computing and high speed internet to design a system forcreating a distributed orchestra. Our solution is easy to im-plement without an additional cost. Demo of design conceptis available at http://www.cs.jhu.edu/˜pramod/ozchi2012