kinect to ipad project

7/29/2019 Kinect to iPad Project

1/4

Julian Joseph 1

University of Texas at Austin, Senior

[email protected]

Date: January 19th, 2012

Submitted to: Office of the Vice President for Research, Attn: URF 2011-2012

Kinect to iPad to Automated Fixtures:

A New Performance Lighting Method with Motion

To create an interface between a motion-tracking device and an automated lighting fixture

that is effective, easy to use, and affordable with direct application in live experimental

performances, but with potential to expand to more conventional settings. In this proposal and

project I will focus on the use of this interface by the performer on stage in a live situation.

Stage lighting is becoming more complex as live entertainment grows in scale and creativity.

With a modern automated light, a lighting technician must control a variety of parameters

including position, color, size, pattern, and speed. However, recent developments in the gaming

and mobile device industry allow for a certain amount of control to be given to performers in

more experimental situations at a relatively low cost. A relationship between the Kinect, iPad,

and automated lighting fixtures in particular has yet to be fully explored.

By creating a way to interface with automated lighting through a Kinect motion-tracking devicein conjunction with an iPad for remote control, the performer will be able to handle all of the

parameters listed previously in a live setting with

relative ease. In fact, musicians today, such as Skrillex,

are already beginning to use the Kinect1. On his 2011

tour, Skrillex enhanced his concert with digital

representations of himself while performing as seen in

Figure 1. This project will explore expanding the use of

a Kinect to include automated lighting fixtures.

Figure 1 - Skrillex1

With this setup, performers can use the Kinect to follow their actions on stage and will be able

to control lights. This means that a light can mimic their movements and/or change its color or

shape based on the performers movements on the stage. Additionally, the Kinect will allow a

light to follow the exact location of the performer. Essentially, the automated light could become

a tracking spotlight, or followspot, without the need for a human operator.


2/4


3/4

Julian Joseph 3


[email protected]

OSC data will be collected from the Kinect using a program called Synapse5 (Figure 3) on a

Mac computer. Once OSC data is received by the computer, it will be manipulated using a

programming language called Pure Data3. Pure Data is

required in order to covert resolution of the Kinect to the

resolution of a light. It is similar to how a laptop computer

converts its display resolution when connecting to an external

monitor or projector.

Figure 2 - Kinect4

After OSC data is collected and converted properly with Pure Data, it will be sent wirelessly

over a network to an iPad running an application called Luminar6 (Figure 4) that can convert

OSC data into lighting commands. The iPad will be wirelessly connected to a router and then an

adapter which feeds directly to an automated lighting fixture. Essentially, the light will then be

receiving commands indirectly from the Kinect. A diagram of

this procedure is shown in Figure 1.

Figure 3 - Synapses Kinect output5

At this point light parameters can begin to be manipulated by

the Kinect. For instance, the XY space that the Kinect sees

can be mapped to the pan and tilt of a light. Taking one step to the left will trigger a numerical

change in data being sent from the Kinect, which will be converted into light language and

trigger the light to respond by moving left. This association of the XY axis of the Kinect to the

pan and tilt on the light can be modified with the iPad to instead control color. So when the user

steps to the left, instead of the light moving, it will turn

more red or more blue. Of course, this can also be

replicated for any other parameter as well. Through

further experimentation, default presets for theseparameters can be established and then easily tweaked

on the spot in a performance by the user.

Figure 4 - Luminar for the iPad6


4/4

Julian Joseph 4


[email protected]

The Kinect and iPad are easy to use, affordable, and have motion-tracking features that allow

their users to control automated lighting on the fly in a way that is visually interesting and unique

to an audience. This method will not likely replace lighting consoles and designers for tradition

entertainment events like theatre, however, it is likely to enhance more experimental forms like

dance, performance art, and live music. This methodology will be tested and used to develop

new techniques for creating visual entertainment. This work will be documented and posted

online so that others can replicate and improve upon it.

Also, the following uses will be explored during this project in the hope of finding new

applications:

Programming a lighting show using body gestures instead of a control board Automatically focusing fixtures on an exact position of a person or object on stage

Followspot control for a performance which automatically tracks the subjects position

Combining real-time digital video from the Kinect with moving lights for a performance

Combining body motion with other interactive devices like Wii controllers

To realize this goal, some equipment must be purchased. Though the most expensive

equipment, a robotic light, has been secured, the Kinect, iPad, Luminar, and several other

intermediary tools are not freely available for this study. Additionally, once this area has been

explored, the information and techniques will be shared with the students currently enrolled in

the Automated Lighting Program at the Department of Theatre and Dance.

An Undergraduate Research Fellowship will allow the purchase of equipment, assure thorough

and accurate testing, and increase likelihood of the success of this project.

1Darren3179. Skrillex Ruffneck Bass @ Palladium Dallas.MP4 - YouTube. Digital image.YouTube - Broadcast Yourself. 29

Sept. 2011. Web. 22 Jan. 2012. .

2"Introduction to OSC." Opensoundcontrol.org. Web. 20 Jan. 2012. .

3Barknecht, Frank. "What Is Pd? PD Community Site."Pure Data PD Community Site. 04 May 2008. Web. 20

Jan. 2012. .

4Terdiman, Daniel.Microsoft Looks to Kinect as Game-changer | E3 2011 - CNET Reviews. Digital image.ProductReviews - Electronics Reviews, Computer Reviews & More - CNET Reviews. Microsoft. Web. 20 Jan. 2012. .

5Challinor, Ryan. "Synapse." Synapse for Kinect. 20 June 2012. Web. 20 Jan. 2012. .

6"Luminair for IPad - Multi-touch Wireless DMX Lighting Control." Synthe FX - Innovative Tools for Artists/designers Film/tv

Architectural. Synthe FX, LLC, 2012. Web. 20 Jan. 2012. .

kinect to ipad project

Documents