1.TAMMUZ DUBNOV, UC BERKELEY, TDUBNOV@GMAIL.COM2014 SPIE ELECTRONIC IMAGINGINTERACTIVE PROJECTION FOR AERIAL DANCE USING DEPTH SENSING CAMERA

2. OUTLINE What is Zuzor? Components How it works Application areas Creative thinking Including video examplesJoint work with Zachary Seldess and Shlomo Dubnov, Qualcomm Institute / Calit2, UCSD 3. ZUZOR An interactive performance system for floor and Aerial Dance using a Microsoft Kinect Detects free movement in space with 3 degree of freedom Gesture tracking and recognition Creates an augmented reality for performances 4. COMPONENTS Kinect IR Camera Tracker Software Gesture Recognition Software Graphic Animation Software Projector Performer with IR Markers 5. KINECT RGB Camera Depth sensor Multi-array microphone Not used in this project Can stream either RGB image or IR image Commonly used in XBOX 6. KINECT CAMERA TRACKING User interface: Receives IR image Receives Depth image Does not receive RGB Controls tilt Can record Many fine settings that can accommodate almost any setting 7. PRE-PROCESSING AND 3D COORDINATES OF IR BLOBS Convert the Kinects raw depth values to meters Thresholding of IR image (user defined) Segmentation of 2D IR blobs Depth estimate Calculate the (x,y) coordinates 8. KINECT CAMERA TRACKING DEPTH VIEW 9. KINECT CAMERA TRACKING IR VIEW 10. GESTURE RECOGNITION Gesture representation as a sequence of xyz coordinates (from the IR markers) Recognition and tracking using modified Hidden Markov Model (HMM) overall likelihood time index and detected speed HMM observation probabilities set by user 11. HIDDEN MARKOV MODEL Left-to-right Markov chain Every time instance corresponds to a state Probability of an observation is represented by a normal distribution Variance set by the user 12. GESTURE RECOGNITION 13. ANIMATION We used Quartz Composer to construct the animation / graphics Communicate with the IR marker tracking and gesture recognition using OSC over UDP OSC = Open Sound Control UDP = User Datagram Protocol 14. SETUP Place IR marker on performer Connect Kinect and Projector to computer Bind ports appropriately between Kinect software, gesture recognition software and Quartz composer. 15. APPLICATION AREAS Performances in an interactive augmented reality Dance A partner piece between the dancers and the reactive animations Circus Graphic emphasis on tricks, such as a spectacle following big tricks Theater A spot-on automated spot light Ideas? 16. CREATIVE THINKING The creation of personalized interactions and augmented reality Direct Mapping Gesture Following Event Detection Actor Immersion 17. DEMO 18. QUESTIONS? If youd like to interact with the system please come to the demonstration session tomorrow at the Continental Ballroom 4 from 5:30 pm to 7:30 pm.