Transformed Human Presence forPuppetry

Keisuke KawaharaUniversity of Tsukuba1-1-1 Tennodai, Tsukuba,Ibaraki, Japankawahara@ai.iit.tsukuba.ac.jp

Ippei SuzukiUniversity of Tsukuba1-2 Kasuga, Tsukuba, Ibaraki,Japan1heisuzuki@gmail.com

Mose Sakashita*Joint first authorshipUniversity of Tsukuba1-2 Kasuga, Tsukuba, Ibaraki,Japanmose.sakashita@gmail.com

Kenta SuzukiUniversity of Tsukuba1-2 Kasuga, Tsukuba, Ibaraki,Japanikuzus.atnek.0626@gmail.com

Amy KoikeUniversity of Tsukuba1-2 Kasuga, Tsukuba, Ibaraki,Japanamy23kik@gmail.com

Yoichi OchiaiUniversity of Tsukuba1-2 Kasuga, Tsukuba, Ibaraki,Japanwizard@slis.tsukuba.ac.jp

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).ACE2016, November 09-12, 2016, Osaka, JapanACM 978-1-4503-4773-0/16/11.http://dx.doi.org/10.1145/3001773.3001813

AbstractWe propose a system for transmitting a human performer’sbody and facial movements to a puppet with audiovisualfeedback to the performer. The system consists of a head-mounted display (HMD) that shows the performer the videorecording of the puppet’s view, a microphone for voice cap-ture, and photoreflectors for detecting the mouth move-ments of the human performer. In conventional puppetry,there is also the need for practice in the manipulation ofthe puppets to achieve good performance. The proposedtelepresence system addresses these issues by enablingthe human performer to manipulate the puppet through theirown body and facial movements. The proposed system isexpected to contribute to the development of new applica-tions of puppetry and expand the interactivity of puppetryand the scope of entertainment.

Author KeywordsAnimatronics; puppet; telepresence; head-mounted display.

ACM Classification KeywordsH.5.2 [Information interfaces and presentation (e.g., HCI)]:User Interfaces

IntroductionPuppetry has a long history, being thought to have origi-nated in 3000 BC [1]. A puppet show is a basic form of per-formance involving the manipulation of inanimate objectswith some semblances to humans or animals. One of thesimplest methods of puppetry involves the movement of thepuppet through the hands or arms of a human puppeteer.The puppet is controlled from above with the rest of thepuppeteer’s body hidden backstage. This, however, poses acoordination challenge and also requires sufficient trainingin manipulating the puppet. To address these issues, manyuser interfaces and novel methods have been developed forcomputational puppet manipulation. For example, a grove-type user interface [7] and a system for recording the limb-controlled movements of the puppet through a webcam [10]have been presented.

(a)

(b)

(c)

Figure 1: (a) Puppetry usingtelepresence user interface. (b)Interaction between a puppet andthe audience. (c) Synchronizationof the performer and puppetthrough the proposed system.

Studies have particularly been conducted on the applicationof telepresence to puppet manipulation. Telepresence is atechnology that enables a user to have a sense of being ina different place. An example of the application of telepres-ence is Huggable [4], a teddy bear robot designed for socialcommunication applications. These developments usedthe controlled puppets for communication and educationpurposes, without particular interest in entertainment. Theparameters that are transmitted to the puppets for manipu-lation are thus often insufficient for the animation of a broadrange of expressions and movements.

The concept of human-to-nonhuman telepresence (Figure2) affords new experiences for performers and observersand has the potential for novel entertainment applications.The telepresence system proposed in this paper transmitsthe body and facial movements of a performer to a puppetwith visual and audio feedback to the performer. In otherwords, it enables the transmission of a human’s presence

to a puppet. The system facilitates easy manipulation ofthe puppet by the performer, as well as the expression ofmore lifelike characteristics. This is because the control isaccomplished by the performer’s own body and facial move-ments, including those of the mouth. Conventional puppetryhas physical limitations, which include the need for the pup-peteer to be at the exact location of the puppet. Telepres-ence eliminates such physical limitations and enables mul-tiple performers at different locations to collaborate in se-quential manipulation of the same puppet. It also enables agiven performer to seamlessly shift between different pup-pets. Further, the remote performer is able to interact withthe audience as if they were at the location of the puppet.This is facilitated by the feature of real-time visual and audiofeedback to the performer. The system enables untrainedhands to perform a puppet show, thus eliminating the spe-cial practice required by conventional puppetry. The per-former’s sense of being on location also promises smoothinteraction between the puppet and the audience, openingnew entertainment frontiers.

Related WorksComputational puppetrySeveral studies have been conducted on the manipulationof puppets, dolls, and toys. PINOKY [11] is a ring-like de-vice that can be used to animate a toy by movement of itslimbs. A grove-type user interface for interactively animat-ing a puppet [7], a system for recording the movements ofa puppet by a webcam [10], and a system for animatronicsstorytelling that enables performers to manipulate puppetsby wearing a mask-type device on their faces [8] are otherrelated developments.

Facial expressionIn our study, we developed a system for transmitting cap-tured human facial expressions and movements to a real

puppet in real time. Hundreds of related virtual reality stud-ies have utilized head-mounted displays (HMDs). However,this creates the difficulty of tracking and capturing the hu-man face because most of the employed facial tracking sys-tems required the captured face to be fully visible. A meansof tracking the eye under an HMD was recently developed[5]. The system uses RGB-D cameras to generate an an-imation of the full face. However, it is heavy and uncom-fortable to carry on the user’s head because the headsetincludes a depth sensor in addition to the HMD. Consider-ing that our proposed system also uses an HMD for visualfeedback to the user, we employed a mask-type device witha photoreflective sensor module to capture the facial ex-pressions. The mask-type device is lighter than a depthcamera, and the employed sensor processing also enablesfaster calculation compared to the use of image processingto capture facial movements.

Objective place

Remote place

TransmissionFeedback

Figure 2: Conventional puppetmanipulation (upper), and theconcept of the proposed system(lower).

TelepresencePresentations have been made regarding the applicationof telepresence to puppet manipulation. An example is ateddy bear robot for social communication applications [4].The robot is remotely controlled through an input devicesuch as a Wii Remote, with the user receiving audiovisualfeedback and being able to communicate with local peo-ple at the location of the robot. Another example is Robot-Phone [9], a robotic user interface (RUI) that allows the userto modify the shapes of the components of the connectedrobot, with the ability to receive feedback on the movementsof the components via the Internet and replicate the samein other RobotPhones. PlayPals [2] is yet another wirelesssystem that offers children a means of playfully commu-nicating with other children elsewhere by manipulating apuppet at the remote location. However, the remote puppetcan only be used for communication and educational pur-poses, and not for entertainment. Conversely, the presently

proposed system enables the user to transmit their voiceand body, facial, and mouth movements to the remote pup-pet, with audiovisual feedback, thus enabling application toentertainment.

ImplementationIn this section, we describe the implementation of the pro-posed system, the configuration of which is shown in Figure3. The employed mask-type device consists of an HMD [6],a microphone for capturing the performer’s voice, and anarray of photoreflectors for detecting the mouth movements.A Kinect camera is used to capture the body movements ofthe performer. The manipulated puppet is equipped with amicrophone for capturing ambient sound, two cameras, andseven servomotors that move its arms, mouth, and neck(see Figure 8).

Head-mounted display and camerasThe eyes of the robot are equipped with two 1.2-million-pixel web cameras (Figure 3 (right)) for streaming live videoto the HMD of the performer. This enables the performerto see through the eyes of the puppet and visualize its sur-roundings (Figures 4 and 5). The distance between thehuman eyes is approximately 65 mm, and this was appliedto the puppet’s eyes.

Facial movementsThe robot has a 2-DOF neck that is controlled in accor-dance with the performer’s facial movements. An inertial-measurement unit (IMU) installed in the HMD is used toread the gyro sensor for measuring the Euler angles of theperformer’s face.

The measured Euler angles are transmitted to the neckservomotors of the robot at 30 fps via a UDP in the softwareof the employed microcontroller (Arduino Micro).

Wire(HDMI, USB) Wire

Wire

Wire

Photoreflector arrays

Microphone

Headphone

Display

IMUMicro controller

Computer 1(Mac)

Computer 2(Windows)

Wireless or Wire(UDP)

WireWire

Wire×2(Computer1)

Wire×2 (USB cameras)

Mask Mask

HMD

Computer 1 (Mac)

Computer 2 (Windows)

Puppet Puppet

Photoreflector arrays

Micro controller

Microphone

Kinect v2Kinect v2

Headphone

Speaker

Microphone

Microphone

Servomotors

Speaker

Micro controller

USB camerasDisplay Raw image

IMU

Transform

Get sensor values

Detection of lip position

Get positions

Servo manager

Calculate angles

Get anglesImages

Angles

Angles

Voices

Position

Ambient sound

Head orientation

Images

Sensor values

Micro controller

Servomotors

USB cameras

Figure 3: Configuration of the proposed system.

Figure 4: Performer’s visualizationof the puppet’s hands through theHMD.

Figure 5: Interaction with theaudience (upper), and visualfeedback (lower).

Detection of mouth state of performerWe developed an array of photoreflectors for detecting theopened or closed state of the performer’s mouth (Figure6 (e)). The sensors were spaced at 6 mm on a board andconnected via a general-purpose input/output (GPIO) to acontrol photo-emitter and the analog-to-digital converter ofthe microcontroller. We employed 10 photoreflector sen-sors (TPR-105; GENIXTEK CORP) with a maximum sens-ing distance of 20 mm. Figure 6 (d) shows sample resultsof the photoreflector sensor measurements for (a) mouthclosed, (b) mouth opened, and (c) mouth partly opened.The program for detecting the lower lip position was imple-mented using a Java processor (Figure 6 (e)). The softwarecould be used to calculate the difference between the mea-surements of the different sensors, with the maximum mea-surement appearing to be the valid indication of the actualposition of the performer’s lower lip.

Based on the detected position, the software controlledthe mouth servomotor of the puppet. This setup synchro-nized the mouth movement of the puppet with the speech of

the performer, giving the impression of the puppet actuallyspeaking to the audience. The positions of the sensors onthe board can be adjusted to fit different users.

Voice transmissionPerformer-to-Puppet: A microphone in the mask is used tocapture the performer’s speech, which is transmitted andreplayed in real-time through a speaker installed in the pup-pet. An effect is used to change the voice of the performerto that of the puppet character, thus affording a more con-vincing and attractive show.

Puppet-to-Performer: Another microphone installed in thepuppet is used to capture ambient sound at the location ofthe puppet, and this is transmitted back to the performerthrough earphones. The performer is thus able to monitoracoustic events around the puppet, such as the reaction ofthe audience and the words of any other puppet characterin the show. This enables the performer to interact with theaudience through the puppet.

Sensing of performer’s armsAs an additional function, a Kinect depth camera is used tocapture the arm movements of the performer to manipulatethose of the puppet. This is based on a previous work [3].The present system thus uses a combination of the bodyand facial movements of the puppeteer to smoothly andsynchronously manipulate the puppet. The tilt angles andpan directions are calculated from the positions of the armscaptured by the Kinect depth sensor, and transmitted by aUDP at 30 fps to the microcontroller. The software of themicrocontroller was developed using SDK version 2.0. Alow-pass filter is used to stabilize the low-resolution andnoisy input data from the Kinect sensor. The two 2-DOFarms of the puppet are driven by two servomotors each inaccordance with the captured motions of the performer’sarms.

0 1 2 3 4 5 6 7 8 9 100

0.10.20.30.40.50.60.70.80.9

Sensor number

Vol

tage

[V]

(a)(b)

(c)

(d)

(a) (b) (c)

10

12

34

56

7

(e) (f)

8

Sensor array

lip9

Figure 6: (a), (b), and (c) showthree states of the performer’smouth, while (d) shows therespective corresponding plots. (e)Lip position detection using thedeveloped sensor array. (f)Software for detecting theperformers lips.

Figure 7: Mask type device thatwe developed.

Mask-type deviceAs already discussed, the mask-type device is used to dis-play images of the puppet’s view to the performer and trackthe head and mouth movements of the performer. It con-sists of an HMD and adaptor (Figure 7). Orientations of theperformers head can also be acquired by the HMD for fur-ther manipulation of the puppet. Because the photoreflectorarray captures the mouth movements of the performer, anadaptor was integrated in the headset.

Puppet bonesWithin the puppet is a skeletal framework to which the ser-vomotors are attached (Figure 8). The framework was de-signed such that the servomotors would not interfere witheach other. Because the framework can be printed by a 3Dprinter, the robot can be manufactured at low cost. Consid-ering that the puppet has two 2-DOF arms, a 1-DOF mouth,and a 2-DOF neck, its entire body has seven DOFs.

DiscussionFigure 9 illustrates the response of the puppet to the pup-peteer’s movements as driven by the servomotors. Al-though the developed telepresence system enables thedemonstration of the mouth movements and facial expres-sions of the puppeteer, there is room for the implementationof additional expressions such as emotion and eye move-ment to further enhance the performance. This would re-quire the utilization of components such as strain gauges todetect the relevant expressions of the performer under theHMD, as has been done in some previous studies. Further,whereas the developed puppet robot has seven DOFs, thisis probably insufficient. Improvements can be achieved byincorporating additional actuators to manipulate the legsand eyes. Additionally, although the weight of the HMD wasreduced by the use of photoreflectors, it may still be tooheavy for the convenience of some users. Hopefully, lighterHMDs would be developed in the future.

Overall, the proposed system overcomes some of the phys-ical limitations of the conventional puppet manipulationmethod, enabling the direct transmission of the actual per-formance of the puppeteer to the puppet. The system hasstrong potential for application to entertainment. For exam-ple, if employed in theme parks, puppets, dolls, and toyscan be manipulated by performers at remote locations,eliminating the practice of wearing special character cos-tumes. Multiple performers at different locations, as men-tioned earlier, may also manipulate the same puppet inturn, and a given performer can move from one puppet toanother. All such shifts can be done almost instantly.

ConclusionWe proposed and described a system for manipulating apuppet using an HMD. The system enables the transmis-sion of the puppeteer’s body and facial movements to the

puppet, with audiovisual feedback to the performer. How-ever, as mentioned in the previous section, the systemhas some limitations and there is room for improvementsthrough further study to achieve more lifelike animation andexplore the potential application to entertainment.

Body yaw

Body pitch

Mouth

LeftArm(2)

LeftArm(1)

RightArm(2)RightArm(1)

Figure 8: Bones and servomotorsinside the puppet.

Figure 9: Time-series photographsof the puppet animation (intervalsof a second).

REFERENCES1. E. Blumenthal. 2005. Puppetry and Puppets: An

Illustrated World Survey. Thames & Hudson.

2. Leonardo Bonanni, Cati Vaucelle, Jeff Lieberman, andOrit Zuckerman. 2006. PlayPals: Tangible Interfaces forRemote Communication and Play. In CHI ’06 ExtendedAbstracts on Human Factors in Computing Systems(CHI EA ’06). ACM, New York, NY, USA, 574–579.

3. Robert Held, Ankit Gupta, Brian Curless, and ManeeshAgrawala. 2012. 3D Puppetry: A Kinect-based Interfacefor 3D Animation. In Proceedings of the 25th AnnualACM Symposium on User Interface Software andTechnology (UIST ’12). ACM, New York, NY, USA,423–434.

4. Jun Ki Lee, R. L. Toscano, W. D. Stiehl, and C.Breazeal. 2008. The design of a semi-autonomousrobot avatar for family communication and education. InRO-MAN 2008 - The 17th IEEE InternationalSymposium on Robot and Human InteractiveCommunication. 166–173.

5. Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei,Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, andChongyang Ma. 2015. Facial Performance SensingHead-mounted Display. ACM Trans. Graph. 34, 4,Article 47 (July 2015), 9 pages.

6. Oculus.inc. 2014. Oculus Rift Develop kit 2 (lastaccessed August 27, 2016). (2014).https://www.oculus.com/en-us/dk2/

7. Emil Polyak. 2012. Virtual Impersonation UsingInteractive Glove Puppets. In SIGGRAPH Asia 2012Posters (SA ’12). ACM, New York, NY, USA, Article 31,1 pages.

8. Mose Sakashita, Keisuke Kawahara, Amy Koike, KentaSuzuki, Ippei Suzuki, and Yoichi Ochiai. 2016. Yadori:Mask-type User Interface for Manipulation of Puppets.In ACM SIGGRAPH 2016 Emerging Technologies(SIGGRAPH ’16). ACM, New York, NY, USA, Article 23,1 pages.

9. Dairoku Sekiguchi, Masahiko Inami, and SusumuTachi. 2001. RobotPHONE: RUI for interpersonalcommunication. In CHI’01 Extended Abstracts onHuman Factors in Computing Systems. ACM, 277–278.

10. Ronit Slyper, Guy Hoffman, and Ariel Shamir. 2015.Mirror Puppeteering: Animating Toy Robots in Front ofa Webcam. In Proceedings of the Ninth InternationalConference on Tangible, Embedded, and EmbodiedInteraction (TEI ’15). ACM, New York, NY, USA,241–248.

11. Yuta Sugiura, Calista Lee, Masayasu Ogata, AnushaWithana, Yasutoshi Makino, Daisuke Sakamoto,Masahiko Inami, and Takeo Igarashi. 2012. PINOKY: ARing That Animates Your Plush Toys. In Proceedings ofthe SIGCHI Conference on Human Factors inComputing Systems (CHI ’12). ACM, New York, NY,USA, 725–734.