A Generative Human-Robot Motion Retargeting Approachusing a Single Depth Sensor

Sen Wang1,2 Xinxin Zuo1,2 Runxiao Wang1 Fuhua Cheng2 Ruigang Yang2

Abstract— The goal of human-robot motion retargeting is tolet a robot follow the movements performed by a human subject.This is traditionally achieved by applying the estimated posesfrom a human pose tracking system to a robot via explicitjoint mapping strategies. In this paper, we present a novelapproach that combine the human pose estimation and themotion retarget procedure in a unified generative framework.

A 3D parametric human-robot model is proposed that hasthe specific joint and stability configurations as a robot whileits shape resembles a human subject. Using a single depthcamera to monitor human pose, we use its raw depth mapas input and drive the human-robot model to fit the input 3Dpoint cloud. The calculated joint angles of the fitted modelcan be applied onto the robots for retargeting. The robot’sjoint angles, instead of fitted individually, are fitted globally sothat the transformed surface shape is as consistent as possibleto the input point cloud. The robot configurations includingits skeleton proportion, joint limitation, and DoF are enforcedimplicitly in the formulation. No explicit and pre-defined jointsmapping strategies are needed.

This framework is tested with both simulations and realrobots that have different skeleton proportion and DoFscompared with human to show its effectiveness for motionretargeting.

I. INTRODUCTION

Nowadays social robots have become more intelligentalong with the progress in object recognition and speechrecognition. However the limited motion generating strategyis one bottleneck that prevents them from more widespreaduse. It is still an active research topic for which many ap-proaches have been proposed. One effective way to generatemotion for robots is to let a robot mimic the movement of ahuman, i.e., human-robot motion retargeting. The goal is todrive a humanoid robot to move in a natural way providedwith the joint movements or positions captured from human.This is a hot topic in both robotics and other areas, as motionretargeting can be widely used in robot simulation [1] andalso in virtual characters animation [2]. In this paper, wepay our attention on how to make robots imitate the humanmotion.

Previously, the joints position obtained from a motioncapture (Mocap) system [3] or Kinect skeleton trackingalgorithm [4] is typically considered as the input of a motionretargeting procedure. After that the joint movements or theend-effector positions are applied onto robots via some pre-defined mapping strategies between human and robots. One

1Northwestern Polytechnical University, Xi’an 710072, P.R.China{wangsen1312, xinxinzuo2353}@gmail.com,wangrx@nwpu.edu.cn

2Univeristy of Kentucky, Lexington, KY 40506, USA{cheng,ryang}@cs.uky.edu

(a)

(b) (c)

Fig. 1: Motion retargeting results of three different poses.The mesh and color image are captured by Kinect V2. Theseposes retargted to NAO robot using our proposed approach.

straightforward way for motion retargeting is to apply inversekinematics (IK) to end-effector positions given the trackedhuman joint positions. Another kind of solution is to preservethe joint angles, which places a major role for definingmapping criteria.

These kinds of methods are not always sufficient sincethe skeleton of a robot usually differs from the skeleton ofa human, such as the proportion of their limbs. Thereforewe need to define the mapping between the target and thesource model and this mapping can not be reused in robotmodels with different configurations. Another drawback forthese methods is that we cannot enforce joint angles andend positions constraints simultaneously. This will causethe dissimilarity of robot movements and human movement.Moreover, they have quite different joint limitations androbots usually have less degree of freedoms (DoFs) thanhuman beings. Stability is another essential problem thatneeds to be dealt with while imitating human motion. Theserequirements pose more challenges for human-robot motionretargeting.

In this paper, rather than dealing with motion trackingand retargeting separately, we present a novel method thatcombines these two procedures in a united framework. Weuse the captured 3D point cloud from a single consumerdepth sensor such as Kinect as our input instead of thecomputed skeleton, and the human subject does not wearany markers.

For the robot model representation, it is usually modeled

as joint or skeleton; in contrast we choose to use a parametrichuman-robot template to represent the joints together with3D surface shape. More specifically, we propose a HUMROBmodel that has the joint configurations of the target robot andthe 3D shape same as the source human subject. The robotis modeled as 3D mesh with bones and joints embeddedinside the surface. Our goal is to compute the joint angles ofthe robot by driving the template model so that the surfaceshape fits as much as possible the point cloud acquired fromdepth sensor. In this way, we can get the joint angles for therobot that maintain the pose similarity. Besides, the stabilityand joint limitations are enforced in the objective functionnaturally. Since we do not need an explicit joint or positionmapping strategies, our method can be applied to robotswith different configurations quite conveniently. We classifyour method as a generative motion retargeting approach ascompared with previous methods that focus on developingvarious skeleton mapping strategies.

As far as we know, our technique is the first one that usesa generative unified framework to achieve motion retargetingwith one single depth sensor. Our algorithm is validatedwith both simulation and real robots with different skeletonproportions from human. Satisfactory and stable motionshave been generated even under some extreme poses. Someexamples are shown in Fig. 1.

II. RELATED WORK

Motion retargeting is a hot topic in robotics that has beenused in gaming and motion generation. For previous motionretargeting methods, two separate procedures are included:motion capture of source and then motion retargeting fortarget. For a detailed review of various algorithms, especiallyfor human-robot motion retargeting, we refer the reader tothe survey [5]. Some other methods have demonstrated highaccuracy performance while considering kinematic and dy-namic constraints such as joint limits, self collisions, torquelimits, balance and so on [6], [7], [8]. In this section, wereview only the most related work focusing on human-robotmotion retargeting including pose estimation and motionretargeting.

A. Human Pose Estimation

Human pose estimation is mainly categorized into markerbased method, inertial based system, and markerless methodregarding to the device that has been used.

Marker based system. Marker based systems require theuser to wear a special suit with markers attached to fixedbody position, such as Vicon™ systems [3] and Optitrack™system.

Inertial based system. Xsens MVN™ motion capturesystem [9], which consisting of inertial sensors attached tothe individual body segments to get the precise position.

Although these two kinds of systems can offer highaccuracy and a rate of frames, precise marker position andhuge cost is an obstacle for widespread use.

Markerless system. Motion capture with only colorcameras is still an unsolved problem for computer vision

community considering the ill-posed condition [10]. Withthe availability of consumer depth sensors such as Kinect,it is easy to get the human skeleton using its SDK [11],[12] which shows more accuracy than color only methods.Apart from the default use with original SDK, researchersfocus on how to get the human pose more precise. Thepapers mainly include discriminative or generative approachfor model based pose estimation. Existing discriminativeapproaches either performed body part detection by iden-tifying salient points of the human body [13], or relied onclassifiers or regression machines trained offline to infer thejoint locations [14]. The generative approaches fit a templateto the observed data. Ye et al. [15] relates the observed datawith model template using Gaussian Mixture Model, withoutexplicitly building point correspondences.

B. Motion Retargeting

The challenge for motion retargeting can be summa-rized into two aspects: 1) Source and target has similaror same topology but with different geometry, that meansthe source/target is expressed as similar or even the samehierarchy link of joints, but the ratio length of bones isdifferent from each other. 2) Source and target has differenttopology, that is a more complex situation and also differentgeometry.

Different geometry. For the different geometry but sametopology, it is common to employ IK solvers integrated withsoft constraints enforced in an object function. Also, somehard constraints are employed to define specific rules for themotion that must be maintained [4], [9], [16].

Different topology. Source and target are identified astopologically different if they do not have the similar skeletonstructure, some methods have been proposed, among whichexampled-based methods [17], [18] are commonly used. Thismethod based on a number of motion mappings, several keyposes and correspondences are pre-defined.

Unlike the traditional separated pose estimation and map-ping procedure, in this paper we propose a unified frameworkcombines the two procedures using a single depth sensor. Wecan deal with motion retargeting both the different geometryand different topology case without special treatment.

III. BACKGROUND

We first introduce some background knowledge that isessential for our approach.

A. Gaussian Mixture Model

We give a brief introduction to GMM and its solution usingEM algorithm, which is the fundamental techniques for ourframework. More details can be found in [19].

Assume that we have N source points, SN∗D =(s1, . . . , sN )T and M target points, TM∗D = (t1, . . . , tM )T .D is the dimensionality of the point set, so D = 3 in thispaper.

We assume that the source points follows the GMM whosecentroids are depend on the target points. The probability of

each source point is expressed as,

p(s) = (1− µ)

M∑m=1

1

Mp(s|tm) + µ

1

N(1)

where

p(s|tm) =1

(2πσ2)32

exp(−||s− tm||2

2σ2) (2)

The outliers are considered as subjecting to the uniformdistribution with µ as the approximation for the percentageof outliers.

Next we reparametrize the GMM centroid locations by aset of parameters φ and estimate them by minimizing thenegative log-likelihood function,

E(φ, σ2) = −N∑n=1

logM+1∑m=1

P (tm)p(sn|tm(φ))

= −N∑n=1

log(

M∑m=1

1− µM

p(sn|tm(φ)) +µ

N)

(3)

The energy function is usually solved using the EMalgorithm in an iterative manner as described in [20]. Wesummarize the E-step and M-step briefly here.

During the E-step, given the values of parameters esti-mated from M-step of the previous iteration, we can thenuse the Bayes’ rules to compute a posteriori probabilitydistribution for the target data points given the source data,

pold(tm(φ)|sn) =exp(− ||sn−toldm (φ)||2

2(σold)2)∑M

i=1 exp(− ||sn−toldi (φ)||22(σold)2

) + c, (4)

where c =(2π(σold)2

) 32 1−µ

µMN .

To simplify the expression, we let pold to stand forpold(tm(φ)|sn) in the following sections.

During the M-step, the parameters are updated via mini-mizing the following complete negative log-likelihood func-tion.

Q(φ, σ2) =1

2σ2

N,M∑n,m=1

pold||sn − tm(φ)||2 +3

2NP logσ2,

(5)Where NP =

∑N,Mn,m=1 p

old. The value of all the parameterswill converge after several iterations.

B. Joint and Vertex transformation

To better represent the posture of whole body, we use theclassical twists exponential map for joint transformation andadopt skeleton-subspace deformation for vertex transforma-tion.

1) Joint transformation: For articulated model such ashuman and robot, pose can be represented by a set of twistsξφ ∈ SO(3), which is the exponential mapping that can begeneralized to the Euclidean group (SE(3)) [21]. We givemore details in the following.

In general, an articulated body transformation is repre-sented as the exponential of the twist T ∈ SE(3) as follows,

T = exp(ξφ) (6)

In homogeneous coordinates, the twist ξ ∈ se(3) can beexpressed as follows,

ξ =

[ω v0 0

]∈ R4×4, (7)

where ω ∈ so(3) is 3×3 skewed-symmetric matrix convertedfrom a 3D vector ω ∈ R3, which is the direction of therotation axis. The 3D vector v determines the location of therotation axis and the amount of translation.

Assume that robot has P joints, including the global trans-formation and rotation, the whole body motion namely thepose is controlled by these parameters Φ := (φ0, φ1 . . . , φP ).With the twists exponential map, final transformation foreach joint k is represented as,

Tk =

P∏p=0

exp(κpk ξpφp), (8)

where ξ0φ0 is the global transformation and rotation for theroot that consists of the six parameters. κpk = 1 if p is thehierarchical parent of the joint k, otherwise κpk = 0 .

2) Vertex transformation: When we get the joint trans-formation under a certain pose Φ, the position of eachvertex v of template mesh is computed by skeleton-subspacedeformation (also known as ”LBS”) method [22],

vi(Φ) =

P∑p=1

ωipTp(Φ)v0p, (9)

where ωip is the skinning parameters attaching the ith vertexto the underlying the joint p, and v0

i stands for the positionfor each vertex in its reset pose.

IV. APPROACH

Our goal is to retarget the motion captured from humansubjects to robots in a generative way without any pre-defined joint mapping. The overall pipeline is shown inFig. 2. There are mainly two steps. First, we present our

HUMROB Model

captureddata

Robot

Fig. 2: System Pipeline. We use captured depth data asinput. First, we build up the HUMROB model, which is thendeformed to fit the captured data and the joint angles arecomputed. Finally, the pose is applied onto the robot.

method to create a parametric HUMROB model in section IV-A; then the template model is deformed to fit the captureddepth map of the source human subject as described insection IV-B. We enforced joint limitations and stabilityconstraints during the deformation. Finally, the joint anglescomputed via the deformation can be applied directly ontothe robots for motion retargeting.

A. HUMROB Model

Considering our goal of fitting robot model to capturedpoint cloud from depth sensor, first we need to build apersonalized 3D Human-Robot model, which is a parametricmodel that can be deformed according to joint angles andpositions. Also, the robot configurations, which are usu-ally different from humans, need to be embedded into itsparametric model. Several steps are performed to get thispersonalized parametric HUMROB model as shown in Fig. 3.

The first step is to build a personalized 3D model forhuman source subjects. Nowadays, there are several success-ful approaches that can be used to get the 3D model. Forexample, we can use multiple depth sensors and then fusethe depth maps captured from multiple view directions [23].Besides, a single depth sensor is also sufficient to providethe complete 3D human model [24].

Then given the 3D model of the human subject, the nextstep is to parametrize the model using the generic parametrichuman model. In more details, we use the personalized 3Dmodel as a target, and the limb length and the geometryof the generic template is adapted to fit the target model,serval methods can be applied, such as [25]. As with theadaptation, the configurations of generic parametric modelare adjusted to make the model align with the 3D person-alized model of the human subject. After the alignment,the skinning weight transfer procedure is followed wherek-nearest neighbor(k-NN) search is implemented betweengeneric template and personalized model to obtain the rawskinning weight. Finally, we use the laplacian smooth tosmooth the transferred skinning weight. By the end of thisstep we get the personalized parametric model automatically.

Finally, we apply the robot configurations including thelimb proportion and its joints limitation to the personalizedparametric model we just have obtained. In this step, the jointpositions for each limb will get adjusted and we will alsochange the DoFs of particular joint according to the robotwe intend to retarget to.

B. Motion Retargeting

Now we have got the HUMROB model, then we canperform motion retargeting under the motion tracking frame-work by fitting this parametric model to captured depthdata. The framework consists of two fundamental techniquesnamely the Gaussian Mixture Model and the joint/vertextransformation described in Sec. III-A and Sec. III-B.

Another thing we need to point out is that we choose to dothe tracking and motion retargeting continuously along thesequence rather than selecting key frames and performingretargeting frame by frame as often used in previous works.

Parametric Personalized

Model

Personalized Model

Robot Configuration

Parametric Generic

Template

Geometry and limb adaptation

HUMROB Model

Fig. 3: Pipeline for building up personalized parmatericHUMROB model

First, suppose that the robot has J joints and we have gotthe pose (the joint angles in our case) at time t denoted asΘt := (θt0, θ

t1 . . . , θ

tJ), we intend to compute the changes for

the pose ∆Θ from time t to t+ 1 as follows,

Θt+1 = Θt + ∆Θ (10)

Under twist-based parametrization, the transformation forany joint k at time t+ 1 can be expressed as,

T t+1k =

J∏j=0

exp(κjk ξk(θtj + ∆θj)) (11)

1) Energy formulation: To compute the pose expressedby Θ, we formulate the energy function as below,

E = E(Θ, σ2) + λ⊥E⊥(Θ, σ2) + λrEr(∆Θ) (12)

The first term is fitting cost that measures the distanceof deformed HUMROB template model from captured pointcloud, which is minimized to enforce pose similarity. Togive more mathematical details, the observed points from thecapture human pose are denoted as (SN∗3 = (s1, . . . , sN )T ),and the template points sampled from the HUMROB modelare represented as a vertex set (VM∗3 = (v1, . . . ,vM )T ). Ifwe take VM∗3 as the centroids of GMM model, then SN∗3can be seen as the observed data that should be generatedfrom the centroids via GMM model as described in Sec.III-A.

Therefore the vertex position fitting cost can be expressedas,

E(Θ, σ2) =1

2σ2

N,M∑n,m=1

pmn||sn − vm(Θ)||2 +3

2NP logσ2,

(13)Similar to pold in Eq. 4, the pmn can be seen as the

probability of sn relating to vm, we give more details insection V-B. vm(Θ) is calculated by substituting the Tj(Θ)in Eq. 9 with Eq. 11, and then we get the position for each

vertex computed as,

vm(Θ) =

J∑j=1

ωmj( J∏j=0

exp(κjk ξk(θtj + ∆θj)))v0m, (14)

The second term in Eq. 12 is another fitting term thatensures the normal consistency between deformed templatemodel and the observed depth data. The cost function forthis term is given as,

E⊥(Θ, σ2) =1

2σ2

N,M∑n,m=1

pmn||s⊥n − v⊥m(Θ)||2 +3

2NP logσ2,

(15)where s⊥n and v⊥m(Θ) are the normal vector for the sn andvm(Θ) respectively.

To reduce the computation cost, we use same σ2 for Eq. 13and Eq. 15.

In addition to the above two fitting terms, we also havethe third term in Eq. 12 that penalize big changes ofthe poses for neighoring frames. This term is necessary topreserve temporal consistency and smooth motion transitionin a sequence.

Er(∆Θ) =∑||∆Θ|| (16)

Another important constraint we have to take into con-sideration while performing motion retargeting is the jointlimitation, as the joint limitation for robots usually differsfrom the human. This constraint can be handled naturallyin our framework by specifying the lower (θmin) and upperbound (θmax) for each joint while performing the optimiza-tion for Eq. 12. This can be seen as the hard constraint of thecost function. So the final energy function can be representedas,

E = E(Θ, σ2) + λ⊥E⊥(Θ, σ2) + λrEr(∆Θ)

Θ ∈ [Θmin,Θmax](17)

We present two examples in Fig. 4 showing the deforma-tion results after minimizing the energy function Eq. 17. Theleft columns is the HUMROB model in initial poses overlaidwith captured 3D mesh data, and the right column is thedeformed HUMROB template achieved by our method.

Fig. 4: Deformed HUMROB model results after applying ourframework.

2) Robot stability: In this paper we assume that the robotis motionless as it is driven at a very low motor speed.This indicates that it only experiences gravitational forces.Therefore instead of using the complex ZMP criteria andwe turn to use floor projection of center of mass (FCoM )

criteria to ensure the stability of the robot. Next we will showhow to integrate this stability constraint into our framework.

Suppose the robot has L links or bones and the vectorspl are the vectors points from the CoM of each individualbones to coordinate origin. The CoM for the whole bodydenoted as pCoM is computed as,

pCoM =

∑Ll=1mlpl∑Ll=1ml

, (18)

where ml is the mass of each bone.The floor projection of the CoM (pFCoM ) is calculated

by taking x and y component of vector pCoM as its x andy component and setting its z component to be zero.

The main criteria is: the motionless humanoid should beable to maintain its balance if pFCoM is in the supportpolygon (SP ), which is formed by the convex hull aboutthe floor support points. In this paper, we use the inscribedcircle of SP as the more strict support polygon, denotedas C SP . We introduce another term Eb express the robotstability, which is computed as,

Eb =

0 ifpFCoM inside C SP∞ ifpFCoM outside SP

o− pFCoM otherwise(19)

where o is center of C SP .Therefore, for each frame we will first find the optimal

pose configuration by minimizing Eq. 17. Then we cancompute Eb which reflects the current stability status. If Ebequals zero, it means we can safely apply the current poseinto the robot with stability already maintained. However,if Eb equals ∞, we will give the opposite offset (0.3 rad)to the hips’ pitch/roll angles. Otherwise, we will record the−→opt = o − pFCoM for the current frame t. Suppose for thethe next frame t + 1, we have got the vector −→opt+1. Thenif cos(−→opt,−→opt+1) > cos(30◦) & ||−→opt+1|| > ||−→opt||, whichindicates that the stability violation will probably occur insuccessive frame, in this case, we will give an opposite offset(0.05 rad) to the hips’ pitch/roll angles.

V. IMPLEMENTATION

A. Data Preprocessing

We capture the depth data of human subject using Mi-crosoft Kinect V2. The depth data has the resolution at 512×424 and the depth stream is captured at 30fps. We assumethat depth sensor is calibrated and the region of interest,namely the human subject, can been roughly segmented withbackground subtraction.

The parameters (Θ) are all coded as rad in order to keepconsistent with the real robot. The algorithm is implementedin Matlab on an ordinary PC with Intel i7 3.6GHz processorand 16GB RAM and it takes about five seconds for eachframe. The pose transferred to real robot is coded withPython through Choregraphe™.

B. Implementation details

Alg. 1 shows how to minimize the energy function Eq. 17to get the pose configurations, namely the parameters Θ andσ2. First, we adopt a linearized surface deformation strategyto simply the constrained nonlinear cost function into a linearone [26]. But the normal fitting term Eq. 15 cannot belinearized, so it is neglected in this stage. The computedlinear solution can then be taken as the initialization for theover-all nonlinear optimization, for which we choose to usetrust-region-reflective algorithm to find a optimal solution forthe nonlinear function Eq. 17.

In the Linear solver part, we only consider position fitting(Eq. 13), then the the posteriors pmn is computed as,

pmn =exp(− ||sn−vold

m (Θ)||22(σold)2

)∑Mi=1 exp(− ||sn−vold

i (Θ)||22(σold)2

) + c, (20)

In the Nonlinear solver part, the vertex and normal areconsidered together (Eq. 13 and Eq. 15), so the posteriorspmn is computed as,

pmn =exp(− ||sn−vold

m (Θ)||2+λf ||s⊥n−v

⊥oldm (Θ)||2

2(σold)2)∑M

i=1 exp(− ||sn−voldi (Θ)+λf ||s⊥

n−v⊥oldm (Θ)||2

2(σold)2) + c

,

(21)And c in Eq. 20 and Eq. 21 can be computed by Eq. 4.

initial pose: linear third order auto-regressionbegin Step One: Linear solver for initialization

while Pose not converged doE-step: Compute posteriors via Eq. 20.M-Step:• Minimize the linearized cost function of

Eq. 13 for (Θ, σ2)• Update template vertices via Eq. 14• parameters are adjusted by enforcing

robot stabilityend

endbegin Step Two: Nonlinear solver for global solution

E-step: Compute posteriors via Eq. 21.M-Step:• Minimize Eq. 17 using trust-region-reflective

algorithm• Apply stability to adjust the parameters

endAlgorithm 1: The parameters optimization procedure

Then, as shown in Alg. 1, the parameters are adjusted inevery M-step to satisfy the stability constraints as describedin section IV-B.2.

Finally, in order to speed up the convergence speed andreduce the computation cost, two strategies are employed.First, we perform sub-sampling for both the HUMROB tem-plate and the captured point cloud. We have tested differentnumbers of sampled points. Without losing key componentsand good performance, we identified the template and point

maps sample number should be 5000 and 2500 respectively.We found these can achieve a good balance. The otherstrategy is that we adopt the poses of previous frames witha third-order linear auto-regression model as a prediction forthe current pose.

C. Parameter settings

In Eq. 13 and Eq. 15, the initial σ2 = 0.022. In Eq. 21,λf = 0.5. In Eq. 17, λ⊥ = 0.5 and λr = 1000. In Alg. 1,the pose converge criteria is that iteration stops when themaximum movement for all joints is less than 0.002rad giventhat the precision for the real robot sensor is 0.1◦.

These values are tuned empirically and remained fixed forall the experiments.

VI. EXPERIMENTS

A. Robot test

We have tested our method on NAO robot that is manufac-tured by Softbank. The robot is 58 cm tall and weighs 5.2 kg.The robot have 23 DoFs in total for whole body (exclude theopen/close hands DoFs), while the generic human skeletonhas 47 DoFs. The specific robot configuration has beenenforced by embedding it into the HUMROB model.

The framework is validated on human subjects both maleand female, with subjects performing various and even someextreme motions. In Fig. 5 shows our retargeting results ofseveral different kinds of postures.

As shown in Fig. 5a, the robot can mimic the hands uppostures performed by the human subjects successfully. InFig. 5b, bow and squat postures can also be executed bythe robot resembling to the human. These postures mainlyfocus on upper/lower body posture with limited stabilityconstraints.

We present more results in Fig. 5c and Fig. 5d with humansubjects leaning forward and kicking. The stability needsto be taken into consideration under these circumstances,especially in one foot support, stability has the highestpriority protecting the robot from falling down. Though somepostures are not executed exactly as the human subjects, theystill share great similarity globally.

B. Stability

The stability check described in section IV-B.2 plays animportant role in the motion retargeting. While we try tomake the robot to perform the poses as similar as possibleto the human subject, we have to take its stability into consid-eration as the robot may fall down in some poses. We presenttwo cases in Fig. 6, that shows the retargeting results withand without our stability constraints. The middle columnsare simulation results without stability requirements, whichare not subjected to gravity force, so the robot acts just likethe human subject. However, if we apply the same pose ontothe real robot, it will fall down. After enforcing the stabilityconstraints, we will get the results as shown in the rightcolumns. We try to achieve the most similar poses whilemaintaining its stability.

(a) Hands up(b) Bow and Squat

(c) Rotation and Lean forward (d) Kick

Fig. 5: Results in different kinds of postures. For each case, the first row shows the original mesh and image, and the secondrow shows the robot executing the pose computed with our proposed approach.

Fig. 6: The effect of robot stability. The left column isoriginal mesh, the middle column is simulation result withoutgravity and stability constraints, the right column is theretargeting on real robot with stability applied.

In the real humanoid test, we also restrict that at least onefoot fully touches the ground. Besides, the support leg of therobot needs to bend a little if it almost reaches the maximummotor torque, as shown in the third pose in Fig. 5d.

(a) Lean forward

(b) Rotation

Fig. 7: Visual comparisons of our results with SDK on twocases. In each case, the left one is the captured image, themiddle left is the result from the SDK while the middleright one is result using our framework, the right one is thesimulation result for motion retargeting.

C. Comparison

We have done some qualitative comparisons between ourmethods and Kinect SDK as shown in Fig. 7.

The skeletons are colored with green and magenta rep-resenting the left and right part of the body, respectively.In Fig. 7a, the human subject leans forwards with one ofher legs severely occluded. The skeleton we get from KinectSDK is clearly wrong for the leg, while we can still get morereasonable result. Fig. 7b shows that when the subject turnsround, the SDK cannot differentiate the front and the backof the human body which cause the flipping of left and rightsides of the body. However, since we take the depth sequenceinto consideration and penalize abrupt changes in successiveframes rather than tracking frame by frame, we are able tohandle these cases.

VII. CONCLUSION

In this paper, we propose a novel generative frameworkfor human-robot motion retargeting with motion trackingand retargeting computed in a single formulation. A para-metric HUMROB template model is built with the robotconfigurations embedded, from which we can formulate thecost function to maintain the similarity between deformedtemplate model and captured data from depth sensor. Thejoint limitation of the robots and stability constraints areenforced in the framework. No pre-defined joints mappingbetween robot and human subject is needed, instead we cantake the similarity for both joint angles and the end positionsimultaneously into consideration. We have validated ourmethod on both simulated and real robots to verify its abilityof motion retargeting for various poses.

Limitations: 1) For some self-interactive motions likehis/her hand touching the head, the interaction may not bemaintained by the robot since we haven’t enforced inter-action constraints explicitly in our formulation. 2) Due tonon-optimized Matlab code and motionless assumption, thewhole process is not real-time. These are two of our majorworks to be improved in the future.

ACKNOWLEDGMENTThis work is partially supported by the National Natural

Science Foundation of China (No. 61603302, 51375390,61332017, 61170324, 61572020), US National ScienceFoundation grants (IIS-1231545, IIP-1543172), Key Indus-trial Innovation Chain of Shaanxi Province Industrial Area(2016KTZDGY06-01, 2015KTZDGY04-01), and the ”111project” of China (No. B13044). Xinxin Zuo and RuigangYang are the co-corresponding authors for this paper.

REFERENCES

[1] M. J. Gielniak, C. K. Liu, and A. L. Thomaz, “Generating human-likemotion for robots,” Int. J. Robot. Res., vol. 32, no. 11, pp. 1275–1301,2013.

[2] S. Guo, R. Southern, J. Chang, D. Greer, and J. Zhang, “Adaptive mo-tion synthesis for virtual characters: a survey,” The Visual Computer,vol. 31, no. 5, pp. 497–512, 2015.

[3] O. E. Ramos, N. Mansard, O. Stasse, C. Benazeth, S. Hak, and L. Saab,“Dancing humanoid robots: Systematic use of osid to compute dynam-ically consistent movements following a motion capture pattern,” IEEERobot. Autom. Mag., vol. 22, no. 4, pp. 16–26, 2015.

[4] L. P. Poubel, S. Sakka, D. Cehajic, and D. Creusot, “Support changesduring online human motion imitation by a humanoid robot usingtask specification,” in Proc. IEEE Int. Conf. Robot. Autom., 2014, pp.1782–1787.

[5] J. P. BANDERA, J. A. RODRIGUEZ, L. MOLINA-TANCO, andA. BANDERA, “A survey of vision-based architectures for robotlearning by imitation,” Int. J. of Hum. Robot., vol. 9, no. 1, pp. 1–40,2012.

[6] B. Dariush, Y. Zhu, A. Arumbakkam, and K. Fujimura, “Constrainedclosed loop inverse kinematics,” in Proc. IEEE Int. Conf. Robot.Autom., 2010, pp. 2499–2506.

[7] G. B. Hammam, P. M. Wensing, B. Dariush, and D. Orin, “Kinody-namically consistent motion retargeting for humanoids,” InternationalJournal of Humanoid Robotics, vol. 12, no. 04, p. 1550017, 2015.

[8] Y. Zhu, B. Dariush, and K. Fujimura, “Kinematic self retargeting: Aframework for human pose estimation,” Computer vision and imageunderstanding, vol. 114, no. 12, pp. 1362–1375, 2010.

[9] J. Koenemann, F. Burget, and M. Bennewitz, “Real-time imitation ofhuman whole-body motions by humanoids,” in Proc. IEEE Int. Conf.Robot. Autom., 2014, pp. 2806–2812.

[10] M. Lee and I. Cohen, “Proposal maps driven mcmc for estimatinghuman body pose in static images,” in Proc. IEEE CVPR, 2004, pp.II–334.

[11] T. Tosun, R. Mead, and R. Stengel, “A general method for kinematicretargeting: Adapting poses between humans and robots,” in Proc.ASME 2014 Int. Mech. Eng. Congress Expo., 2014.

[12] Y. Ou, J. Hu, Z. Wang, Y. Fu, X. Wu, and X. Li, “A real-timehuman imitation system using kinect,” International Journal of SocialRobotics, vol. 7, no. 5, pp. 587–600, 2015.

[13] C. Plagemann, V. Ganapathi, D. Koller, and S. Thrun, “Real-timeidentification and localization of body parts from depth images,” inProc. IEEE Int. Conf. Robot. Autom., 2010, pp. 3108–3113.

[14] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore,A. Kipman, and A. Blake, “Real-time human pose recognition in partsfrom single depth images,” in Proc. IEEE CVPR, 2011, pp. 1297–1304.

[15] M. Ye and R. Yang, “Real-time simultaneous pose and shape esti-mation for articulated objects using a single depth camera,” in Proc.IEEE CVPR, 2014, pp. 2353–2360.

[16] Y. Yang, V. Ivan, and S. Vijayakumar, “Real-time motion adaptationusing relative distance space representation,” in Proc. IEEE ICAR,2015, pp. 21–27.

[17] Y. Seol, C. O’Sullivan, and J. Lee, “Creature features: online motionpuppetry for non-human characters,” in Proc. ACM SCA, 2013, pp.213–221.

[18] K. Yamane, Y. Ariki, and J. Hodgins, “Animating non-humanoidcharacters with human motion data,” in Proc. ACM SCA, 2010, pp.169–178.

[19] A. Myronenko and X. Song, “Point set registration: Coherent pointdrift,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 12, pp.2262–2275, 2010.

[20] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihoodfrom incomplete data via the em algorithm,” Journal of the royalstatistical society, pp. 1–38, 1977.

[21] R. M. Murray, Z. Li, and S. S. Sastry, A Mathematical Introductionto Robotic Manipulation. Boca Raton, FL, USA: CRC Press, Inc.,1994.

[22] J. P. Lewis, M. Cordner, and N. Fong, “Pose space deformation: a uni-fied approach to shape interpolation and skeleton-driven deformation,”in Proc. ACM SIGGRAPH, 2000, pp. 165–172.

[23] J. Tong, J. Zhou, L. Liu, Z. Pan, and H. Yan, “Scanning 3d full humanbodies using kinects,” IEEE Trans. Vis. Comput. Graph, vol. 18, no. 4,pp. 643–650, 2012.

[24] Q. Zhang, B. Fu, M. Ye, and R. Yang, “Quality dynamic human bodymodeling using a single low-cost depth camera,” in Proc. IEEE CVPR,2014, pp. 676–683.

[25] T. Helten, A. Baak, G. Bharaj, M. Muller, H. Seidel, and C. Theobalt,“Personalization and evaluation of a real-time depth-based full bodytracker,” in Proc. IEEE 3DV, 2013, pp. 279–286.

[26] M. Ye, Y. Shen, C. Du, Z. Pan, and R. Yang, “Real-time simultaneouspose and shape estimation for articulated objects using a single depthcamera,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp.1517–1532, 2016.