dynamic skin deformation simulation using musculoskeletal model … · 2018. 5. 17. · skeleton...
Embed Size (px)
Computational Visual MediaDOI 10.1007/s41095-016-0065-1 Vol. 3, No. 1, March 2017, 49–60
Dynamic skin deformation simulation using musculoskeletalmodel and soft tissue dynamics
Akihiko Murai1(�), Q. Youn Hong2, Katsu Yamane3, and Jessica K. Hodgins3
c© The Author(s) 2016. This article is published with open access at Springerlink.com
Abstract Deformation of skin and muscle is essentialfor bringing an animated character to life. Thisdeformation is difficult to animate in a realistic fashionusing traditional techniques because of the subtlety ofthe skin deformations that must move appropriatelyfor the character design. In this paper, we presentan algorithm that generates natural, dynamic, anddetailed skin deformation (movement and jiggle) fromjoint angle data sequences. The algorithm has twosteps: identification of parameters for a quasi-staticmuscle deformation model, and simulation of skindeformation. In the identification step, we identify themodel parameters using a musculoskeletal model and ashort sequence of skin deformation data captured viaa dense marker set. The simulation step first usesthe quasi-static muscle deformation model to obtainthe quasi-static muscle shape at each frame of thegiven motion sequence (slow jump). Dynamic skindeformation is then computed by simulating the passivemuscle and soft tissue dynamics modeled as a mass–spring–damper system. Having obtained the modelparameters, we can simulate dynamic skin deformationsfor subjects with similar body types from new motiondata. We demonstrate our method by creating skindeformations for muscle co-contraction and externalimpacts from four different behaviors captured asskeletal motion capture data. Experimental resultsshow that the simulated skin deformations arequantitatively and qualitatively similar to measuredactual skin deformations.
1 National Institute of Advanced Industrial Scienceand Technology, Tokyo, 135-0064, Japan. E-mail:[email protected] (�).
2 Carnegie Mellon University, USA.3 Disney Research, USA.Manuscript received: 2016-08-31; accepted: 2016-10-23
Keywords three-dimensional graphics and realism;musculoskeletal model; quasi-staticmuscle model; dynamic skin deformation
Skin deformation of animated characters must benatural, dynamic, and detailed if the characters areto appear realistic and lifelike. This level of realismis particularly important in scenes of rich naturalenvironments such as those in The Jungle Book andrealistic special-effect shots such as those in Planetof the Apes. These deformations are essential forcreating a sense of life: tension in the muscles andjiggle of the underlying muscle and soft tissue conveythe exertion of the character and the dynamics of themotion.
A number of algorithms have been created forgenerating plausible skin deformation [1–3]. Mostof these approaches drive the skin motion fromthe skeletal structure. Linear blending, spring–damper systems, and volumetric models have beenused to create natural skin deformations. Theseapproaches, however, do not accurately representthe complex underlying structure of bones, tendons,muscles, and interstitial tissue that are present inthe human body. Recently, more effort has beenexpended in making anatomically grounded models.For example, Si et al.  used an anatomicallycorrect human musculoskeletal model to simulatequasi-static skin deformation for the upper bodybased on detailed geometry and deformation modelsof the muscles.
In this paper, we present an algorithm thatgenerates detailed skin deformation (movementand jiggle) from a skeleton animation based on
50 A. Murai, Q. Y. Hong, K. Yamane, et al.
standard motion capture joint angle data andthree models: (i) a quasi-static muscle model,(ii) a muscle dynamics model, and (iii) a softtissue dynamics model. Our approach has twomain steps: identification of quasi-static musclemodel parameters, followed by simulation ofmuscle and soft tissue dynamic deformation. In theidentification step, which is performed only oncefor each body type, we compute subject-specificmuscle shape parameters using a musculoskeletalmodel  and a short sequence of skin deformationdata (slow jump) captured with a dense markerset from Ref. . The quasi-static muscle modelrelates the quasi-static muscle shape to musclelength and tension, which can be obtained bycomputing the inverse kinematics and dynamicsusing a musculoskeletal model and joint angledata. Having obtained a muscle deformation model,we can simulate dynamic muscle deformations usingonly joint angle data. These can be obtained fromskeletal motion capture (with 50–60 markers) orfrom a physically plausible keyframe animation. Thesimulation step first uses the quasi-static muscledeformation model identified in the previous step toobtain the quasi-static muscle shape for the givenmotion sequence. It then computes the dynamicskin deformation by simulating the passive muscleand soft tissue dynamics modeled as a mass–spring–damper system. To demonstrate the efficiency ofthis approach, we use skeletal motion capture dataand electromyography (EMG) data to generate skindeformations that show muscle co-contraction andexternal impacts. Four motions, co-contraction,jumping, running, and impact, are created in thisway using parameters identified from a slow jump.
We realize a detailed skin deformation model basedon a human musculoskeletal model and measuredskin surface deformation data. The contributionsof our work include: (i) a method for identifyingmuscle deformation model parameters from a shortsequence of skin deformation data (using a slowjump) measured by motion capture using a densemarker set (400–450 markers), (ii) a method forapplying the muscle and skin deformation modelto joint angle data (for the motions co-contraction,jumping, running, and impact) recorded with 50–60 markers to create new sequences with detailed
skin and muscle deformation. Our approach appliesa parametric model for muscle deformation anda simple spring–damper model for soft tissuesimulation for its simplicity and accuracy.
Simulation-based methods tend to only handlerelatively static body parts and motions. Data-basedmethods find it difficult to handle novel motions.Our method combines simulation-based and data-driven approaches: our model has the ability togeneralize to different types of motions that arenot used for model identification, and the ability togeneralize to different body types, e.g., with differentfat thickness.
2 Related work
Skin deformation and dynamics are required for arealistic and natural-looking character, and thereforemany approaches have been developed to generatethis motion. One of the most common approachesis linear-blend skinning in which each skin vertexposition is computed using a weighted sum of thepositions of nearby joints. However, when linear-blend skinning is applied to a human character, theskin often lacks realism because of artifacts andbecause small details in the skin deformation aremissing. A number of algorithms have been createdto overcome these problems [2, 7].
More realistic models fall into two broad classes:simulated and data-driven. To model human-likecreatures, researchers have proposed a layeredapproach in which the skin is driven by interactionsbetween multiple underlying layers with differentproperties, based on anatomy. Chadwick et al. introduced a deformable layer between the rigidskeleton and the skin surface to represent musclesand soft tissues. Various parts of the human bodyhave been modeled in detail, such as the face ,torso , hands , and upper body . With thisapproach, the research focus has been on modelingthe shape and deformation of muscles to reduceartifacts and express details in skin deformation.The muscle shape has been approximated usingellipsoids , or learned from anatomical dataas in BodyParts3D . Deformation of musclesand soft tissues is often simulated by physics-basedmodels such as mass–spring–damper models and volumetric models such as the finite element
Dynamic skin deformation simulation using musculoskeletal model and soft tissue dynamics 51
method [15–17] or the finite volume method .To avoid issues with stability and computation,
data-driven approaches model skin deformationdirectly from data rather than simulating thebehavior of each layer in the musculoskeletalstructure. Anguelov et al.  used SCAPE tobuild a pose deformation model and a body shapevariation model from three-dimensional scan data,and to generate skin deformation for a new posture.Park and Hodgins  used motion capture to collectdata, and then trained the parameters of a mass–spring model from motion captured data to generateskin deformation from skeletal motion capture data.Their mass–spring model treated the body parts asa homogeneous medium rather than having separatemodels for muscles, fat, and interstitial tissue aswe do. Wang et al.  combined a mass–springsystem and a data-driven wrinkle model to achievewrinkling cloth simulation that was both efficientand physically plausible. Their wrinkle model relatesthe cloth wrinkles to joint angles. While we couldbuild a similar model for muscle deformation, itwould need to take joint angles, velocities, andaccelerations as inputs, making the database sizesignificantly larger than ours.
In biomechanics and robotics, manymusculoskeletal models have been developedfor simulation and analysis of human bodydynamics [5, 22]. We use one of the musculoskeletalmodels developed in this field to obtain musclepathways and tension . However, these modelsfocus on accurate simulation and analysis of humanmotion and do not include computation of the skinor muscle shape needed for animated characters.
During dynamic motion, actuated muscles causebone motion as well as muscle bulging due totension and length changes. Additionally, musclesand soft tissue, including skin, fat, and viscera,deform passively in response to the bone andmuscle movement as well as external forces fromthe environment. Accordingly, our skin deformationmodel comprises three sub-models (Fig. 1):1. A quasi-static muscle model that relates muscle
length and tension to the quasi-static muscleshape. This model represents muscle bulging and
Fig. 1 The three models used in our skin deformation model. Left:the quasi-static muscle model that determines the quasi-static muscleshape. Middle: the muscle dynamics model that determines thedynamic deformation of muscles. Right: the soft tissue dynamicsmodel that determines the dynamic deformation of fat and viscera.
relaxation due to different activation levels. Weuse a musculoskeletal model to estimate musclelength and tension from motion capture data.
2. A muscle dynamics model that describes thepassive dynamics of muscles using a mass–spring–damper system. The model consists of pointmasses placed at the vertices of a muscle polygonmodel, connected by springs and dampers. Eachpoint mass is also connected to the correspondingvertex of the quasi-static muscle by a spring anddamper.
3. A soft tissue dynamics model that describes thepassive dynamics of the skin and subcutaneous fatusing a mass–spring–damper system. The modelconsists of point masses placed at the vertices ofthe polygons on the skin surface with springs anddampers connecting them to neighboring verticesand to the point masses on the dynamic muscleor bone surfaces.
The quasi-static muscle model includes a numberof parameters that would be tedious to adjustby hand. We therefore provide a method toidentify the quasi-static muscle model parametersfrom a sequence of measured skin geometry data.Although this method requires skin deformationdata measured with a dense set of markers (400–450), the identification process is performed onlyonce for each subject or body type. Havingidentified the parameters, our system can simulateskin deformation from novel motion data capturedusing standard marker sets (i.e., joint angle data).We describe the details of the modeling andidentification processes in Section 3.2.
Figure 2 gives block diagrams of the identificationand simulation processes, where the blocks withred borders are the new components developed
52 A. Murai, Q. Y. Hong, K. Yamane, et al.
Fig. 2 Block diagram of the identification and simulation processes.The red borders depict the new components developed in this work.The identified model parameters are used in the simulation of motionsequences.
in this work. The details of each process areexplained in the following subsections. In Section 3.1,we review the skin deformation data collectionprocess  and the musculoskeletal model . Wethen present the quasi-static muscle model andparameter identification process in Section 3.2. InSection 3.3, we describe the algorithm to simulatedynamic skin deformation using muscle and softtissue dynamics models.
3.1 Skin deformation data andmusculoskeletal model
Identifying the quasi-static muscle model parametersrequires sample skin deformation data. We use thedata recorded by Park and Hodgins  usingan optical motion capture system with 400–450reflective markers. The marker trajectories wererecorded by 16 near-infrared Vicon MX-40 camerasat a rate of 120 fps. To identify the quasi-staticmuscle model parameters, we use a slow jump motionthat is approximately 300 frames (2.5 s) in length.We intentionally use a slow sequence in which softtissue dynamics do not play a big role in skindeformation.
In order to obtain the input data for the quasi-
static muscle model, we apply inverse kinematicsand dynamics algorithms from a musculoskeletalmodel  using the trajectories of 60 markersmanually chosen from the full set of 400–450. Themusculoskeletal model used in our work  consistsof skeleton and musculo–tendon network models.The skeleton model has 155 degrees of freedom(DOF), and the inertial parameters (mass, inertia,and local center of mass) of the bone segments arecomputed from an average human model . Themusculo–tendon network model includes 989 musclesto drive the skeleton, as well as 50 tendons, 177ligaments, and 34 cartilages to passively constrainthe skeleton. Each of the muscles, tendons, andligaments is represented by two end points (originand insertion points), any number of via points,and straight pathways between them. Each origin,insertion, or via point is fixed with respect to a bone,and their locations are computed by solving forwardkinematics (see Fig. 3).
We first obtain the joint angles of the skeletonmodel for each frame with an iterative inversekinematics algorithm using the positions of the60 markers as soft constraints . Then, thejoint torques required to execute the measuredmotion are computed by applying a recursive inversedynamics algorithm for articulated rigid bodies .Finally, we compute the muscle tensions required toproduce the joint torques. The number of musclesis much larger than the number of joint torques,and this redundancy is resolved with mathematicaloptimization that minimizes the quadratic sum ofthe muscle tensions. The detailed algorithm is givenin Refs. [5, 23, 27]. If EMG data are recorded at the
Fig. 3 Muscle shape and its local coordinate system. The red linerepresents the muscle pathway that connects the origin point, one ormore via points, and the insertion point. x is the location along themuscle pathway, θ is the angle from the polar axis, and rm is thedistance from the pathway.
Dynamic skin deformation simulation using musculoskeletal model and soft tissue dynamics 53
same time, we can obtain physiologically plausiblemuscle tensions for actions that are not observablefrom the motion, such as co-contraction .
3.2 Modeling and identification of quasi-static muscle model
We next develop a quasi-static muscle model thatcomputes the quasi-static muscle shape from themuscle length and tension. The length terms realizethe muscle volume constraint, and the tensionterms realize the muscle bulging even for isometriccontraction. We first choose about 300 surfacemuscles from the 989 muscles in Ref.  because wecannot identify the parameters of the inner musclesfrom surface data. The remaining 700 muscles arestill used for inverse dynamics because their tensionsdo affect the tensions of the surface muscles. Weconstruct the following quasi-static muscle modelaround the pathway of these 300 muscles.
Because most skeletal muscles have spindle-like shapes, we approximate the quasi-staticmuscle surface with a spindle whose cross-sectionperpendicular to the pathway is an ellipse, thesize of which varies along the pathway accordingto a sigmoid function (see Fig. 3). The pennatemuscles such as pectoralis major, whose cross-sectional shapes are quite different from an ellipsoidshape, are modeled with multiple thin spindle-shaped wires. The sigmoid parameters and theeccentricity are represented as functions of themuscle length and tension. In addition, we dividesome muscles at their center point into two partswith different sets of sigmoid function parametersto represent asymmetric muscles such as the soleus.The same identification and simulation method canbe applied to any muscle shape if the pathway of themuscle is given. In the following equations, we omitthe muscle index for clarity.
We represent the quasi-static muscle surface shapein a cylindrical polar coordinate system for each partwhose longitudinal axis is the muscle pathway (seeFig. 3). For a point on the m-th (m = 1, 2) partof a muscle, the distance from the pathway, rm, isdescribed by the location along the pathway x, theangle from the polar axis θ, and the current framenumber t (t = 1, 2, . . . , T ) as
rm(x, θ, t) =(
1− ε2(t) sin2 θ (1)
where sigmoid function parameters km,n(t) (m =1, 2, n = 1, 2, 3, 4) and the eccentricity ε(t) arefunctions of the muscle length l(t) (the distancebetween its end points through its via points) andtension τ(t) that are computed by inverse kinematicsand dynamics and mathematical optimization [5, 23,27]:
km,n(t) = αm,nl(t) + βm,nτ(t) + γm,n(n = 1, 2, 3, 4) (2)
ε(t) = α5l(t) + β5τ(t) + γ5 (3)In our implementation, the x axis is normalizedfor each part so that x = 0 represents the originor insertion of a muscle and x = 1 representsthe center point. The local coordinate system ofeach part is defined with respect to the closestbone’s local coordinate system at the initial skeletonposture. Therefore, the local coordinate system doesnot change discontinuously as long as the skeletonmotion is continuous, and there will not be asudden jump between the coordinate systems of twoconsecutive parts.
In this model, the total number of parametersto identify is 27 (αm,n, βm,n, γm,n (m = 1, 2, n =1, 2, 3, 4), α5, β5, γ5) for each muscle. We determinethese parameters at each muscle independently sothat the muscle shape fits the skin deformationaround the muscle during the motion capturesequence.
Let us define a muscle segment as a section of amuscle between two neighboring origin, insertion,or via points along the pathway, and denote thenumber of segments in a muscle by L. At each motioncapture frame t, we find a user-defined number ofmarkers closest to the pathway that belongs to eachsegment and represent their positions in the localcylindrical polar coordinate system of the muscle as(r̂k,t, θ̂k,t, x̂k,t) (k = 1, 2, . . . , L). The closest markerof a segment may differ across frames, but thenumber of markers used is determined once for eachbody part.
We then solve an optimization problem to adjustthe model parameters so that the total distancebetween the muscle surface and the positions of theclosest markers is minimized. We used a gradientalgorithm based on the interior reflective Newtonmethod  to minimize the following quadratic costfunction:
Z = 12 (Zr + avZv + atZt + asZs) (4)
54 A. Murai, Q. Y. Hong, K. Yamane, et al.
where a∗ are user-defined positive weights.The first term Zr represents the total squared
distance between the muscle surface and measuredmarker data and is formulated as
∆rk,t = r̂k,t − rm(x̂k,t, θ̂k,t, t)/β (6)where m represents the part containing segment kand β is a manually chosen fat thickness parameterdefined in Fig. 4. We use β = 1.0 in the experiment.The second term Zv represents the variance of themuscle volume across the entire motion sequence,formulated by
(V1(t) + V2(t)− V̄
where V̄ is the average of the total volume duringthe whole motion sequence and Vm(t) is the volumeof part m at frame t computed by
Vm(t) = li(t)∫ 1
0r2m(x, 0, t)π
√1− ε2(t)dx (8)
This term represents the conservation of musclevolume .
The third term Zt is added to constrain the radiusat the origin and insertion so that the muscle issmoothly connected to the tendons at the ends. Ztis formulated as
[(w − r1(0, 0, t))2 + (w − r2(0, 0, t))2
where w is a manually chosen tendon radius. We use
Fig. 4 Spring–damper connections between skin, muscle, and bonevertices, where r is the distance from each skin vertex to the nearestvertex on the muscles or bones. Muscle vertices (orange dots) orbone vertices (black dots) within α+ r (α > 0) are connected to theskin vertex by springs and dampers. β represents the average relativethickness of muscles with respect to soft tissue.
w = 0.01m in the experiment.The last term, Zs, represents the difference
between the radii at the ends of the two parts and isformulated as
t=1(r1(1, 0, t)− r2(1, 0, t))2 (10)
This term ensures that the two parts are connectedsmoothly at the boundary. We set the weights forEqs. (9) and (10) (at and as) high so that the muscleshape is smooth after optimization (av = 100, at =102, as = 102).
3.3 Muscle and soft tissue dynamicdeformation
Having identified the quasi-static muscle modelparameters, we can use the same set of parametersto simulate the dynamic deformation of the musclesand the soft tissue for new joint angle data sequences.Here, the shape of the quasi-static muscle modeldefines the rest shape of the dynamic muscle modelfrom its length and tension. Our method models thebones, the quasi-static muscles, the dynamic muscle,and the skin surface as polygonal surfaces. Let Psdenote the set of skin vertices, Pqm the vertices onthe quasi-static muscle surfaces, Pdm the vertices onthe dynamic muscle surfaces, and Pb the verticeson the bone surfaces. In the soft tissue dynamicsmodel (see Fig. 4, top), each skin vertex ps ∈ Ps isconnected to:1. the adjacent skin vertices,2. a set of nearby muscle vertices: those within the
hemisphere whose center is at ps and radius isα+ r (here, α = 2.0 cm), where r is the distancebetween ps and its nearest vertex in Pdm∪Pb andα (> 0) is the offset, and
3. the bone vertices included in that hemisphere.Note that a skin vertex may be connected to
multiple muscles. These connections allow the skinto slide over the muscle surface to the extent allowedby the spring stiffness. In the muscle dynamicsmodel (see Fig. 4, bottom), each muscle vertex pdm ∈Pdm is connected to:1. the adjacent dynamic muscle vertices,2. the skin vertices that have been connected to pdm,
and3. the corresponding quasi-static muscle vertex pqm.
As a result, the muscle deforms not only becauseof the skeleton motion but also based on the changein the quasi-static muscle shape due to muscle
Dynamic skin deformation simulation using musculoskeletal model and soft tissue dynamics 55
activation.If pi is connected to pj via a spring and damper
pair, the force applied to vertex pi from pj , fij , iscomputed by
fij = kij(||xij ||−lij)xij||xij ||
where xi and vi are the position and velocity ofvertex pi, xij = xj − xi, vij = vj − vi, and kij andcij are the stiffness and damping coefficients of thespring connecting vertices pi and pj .
The spring and damper coefficients have to bechosen so that the simulation results are stableand realistic. We use different spring and dampercoefficients for the springs between skin vertices,between muscle vertices, and between skin andmuscle vertices because they have different materialproperties. The skin has high stiffness whereas thesoft tissue has lower stiffness so that it moves moredynamically . The individual spring coefficientsare determined based on a few manually selectedglobal spring parameters shown in Table 1. Theseparameters are selected such that the skin becomesstiffer at locations closer to the bones such as aroundthe elbow and ankle, and more compliant at otherplaces, to emulate the effect of thick soft tissue andmuscle layers. To compute the spring coefficientsof individual springs, Kss, Kmm, and Kdqm arescaled by the size of the polygon that vertices belongto, whereas Ksmb is determined to be inverselyproportional to the distance between the skin andbone vertices. In all cases, the damping coefficientis set to d =
√k/50 for a connection with a spring
coefficient of k. While these parameters are manuallychosen, it is easy to find a set of values that yieldreasonable simulation results.
We add all the forces for each vertex in Ps ∪Pdm,and compute its acceleration by dividing by its mass.We use the velocity Verlet integration method  toupdate the positions and velocities of the skin andmuscle surface vertices. This method allows us toachieve high stability at no significant computationalcost over the explicit Euler method. Although an
Table 1 Types of springs and their global parameters
Parameter Springs between vertices of ValueKss Skin–skin 104
Kmm Dynamic muscle–muscle 105
Kdqm Dynamic muscle–quasi-static muscle 102
Ksmb Skin–dynamic muscle 107
implicit integration method  would allow a largertime step than explicit integration, that class ofmethod is not suitable for our application becauseit adds extra damping that diminishes the jiggling ofthe surface of the skin that we are modeling.
The sample skin deformation data used for quasi-static muscle model identification were recorded with400–450 reflective markers using 16 near-infraredVicon MX-40 cameras at a rate of 120 fps .We used 300 frames of a slow jump motionsequence to identify the quasi-static muscle modelparameters. The identified parameters are usedfor co-contraction, jumping, running, and impactexamples. In contrast, Ref.  used severalthousands frames to build a static and dynamicdeformation model. We need many fewer framesbecause our model generalizes to different motiontypes. The motion data used for the simulations(co-contraction, jumping, running, and impact)were recorded with 60 reflective markers using thesame motion capture system. We also recorded thecontact force between the subject and the groundusing two AMTI AccuSway PLUS force plates, eachof which can measure the six-axis contact force andmomentum at a rate of 1 kHz. An Aurion ZeroWiresystem with 16 pairs of electrodes was used tocapture electromyography (EMG) data for musclesbeneath the electrodes at a rate of 5 kHz. TheEMG data were processed by mean subtraction,rectification, and a Butterworth bandpass filter witha cut-off frequency of 10–1000 Hz. The force platesand the EMG sensors are optional, although theyallow us to consider the internal force and muscle co-contraction respectively. A high-speed video camerawas also used for some of the motions to capturedynamic skin deformation at 1 kHz, for ground truth.4.1 Evaluation of identified quasi-static
We first demonstrate the advantage of usingmusculoskeletal and muscle deformation models toobtain the underlying muscle shapes. As mentionedin Section 3, we use skin deformation data capturedby markers densely placed on the skin to identifythe muscle parameters. The distance between amarker and the closest point on the simulated skin
56 A. Murai, Q. Y. Hong, K. Yamane, et al.
surface indicates how well the muscle deformationmodel matches the actual skin deformation. Weevaluated the quasi-static muscle deformation usingtwo motion sequences: a slow jump motion usedfor identification, and a slow walk motion for crossvalidation. The active deformation of the muscleis several millimeters even at the maximum muscleactivity, which is much smaller than the deformationcaused by the skeleton motion. As shown in Table 2,nonetheless, the means and standard deviations ofthe distances are smaller with muscle deformationin both motions. Specifically, applying dynamicdeformation to the quasi-static muscle deformationmodel results in larger improvement on average(∼2 mm) than the dynamic deformation used inRef.  (∼1 mm). We also qualitatively comparedour results with Ref. , and there is no significantvisual difference between them.
4.2 Simulation results
We now show simulated deformation of differentparts of the skin for various motions (co-contraction,jumping, running, and impact) to demonstrate ourmethod. These motions are measured from adifferent subject to the one used for identifying thequasi-static muscle model using a standard markerset, force plates, EMG, and a high-speed camerarecording for reference. Video clips of the simulatedand recorded skin deformations are shown in theElectronic Supplementary Material (ESM).
Figure 5 shows a co-contraction motion that isa body building pose with tensing of the upperarm muscles. The top row represents the simulatedskin deformation, the middle row represents theincrease in the upper arm perimeter from the initialstate, where the color changes from yellow to redas the perimeter increases, and the bottom rowrepresents the corresponding snapshots from thehigh-speed video camera. The bottom graph showsthe normalized activities of the biceps brachii andtriceps brachii obtained by post-processing the EMGdata. The result shows that our method effectively
Table 2 Means and standard deviations of the distances betweenmeasured markers and their closest points on the skin surface withand without quasi-static muscle deformation
Motion With deformation Without deformationMean Std Mean Std
Slow jump 16.2 mm 2.3 mm 18.4 mm 2.5 mmSlow walk 18.6 mm 2.4 mm 20.2 mm 2.5 mm
Fig. 5 Skin deformation simulation with muscle co-contraction.Top: skin deformation simulation. Middle: the increase in the upperarm perimeter. Yellow and red colors indicate the amount of musclebulging. Bottom: activities of the biceps brachii and triceps brachiimeasured by EMG.
simulates the bulging of the muscles during co-contraction, which is mainly detected by the EMGdata because co-contraction of antagonistic musclesdoes not appear as joint motion. The skin jittersobserved in the ESM come from noise in theEMG signal, which remains even after Butterworthbandpass filtering.
Figure 6 shows the simulation results for a runningmotion and Fig. 7 represents the simulation resultsfor a jumping motion. In this motion capture session,we attached several markers in a grid pattern toquantitatively compare the actual and simulated skindeformations.
The simulated skin deformation with ouralgorithm, the simulated skin deformation withoutthe passive dynamics of muscles and skin, and thecorresponding movie from the high-speed camerarecording are shown in the ESM. Figure 7 plotsthe average trajectories of the 16 markers fromthe measured and simulated skin deformations.Each trajectory is represented in a local coordinatesystem fixed to the lower leg bone. The blue andred lines represent the measured and simulated
Dynamic skin deformation simulation using musculoskeletal model and soft tissue dynamics 57
Fig. 6 Skin deformation simulated by our method (top) and corresponding images from high-speed video recording (bottom). Similar skinfolds are to be seen, especially in the white circle.
Fig. 7 Trajectories of the measured and simulated markers duringthe jumping motion. Each trajectory is represented in the localcoordinate system fixed to the lower leg bone. The graphs only showfour selected markers for clarity, but all other markers show similartrajectories. The three axes are defined as shown in the upper-left.
trajectories of four selected markers, which are onthe softest part of the leg. The green dotted linesrepresent the marker trajectories that are simulatedwithout muscle or soft tissue dynamics. Thedifference between the simulated skin deformationwith muscle and soft tissue dynamics and theone without them is obvious. The third graphrepresents the y-direction trajectory, post-processedby a Butterworth high-pass filter with a cut-off
frequency of 100 Hz. This graph shows that theamplitude, frequency, and duration of the jigglesin the simulated skin deformation with muscle andsoft tissue dynamics are similar to those in themeasured motion, especially in the y direction justafter landing. This effect would not be realizedwithout muscle or soft tissue dynamics.
In Fig. 8, the subject hits an object with hisarm. The top row represents the simulated skindeformation, and the middle row represents thecorresponding snapshots from a high-speed camera.The high-speed camera shows the skin wrinklearound the elbow caused by the impact, which is alsoseen in our simulation. We include a parameter thatdetermines the average relative thickness of muscleswith respect to soft tissue (β in Fig. 4, bottom) tosimulate different body types. The third row of Fig. 8shows the skin deformation when this parameter setto β = 0.75, so that the model is 25% less muscularthan the one in the top row. The amplitude of theskin jiggle becomes larger than that observed in themuscular model as expected.
In this paper, we developed a new algorithmfor simulating skin deformation in novel motionsequences based on an anatomical model of themusculoskeletal system and a passive dynamicsmodel of soft tissue. This algorithm directly
58 A. Murai, Q. Y. Hong, K. Yamane, et al.
Fig. 8 Skin deformation simulation of an arm with an externalimpact. The bottom row shows a simulation result with a lessmuscular model.
generates the skin deformation from skeletal motiondata.• The quasi-static muscle model allows us to
compute the quasi-static muscle shape frommuscle length and tension information for a widerange of motions. The resulting muscle shapeis consistent with the dynamics of the motionbecause it is based on muscle pathway and tensiondata obtained by inverse kinematics and dynamicsalgorithms for a musculoskeletal model.• The passive dynamics of the soft tissue effectively
describes the interaction between the skin andinternal bones and muscles. Our model cansimulate skin deformations that depend on theunderlying structure, such as different jigglingpatterns when the skin hits the front side (tibia)and the calf side of the lower leg.• This algorithm can simulate physiologically
realistic skin deformations that are difficult toestimate only from standard motion capture dataif EMG data are recorded along with the motiondata. An example is muscle co-contraction,which cannot be estimated only from motion databecause the activations of antagonistic pairs ofmuscles do not cause joint motion.• The model can be generalized to a variety of body
types by applying our identification method.
Earlier simulation-based methods tend to handlerelatively static body parts and motions, whiledata-based methods find it difficult to handle novelmotions. Our method combines simulation-basedand data-driven approaches: simulation allows us toobtain realistic results for a wide variety of motions,while a small set of data can be used to adapt themodel to different body types.
We model the muscle and soft tissue dynamicswith a mass–spring–damper system. This mass–spring–damper system is based on a realistic bodyshape created by a modeler, and a simple spindle-like muscle shape is only used to indicate how thedetailed skin shape should deform. We chose to usea mass–spring–damper model because finite elementmethod (FEM) would require significantly moreparameter tuning for a similar simulation resolution,even if, as shown in Ref.  FEM may be appliedin principle. Our experimental results show thatour simple mass–spring–damper model with well-optimized parameters realizes a skin deformationsimulation that corresponds well to experimentallymeasured data. There are oscillation artifacts seenin our simulation that may be caused by the explicitintegration. Applying implicit integration  maydecrease these artifacts.
Our method has several limitations. The quasi-static muscle model parameter identification processrequires some frames of skin deformation datacaptured with a dense set of markers. Settingup such a motion capture session is cumbersomeand may not be possible for every subject. Asan alternative to measured skin deformation data,a modeler could provide the skin shapes at a fewframes in a motion sequence. It is also possible thatmodern depth cameras such as the Kinect 2 couldbe used to provide this data. The identificationprocess described in Section 3.2 could proceed inexactly the same way with this sparser data set.The other limitation is that we identified the quasi-static muscle model parameters assuming that themeasured skin deformation data are not affected bythe soft tissue dynamics. We chose to use slowmotions for identification to minimize this potentialcoupling. We then manually adjusted the passivedynamics model parameters (the spring and dampercoefficients) to obtain desirable skin deformations. Itmay be possible to identify the two sets of parameterssimultaneously using dynamic skin deformation datawhich might provide more accurate results.
Dynamic skin deformation simulation using musculoskeletal model and soft tissue dynamics 59
Electronic Supplementary Material Supplementarymaterial is available in the online version of this article athttp://dx.doi.org/10.1007/s41095-016-0065-1.
 Autodesk. Maya 2015. 2015. Lewis, J. P.; Anjyo, K.; Rhee, T.; Zhang, M.; Pighin,
F.; Deng, Z. Practice and theory of blendshape facialmodels. In: Proceedings of Eurographics 2014—Stateof the Art Reports, 2014.
 Pons-Moll, G.; Romero, J.; Mahmood, N.; Black, M.J. Dyna: A model of dynamic human shape in motion.ACM Transactions on Graphics Vol. 34, No. 4, ArticleNo. 120, 2015.
 Si, W.; Lee, S.-H.; Sifakis, E.; Terzopoulos, D.Realistic biomechanical simulation and control ofhuman swimming. ACM Transactions on GraphicsVol. 34, No. 1, Article No. 10, 2014.
 Murai, A.; Takeichi, K.; Miyatake, T.; Nakamura, Y.Musculoskeletal modeling and physiological validation.In: Proceedings of 2014 IEEE International Workshopon Advanced Robotics and its Social Impacts, 108–113,2014.
 Park, S. I.; Hodgins, J. K. Capturing and animatingskin deformation in human motion. ACM Transactionson Graphics Vol. 25, No. 3, 881–889, 2006.
 Vaillant, R.; Barthe, L.; Guennebaud, G.; Cani, M.-P.; Rohmer, D.; Wyvill, B.; Gourmel, O.; Paulin, M.Implicit skinning: Real-time skin deformation withcontact modeling. ACM Transactions on Graphics Vol.32, No. 4, Article No. 125, 2013.
 Chadwick, J. E.; Haumann, D. R.; Parent, R.E. Layered construction for deformable animatedcharacters. In: Proceedings of the 16th AnnualConference on Computer Graphics and InteractiveTechniques, 243–252, 1989.
 Tsoli, A.; Mahmood, N.; Black, M. J. Breathing lifeinto shape: Capturing, modeling and animating 3Dhuman breathing. ACM Transactions on Graphics Vol.33, No. 4, Article No. 52, 2014.
 Sueda, S.; Pai, D. K. Dynamic simulation of the hand.In: The Human Hand as an Inspiration for RobotHand Development. Balasubramanian, R.; Santos, V.J. Eds. Springer International Publishing, 267–288,2014.
 Lee, S.-H.; Sifakis, E.; Terzopoulos, D. Comprehensivebiomechanical modeling and simulation of the upperbody. ACM Transactions on Graphics Vol. 28, No. 4,Article No. 99, 2009.
 Pratscher, M.; Coleman, P.; Laszlo, J.;Singh, K. Outside-in anatomy based character rigging.In: Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation,329–338, 2005.
 Mitsuhashi, N.; Fujieda, K.; Tamura, T.; Kawamoto,S.; Takagi, T.; Okubo, K. BodyParts3D: 3D structuredatabase for anatomical concepts. Nucleic AcidsResearch Vol. 37, D782–D785, 2009.
 Zordan, V. B.; Celly, B.; Chiu, B.; DiLorenzo, P.C. Breathe easy: Model and control of simulatedrespiration for animation. In: Proceedings of the2004 ACM SIGGRAPH/Eurographics Symposium onComputer Animation, 29–37, 2004.
 Koch, R. M.; Gross, M. H.; Carlsy, F. R.; vonBüren, D. F.; Fankhauser, G.; Parish, Y. I. H.Simulating facial surgery using finite element models.In: Proceedings of the 23rd Annual Conference onComputer Graphics and Interactive Techniques, 421–428, 1996.
 Blemker, S.; Teran, J.; Sifakis, E.; Fedkiw, R.; Delp,S. Fast 3D muscle simulations using a new quasistaticinvertible finite-element algorithm. In: Proceedings ofInternational Symposium on Computer Simulation inBiomechanics, 2005.
 Webb, J. D.; Blemker, S. S.; Delp, S. L. 3D finiteelement models of shoulder muscles for computing linesof actions and moment arms. Computer Methods inBiomechanics and Biomedical Engineering Vol. 17, No.8, 829–837, 2014.
 Fan, Y.; Litven, J.; Pai, D. K. Active volumetricmusculoskeletal systems. ACM Transactions onGraphics Vol. 33, No. 4, Article No. 152, 2014.
 Anguelov, D.; Srinivasan, P.; Koller, D.; Thrun, S.;Rodgers, J.; Davis, J. SCAPE: Shape completion andanimation of people. ACM Transactions on GraphicsVol. 24, No. 3, 408–416, 2005.
 Park, S. I.; Hodgins, J. K. Data-driven modeling ofskin and muscle deformation. ACM Transactions onGraphics Vol. 27, No. 3, Article No. 96, 2008.
 Wang, H.; Hecht, F.; Ramamoorthi, R.; O’Brien,J. Example-based wrinkle synthesis for clothinganimation. ACM Transactions on Graphics Vol. 29,No. 4, Article No. 107, 2010.
 Delp, S. L.; Anderson, F. C.; Arnold, A. S.; Loan, P.;Habib, A.; John, C. T.; Guendelman, E.; Thelen, D. G.OpenSim: Open-source software to create and analyzedynamic simulations of movement. IEEE Transactionson Biomedical Engineering Vol. 54, No. 11, 1940–1950,2007.
 Nakamura, Y.; Yamane, K.; Fujita, Y.;Suzki, I. Somatosensory computation for man-machine interface from motion-capture data andmusculoskeletal human model. IEEE Transactions onRobotics Vol. 21, No. 1, 58–66, 2005.
 AIST. Japanese body dimension data 1997–98. 1998.Available at http://www.dh.aist.go.jp/database/97-98/e index.html.
 Yamane, K.; Nakamura, Y. Natural motion animationthrough constraining and deconstraining at will. IEEETransactions on Visualization and Computer GraphicsVol. 9, No. 3, 352–360, 2003.
 Luh, J. Y. S.; Walke, M. W.; Paul, R. P. C. On-linecomputational scheme for mechanical manipulators.Journal of Dynamic Systems, Measurement, andControl Vol. 102, No. 2, 69–76, 1980.
60 A. Murai, Q. Y. Hong, K. Yamane, et al.
 Nakamura, Y.; Yamane, K.; Murai, A. Macroscopicmodeling and identification of the humanneuromuscular network. In: Proceedings of the28th Annual International Conference of the IEEE,99–105, 2006.
 Yamane, K.; Fujita, Y.; Nakamura, Y. Estimationof physically and physiologically valid somatosensoryinformation. In: Proceedings of the 2005 IEEEInternational Conference on Robotics andAutomation, 2624–2630, 2005.
 Coleman, T. F.; Li, Y. On the convergence of interiorreflective Newton methods for nonlinear minimizationsubject to bounds. Mathematical Programming Vol. 67,No. 1, 189–224, 1994.
 Kardel, T. Niels Stensen’s geometrical theory ofmuscle contraction (1667): A reappraisal. Journal ofBiomechanics Vol. 23, No. 10, 953–965, 1990.
 Howell, J.; Chleboun, G.; Conatser, R. Musclestiffness, strength loss, swelling and soreness followingexercise-induced injury in humans. The Journal ofPhysiology Vol. 464, No. 1, 183–196, 1993.
 Swope, W. C.; Andersen, H. C.; Berens, P. H.;Wilson, K. R. A computer simulation method for thecalculation of equilibrium constants for the formationof physical clusters of molecules: Application to smallwater clusters. The Journal of Chemical Physics Vol.76, 637–649, 1982.
 Baraff, D.; Witkin, A. Large steps in cloth simulation.In: Proceedings of the 25th Annual Conference onComputer Graphics and Interactive Techniques, 43–54, 1998.
Akihiko Murai received his B.S.degree in mechanical engineering, andM.S. and Ph.D. degrees in mechano-informatics, in 2003, 2005, and 2009respectively from the University ofTokyo, Japan. He is currently a researchscientist at the National Instituteof Advanced Industrial Science and
Technology (AIST). Prior to joining AIST, he was apostdoctoral researcher at Disney Research, Pittsburgh,and a project assistant professor at the University ofTokyo. Dr. Murai is a recipient of Young InvestigationExcellence Award of RSJ (2016), Robotics-MechatronicsDivision Annual Prize (2010), SICE Research Award (2009),and JSPS Research Fellowship for Young Scientists (2009).His research interests include anatomical human modeling,the human neuro-musculoskeletal system, human motionmeasurement and analysis, and character animation.
Q. Youn Hong received her bachelordegree from Seoul National University,Seoul, Repulic of Korea, in 2006. Sheis currently working toward a graduatedegree at Carnegie Mellon University.Her research interests include characteranimation and skin deformation.
Katsu Yamane received his B.S.,M.S., and Ph.D. degrees in mechanicalengineering in 1997, 1999, and 2002respectively from the University ofTokyo, Japan. He is currently a seniorresearch scientist at Disney Research,Pittsburgh, and an adjunct associateprofessor at the Robotics Institute,
Carnegie Mellon University. Prior to joining Disney, hewas an associate professor at the University of Tokyo. Dr.Yamane is a recipient of numerous awards including King-Sun Fu Best Transactions Paper Award and Early AcademicCareer Award from the IEEE Robotics and AutomationSociety, and Young Scientist Award from the Ministry ofEducation, Japan. He has served as an associate editorof IEEE Transactions on Robotics and an area chair ofthe Robotics: Science and Systems Conference, as wellas program committee member of various internationalconferences in robotics and graphics. His research interestsinclude humanoid robot control and motion synthesis,human–robot interaction, character animation, and humanmotion simulation.
Jessica K. Hodgins is a professor inthe Robotics Institute and ComputerScience Department at Carnegie MellonUniversity and a director of DisneyResearch, Pittsburgh. Prior to movingto Carnegie Mellon in 2000, she was anassociate professor and assistant deanin the College of Computing at Georgia
Institute of Technology. She received her Ph.D. degree incomputer science from Carnegie Mellon University in 1989.Her research focuses on computer graphics, animation, androbotics with an emphasis on generating and analyzinghuman motion. She has received NSF Young InvestigatorAward, Packard Fellowship, and Sloan Fellowship. She waseditor-in-chief of ACM Transactions on Graphics from 2000to 2002 and ACM SIGGRAPH Papers Chair in 2003. In2010, she was awarded the ACM SIGGRAPH ComputerGraphics Achievement Award.
Open Access The articles published in this journalare distributed under the terms of the CreativeCommons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permitsunrestricted use, distribution, and reproduction in anymedium, provided you give appropriate credit to theoriginal author(s) and the source, provide a link to theCreative Commons license, and indicate if changes weremade.
Other papers from this open access journal are available freeof charge from http://www.springer.com/journal/41095.To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.