How to Use SMPL Model in Unity

Download Model & Scripts

About the SMPL Unity Download Package

How-to Run the Included Sample Project Play the Sample Game

How-to Setup SMPL Model in a New Scene Import Settings for SMPL Models Loading Models into a New Scene Adding Motions Loading the SMPL Blendshapes Script Adding the Joint Regressor JSON Creating new Body Shapes Optimizing for Faster Framerates Adding Textures

Comparison to other SMPL Download Packages

Download Model & Scripts Download the SMPL for Unity files from the SMPL website: http://smpl.is.tue.mpg.de/downloads By using this software you agree to the terms of the SMPL Model license at: http://smpl.is.tue.mpg.de/license

About the SMPL Unity Download Package The Unity Download includes the male & female SMPL model FBX files & scripts to support using SMPL in Unity. This download is a zipped archive called ‘SMPL_unity_v.1.0.0.zip' which contains the following files and folders:

● SMPL directory: ○ Models directory

■ Male & Female SMPL models as Unity-compatible FBX files - ../SMPL/Models/SMPL_f_unityDoubleBlends_lbs_10_scale5_207_v1.0.0.fbx - ../SMPL/Models/SMPL_m_unityDoubleBlends_lbs_10_scale5_207_v1.0.0.fbx

○ Samples directory ■ Male & Female sample texture-maps

1

- ../SMPL/Samples/Materials/SMPL_sampleTex_f.jpg - ../SMPL/Samples/Materials/SMPL_sampleTex_m.jpg

■ Male & Female sample shape-parameters as JSON files - ../SMPL/Samples/Betas/SMPL_sampleShape1_f.json - ../SMPL/Samples/Betas/SMPL_sampleShape2_f.json - ../SMPL/Samples/Betas/SMPL_sampleShape1_m.json - ../SMPL/Samples/Betas/SMPL_sampleShape2_m.json

○ Scripts directory ■ Male & Female joint-regressors as JSON files

- ../SMPL/Scripts/MPI/jnt_regressors/SMPL_jReg_f_lbs_v1.0.0.json - ../SMPL/Scripts/MPI/jnt_regressors/SMPL_jReg_m_lbs_v1.0.0.json

■ C# scripts for using SMPL model in Unity - ../SMPL/Scripts/MPI/SMPLBlendshapes.cs - ../SMPL/Scripts/MPI/SMPLJointCalculator.cs - ../SMPL/Scripts/MPI/SMPLModifyBones.cs - ../SMPL/Scripts/MPI/SMPLOptimalPoseBlends.cs

■ Third party C# scripts for reading JSON files & matrix manipulation (including relevant licenses):

- ../SMPL/Scripts/ThirdParty/Matrix - ../SMPL/Scripts/ThirdParty/JSON

● SMPL_mecanim directory containing a sample Unity project ● README.txt file containing installation instructions ● How-to_SMPLinUnity.pdf file containing a step-by-step guide to setup and use SMPL model in Unity

Each FBX files contains:

● Default mesh (male or female, depending on the loaded FBX file). This is the average body shape as defined by the SMPL parameters.

● Rigged skeleton for the default mesh. The skeleton is bound to the mesh using LBS (linear blend skinning).

● 10 shape blendshapes for changing SMPL Shape parameters. These 10 blendshapes can be changed to create a new body shape for the mesh.

● 414 corrective pose blendshapes. These blendshapes apply the pose-dependent deformations to the body.

The included scripts implement the following functionalities:

● When shape blendshapes are changed to create a new body, the skeleton needs to be updated to match the new body shape.

● When the model is reposed to a new pose, the corresponding pose-dependent blendshapes (aka pose blendshapes) need to updated so that deformations, such as fat and muscle bulging, can be applied to the body. Pose blendshapes also fix the skinning artifacts typically observed in LBS-skinned rigs, such as candy-wrapper/joint collapse and doughy elbows.

● Optimizing the model for faster framerate Extract the contents of this archive to a location where you would like to store the SMPL in Unity files.

2

How-to Run the Included Sample Project Navigate to the ‘SMPL_mecanim’ directory. This is a sample Unity project created using Unity’s Mecanim Animation System: https://unity3d.com/learn/tutorials/topics/animation/animate-anything-mecanim After you’ve unzipped the archive, open the SMPL_mecanim scene in Unity, located here (you can double click the following file to open or navigate to it from inside Unity to open the scene): <unzip_dir>/SMPL_mecanim/Assets/SMPL_mecanim.unity

Play the Sample Game The scene already contains the following objects:

● ‘SMPL_f_unityDoubleBlends_lbs_10_scale5_207_v1.0.0 (female SMPL model) ● ‘SMPL_m_unityDoubleBlends_lbs_10_scale5_207_v1.0.0’ (male SMPL model) ● ‘Main Camera’ (the primary camera in the scene) ● ‘Directional Light’ & ‘Directional Light_Sun’ (2 kinds of lighting for the scene) ● ‘Plane’ (ground plane)

The opened scene should look something like below:

3

If you click the Play button on the scene, the game engine should play a sample scene. Use the following keys to move/interact with the models in this scene:

● Arrow Keys ⇒ Walk Forward/ Left, Right / Backward ● Spacebar (while walking forward) ⇒ Jump ● Control / Command Key (while standing idle) ⇒ Crouch

The following images show the actions (walking, jumping and crouching) in the play mode:

How-to Setup SMPL Model in a New Scene If you want to set up your own scene with the SMPL model in it, the following steps will give you a simple walkthrough for how to do it. We’ve also included brief explanations about the different components in the SMPL model to help you better understand how each component is working. We use the included sample project as an example here. To begin, after you’ve created your own new project, extract the zipped archive named ‘SMPL.zip’ into your project’s assets directory, e.g. in the SMPL_mecanim project this is in: <unzip_dir>/SMPL_mecanim/Assets/SMPL

Import Settings for SMPL Models Once you place the files in your assets-directory, the model and script files should automatically get loaded into your Unity scene once you go back to the scene window. Loading might take a moment. Once loaded, you should be able to view the SMPL folder in your Project view. Here, navigate to either of the SMPL model FBX files and change the import settings to make sure they match those shown in the Inspector view above. Namely, these are the settings that might need to be changed:

● Model: ○ Meshes

■ Mesh Compression: Off ■ Read/Write Enabled: On

4

■ Optimize Mesh: Off ■ Import Blendshapes: On ■ Keep Quads: Off ■ Weld Vertices: Off

○ Normals & Tangents ■ Normals: Calculate ■ Smoothing Angle: recommended value=180 (higher numbers help with faster and

smoother rendering) ■ Tangents: None

○ Materials ■ Import Materials: off

● Rig: ○ Animation Type: Humanoid

Do the above for both the male and female SMPL FBX model. Please note that it might take a moment to apply the settings on the SMPL model.

Loading Models into a New Scene Once these settings are applied, the SMPL model is ready to be placed into the scene. In the sample Unity project SMPL_mecanim, you will see that the SMPL model’s main node in the Hierarchy view has a couple of components attached to it. These components were added using the Unity Mecanim tutorials. The components define game logic for movement and collision detection, namely:

● Animator: to add the animation tree from Unity’s Animator view ● Rigidbody: to make the models rigid objects that have collision ● Capsule Collider: to detect collision in a capsule area around the model ● Player Walk (Script): to define animation behavior for keyboard events

5

Further details about these and other game logic related options in Unity can be found in the tutorials mentioned in the next section.

For the Mesh node of the SMPL model in Hierarchy view it is recommended to set the Quality setting to use ‘4 Bones’. Since the skinning of SMPL model is defined using four bones, using this setting gives better results.

Adding Motions The existing animations in the sample project scene are applied to the SMPL models using mocap animations downloaded from Unity’s Asset Store: https://www.assetstore.unity3d.com/en/#!/content/5330 You can add your own animations to the SMPL model just like you would for any other humanoid model following the Mecanim Tutorial here: https://unity3d.com/learn/tutorials/topics/animation/humanoid-avatars?playlist=17099 Or by following a basic beginners’ tutorial found on this blog: http://quit-your-job-to-build-iphone-app.blogspot.de/2015/03/unity-50-step-by-step-animation-using.html

Loading the SMPL Blendshapes Script The SMPL Blendshapes C# script creates the module to be attached to the SMPL mesh inside the Unity scene. This script controls both the 'Shape blendshapes' and 'Pose blendshapes' for the SMPL model. The script allows you to use the following functionalities of the SMPL model:

● Reshaping the model to create new bodies

6

● Applying pose-dependent deformations with each new pose to create more realistic joint creases, muscle-bulging effects

● Optimizing the model for faster framerate To add the SMPL Blendshapes module to the SMPL model in your scene:

⋟ In the scene’s ‘Hierarchy’ view, click on the dropdown arrow next to the male or female model ⋟ Click on the mesh object of the model (called ‘m_avg’ or ‘f_avg’) ⋟ Here click ‘Add Component’, and type SMPL Blendshapes in the component search field. ⋟ Add the script as a component. You should see it now in the Inspector view of the mesh object as

‘SMPL Blendshapes (Script)’.

Adding the Joint Regressor JSON The script requires one of the joint-regressor files to run. These files are provided with the scripts in the following location: Female:<unzip_dir>/SMPL_mecanim/Assets/SMPL/Scripts/MPI/jnt_regressors/SMPL_jReg_f_lbs_v1.0.0.json

Male:<unzip_dir>/SMPL_mecanim/Assets/SMPL/Scripts/MPI/jnt_regressors/SMPL_jReg_m_lbs_v1.0.0.json

To add the joint-regressor file to the script, click on the ‘Joint Regressor JSON’ field (click the circle-dot icon on the right). Pick the regressor file corresponding to the gender of the model.

7

Now you are setup to use the SMPL model with your own animations and game elements in Unity!

Creating new Body Shapes A couple of sample files are provided to show how to load new body shapes into the SMPL model. Here’s how to try one of these shape samples:

⋟ In the SMPL Blendshapes (Script) component of the mesh object, click on the Shape Parms JSON field ⋟ Load one of the sample shape files corresponding to the gender of the model you’re setting the shape

for (gender is denoted by the ‘_m’ or ‘_f’ in all the model and script files) ⋟ Click play

Sample shape files are located here: <unzip_dir>/SMPL_mecanim/Assets/SMPL/Samples/Betas/SMPL_sampleShape*.json

8

Below are images of using the 2 sample shapes for the male model:

To create other body shapes, new shape JSON files can be created easily. You can get started by trying other random numbers in the sample shape files, or by manually varying the shape blendshapes of the FBX files. In the publicly available SMPL model there are 10 shape shape parameters that can be set to a positive or negative value. These are the 10 values present in the sample shape files (e.g. SMPL_sampleShape1_m.json). In the SMPL FBX model in Unity, each of the shape parameters is divided into 2 blendshapes, one setting a positive value of the shape parameter, the other for setting a negative value for the shape parameter . 1

There are a couple of things to note about the values in the JSON file versus how they are used in the Unity FBX file:

● When loading the values from the JSON file, the value is set to either the positive blendshape of the corresponding shape parameter or the negative blendshape (not both).

● The values in the JSON file need to be multiplied by 20.0 before being in the Unity model - basically, a multiplication by 100.0 and division by 5.0 for the explanation below

Unity blendshape can range only from 0-100, while for the original python model the shape blendshape values typically range from -5 to 5. So the SMPL model for Unity has the original shape blendshapes rescaled so that:

● Ranges of 0 to +5 lie between 0 to 100 for each shape parameter’s positive side of the blendshape. ● Ranges of 0 to -5 lie between 0 to 100 for each shape parameter’s negative side of the blendshape.

1 only one of these should be used at a time. This division of blendshapes into positive and negative halves is an optimization to allow Unity to ignore blendshapes that are not being used, by letting their value remain at 0.

9

Here is what the 10 shape parameters (totalling 20 blendshapes) look like in Unity:

The shape parameters are the principal components of the shape variation learned from 3D scans of thousands of people. So there is no specific definition of what body part each of the shape parameters changes. Here’s a rough layout for the first few:

- Shape000: height / overall scale of the model - Shape001: weight - Shape002: torso height + shoulder width - Shape003: chest breadth + neck height - ...

To gain a better understanding for how the shape parameters work, please look at the description in the SMPL paper (http://smpl.is.tue.mpg.de).

Optimizing for Faster Framerates If you encounter slower framerates using the SMPL model with its full set of blendshapes, the model can be optimized to run faster. Simply click the checkbox next to ‘Optimize Pose Blends’ to turn this feature on. The pose blendshapes will now be optimized to use only the first 40 most significant pose blendshapes (out of the total 207 pose blendshapes).

10

The total number of optimized pose-blendshapes can be changed to be even lower, or increased to suit your needs by simply changing the ‘_numPoseBlendsToSet’ variable in the SMPLBlendshapes.cs script file. For further details about it, please look at the script. <unzip_dir>/SMPL_mecanim/Assets/SMPL/Scripts/MPI/SMPLBlendshapes.cs

Adding Textures The SMPL FBX model files also contain UV texture coordinates to set a texture for the model. The sample project and zipped archive ‘SMPL.zip’ also contains a sample male and female texture image that can be applied to the SMPL FBX models, here: Female:

<unzip_dir>/SMPL_mecanim/Assets/SMPL/Samples/Materials/SMPL_sampleTex_f.jpg

Male:

<unzip_dir>/SMPL_mecanim/Assets/SMPL/Samples/Materials/SMPL_sampleTex_m.jpg

To use these texturemaps, simply create a new Material in your project and set the Albedo to one of the sample texturemap JPG files. Now you can use this material on the mesh element of the SMPL model in your scene.

Comparison to other SMPL Download Packages We provide separate download packages for different animation environments. At the moment we have 3 separate packages that provide the SMPL model:

1. SMPL for Python 2. SMPL for Maya 3. SMPL for Unity

The SMPL models provided with this download are in the form of FBX files that are compatible with Unity. The FBX files provided with the Maya download package can also be loaded into Unity however, there is a difference. The FBX files in the Maya download package and those in the Unity download package differ in the following ways:

11

● Since Unity has a different way of addressing blendshapes, the Pose & Shape blendshapes in the

Maya downloads may not work correctly inside Unity. ○ In Unity blendshapes range from 0 to 100, and start at 0 by default, while in Maya and the

original python version of the SMPL model, the blendshapes are assumed to be centered at 0, and extend in both the positive and negative direction. They range from -1 to +1 for pose blendshapes and can range from up to -5 to +5 in shape blendshapes in the original SMPL model.

● The Unity-compatible FBX provided also differs from the Maya version of the FBX in the number of blendshapes included. The blendshapes for Unity had to be divided into positive and negative set of values for each blendshape. Which means the total number of blendshapes (217 in Maya) is doubled when used in Unity (totalling 434 blendshapes).

● Contrary to Maya and our original python version of the model, which both assume a right-handed coordinate system, Unity utilizes a left-handed system. This issue is not addressed in the unity-compatible FBX files per se, but it is addressed in the scripts provided for Unity.

12