Modeling the Shape of People from 3D Range Scans

Dragomir AnguelovAI Lab

Stanford University

Joint work with

Praveen Srinivasan, Hoi-Cheung Pang, Daphne Koller, Sebastian Thrun, James Davis

The Dataset

Cyberware Scans 4 views, ~125k polygons ~65k points each

Problems Missing surface Drastic pose changes

70 scans

48 scans

Modeling Human ShapePose

vari

ati

on

Body-shape variation

Space of Human Shapes Movie

[scape movie]

Talk outline Data processing pipeline

Registration Recovering an articulated skeleton

Modeling the space of human deformations Pose deformations Body shape deformations

Shape completion Partial view completion Animating motion capture sequences

Talk outline Data processing pipeline

Registration Recovering an articulated skeleton

Modeling the space of human deformations Pose deformations Body shape deformations

Shape completion Partial view completion Animating motion capture sequences

Data Processing Pipeline

Registration - CC Algorithm [Anguelov et al. 2004]

Input:Pair of scans

Output: Correspondences

Correlated Correspondence Algorithm

ZX

ZX

Computes an embedding of mesh Z into mesh X

1. Defines a discrete Markov Net M over correspondence variables of mesh Z

Markov Net potentials enforce• Minimal surface deformation• Similar local surface appearance• Preservation of geodesic distance

2. Embedding of Z into X is computed by performing Loopy Belief Propagation on M

Related work: [Huttenlocher et al 00] [Coughlan 02]

Results: Human poses datasetModel

• 4 markers were used on each scan to avoid the need for multiple initializations of Loopy-BP

Cyberware scans

Registrations

Recovering articulation

Input: models, correspondences

Output: rigid parts, skeleton

Recovering articulation [Anguelov et al’04]

Stages of the process1. Register meshes using Correlated Correspondences

algorithm2. Initialize

1. break template surface into N arbitrary components

3. Cluster surface into rigid parts4. Estimate joints

Related work: [Cheung et al’03]

1 2 3 4

Probabilistic Generative Model

Transformations

T

TransformedModel

y1 yN…

Modela1

x1

aN

xN

…

…

Part labels

Points

zK

bK

…z1

b1Instanc

e

Point corrs

Points

Contiguity Prior Parts are preferably contiguous regions

Adjacent points on the surface should have similar labels

Enforce this with a Markov network:

a1 a2

a3

Clustering algorithm

Algorithm Given transformations , perform min-cut* inference

to get

Given labels , solve for rigid transformations

Objective

Contiguity prior

Data Likelihood

* [Greig et al. 89], [Kolmogorov & Zabih 02]

Results: Puppet articulation

Results: Arm articulation

Results: 50 scans of a human

Tree-shaped skeletonfound

Rigid parts found

Talk outline Data processing pipeline

Registration Recovering an articulated skeleton

Modeling the space of human deformations Pose deformations Body shape deformations

Shape completion Partial view completion Animating motion capture sequences

Modeling Pose Deformationinput

Deformations

output

Regression function

Joint angles

Modeling Pose Deformation

template

Predict independently for each triangle

Reconstructcomplete shape

Related work: [Allen et al ‘03][Sumner+Popovic 2004]

Representation of pose deformation

Pose deformation

Rigid articulateddeformation

Given estimates of R, Q, synthesizing the shape is straightforward :

Learning pose deformation

For each polygon, predict entries of from rotations of nearest 2 joints (represented as twists ).

Linear regression parameters :

Obtaining values of in the first place:

Twists and exponential maps

Twist

From twist to rotation matrix

Joint angles

Pose deformation space

Learning body-shape deformation Include also change in shape due to different

people:

Do PCA over body-shape matrices :

Getting estimates of :

PCA over body shape

Combining pose and body shape spaces

Talk outline Data processing pipeline

Registration Recovering an articulated skeleton

Modeling the space of human deformations Pose deformations Body shape deformations

Shape completion Partial view completion Animating motion capture sequences

Shape completion Find surface Y from our space which matches a set

of markers Z

Y[Z] : completed mesh deforms out of space spanned by , R to match Z

Y’[Z]: predicted mesh constrained to be in space spanned by , R

Target optimized by iteratively solving for , R orY while holding the others fixed

Partial view completion

Process: Add a few markers (~6-8) Run CC algorithm to get >

100 markers Optimize to find Y[Z]

Shape completion from motion capture data

Conclusions Presented a data-driven method of modeling human

deformations induced by Pose Body shape

Extending the model Nonlinear prediction of pose deformation (ongoing)

Shape complete original scans using current model Acquire and learn from the entire pose-bodyshape matrix Prior on likely joint angles, e.g. [Popovic + et al ’04] Enforce temporal consistency in tracking applications

Extending the possible applications Markerless motion capture (shape completion in shape-from-

silhouette data) Modeling other beasts