learning a correlated model of identity and pose-dependent body shape variation for real-time...
TRANSCRIPT
Learning a correlated model of identity and pose-dependent body shape variation
for real-time synthesis
Brett Allen1,2, Brian Curless1, Zoran Popović1, and Aaron Hertzmann3
1 University of Washington2 Industrial Light & Magic
3 University of Toronto
Motivation
movies games
telepresence design
Goal
• We would like to be able to generate body models of any individual in any pose.
Identity +
pose shape
- want to synthesize models in real-time
- model should be learnable from real data
Data
• CAESAR data set: 44 subjects in 2 poses• Multi-pose data set: 5 subjects in 16 poses• Dense-pose data set: 1 subject in 69 poses
[Anguelov et al. 2005]
Related work
Anatomical methods
Aubel 2002
Chadwick et al. 1989Turner and Thalmann 1993Scheepers et al. 1997Wilhelms and Van Gelder 1997…
Example-based methods
7000
7000
7000
0
0
0
z
y
x
z
y
x
v
),( Φpv f
v = shape vectorp = example parameters = function parameters
Given: n examples v0…vn and n sets of parameters p0…pn (optional)find .
Scattered data interpolation
Allen et al. 2002
Lewis et al. 2000Sloan et al. 2001Kry et al. 2002
i
iif φΦp ),(columns of = key shapesi = reconstruction weights from applying k-nearest neighbors or RBFs on p
1
2
43
1
3
2
4
An aside on enveloping
Enveloping + scattered data interpolation= “corrective enveloping”
Latent variable modelsvWxvWx }),{,(f
x = latent variable (component weights)W = components in columnsv = average shape
Blanz & Vetter 1999
Allen et al 2003Seo et al 2003Anguelov et al 2005…
Pose variation vs body variation
Sloan et al. 2001
Pose variation vs body variation
Anguelov et al. 2005
KωvWxv
Our approach
iii
k
φv
b
φ
φ
c
cWxc
1
c = “character vector”: all information needed to put a character in any posev = shape in a particular pose
Intrinsic skeletonparameters: bonelengths and carrying angles
Two Problems
1. Scans might not be at “key” poses
2. Scans are not complete
Maximize: p(c | {e()})
…actually, we don’t know the pose or skinning weights either:
Maximize: p(c, s, q{} | {e()})
Maximum a posteriori estimation
})({log)(log)(log
2
1)2log(5.1log
})({)()(),,|(
}){|}{,,(
1 1
2)()(2
2
1 1
)(
)(
qsc
ve
qscqsc
eqsc
v
v
vvv
ppp
nnP
pppepP
pP
n n
iii
n n
ii
Results
Going to multiple characters
• One possibility: Learn several character vectors separately, then run PCA.
• Two problems:– the character vector contains values that have
different scales (rest positions, offsets, bone lengths)
– we don’t have enough data!
Identity variation
),0();,(),,( 2),(
NNg ii I0xqsce
cWxc
~ ~
c is the character vector of the th example persong(c,s,q) applies skinning and pose space deformations
Alternating optimization
• We initialize the {x} with the weights from running PCA on estimated skeleton parameters.
• We then optimize W, c, s, q.
• Then we optimize for {x}.
• Repeat…
Results
Results
(video)
Conclusions
We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two.+ very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves+ the learned model can generate body shapes very quickly (over 75 fps)
Limitations
• You need a lot of data! Our data set was too sparse in some areas.
• Some poses are hard to capture.• It’s very hard to compensate for the
skinning artifacts.• The shape matching could be improved
(high-frequency details are lost if the matching is poor).
Acknowledgements
• UW Animation Research Labs• Washington Research Foundation• National Science Foundation, NSERC, CFI• Microsoft Research, Electronic Arts, Sony, Pixar• Kathleen Robinette and the AFRL lab• Dragomir Anguelov• Domi Pitturo