brett allen 1,2 , brian curless 1 , zoran popovi ć 1 , and aaron hertzmann 3
DESCRIPTION
Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis. Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3 1 University of Washington 2 Industrial Light & Magic 3 University of Toronto. movies. games. telepresence. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/1.jpg)
Learning a correlated model of identity and pose-dependent body shape variation
for real-time synthesis
Brett Allen1,2, Brian Curless1, Zoran Popović1, and Aaron Hertzmann3
1 University of Washington2 Industrial Light & Magic
3 University of Toronto
![Page 2: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/2.jpg)
Motivation
movies games
telepresence design
![Page 3: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/3.jpg)
Goal
• We would like to be able to generate body models of any individual in any pose.
Identity +
pose shape
- want to synthesize models in real-time
- model should be learnable from real data
![Page 4: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/4.jpg)
Data
• CAESAR data set: 44 subjects in 2 poses• Multi-pose data set: 5 subjects in 16 poses• Dense-pose data set: 1 subject in 69 poses
[Anguelov et al. 2005]
![Page 5: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/5.jpg)
Related work
![Page 6: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/6.jpg)
Anatomical methods
Aubel 2002
Chadwick et al. 1989Turner and Thalmann 1993Scheepers et al. 1997Wilhelms and Van Gelder 1997…
![Page 7: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/7.jpg)
Example-based methods
7000
7000
7000
0
0
0
z
y
x
z
y
x
v
),( Φpv f
v = shape vectorp = example parameters = function parameters
Given: n examples v0…vn and n sets of parameters p0…pn (optional)find .
![Page 8: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/8.jpg)
Scattered data interpolation
Allen et al. 2002
Lewis et al. 2000Sloan et al. 2001Kry et al. 2002
i
iif φΦp ),(columns of = key shapesi = reconstruction weights from applying k-nearest neighbors or RBFs on p
1
2
43
1
3
2
4
![Page 9: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/9.jpg)
An aside on enveloping
Enveloping + scattered data interpolation= “corrective enveloping”
![Page 10: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/10.jpg)
Latent variable modelsvWxvWx }),{,(f
x = latent variable (component weights)W = components in columnsv = average shape
Blanz & Vetter 1999
Allen et al 2003Seo et al 2003Anguelov et al 2005…
![Page 11: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/11.jpg)
Pose variation vs body variation
Sloan et al. 2001
![Page 12: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/12.jpg)
Pose variation vs body variation
Anguelov et al. 2005
KωvWxv
![Page 13: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/13.jpg)
Our approach
iii
k
φv
b
φ
φ
c
cWxc
1
c = “character vector”: all information needed to put a character in any posev = shape in a particular pose
Intrinsic skeletonparameters: bonelengths and carrying angles
![Page 14: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/14.jpg)
Two Problems
1. Scans might not be at “key” poses
2. Scans are not complete
Maximize: p(c | {e()})
…actually, we don’t know the pose or skinning weights either:
Maximize: p(c, s, q{} | {e()})
![Page 15: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/15.jpg)
Maximum a posteriori estimation
})({log)(log)(log
2
1)2log(5.1log
})({)()(),,|(
}){|}{,,(
1 1
2)()(2
2
1 1
)(
)(
qsc
ve
qscqsc
eqsc
v
v
vvv
ppp
nnP
pppepP
pP
n n
iii
n n
ii
![Page 16: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/16.jpg)
Results
![Page 17: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/17.jpg)
Going to multiple characters
• One possibility: Learn several character vectors separately, then run PCA.
• Two problems:– the character vector contains values that have
different scales (rest positions, offsets, bone lengths)
– we don’t have enough data!
![Page 18: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/18.jpg)
Identity variation
),0();,(),,( 2),(
NNg ii I0xqsce
cWxc
~ ~
c is the character vector of the th example persong(c,s,q) applies skinning and pose space deformations
![Page 19: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/19.jpg)
Alternating optimization
• We initialize the {x} with the weights from running PCA on estimated skeleton parameters.
• We then optimize W, c, s, q.
• Then we optimize for {x}.
• Repeat…
![Page 20: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/20.jpg)
Results
![Page 21: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/21.jpg)
Results
(video)
![Page 22: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/22.jpg)
Conclusions
We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two.+ very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves+ the learned model can generate body shapes very quickly (over 75 fps)
![Page 23: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/23.jpg)
Limitations
• You need a lot of data! Our data set was too sparse in some areas.
• Some poses are hard to capture.• It’s very hard to compensate for the
skinning artifacts.• The shape matching could be improved
(high-frequency details are lost if the matching is poor).
![Page 24: Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3](https://reader031.vdocuments.net/reader031/viewer/2022020417/56814491550346895db12f03/html5/thumbnails/24.jpg)
Acknowledgements
• UW Animation Research Labs• Washington Research Foundation• National Science Foundation, NSERC, CFI• Microsoft Research, Electronic Arts, Sony, Pixar• Kathleen Robinette and the AFRL lab• Dragomir Anguelov• Domi Pitturo