![Page 1: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/1.jpg)
Machine Learning for Computer Graphics
Aaron HertzmannUniversity of TorontoCIAR summer school
July 16, 2005
![Page 2: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/2.jpg)
CG is maturing …
![Page 3: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/3.jpg)
… but it’s still hard to create
![Page 4: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/4.jpg)
… it’s hard to create in real-time
![Page 5: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/5.jpg)
Petersen, dir: The Perfect Storm, 2000
![Page 6: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/6.jpg)
One shot of The Perfect Storm
1. “Ocean scout” looks through 1000-frame simulation, puts camera in position (days)
2. Animator refines camera motion, adds boats (days)
3. TD adds lighting, particle simulation, bow wake (8 weeks)
4. Compositor puts it all together (2 weeks)
![Page 7: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/7.jpg)
Altogether…
• 250-300 shots (~5 seconds each)– 75 TDs– 12 animators– 6 “ocean scouts”– 15 “match movers”– 15 compositors
• All in all: close to 100,000 man-hours!
![Page 8: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/8.jpg)
Thus, the problem
The process for creating CG doesn’t scale:– Good models are hard to come by– Need a human expert to create each one
This is fine for big studios, but not for independent artists, home users, researchers, etc.
![Page 9: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/9.jpg)
Data-driven computer graphics
What if we can get models from the real world?
![Page 10: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/10.jpg)
Overview
Intro (10 mins)Character Animation (80 mins)• Motion textures• Probabilistic kinematic models• Biomechanical ModelsTexture (60 mins)• Texture synthesis and extensions• Image Analogies
![Page 11: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/11.jpg)
Key themes
1. Black-box models vs. “strong” models
• “Strong” models leverage relevant domain knowledge
• Require less data to be predictive (sometimes much less)
• Can be more efficient• But require more effort to design
![Page 12: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/12.jpg)
Key themes
2. Algorithms driven by the application, not the tools
• Formulate the problem, then design a model and an algorithm
• Does the algorithm solve the real problem?• Don’t just apply your favorite learning
algorithm uncritically
![Page 13: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/13.jpg)
Key themes
3. Symbiosis of graphics and vision
Graphics Vision/Learning
Hyperparameters
Scene
Images
![Page 14: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/14.jpg)
Key themes
4. Practical issues matter
• As researchers, we want to study the fundamentals of a problem
• But complicated and slow techniques are much less likely to be adopted
• (This contradicts theme #1)
![Page 15: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/15.jpg)
Character animation
![Page 16: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/16.jpg)
Body parameterization
Pose at time t: qt
Root pos./orientation (6 DOFs)
Joint angles (29 DOFs)
MotionX = [q1, …, qT]
![Page 17: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/17.jpg)
Forward kinematics
Pose to 3D positions:
qt
[xi,yi,zi]t
FK
![Page 18: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/18.jpg)
Keyframe animation
![Page 19: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/19.jpg)
Keyframe animation
http://www.cadtutor.net/dd/bryce/anim/anim.html
q1 q2 q3
q(t) q(t)
![Page 20: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/20.jpg)
Keyframe animation
• Define a set of key poses: [q1,…,qT]• Interpolate to produce q(t)
– typically, with spline curves
![Page 21: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/21.jpg)
Summary of keyframing
• Pros:– very expressive. total control to the artist
• Cons:– very labor intensive– hard to create physical realism– hard to match individual style
• Uses:– potentially, everthing except complex
physical phenomena (e.g., smoke)
![Page 22: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/22.jpg)
Motion capture
[Images from NYU and UW]
![Page 23: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/23.jpg)
Motion capture
![Page 24: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/24.jpg)
Mocap is not a panacea
![Page 25: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/25.jpg)
Motion capture
Demo
![Page 26: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/26.jpg)
Summary of motion capture
• Pros:– captures specific style of real actors
• Cons:– often not expressive enough (!)– time-consuming and expensive
• lots of equipment, space, actors• manual clean-up
– hard to edit
• Uses:– character animation– medicine (kinesiology, biomechanics)
![Page 27: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/27.jpg)
Data-driven animation
Can we learn motion style from examples• How do we model and represent style?• Representation will directly affect
quality of the results
![Page 28: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/28.jpg)
Animation
Off-Line Learning
Motion Learning Style
Synthesis
Pose
Constraints
Data-driven animation
![Page 29: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/29.jpg)
Probabilistic motion models
• Given input motion sequences, learn PDF over motions
• Generating motions: constrained sampling from PDF
![Page 30: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/30.jpg)
Data “learning”
Probabilistic motion models
Walk cycle in pose/velocity space
HMM
Brand and Hertzmann, SIGGRAPH 2000, “Style machines”
![Page 31: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/31.jpg)
Style machines demo
![Page 32: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/32.jpg)
Summary of Style machines
Pros: – generate novel sequences of motions– parameterized style model
Cons:– HMM is a poor model for continuous motion– No kinematic control yet– No physical model
• e.g., ground contact is smoothed out
– Our learning strategy could be improved…
![Page 33: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/33.jpg)
Mark V. Chaney
[Shannon 48] proposed a way to generate English-looking text using N-grams:– Assume a generalized Markov model– Use a large text to compute probability distributions of
each letter given N-1 previous letters • precompute or sample randomly
– Starting from a seed repeatedly sample this Markov chain to generate new letters
– One can use whole words instead of letters too:
WE NEED TO EAT CAKE
![Page 34: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/34.jpg)
Mark V. ChaneyResults (using alt.singles corpus):
– “As I've commented before, really relating to someone involves standing next to impossible.”
– "One morning I shot an elephant in my arms and kissed him.”
– "I spent an interesting evening recently with a grain of salt"
![Page 35: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/35.jpg)
Motion graphs
• Idea: cut-and-paste motion capture to create new motion
• Four papers introduced the idea at SIGGRAPH 2002
• Inspired by texture synthesis algorithms (next hour)
• I’ll outline one of them: J. Lee et al., Interactive Control of Avatars
![Page 36: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/36.jpg)
Motion graphs
Input: raw motion capture
“Motion graph”
![Page 37: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/37.jpg)
Distance between FramesDistance between Frames
),(),(),( jiji vvdppdjiD α+=
Weighted differencesWeighted differencesof joint anglesof joint angles
Weighted differencesWeighted differencesof joint velocitiesof joint velocities
![Page 38: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/38.jpg)
Pruning Transition
Contact state: Avoid transition to dissimilar contact state
Likelihood: User-specified threshold
Similarity: Local maxima
Avoid dead-ends: Strongly connected components
![Page 39: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/39.jpg)
Run-time graph search
Best-first graph traversal• Path length is bounded• Fixed number of frames at each frame
Comparison to global search• Intended for interactive control• Not for accurate global planning
![Page 40: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/40.jpg)
Global vs. Local Coordinates
Local, moving,body-relative
coordinates
Global, fixed,object-relative
coordinates
![Page 41: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/41.jpg)
Demo
![Page 42: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/42.jpg)
Motion synthesis with annotations
• Arikan et al., SIGGRAPH 2003
![Page 43: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/43.jpg)
Summary of motion textures
Pros:– very realistic– easy to understand and implement– real-time synthesis
Cons:– no generalization to new poses or new styles– e.g., no kinematic/keyframe control
![Page 44: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/44.jpg)
Style-Based Inverse Kinematics
with: Keith Grochow, Steve Martin, Zoran Popović
![Page 45: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/45.jpg)
Motivation
![Page 46: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/46.jpg)
Problem Statement
• Generate a character pose based on a chosen style subject to constraints
Constraints
Degrees of freedom (DOFs) q
![Page 47: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/47.jpg)
Style Representation
• Objective function– given a pose evaluate how well it matches a style – allow any pose
• Probability Distribution Function (PDF)– principled way of automatically learning the style
![Page 48: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/48.jpg)
Goals for the PDF
• Learn PDF from any data
• Smooth and descriptive
• Minimal parameter tuning
• Real-time synthesis
![Page 49: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/49.jpg)
Mixtures-of-Gaussians
![Page 50: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/50.jpg)
SGPLVM
Scaled Gaussian Process Latent Variable Model
based on [Lawrence 2004]
– automatic parameter scaling– extensions for real-time synthesis– style interpolation
![Page 51: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/51.jpg)
Gaussian Processes
Let g(x) be a nonlinear mapping, e.g., RBFs:– y = g(x) = ∑i wi φi(x)– Gaussian prior on wi
We can marginalize out w explicitlyp(ynew | X, Y) = ∫ p(ynew, w | X, Y) dw
![Page 52: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/52.jpg)
Gaussian Processes
![Page 53: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/53.jpg)
Gaussian Processes
![Page 54: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/54.jpg)
y(q) = q orientation(q) velocity(q)[ q0 q1 q2 …… r0 r1 r2 v0 v1 v2 … ]
Features
![Page 55: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/55.jpg)
GPLVM
y1
y2
y3
g(x)
x1
x2
Latent Space Feature Space
x ~ N(0,I)
![Page 56: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/56.jpg)
Precision in Latent Space
σ2(x)
![Page 57: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/57.jpg)
CPose Synthesis
y1
y2
y3
x1
x2
y
xx
arg minx,q LIK (x,y(q); θ)s.t. C(q) = 0
![Page 58: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/58.jpg)
);(ln2);(2
);();(L 2
2
2
IK θxθxθ)f(xyW
θyx, θ σσ
D+
−=
SGPLVM Objective Function
y1
y2
y3
x1
x2 θ)f(x;y
xx
C
![Page 59: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/59.jpg)
Style Learning
y1
y2
x1
x2
![Page 60: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/60.jpg)
Style Learning
y1
y2
y3
x1
x2
![Page 61: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/61.jpg)
Different styles
Jump ShotTrack StartBaseball Pitch
![Page 62: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/62.jpg)
Style interpolationGiven two styles θ1 and θ2, can we
“interpolate” them?
));(exp()(1 1θyy IKLp −∝
Approach: interpolate in log-domain
));(exp()(2 2θyy IKLp −∝
![Page 63: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/63.jpg)
Style interpolation
));(exp()( 22 θyy IKLp −∝));(exp()(1 1θyy IKLp −∝
(1-s) s
)(s)()s1( 21 ypyp +−
![Page 64: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/64.jpg)
Style interpolation in log space
));(exp( 1θyIKL− ));(exp( 1θyIKL−
(1-s) s
));(s);()s1((exp( 21 θyθy LL +−−
![Page 65: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/65.jpg)
Applications
![Page 66: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/66.jpg)
Interactive Posing
![Page 67: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/67.jpg)
Multiple motion style
![Page 68: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/68.jpg)
Style Interpolation
![Page 69: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/69.jpg)
Trajectory Keyframing
![Page 70: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/70.jpg)
Summary of Style IK
Pros:– Arbitrary kinematic constraints– Minimal parameter tuning
Cons:– Weak temporal model– Some optimization issues, particularly with
large datasets– Purely kinematic (no physical model)
![Page 71: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/71.jpg)
Learning Biomechanical Models of Human Motion
Aaron HertzmannUniversity of Toronto
with: Karen Liu, Zoran PopovićUniversity of Washington
![Page 72: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/72.jpg)
What determines how we move?
Individual Style:BiologyPhysicsIntentionEmotion…
Can we build realistic and accurate models?
![Page 73: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/73.jpg)
Goals for the model
1. Physically plausible
2. Generative models of human motion
• Predictive, synthesis-quality
3. Generalize from a small dataset
![Page 74: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/74.jpg)
Biomechanical principles
![Page 75: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/75.jpg)
Optimality Theory
Hypothesis:We optimize for efficiency, both in our body structure and movement
![Page 76: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/76.jpg)
Criticisms of Optimality Theory
• We’re not really globally optimal– The objective function constantly changes– We never really converge– We may be locally optimal
• Non-optimal variation, e.g., genetic drift • Hard to build realistic models
![Page 77: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/77.jpg)
Optimality Theory
This is controversial among biologists
Use optimization to test our assumptions about organisms
“Optimization is the only approach biology has for making predictions from first principles.”
– W. Sutherland, Nature, June 2005
![Page 78: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/78.jpg)
Not like this:
How do you walk?
All joints actively actuated
![Page 79: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/79.jpg)
Passive Dynamic Walking
Walking on a ramp without any muscles
![Page 80: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/80.jpg)
Walking robots
Collins et al, Science 2005
![Page 81: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/81.jpg)
Efficiency of walking
Dimensionless Cost of Transport = Energy cost / (Body Weight * Distance)
DCT: 1.6 DCT: 0.055DCT: 0.05
![Page 82: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/82.jpg)
Musculoskeletal structure
• Agonist/antagonist muscles• Passive properties of muscles
and tendons:– Elastic– Damping
![Page 83: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/83.jpg)
Muscle stiffness and springiness
Stiffness improves stability/robustness– Muscle stiffness varies for different tasks
Passive elements help conserve energy– saves 20-30% of energy during running
![Page 84: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/84.jpg)
Relative muscle preferences
• Some muscles are stronger than others• Some muscles are more efficient
– muscle attaches to bone in different ways
• Some body parts may be more prone to damage
![Page 85: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/85.jpg)
Physics-based motion style
![Page 86: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/86.jpg)
Body parameterization
18 body nodesPose at time t: qt
Root pos./orientation (6 DOFs)
Joint angles (29 DOFs)
Using exponential mapsMotion
X = [q1, …, qT]
![Page 87: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/87.jpg)
Forces at a joint
Assumption: constant stiffness at each joint
![Page 88: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/88.jpg)
Equations of motion at a joint
(Considering only aggregate forces)
tr(dWi/dqj Mi W’’iT) = Qmj + Qgj + Qpj + Qcj + Qsj
“F = ma”:
![Page 89: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/89.jpg)
Muscle preferences
Muscle preferences: αj
![Page 90: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/90.jpg)
Physical style
Parameter vector θ includes– spring constants ks1, ks2 (two per DOF)– rest angles q (one per DOF)– shoe spring constants kshoe (two)– damping constants kd (one for DOF)– muscle preferences αj (one per DOF)
Total: 147 dims. in θA vector θ defines a physical style
(Skeleton and masses from preprocessing)
![Page 91: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/91.jpg)
Constraints on motion
Foot contact constraints C
![Page 92: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/92.jpg)
Objective function for motion
Weighted sum of magnitudes of all muscle forces:
E(X; θ) = ∑j,t αj (Qm,j(t, X, θ))2
Qm,j(t,X,θ) determines muscle force at time t from X and θ (closed-form)
![Page 93: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/93.jpg)
Generating motion
X* = arg minX ∈ C E(X; θ)
X
E(X
; θ)
![Page 94: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/94.jpg)
Problem
θ is 147-dimensionalImpossible to tune by handHow can we acquire it from data?
![Page 95: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/95.jpg)
Nonlinear Inverse Optimization
![Page 96: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/96.jpg)
Problem statement
Given an optimal motion, what was the energy function?
Given– mocap XT
– constraints C
Determine θ
XT
E(X
; θ)
![Page 97: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/97.jpg)
What doesn’t work
Least-squares– ||XT – arg min E(X; θ) ||2
– not robust– very hard to optimize
Maximum likelihood– Gibbs distribution: p(X|θ, C) = e-E(X; θ, C)/Z(θ, C)
– Intractable– … even with Contrastive Divergence
![Page 98: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/98.jpg)
Idea
Goal:
E(XT; θ) = minX ∈ C E(X; θ)Learning objective function:
G(θ) = E(XT; θ) - minX ∈ C E(X; θ)
Constraints: ∑j αj = 1, αj >= 0 – to prevent E=0 everywhere
![Page 99: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/99.jpg)
Learning
G(θ) = E(XT; θ) - minX ∈ C E(X; θ)
XS = arg minX ∈ C E(X; θ)H(θ) = E(XT; θ) - E(XS; θ)
dG/dθ ≈ dH/dθ= dE/dθ|XT – dE/dθ|XS
![Page 100: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/100.jpg)
Learning
initialize θwhile not done do
XS = arg minX ∈ C E(X; θ)∆θ = dE/dθ|XT – dE/dθ|XS
β = arg minβ G(θ - β∆θ)θ = θ - β ∆θ
![Page 101: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/101.jpg)
XS
Learning
XT
E(X
; θ)
XS
XS = arg minX ∈ C E(X; θ) ∆θ ≈ dE/dθ|XT – dE/dθ|XS
![Page 102: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/102.jpg)
Nonlinear Inverse Optimization
• Solve for optimization parameters for any differentiable energy function
• Works with hard constraints• All you need is a forward optimizer• Degenerate form of CD• Related to energy-based models of
[LeCun and Huang 2005]
![Page 103: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/103.jpg)
A basic walk
![Page 104: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/104.jpg)
New constraints
![Page 105: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/105.jpg)
Warping vs. ground truth
![Page 106: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/106.jpg)
Comparison to mocap
![Page 107: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/107.jpg)
A “sad” style
![Page 108: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/108.jpg)
Walking uphill
![Page 109: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/109.jpg)
Motion warping vs. ground truth
![Page 110: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/110.jpg)
Comparison to mocap
![Page 111: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/111.jpg)
Running
![Page 112: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/112.jpg)
Springier shoes
![Page 113: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/113.jpg)
“Powerwalking”
![Page 114: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/114.jpg)
A different subject
![Page 115: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/115.jpg)
![Page 116: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/116.jpg)
Another subject
![Page 117: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/117.jpg)
![Page 118: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/118.jpg)
Summary of physics-based style
Pros:– Generalizes to new physical situations– Small training sets
Cons:– expensive optimizations– physical model is incomplete (so far)
Future work:– does it work for broader classes of motions
and styles?– model control, space of styles, etc.
![Page 119: Machine Learning for Computer Graphicshertzman/CIAR2005.pdf · 2005-07-16 · Machine Learning for Computer Graphics Aaron Hertzmann University of Toronto ... Symbiosis of graphics](https://reader033.vdocuments.net/reader033/viewer/2022041621/5e3f4e11fb7ecc7d01216dff/html5/thumbnails/119.jpg)
Summary of animation
Motion graphs– simple easy fast– can’t generate new poses at all
Probabilistic kinematic models– not quite as fast– no physics
Physics-based style– very slow– potentially, very powerful