prediction of non-linear aging trajectories of faces k. scherbaum, m. sunkel, v. blanz and h.-p....
TRANSCRIPT
![Page 1: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/1.jpg)
Prediction of Non-Linear Aging Trajectories of Faces
K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel
[ 2007/5/9, Eurographics 2007, Prague ]
![Page 2: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/2.jpg)
Kristina Scherbaum [email protected]
Motivation / Goal
automated growth-prediction system applications
photofit-pictures of missing children automated animation, art
![Page 3: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/3.jpg)
Kristina Scherbaum [email protected]
9 years
10 years 11 years
Age Progression – Optimal Case
face space
child 1
child 2
child 3
![Page 4: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/4.jpg)
Kristina Scherbaum [email protected]
Real Case – Support Vector Regression
face space
9 years
10 years 11 years
only 1 sample per person no longitudinal study
find isosurfaces and gradients
Runge-Kutta Integration
9 years
10 years
11 years
![Page 5: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/5.jpg)
Kristina Scherbaum [email protected]
Main Assumption - Curved Trajectories
use machine learning non-linear Support Vector Regression
integration of local age-gradient
growing faces transform along curved trajectories
![Page 6: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/6.jpg)
Kristina Scherbaum [email protected]
Challenges
learn change over time of individual facesnon-linear dependency on time, curved trajectory
learn how the change depends on individual facenon-linear dependency in face space
sparse dataset, no longitudinal study
![Page 7: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/7.jpg)
Kristina Scherbaum [email protected]
3D Morphable Facemodel
System is based on a Morphable 3D Facemodel [Blanz,Vetter‘99] Built from 200 3D-face-scans of adults
![Page 8: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/8.jpg)
Kristina Scherbaum [email protected]
3D Morphable Facemodel
vector space of faces
vectors with point-to-point correspondence
Shape
Texture
![Page 9: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/9.jpg)
Kristina Scherbaum [email protected]
Representation of Faces - Face Spaces
PCA to reduce dimensionality (yields coefficients)
![Page 10: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/10.jpg)
Kristina Scherbaum [email protected]
Extended Morphable Model
Extension by … plus ~238 facemodels of teenagers 3 simultaneous laser scans per face
Correspondence by … top-down approach fitting Morphable Model to new 3D faces merging original data and best fit
![Page 11: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/11.jpg)
Kristina Scherbaum [email protected]
Fitting the Morphable Model to 3D Scans
no optical flow because scans are often incomplete
best fit of the morphable model
merged result3D laser scans
![Page 12: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/12.jpg)
Kristina Scherbaum [email protected]
learn function that maps any face x to a scalar age y
to learn this function we use …non-linear Support-Vector-Regression on training sets of l pairs
Age Progression Algorithm1
![Page 13: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/13.jpg)
Kristina Scherbaum [email protected]
Fitting a Regression Curve
for a given set of samples find f(x) such that all samples are within an -tube preselect and tradeoff between smoothness and errors of outliers
2
x
y
Linear: f(x) = wx + b Non-linear: f is sum of Gaussian RBF kernels K(x-xi)
![Page 14: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/14.jpg)
Kristina Scherbaum [email protected]
Gaussian RBF (Radial Basis Function) as kernel
we applied grid search using cross validation to optimize parameters such as (Kernelwidth) i and b are determined by SVM training
using LIBSVM for -Support Vector Regression
Non-Linear SVM Regression2
![Page 15: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/15.jpg)
Kristina Scherbaum [email protected]
Isosurfaces are defined in PCA space Gradient gives shortest path to next isosurface
Along the gradient … many facial changes due to aging almost no other changes
(known technique, Blanz et al. 99)
Thus: Compute growth along the gradient!
Local Aging3
![Page 16: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/16.jpg)
Kristina Scherbaum [email protected]
Gradient Example - Facial Attributes
gender manipulation
original
3
femalemale
![Page 17: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/17.jpg)
Kristina Scherbaum [email protected]
Growth Simulation: New Approach3
growth curve with given face x0 at time t currently we compute the local gradient and walk along this gradient instead we should compute the curved trajectory
![Page 18: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/18.jpg)
Kristina Scherbaum [email protected]
Solve differential equation … to compute curved trajectories integrate the differential equationusing Runge-Kutta algorithmperform small steps
Runge Kutta Integration4
![Page 20: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/20.jpg)
Kristina Scherbaum [email protected]
Reducing Complexity
we did not train on all principle components speedup of SVM training we experimented with 20, 40 or 80 PCs
Justification …
growth leads to overall change of facial size significant changes are represented
by the first PCs [ large variance ] facial growth should happen in the first PCs
4
![Page 21: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/21.jpg)
Kristina Scherbaum [email protected]
Growth Example
growth simulation for both, shape and texture
12 14 16 18 20 years
22 24 26 28 30 years
![Page 22: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/22.jpg)
Kristina Scherbaum [email protected]
More Examples
10 years
15 years
20 years
30 years
12 13 12 10 14
3D laser scans,original
age
![Page 23: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/23.jpg)
Kristina Scherbaum [email protected]
Background, Haircut
Face
Pose, Light
3D reconstruction and aging
Rendering the Result into Images [EG’04]
Composed Result
![Page 24: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/24.jpg)
Kristina Scherbaum [email protected]
Input at the age of 11
Photofit Picture Example
Possible appearances at the age of 17
![Page 25: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/25.jpg)
Kristina Scherbaum [email protected]
Aging in Images - Example
Picture (1999)
Ground truth pictures (2005)
Different prediction renderings
3D reconstruction and aging
![Page 26: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/26.jpg)
Kristina Scherbaum [email protected]
Linear vs. Non-Linear
Linear age progression perform linear regression (yields a function)
[ straight-forward least squares fit ] transform faces also along the gradient
Disadvantages … the gradient is constant [ linear function ] each face moves along
the same straight trajectory
![Page 27: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/27.jpg)
Kristina Scherbaum [email protected]
Linear vs. Non-Linear
comparison of age estimation error (in months) mean squared training and generalization errors
non-linear (RBF)
38.90 29.35
linear 67.66 62.87
non-linear (RBF)
32.68 18.12
linear 66.14 60.05
non-linear SVM regression behaves superior! generalization indicates: no overfitting
![Page 28: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/28.jpg)
Kristina Scherbaum [email protected]
Remember the Challenges
Are growth trajectories curved?Mean angle between start- and target-tangent
the trajectories are curved, not linear
10.3º 30.0º
Have different faces distinct trajectories?Mean angle of trajectories of different faces
15.7º 33.5º
the trajectories are different
![Page 29: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/29.jpg)
Kristina Scherbaum [email protected]
Conclusions
Results … aging involves non-linear components trajectories are distinct for different individuals linear systems are a reasonable approximation technique works without longitudinal data
But … more data would be helpful longitudinal data would allow for exact evaluation
![Page 31: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/31.jpg)
Kristina Scherbaum [email protected]
Representation of Faces - Face Spaces
arbitrary faces by linear combinations of examples
PCA to reduce dimensionality (yields coefficients)
![Page 32: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/32.jpg)
Kristina Scherbaum [email protected]
Main Idea … compute aging trajectories z(t) locally along gradient of the aging function f(x) and going through a start vector or face x0:
Aging Trajectories4
![Page 33: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/33.jpg)
Kristina Scherbaum [email protected]
Aging Information
extracted from the database of 200 adult face scans and new database of 238 face scans of teenagers
teenager overview
![Page 34: Prediction of Non-Linear Aging Trajectories of Faces K. Scherbaum, M. Sunkel, V. Blanz and H.-P. Seidel [ 2007/5/9, Eurographics 2007, Prague ]](https://reader035.vdocuments.net/reader035/viewer/2022062716/56649dc35503460f94ab5b98/html5/thumbnails/34.jpg)
Kristina Scherbaum [email protected]