CS4243 Project: Face Morphing 7

Caitlin, Michael, Wai Tsun, Wei Qi

Objectives• Achieve a seamless transition between faces

regardless of face shape• Features should be aligned while being warped

Requirements• Landmark points should be marked rigorously

(each important feature should have one)• Images can be frontal/non-frontal

Requirements• Landmark points should be marked rigorously

(each important feature should have one)• Images can be frontal/non-frontal

Algorithm• 1. Landmark points/tesselation• 2. Piecewise warp two-way• 3. Color interpolation• 4. Remapping using linear interpolation

Landmark points• Marked manually• Tesselate using Delaunay triangulation• To facillate more control of transformation

Piecewise warp• For each triangle:• Interpolate src and dest points according to ratio

t: in_between. Transform source image by doing a transformation from src to in_between. Transform source image by doing a transformation from in_between to dest.

• Helps to align features compared to only warping one way

Color interpolation• Blend the two intermediate images according to

ratio t.• Interpolation function is quadratic... so that it will

stay at the source face for a longer period of time.

• But the interpolation function can be adjusted for other effects

Remapping• Backward mapping using linear interpolation to

determine final color• Because coordinates may not be integer after

transform

Choice of transformation

function• At first, tried Affine: features don’t align well• Polynomial: overfitting• Piecewise polynomial: couldn’t get it to work• Piecewise thin-plate spline: works well!

Thin-plate spline• Modeled after thin-metal plate• So should be a good model for face• Borrowed Python snippet and verified using

Bookstein ’89 “Principal Warps: Thin-Plate Splines and the Decomposition of Deformations”

Sample models• Only compare two in the slides; more in video

Landmark points• Only on face; can be improved

Delaunay triangulation• Add corners so that whole image is transformed;

Global Affine• Features not aligned! Background warped

Global Polynomial• Features aligned, but strange deformation;

Background warped• Quite good for global though

Global Thin-Plate Spline

• Similar to global polynomial;

Piecewise Polynomial• Oops!

Piecewise Thin-plate Spline (Non-Frontal)

• Breaks…

Piecewise Thin-plate Spline (Frontal)

• Looks seamless! And background doesn’t warp• Didn’t put landmark points on garb…

Videos• More faces morphed in the videos:

o Affine: http://www.youtube.com/watch?v=IxhHRjTqPpUo Poly: http://www.youtube.com/watch?v=SkBH9mDVC80o TPS: http://www.youtube.com/watch?v=nC_6DDG9a3Uo Piecewise TPS(non-frontal):

http://www.youtube.com/watch?v=ZRmIsB-gpsco Piecewise TPS (frontal): http://

www.youtube.com/watch?v=SMbzd7FmAFY

Met Objectives?• Achieve a seamless transition between faces

regardless of face shape• Features should be aligned while being warped;

but blending of hair etc could be better

Met Requirements• Landmark points should be marked rigorously (each

important feature should have one); should mark non-face features as well

• Images can be frontal/non-frontal

Credits• Faces: http://pics.psych.stir.ac.uk/2D_face_sets.htm• Landmark points marking: from lecturer• Transformation matrices:

o Affine: using OpenCV library (Geometric Transforms)o Polynomial: self-implemented using numpy to multiply matrices, because the

one from scikit-image didn’t work properlyo Piecewise thin-plate spline:

• Delaunay triangulation done using scipy.spatial library• Thin-plate spline adapted from snippet by Zachary Pincus, and verified

from original paper by Bookstein ’89 “Principal Warps: Thin-Plate Splines and the Decomposition of Deformations”

• Transformation:o remapping done using OpenCV library (Geometric Transforms)

• Color interpolation:o OpenCV’s AddWeighted