forward vs. inverse warping - computer sciencelazebnik/research/fall08/lec08_faces.pdf · how do we...

Review: Image warping and morphing• Forward vs. inverse warping

f(x,y) g(x’,y’)

Forward warping

Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image

x x’

T(x,y)

Q: what if pixel lands “between” two pixels?

y y’

Slide credit: S. Seitz

f(x,y) g(x’,y’)

Forward warping

Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image

x x’

T(x,y)

Q: what if pixel lands “between” two pixels?

y y’

A: distribute color among neighboring pixels (x’,y’)– Known as “splatting”– Check out griddata in Matlab Slide credit: S. Seitz

f(x,y) g(x’,y’)xy

Inverse warping

Get each pixel g(x’,y’) from its corresponding location (x,y) = T-1(x’,y’) in the first image

x x’

Q: what if pixel comes from “between” two pixels?

y’T-1(x,y)


f(x,y) g(x’,y’)xy

Inverse warping

Get each pixel g(x’,y’) from its corresponding location (x,y) = T-1(x’,y’) in the first image

x x’

T-1(x,y)

Q: what if pixel comes from “between” two pixels?

y’

A: Interpolate color value from neighbors– nearest neighbor, bilinear, Gaussian, bicubic– Check out interp2 in Matlab Slide credit: S. Seitz

Forward vs. inverse warpingQ: which is better?

A: usually inverse—eliminates holes• however, it requires an invertible warp function—not always possible...


Review: Image morphing

Michael Jackson’s “Black or White” (1991)

How do we create a morphing sequence?

1. Define corresponding control points on both images2. For each intermediate image:

1. Create an intermediate shape (by interpolation)2. Warp both images towards it3. Cross-dissolve the colors between the newly warped

images

morphing

Morphing example

image #1 image #2

warp warp

Morphing example

Fun with faces

How do we average faces?

http://www2.imm.dtu.dk/~aam/datasets/datasets.html

Sir Francis Galton: Composite Portraits

Composite portraits showing "features common among men convicted of crimes of violence," by Francis Galton, 1885

Sir Francis Galton: Composite portraits

“The Jewish Type”

Jason Salavon: “The Late Night Triad”

http://www.salavon.com/

Jason Salavon: “100 Special Moments”

Jason Salavon: “Every Playboy Centerfold”

Is there a better way to average faces?

Average Face

1. Warp to mean shape2. Average pixels

http://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/ Slide credit: A. Efros

Representing the shape of a face

Slide credit: A. Efros

Representing the shape of a face

Vector of normalized landmark coordinates (or inter-landmark distances)


Subpopulation means

• 250 control points per face manually specified

Average male(32 faces)

Average female(64 faces)

http://www.beautycheck.de

Manipulating faces

• How can we make a face look more male/female, young/old, happy/sad, etc.?1. Obtain the two prototypes spanning the desired axis

(gender, age, expression, etc.) and find the difference vector between them

2. Add scaled versions of this vector to a given image

Prototype 1

Prototype 2Current face

D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995

Changing gender

Deform shape and/or color of an input face in the direction of “more female”

original shape

color both



Enhancing gender

more same original androgynous more opposite



Changing age

Face becomes “rounder” and “more textured” and “grayer” original shape

color both



Which face is more attractive?



attractive0.5(attractive + average)


0.5(adult+child) adult

Effect of skin texture

average shape + unattractive texture

average shape + average texture

Digital beautification – Video http://www.youtube.com/watch?v=iYhCn0jf46U

Data-Driven Enhancement of Facial Attractiveness

T. Leyvand, D. Cohen-Or, G. Dror, and D. Lischinski

SIGGRAPH 2008

System Overview

Feature extraction• Active shape model (Cootes et al. 1995)

• Represents distribution of shape (facial features represented by splines) and appearance (pattern of local gradients)

• Initialized from manually labeled training data• At test time, need good initial estimate of face location, scale

and orientation – use a face detector• 84 keypoints are converted into a planar graph of 234 inter-

point distances

Average and scatterfrom training set

Results of ASM search

Graph of distances

Regression of facial attractiveness model• Input: training set of distance vectors of faces rated

on a scale of 1 to 7 by human subjects (Eisenthal et al. 2006)• 92 female and 33 male photographs

• Support vector regression: learn smooth function f(v) to estimate attractiveness ranking

Beautification• Problem: given a vector v and estimated attractiveness

rating f(v), find nearby vector v’ such that f(v’) > f(v)• Method 1 (KNN): move the face vector in the direction of the

beauty-weighted average of K nearest neighbors of that face:

• Method 2 (SVR): perform gradient ascent of f starting at v• Results

• NN-based beautification worked better on male faces, but SVR-based beautification worked better on female faces

• On average, estimated beauty score was increased by 30%

,beauty score of the ith neighbor

Choosing K in KNN beautification

Comparison of KNN and SVR beautification

Additional details• Distance embedding

• Given a vector of distances v’ = {dij}, find a set of points {qi} that conform to these distances:

• Avoiding pupil distortion• Add post-process to enforce similarity transform for feature

points on the eyes

• Warping: Multilevel Free-Form Deformation (Lee et al. 1996)• Uses B-splines, avoids fold-overs

Interactive system: Video


originalbeautified


original beautified


beautified original


original beautified


beautified original

Results of user study• 92 face pairs (47 female, 45 male)• 68 raters, males and females aged between 25 and 40

• “For all 47 portraits of females, the beautified versions were chosen as more attractive by most raters, and in half of the cases the beautified versions were preferred by more than 80 percent of the raters. As for the male portraits, 69 percent of the beautified versions were chosen as more attractive.”

Application: Construction of facial collages

initial collage Poisson blending beautification

Face Swapping: Automatically Replacing Faces in Photographs

D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, S. K. Nayar

SIGGRAPH 2008

Outline• Offline: create face library• Given a new query:

• Detect faces• For each face:

– Retrieve candidate replacement faces from the library– Apply recoloring and relighting– Perform face replacement– Rank replacement results

Creating a face library• Download images from the Internet and process them

using a commercial face detector• Approximately 33,000 faces found• Detector returns face pose (yaw, pitch, roll) and six fiducial points• Bin faces according to yaw and pitch angles, align to a common

coordinate system using fiducial points

Candidate selection• Detect faces in new image, estimate their pose

angles and fiducial points• Matching criteria:

• Pose: yaw and pitch angles• Resolution: inter-eye distance • Blur: histogram of gradient magnitudes in the eye region• Lighting: approximate the face by a Lambertian cylinder

under orthographic projection, estimate face albedos and coefficients for an orthonormal illumination basis

• Seam signature: ribbon of pixels within a small radius of the replacement boundary (resampled, intensity normalized)

• Retrieval• SQL query for pose, resolution, and albedo• Further prune using blur and lighting distance• Select top 50 final candidates using seam distance=> Less than 1 sec to generate list of candidate replacements

Candidate selection

• Approximate face images:

• Relit replacement:

• After relighting, match the color histograms of the two faces

Relighting and recoloring

albedo

illumination coefficient

illumination basis image

Relighting and recoloring

Face replacement: Final steps• Rank replacements by color difference along the

overlap region• Perform feathering along the overlap region

Face replacement results

Face de-identification

More results

Improving family portraits

Limitations

User study• 50 face images, 12 subjects, 5 seconds per image

Aside: Detecting composites of faces

M.K. Johnson and H. Farid, “Exposing Digital Forgeries Through SpecularHighlights on the Eye,” 9th International Workshop on Information Hiding, 2007

More fun with faces

Example-Based Cosmetic Transfer

W.-S. Tong, C.-K. Tang, M. Brown, Y.-Q. XuPacific Graphics 2007

More fun with faces

A Conditional Random Field for Automatic Photo EditingM. Brand, P. Pletscher

CVPR 2008

More fun with faces

Expressive Expression Mapping with Ratio ImagesZ. Liu, Y. Shan, Z. Zhang

SIGGRAPH 2001

Original image Expression transfer by warping

Proposed method

forward vs. inverse warping - computer sciencelazebnik/research/fall08/lec08_faces.pdf · how do we...

Documents