by: david gelbendorf, hila ben-moshe supervisor : alon zvirin 19.10.2015

AUTOMATIC 3D FACE PRINTING

By:David Gelbendorf,

Hila Ben-MosheSupervisor: Alon Zvirin

19.10.2015

Face Printing?

- GIP sensor: Today we are capable to capture 3D images

- We are able to print in 3D, with the right printer

- Everyone has a passport photo on he’s I.D card

- Why not printing a face in 3D?

Face Printing!

Face Printing - Goal

First Stage: Developing a tool for processing a facial video

and printing a 3D face automatically Input: a 3D record of a frontal face Output: a 3D printed model of the face

Second Stage: Wrap the tool in a easy-to-use GUI

Face Printing – The challenge

- Printing a 3D object requires:- A good 3D image, which contains all face features (eyes, nose and mouth) and with high density


- Printing a 3D object requires:- Smoothness and fullness

- The 3D frames are not complete. Many pixels has NAN values and missing data (non-Continuity)

- Choosing bounding box


- Printing a 3D object requires:- Transformation of the 3D GIP file to a 3D printer standard format (for instance obj, stl)

Face Printing – Block Diagram

Recoding a video

Choosing Best

Frame

Processing frame (creating model)

Converting to STL

Model ready to

print

Face Printing – A good Image

- How to choose a good frame? - Solution: Use Viola Jones face

detection algorithm:- Use VJ for searching facial features- Use VJ for detecting movement of facial

features

Choosing Best Frame


Profile Face (no detections) True positives + False Positives (double detection of mouth and left eye)

Semi-Profile Face with low detection results

Choosing Best Frame

Detected face frame: 216Detected face frame: 84Detected face frame: 174


Double detection of mouth,no detection of left/right eye

No detection of pair of eyes Perfect match (one match for each detector)

Choosing Best Frame

Detected face frame: 78Detected face frame: 47Detected face frame: 111


Facial features detection: - Scanning each frame for a face, pair of

eyes, left eye, right eye, nose and mouth- Sorted by importance :

- 2: Face and pair of eyes:- 1: left eye, right eye, *nose- 0.5: Mouth- *0.75: if found more than one nose, there will be penalty

- *0.5: if found more than 2 pair of eyes- Grade: normalized to [0..1]

Choosing Best Frame


Movement Detection: - Examining difference of VJ detections in adjacent frames (previous and next frames) :- Pair of eyes, left eye, mouth, nose

- Calculating the distance of adjacent detections frames (Euclidean metric)

- Calculating a grade according to the results. A perfect grade (1.00) will be achieved only if perfect match is found between previous and next frames in all examined features.

Choosing Best Frame


Movement Detection: - Examining difference of VJ detections in adjacent

frames (previous and next frames) for 4 detectors: Pair of eyes, left eye, mouth, nose

- Calculating the distance of adjacent detections frames (Euclidean metric):- Let - Let - The delta distance between adjacent frames is

calculated as follows:

- The total movement grade is calculated according to d (bigger d results with lower grade)

Choosing Best Frame


- The final grade combined from both methods is calculated as:

- *VJDetecionGrade + (1-)*movementGrade- Testing different values has proven to be

most effective- The grading m

- is a parameter and therefore can be changed- Other grading methods could be added or

changed easily

- The chosen frame of the video would be the one with best score.

Choosing Best Frame


- Grades examples:

Choosing Best Frame

0.910 0 0.710.82

Face Printing – Image Processing

The chosen image is a raw 3d image, contains noise, holes and discontinuities.

Goal: automatic filtering of the image in order to create a 3D complete, continuous model

Choosing Best Frame Processing Frame


Challenge: choosing a right bounding box for the desired model (for instance, without background)

Solution: choosing the bounding box automatically with optimization



Bad Bounding Box (Noise is cut. Also, in the back of the head there is too much information)



Good Bounding Box:



Optimization of bounding box: Specially important for Z axis (depth) Iterative “exhaustive search” for optimal max

Z: Starting from a min size of max Z, and

progressing with max of Z axis Dividing the min – max range into levels,

increasing the max Z value in each iteration Calculating a grade for each max Z value Searching for minimum grade

Second option: run constrained optimization to find optimal min and max value for x, y, z (6 variables)Choosing Best Frame Processing Frame


Searching for minimum Weighted grade according to: Length of Contour 2D: 0.2 Length of Contour 3D: 0.2 Number of pixels: 0.3 Quadratic Shape of Contour: 0.7




-0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

maxZ

grad

e

num pixels

quadratic rms

2D curve length3D curve length

wighted grade


Filtering: 3D filter:

A chosen filter (median, average, raw, beltrami) Clean Margins Remove pixels far from the median of their

neighbors Remove pixels with few neighbors Choose K largest connectivity components

Filling Holes: cubic interpolation of each 2D connected

component



Problem: Dark eyebrows: risk for missing data for

building the model

Solution: Detect and fix holes in eyebrows:

Detect eyebrows area with Viola Jones algorithm (by detecting pair of eyes position).

If holes detected, try to fix by moderate filtering.



Fixing Eyebrows - Detection: Let (rectangle of pair of eyes position. Let (w,h) be the corresponding width and

height. The eyebrows position is a shifting and

enlargement transform as follows:

Missing pixels fraction of eyebrows area is calculated (on mask of valid pixels): If fraction > 8% than try to fix eyebrows.



Fixing Eyebrows – Examples (Detect): Example 1: eyebrows detection returns true negative

(0% missing pixels):

Example 2: eyebrows detection returns true positive (28% missing pixels):



Fixing Eyebrows –Repair: Trying to moderate image filtering

parameters: Changing threshold of minimal neighbors from 4

to 2. Iteratively compute the distance threshold

between a pixel and it’s median with is neighbors. Increasing threshold until fraction of missing data <8% or after reaching maximum threshold value.



Fixing Eyebrows – Examples (Detect and fix): Example 1: eyebrows detection returns true negative

(0% missing pixels):

Example 2: eyebrows detection returns true positive (28% missing pixels):

After fixing,missing pixelsfraction decreasesto 5.7%



Fixing Eyebrows – Examples: Example 3: Final Model with and without fixing



Fixing Eyebrows – drawbacks: If Viola Jones algorithm fails to detect

pair of eyes detection, eyebrows detection will fail as well.

Moderate filtering may result with a distorted model.



After bounding box optimization and filtering: The 3D model is prepared The Contour of the model is prepared


Face Printing – Writing to STL

Face model is ready, we can transform the model into 3D STL format which is a known standard for 3D models In the writing process, the 3D model’s points

are re-arranged in triangles and their corresponding normals.

Choosing Best Frame Processing Frame Writing STL file

Face Printing Application– Drawbacks

Automatic bounding box may result in a cut face

Automatic filtering process (which is not configured to each face individually) may result with a less accurate model, compared to a model which was filtered manually.

Run-time of the all automatic modules may take a couple of minutes

Face Printing – future improvements

Improving run-time of bottlenecks : Run-time for calculating grade for each frame Run-time for calculating bounding box Run-time for writing STL modules

Improving filtering by adjusting parameters for each frame (change some parameters to be dynamic)

Building a better model from several different frames

Face Printing - GUI

Face Printing - GUI

All option can be easily operated from the GUI: Choosing a video First Option:

Getting a 3D model to print automatically Second Option:

Edit parameters for each module:- Choose best frame- Choose bounding box (option for manual or

automatic bounding box)- Choose file format and save the model to print

Advanced menu to change several parameters

Bibliography

Viola Jones in Matlab: http://www.mathworks.com/help/vision/ref/

vision.cascadeobjectdetector-class.html

by: david gelbendorf, hila ben-moshe supervisor : alon zvirin 19.10.2015

Documents

d face printing

box face printing

d face automaticallyinput

d frames

high density face printing

easytouse gui face printing

d object

face features eyes