exchanging faces in images siggraph ’04 blanz v., scherbaum k., vetter t., seidel hp. speaker:...
Post on 19-Dec-2015
215 views
TRANSCRIPT
![Page 1: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/1.jpg)
Exchanging Faces in Images
SIGGRAPH ’04Blanz V., Scherbaum K., Vetter T., Seidel
HP.Speaker: Alvin
Date: 21 July 2004
![Page 2: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/2.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 2
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 3: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/3.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 3
Introduction
Pasting somebody’s face into an existing image.A novel type of image manipulation:
Always needs pairs of images with the same viewpoint and the same illumination.The system only need one image, and can across large differences in viewpoint and illumination.
![Page 4: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/4.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 4
Introduction (cont.)
Manual interaction:Click on a set of about 7 feature points.Mark the hairline in the target image.
ExampleTwo applications:
Virtual try-on for hairstylesFace recognition
![Page 5: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/5.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 5
Previous Works
![Page 6: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/6.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 6
Previous Works (cont.)
![Page 7: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/7.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 7
Previous Works (cont.)
![Page 8: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/8.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 8
Previous Works (cont.)
![Page 9: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/9.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 9
Previous Works (cont.)
![Page 10: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/10.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 10
Previous Works (cont.)
![Page 11: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/11.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 11
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 12: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/12.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 12
Morphable Models
A vector space of 3D shapes and textures.Derived from 200 texture Cyberware (TM) laser scans.
100 male and 100 female.In a cylindrical representation with radii r(h, Φ) of surface points
512 equally-spaced angles Φ.512 equally-spaced vertical steps h.
![Page 13: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/13.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 13
Morphable Models (cont.)
Dense correspondence is computed automatically with an algorithm derived from optical flow.
![Page 14: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/14.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 14
Morphable Models (cont.)
After performing a PCA
m = 149
![Page 15: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/15.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 15
Fitting
![Page 16: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/16.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 16
Light Direction And Intensity Estimation
![Page 17: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/17.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 17
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 18: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/18.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 18
EstimationAll parameters are estimated simultaneously in an analysis-by-synthesis loop.
All scene parameters are recovered automatically, starting from a frontal pose in the center of the image, and at frontal illumination.
![Page 19: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/19.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 19
Estimation (cont.)
Cost Function
![Page 20: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/20.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 20
Estimation (cont.)
The optimization is performed with a Stochastic Newton Algorithm.The linear combination of texture Ti cannot reproduce all local characteristics of the novel faces.Extract the texture by an illumination-corrected texture extraction method.
![Page 21: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/21.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 21
References
A morphable model for the synthesis of 3D faces. SIGGRAPH’99, pp. 187–194.Face recognition based on fitting a 3D morphable model. IEEE Trans. on Pattern Analysis and Machine Intell. 25, 9 (2003), 1063– 1074.
![Page 22: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/22.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 22
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 23: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/23.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 23
Exchanging FacesBoth 3D shapes are aligned to each other in 3D with 3D Absolute Orientation Algorithm.Both textures have similar illumination.
Illumination-corrected Texture Extraction Algorithm.
Render the face that was reconstructed from the source image with the rendering parameters that were estimated from the target image.
![Page 24: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/24.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 24
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 25: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/25.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 25
Compositing
![Page 26: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/26.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 26
Background LayerThe scene of target image, and the original person’s face, hair and body.The novel face may be smaller than the original.
Solved by a background continuation method.
Based on a reflection of pixels beyond the original contour into the face area.
![Page 27: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/27.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 27
Face LayerThe silhouette of this region:
Occluding contours.Boundaries of hair regions that occlude the skin.Mesh boundaries at the neck and the forehead.
Skin may be partly covered by hair. This hair would be mapped on the face as a texture.
![Page 28: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/28.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 28
Hair Layer
Drawn in front of the face.Can be used for all faces.Automated classification of pixels into skin and hair is a difficult task.Manually define alpha values for opacity.
![Page 29: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/29.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 29
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 30: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/30.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 30
ApplicationsCurrent systems are restricted to frontal view of faces.
![Page 31: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/31.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 31
Applications (cont.)
![Page 32: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/32.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 32
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 33: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/33.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 33
Results
![Page 34: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/34.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 34
Results (cont.)
![Page 35: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/35.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 35
Results (cont.)
![Page 36: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/36.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 36
OutlineIntroductionMorphable ModelsEstimationExchanging FacesCompositingApplicationResultsConclusions
![Page 37: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/37.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 37
Conclusions
A novel way of processing images on a high level.Only needs simple manual processing steps.For a wide range of applications.Transferring technology from CG to CV.Combines the benefit of image-based method with the versatility of 3D graphics.
![Page 38: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/38.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 38
Future Works
Fully automated:Detecting facial features.Hair Segmentation.
Exchange faces in video sequences.
Tracking head motion.
![Page 39: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/39.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 39
Thank you for your patience
![Page 40: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/40.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 40
Example
![Page 41: Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004](https://reader030.vdocuments.net/reader030/viewer/2022032800/56649d355503460f94a0d4fb/html5/thumbnails/41.jpg)
Alivn/GAME Lab./CSIE/NDHU
Exchanging Faces in Images 41
Feature Points