"leveraging computer vision and machine learning to power the visual commerce revolution,"...
TRANSCRIPT
Transforming
Beauty & Fashionusing Computer Vision & Machine Learning
Satya Mallick, Ph.D.Co-Founder, Sight Commerce
Nothing is as powerful as an idea whose time has come
Vision Graphics Learning
Beauty & Fashion
DISRUPT
Why?
Pour quoi?
Por quê?
Zergatik?
Per ché?
Warum?
क्यों ?
Почему?
Beauty & Fashion Shopping is Visually Inspired
Intel
• Speed : 3.50 GHz
• Cache : 20MB
• Number of Cores : 4
• Instruction Set : 64-bit
• Power : 140 W
Intel
• Speed : 3.50 GHz
• Cache : 20MB
• Number of Cores : 4
• Instruction Set : 64-bit
• Power : 140 W
Victoria’s Secret
Intel
• Speed : 3.50 GHz
• Cache : 20MB
• Number of Cores : 4
• Instruction Set : 64-bit
• Power : 140 W
Victoria’s Secret
Help create Beautiful Imagery
Communication is Visual
1.8B Photos / Day
Communication is Visual
Communication is Visual
Image data is abundant!
Extreme Personalization
of the shopping experience is Inevitable
Extreme Personalization
of the shopping experience is Inevitable
Image based Recommendations
CVML @Sight Commerce • Virtual Makeover
• Jewelry try-on
• Virtual Nails
• Virtual Clothing for ecommerce
• Product recommendation based on facial analysis
• Apparel recommendation based on photo
Photo Enhancement
• Lipstick • Lipgloss • Lipliner • Blush • Bronzer • Foundation • Concealer • Eyeliner • Macara • Eyeshadow • Contacts
Virtual Makeover
• Lipstick • Lipgloss • Lipliner • Blush • Bronzer • Foundation • Concealer • Eyeliner • Macara • Eyeshadow • Contacts
Virtual Makeover
User Uploads a Photo
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
Facial Contour Estimation Optimization problem based on detected landmarks and image gradient
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
Facial Contour Estimation Optimization problem based on detected landmarks and image gradient
Photo Quality Estimation 1. Noise 2. Illumination
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
Facial Contour Estimation Optimization problem based on detected landmarks and image gradient
Photo Quality Estimation 1. Noise 2. Illumination
Skin Detection Samples of skin taken from detected face region and a matting problem is set up.
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
Facial Contour Estimation Optimization problem based on detected landmarks and image gradient
Photo Quality Estimation 1. Noise 2. Illumination
Skin Detection Samples of skin taken from detected face region and a matting problem is set up.
Render Makeup1.Specular diffuse separation2.Illumination preserving rendering that
approximates BRDF of makeup / skin.
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
Facial Contour Estimation Optimization problem based on detected landmarks and image gradient
Photo Quality Estimation 1. Noise 2. Illumination
Skin Detection Samples of skin taken from detected face region and a matting problem is set up.
Render Makeup1.Specular diffuse separation2.Illumination preserving rendering that
approximates BRDF of makeup / skin.
Hair Segmentation1. Bayesian Matting2. Poisson Matting3. Spectral Matting.
User Uploads a Photo
Facial Feature Detection1. Active Appearance Model2. Supervised Descent Method3. Deep Learning based
Head Pose Estimation Facial features used to estimate pose relative to a canonical 3D shape
Facial Contour Estimation Optimization problem based on detected landmarks and image gradient
Photo Quality Estimation 1. Noise 2. Illumination
Skin Detection Samples of skin taken from detected face region and a matting problem is set up.
Render Makeup1.Specular diffuse separation2.Illumination preserving rendering that
approximates BRDF of makeup / skin.
Hair Segmentation1. Bayesian Matting2. Poisson Matting3. Spectral Matting.
Hair coloring Matching color histograms in the PCA space
Automatic Facial Feature Detection
Interactive Alpha Matting
Interactive Alpha Matting
Interactive Alpha Matting
Virtual Hairstyle
Virtual Makeover for E-Commerce
Product images are not enough!
Compelling Imagery Sells Products
Create a Look
Create a look
10 models x 100 lip products x 100 eye products x 10 blush
1,000,00010 models x 100 lip products x 100 eye products x 10 blush
1,000,00010 models x 100 lip products x 100 eye products x 10 blush
Sight Commerce Works
Beyond Makeup
Virtual Nails
Virtual Sunglasses
Virtual Jewelry
Virtual Clothing
Virtual Accessories
Image Based Recommendation
“35 percent of product sales result from recommendations” — Amazon
Photo Analysis + Shopping Data =
Extremely Personalized Recommendation
User Photo Analysis• Skin Tone • Eye Color • Lip Color • Hair Color • Ethnicity • Face Shape • Wearing Glasses ?
Color & Pattern based Apparel Recommendation
Wish List Face Processor
Landmark detector Skin detection
Color MeasurementInexpensive
Pattern Classification
Sight Commerce
Color & Pattern based Apparel Recommendation