assessing product image quality for online shopping
TRANSCRIPT
eBay Confidential
Assessing product image quality for online shopping
Anjan Goswami, Sung H. Chung, Naren Chittar and Atiq Islam
eBay Confidential
Outline
• Motivation• Definition • Features• Experiments• Image quality models.• What’s next?
eBay Confidential
Motivation
eBay Confidential
Motivation
• Better Aesthetics.• Better customer experience?• Better conversion?• Better branding?• Better looking merchandising pages.
eBay Confidential
Applications
• Ranking.• Improve browsing experience.• Guidelines for sellers.• Merchandising.• Many more.
eBay Confidential
What is product image quality?• We care mainly about product images.• Product images have specific characteristics.
eBay Confidential
Computing image quality
• Machine learning problem.• Factors from images to construct feature
vectors.• Label the data points as one of the classes.• Build a classifier to get the class probabilities.• Alternatively, make a regression model from
human judgment data.
eBay Confidential
What factors? How do we compute?
• Size factors such as area, aspect ratio.• Image attributes such as brightness,
saturation, colorfulness, contrast, dynamic range.
• Factors based on background and foreground segmentation.
eBay Confidential
BrightnessAverage grayscale intensity of the image.
0.299*red+0.587*green+0.114*blue; 0.18
0.58
eBay Confidential
Colorfulness
• Difference between a color against gray. • Many empirical notions.• We use an empirical expression designed in
Natural Color Space. (NCS)• This space has a concept of rg and yb
coordinates.• stdev (rg,yb)+ 0.3 mean (rg,yb)• We compute this globally and for foreground.
eBay Confidential
ExampleGlobal FG20 8
60 64
eBay Confidential
Saturation
• Many notions and empirical formula.• Average of (max – min) in RGB space. (Used
mainly in photography)
eBay Confidential
Example of SaturationScore86
255
eBay Confidential
Dynamic range
• Variants of expressions in photography literature and in computer vision.
• We are using a simpler definition used in photography based on range of gray scale intensity.
eBay Confidential
Example of Dynamic RangeScore32
100
eBay Confidential
Contrast
• Michelson contrast : Range /(max + min) in a color space.
• RMS contrast : stdev of spatial intensities.
eBay Confidential
Example of Michelson ContrastScore25
99
eBay Confidential
Segmentation
• Grabcut with a heuristic for automated segmentation.
eBay Confidential
Segmentation
eBay Confidential
Background and foreground area ratio
• Use segmented image.• An approximation is used by using ratio of
pixels in the foreground and in the background.
eBay Confidential
ExampleScore12
83
eBay Confidential
background and foreground differences.
• Brightness L1 distance.• Contrast L2 distance.
eBay Confidential
Properties of background
• stdev of lightness (distance from white in RGB.)
• Mean of lightness. (RGB)
• A score on uniformity of background intensity that approximates texture properties.
eBay Confidential
An example
Image Colorfulness
RMS Contrast
Brightness diff
Area ratio
Stdev of lightness of background
Uniformity Whiteness of background
Colorfulness of foreground
38.85
0.30 142.33
0.75 4.22 2.63 100 35.76
44.77
0.086 28.6808
0.66 27.27 16.37 0 39.73
eBay Confidential
Data Collection
• Crowdsourcing (52K images)• Professional judgments. (6K images)
eBay Confidential
Crowdsourcing
• Has its own challenges.• Require thoughts for framing questions.• Require thoughts for conducting the
experiment.• Cheap labelers can attempt cheating.• Classifier result can be different based on
voting techniques used to find the label.• More judgments are better.
eBay Confidential
Image Quality Classes
• Good• Fair• Poor
eBay Confidential
Professional Quality Images
• Mostly white, light or uniform background.• Image is free from compression artifact such
as blurring.• Professionally photographed in proper lighting
condition .• Subject has a reasonable size and is in focus.• Example of such images can be seen in
branded retail websites.
eBay Confidential
Professional Quality
eBay Confidential
Poor images
• Poor or dark background• Can have incomprehensible texture.• Subject small.• Subject unclear.• Bad aspect ratio.• Poor resolution and photography.
eBay Confidential
Poor Images
eBay Confidential
Fair images
• These are images that are not poor. However, they are not as clean as professional looking photos. (Add examples)
eBay Confidential
How do we develop the model
• Multi-class classification with two data sources.
• Direct Regression with crowd-sourced data.• Used Gradient boosted tree.
eBay Confidential
Factor importance in Quality Classifier (Classifier)
• Background lightness • Brightness • Aspect ratio • Dynamic range • Background foreground area ratio• Michelson contrast
eBay Confidential
Error Rates
• Poor misclassification 10%• Fair misclassification 50% [BAD]• Good misclassification 7%
• However, our training data is so far not perfect.
eBay Confidential
Quality Score for Classification
• The quality score is the expected average of the class weights using the class probabilities.
• Currently class weights are simple linear function that maps poor, fair and good to 1,2,3.
eBay Confidential
Quality Score201
244
102
119
eBay Confidential
Quality Score 177
137
153
eBay Confidential
Quality Score122
102
105
eBay Confidential
Factor Importance in Regression
• Background lightness• Saturation• Aspect ratio• RMS contrast• Colorfulness
eBay Confidential
Comparison of ModelsImage Classifier Regression
98 86
100 57
231 235
133 96
eBay Confidential
What’s Next (Semantics)
• Object recognition/relevance.
• Text/watermark detection.
• Human model detection.
>
eBay Confidential
Q & A
Thanks!Ideas & questions?