assessing product image quality for online shopping

eBay Confidential

Assessing product image quality for online shopping

Anjan Goswami, Sung H. Chung, Naren Chittar and Atiq Islam

eBay Confidential

Outline

• Motivation• Definition • Features• Experiments• Image quality models.• What’s next?

eBay Confidential

Motivation

eBay Confidential

Motivation

• Better Aesthetics.• Better customer experience?• Better conversion?• Better branding?• Better looking merchandising pages.

eBay Confidential

Applications

• Ranking.• Improve browsing experience.• Guidelines for sellers.• Merchandising.• Many more.

eBay Confidential

What is product image quality?• We care mainly about product images.• Product images have specific characteristics.

eBay Confidential

Computing image quality

• Machine learning problem.• Factors from images to construct feature

vectors.• Label the data points as one of the classes.• Build a classifier to get the class probabilities.• Alternatively, make a regression model from

human judgment data.

eBay Confidential

What factors? How do we compute?

• Size factors such as area, aspect ratio.• Image attributes such as brightness,

saturation, colorfulness, contrast, dynamic range.

• Factors based on background and foreground segmentation.

eBay Confidential

BrightnessAverage grayscale intensity of the image.

0.299*red+0.587*green+0.114*blue; 0.18

0.58

eBay Confidential

Colorfulness

• Difference between a color against gray. • Many empirical notions.• We use an empirical expression designed in

Natural Color Space. (NCS)• This space has a concept of rg and yb

coordinates.• stdev (rg,yb)+ 0.3 mean (rg,yb)• We compute this globally and for foreground.

eBay Confidential

ExampleGlobal FG20 8

60 64

eBay Confidential

Saturation

• Many notions and empirical formula.• Average of (max – min) in RGB space. (Used

mainly in photography)

eBay Confidential

Example of SaturationScore86

255

eBay Confidential

Dynamic range

• Variants of expressions in photography literature and in computer vision.

• We are using a simpler definition used in photography based on range of gray scale intensity.

eBay Confidential

Example of Dynamic RangeScore32

100

eBay Confidential

Contrast

• Michelson contrast : Range /(max + min) in a color space.

• RMS contrast : stdev of spatial intensities.

eBay Confidential

Example of Michelson ContrastScore25

99

eBay Confidential

Segmentation

• Grabcut with a heuristic for automated segmentation.

eBay Confidential

Segmentation

eBay Confidential

Background and foreground area ratio

• Use segmented image.• An approximation is used by using ratio of

pixels in the foreground and in the background.

eBay Confidential

ExampleScore12

83

eBay Confidential

background and foreground differences.

• Brightness L1 distance.• Contrast L2 distance.

eBay Confidential

Properties of background

• stdev of lightness (distance from white in RGB.)

• Mean of lightness. (RGB)

• A score on uniformity of background intensity that approximates texture properties.

eBay Confidential

An example

Image Colorfulness

RMS Contrast

Brightness diff

Area ratio

Stdev of lightness of background

Uniformity Whiteness of background

Colorfulness of foreground

38.85

0.30 142.33

0.75 4.22 2.63 100 35.76

44.77

0.086 28.6808

0.66 27.27 16.37 0 39.73

eBay Confidential

Data Collection

• Crowdsourcing (52K images)• Professional judgments. (6K images)

eBay Confidential

Crowdsourcing

• Has its own challenges.• Require thoughts for framing questions.• Require thoughts for conducting the

experiment.• Cheap labelers can attempt cheating.• Classifier result can be different based on

voting techniques used to find the label.• More judgments are better.

eBay Confidential

Image Quality Classes

• Good• Fair• Poor

eBay Confidential

Professional Quality Images

• Mostly white, light or uniform background.• Image is free from compression artifact such

as blurring.• Professionally photographed in proper lighting

condition .• Subject has a reasonable size and is in focus.• Example of such images can be seen in

branded retail websites.

eBay Confidential

Professional Quality

eBay Confidential

Poor images

• Poor or dark background• Can have incomprehensible texture.• Subject small.• Subject unclear.• Bad aspect ratio.• Poor resolution and photography.

eBay Confidential

Poor Images

eBay Confidential

Fair images

• These are images that are not poor. However, they are not as clean as professional looking photos. (Add examples)

eBay Confidential

How do we develop the model

• Multi-class classification with two data sources.

• Direct Regression with crowd-sourced data.• Used Gradient boosted tree.

eBay Confidential

Factor importance in Quality Classifier (Classifier)

• Background lightness • Brightness • Aspect ratio • Dynamic range • Background foreground area ratio• Michelson contrast

eBay Confidential

Error Rates

• Poor misclassification 10%• Fair misclassification 50% [BAD]• Good misclassification 7%

• However, our training data is so far not perfect.

eBay Confidential

Quality Score for Classification

• The quality score is the expected average of the class weights using the class probabilities.

• Currently class weights are simple linear function that maps poor, fair and good to 1,2,3.

eBay Confidential

Quality Score201

244

102

119

eBay Confidential

Quality Score 177

137

153

eBay Confidential

Quality Score122

102

105

eBay Confidential

Factor Importance in Regression

• Background lightness• Saturation• Aspect ratio• RMS contrast• Colorfulness

eBay Confidential

Comparison of ModelsImage Classifier Regression

98 86

100 57

231 235

133 96

eBay Confidential

What’s Next (Semantics)

• Object recognition/relevance.

• Text/watermark detection.

• Human model detection.

>

eBay Confidential

Q & A

Thanks!Ideas & questions?