Creating Better Thumbnails

Chris Waclawik

Project Motivation

• Thumbnails used to quickly select a specific a specific image from a set (when lacking appropriate metadata), but…

• Larger sets require smaller, less detailed thumbnails.

Thumbnail effectivenessCan you find the picture with the dog in it?

Thumbnail effectivenessCan you find the picture with the dog in it?

Thumbnail effectivenessCan you find the picture with the dog in it?

Problem Statement

• How can thumbnails be made more effective?

Show only the salient parts

Two subproblems

• How do we determine the salient portions of the image?

• Knowing the above, how should we crop the image?

Determining Image Saliency

• F. Liu, M. Gleicher, Region enhanced scale-invariant saliency detection, in: Proceedings of IEEE International Conference on Multimedia & Expo (ICME), 2006, pp. 1477-1480.

• Two-part method:– Scale-invariance makes this method suitable for

determining salient features of many sizes.– Region-enhancement aids in making logical crops

later on.

Scale-invariant Saliency

• Image transformed to perceptually uniform color space (Lu*v*).– Difference in color values proportional to perceived difference.

• Gaussian pyramid created and a contrast map is calculated for each level.– The contrast of each pixel is the sum of its distances from its

neighbors (in Lu*v* space).– Pixels closer to the center are given more weight.

• The contrast maps are summed to find scale invariant saliency.

• Algorithm implemented in Matlab.

Region-enhanced Saliency

• Regions extracted from the image using any preferred method.– Mean shift method used in paper (and in this

project).– C++ code from EDISON, Matlab wrapper function

from shawnlankton.com.• Scale-invariant saliency is averaged over each

region.

Saliency-based Cropping

• B. Suh, H. Ling, B. B. Bederson, and D. W. Jacobs, "Automatic Thumbnail Cropping and Its Effectiveness," in 16th Annual ACM Symposium on User Interface Software and Technology. 2003, pp. 95-104.

• What is the smallest crop that exceeds a specified saliency threshold (proportion of the total image saliency)? – Can be calculated by brute force, but inefficient

A greedy approach

• User specifies a target output size and saliency threshold.

• Select the most salient region.• Until we reach our threshold:

– Expand selection to include next most salient region.– Adjust selection to match aspect ratio of output size.

• Not guaranteed optimal crop– But much faster– Guaranteed to include whole regions

Demonstration

• Bounding box in red, correct aspect ratio in green

Determining Saliency Threshold Automatically

• Find the point of diminishing returns: when does increasing the crop area not become worth it?

• One simple method: select the image coming after the biggest increase in saliency.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Saliency vs. Area

Saliency (percent of total)

Are

a (p

erce

nt o

f to

tal)

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Results

Original Thumbnail

Further possible improvements

• Usability testing– Test users on how quickly they can locate a

specified image in a set given a table of thumbnails.

• Tweak region parameters• Saliency threshold selection

– Try out more sophisticated metrics• Basic feature recognition

– Face detection