IMAGE RETARGETING: SEAM CARVING ECE 533 Image Processing Project 12/20/2012 University of Wisconsin-Madison Shiwei Zhou

1

1. Introduction

Nowadays with the increasing number of display devices, web contents can be displayed with

various sizes for different devices. This imposes new demands on digital media which require

designers to create different alternatives and design different layouts for these devices [1].

Currently, page layout and text can be dynamically changed according to the size of the display

devices through some standards like HTML. However, it is still almost impossible to deform

images to fit for different layouts, due to its rigidity in size. Therefore, there is great need for

displaying images on various media like cell phones or PDAs without distortion.

Normal scaling does not work very well because content of the image have been distorted, which

makes the image less visually pleasing. Cropping also has certain drawbacks since some of the

important image contents could have been discarded. Hence, it is necessary to develop a more

effective resizing approach which considers the image contents instead of geometric constraints

[1]. Avidan and Shamir (2007) has proposed such an effective algorithm for image resizing that

tries to get rid of or add to regions of the image which are not content-aware. Figure 1.1

illustrates an example of different approaches to resize an image.

Original Image Normal Scaling

Cropping Seam Carving

Figure 1.1 Different Approaches for Image Resizing [2]

2

2. Approach

The essence of seam carving is to find the optimal seam that is least content-aware in the image.

For image of size n × m, a vertical seam is defined as a set of pixels of the form 1

,n

ii S i

where S is a mapping S: [1, … , n] → [1, … , m], such that 1 1S i S i for all i, i.e. all the

pixels along the path are 8-connected. Similarly, horizontal seam is defined as the set

1

,m

iS i i

such that 1 1S i S i for all i. The process on vertical seams and horizontal

seams are similar, hence to avoid redundancy, the following parts will mainly focus on vertical

seams.

To find a continuous path across the image that can go through areas of image that are less

content-aware and avoid major content of the image, it is necessary to search for seams with

minimal energy. The basic energy function commonly used for an image is defined as the total

gradient of the image: 1

I Ie I

x y

, which can be approximated for each pixel (i,j) by

, 1, , , 1 ,e i j x i j x i j x i j x i j . There are several possible image importance

measures, such as entropy, segmentation, Histogram of Gradients (HoG), or other norms of total

gradient [3]. For this project, I will use e1 as the energy function. e(i,j) is also known as the cost

of a single pixel at (i,j), hence the cost of a seam is defined as ,ke k S k for vertical seam

and ,ke S k k for horizontal seam. Therefore, given an energy function e, the optimal seam

is the one that minimizes the cost of the seam.

The optimal seam can be found using dynamic programming. Given e and a direction, say

vertical, we define the cumulative minimum energy M as , min ,end

S i j k iM i j e k S k

,

which is the minimum cost of the vertical seam starting from pixel (i,j) and going down to the

bottom of the image. Thus the minimum entry of the first row of M indicates the final pixel of

the optimal seam that we are searching for. In order to find the whole optimal seam, we can

simply trace back using the following dynamic programming algorithm:

1) The last row of M is equal to the last row of e, i.e. , ,M n j e n j

2) For 1 ≤ i ≤ n – 1 and 1 ≤ j ≤ m,

, , min 1, 1 , 1, , 1, 1M i j e i j M i j M i j M i j

3) For pixels at edges of the image, we can replace the above equation with

,1 ,1 min 1,1 , 1,2M i e i M i M i

or ,1 ,1 min 1, , 1,1 , 1,2M i e i M i j M i M i

3

Once we have found the optimal seam, we can either shrink the image or enlarge the image

without distorting the major content of image. For shrinking the image, we just need to search

the optimal seams and remove them from the image. For stretching the image, it is necessary to

first find the optimal seams that need to be removed, then duplicate them and insert at the right

side of those seams. Details about the implementation will be discussed in later sections.

3. Work Performed

For MATLAB implementation, the main function of the whole program is called ImageRetarget

and all other functions are called in this function. Firstly the energy function named energy

should be implemented. Here I calculate the total gradient using a function called gradient. Also

since the original image is RBG image, a function called rgb2gray is used to convert it into gray

image, so that the computation complexity is reduced. The following MATLAB codes show how

to implement the energy function. Figure 3.1 displays the gradient of an example image. Dark

areas mean low gradient, and bright areas mean high gradient, which represent relatively

important regions.

Function energy:

Figure 3.1 Energy/Gradient of the Image

function E = energy(img)

% this function calculates the energy/total gradient of the image

[fx fy] = gradient(rgb2gray(img));

E = abs(fx)+abs(fy);

4

The next step is to calculate the cumulative minimum energy M and find out the optimal seam.

Construct a new function called vertical, which takes in the image and the number of seams need

to remove or insert as input parameters. Then most part of the image retargeting implementation

will be in this function.

Shrinking Images

First consider the case of removing seams. For each iteration, the program needs to calculate the

updated energy e of the image, and derives M from e using the algorithm in Section 2. However,

even if we have the minimum cost of the best seam, it does not necessarily mean that we have

the seam. Therefore, it is important to store whether we have chosen j – 1, j, or j + 1 when

minimizing M(i,j) from the row below. To achieve this, I implement a matrix called Seammat

that stores the decision for each M(i,j). Notice that special attention should be paid to pixels at

the edge of the image. This can be illustrated in the following code fragments. To keep the

program concise, I put it into a function named findSeamVert. Figure 3.2 shows the visualization

of M in vertical direction, in which dark areas mean low cumulative energy cost and bright areas

mean high cumulative energy cost. From the figure we can see that the optimal vertical seam

might be at the right edge of the image since that area is darkest in general.

Function findSeamVert:

for i = 1:R-1 % for pixels at the first column M(R-i,1) = M(R-i,1)+min([M(R-i+1,1),M(R-i+1,2)]); if M(R-i+1,1) < M(R-i+1,2) Seammat(R-i,1) = 0; elseif M(R-i+1,1) > M(R-i+1,2) Seammat(R-i,1) = 1; end % for pixels at the last column M(R-i,C) = M(R-i,C)+min([M(R-i+1,C-1),M(R-i+1,C)]); if M(R-i+1,C-1) < M(R-i+1,C) Seammat(R-i,C) = -1; elseif M(R-i+1,C-1) > M(R-i+1,C) Seammat(R-i,C) = 0; end % for other pixels in between for j = 2:C-1 M(R-i,j) = M(R-i,j)+min([M(R-i+1,j-1),M(R-i+1,j),M(R-i+1,j+1)]); if min([M(R-i+1,j-1),M(R-i+1,j),M(R-i+1,j+1)]) == M(R-i+1,j-1) Seammat(R-i,j) = -1; elseif min([M(R-i+1,j-1),M(R-i+1,j),M(R-i+1,j+1)]) == M(R-i+1,j) Seammat(R-i,j) = 0; elseif min([M(R-i+1,j-1),M(R-i+1,j),M(R-i+1,j+1)]) == M(R-i+1,j+1) Seammat(R-i,j) = 1; end end end

5

Figure 3.2 Seam Costs of Entire Image in Vertical Direction

Given M, it is then easy to find the optimal seam. Following the algorithm mentioned in Section

2, we can find the minimal value of the top row of M and its location in the row. This location is

the first entry of the seam. Then we can trace back to find all pixels of the whole seam using

following code fragments. Figure 3.3 shows the first optimal vertical and horizontal seam found

using above algorithm. As expected, this seam is at the right/upper side of the image, which

means the above codes work properly.

Segments of function Vertical:

After finding the seam, it is necessary to remove it from the image. To achieve this, I

implemented a function called s_remove, which takes in the image, the seam, and the direction as

input parameters. The basic idea of removing seam is to create a mask in which only the pixels at

the seam are defined as false while other pixels are defined as true. Then initialize a new image

matrix of size that is one column less than the original image, and set the values of the matrix to

the corresponding pixel values of the image where mask shows true. As a result, the seam pixels

will be discarded. Also in order to implement with RGB images, the function called repmat is

used to duplicate the mask to three layers. However, due to the inside algorithm of MATLAB,

repmat can only be applied to horizontal seam removal, while for vertical seams, manual

implementation is needed, which can be shown as the following codes. Repeating the above

procedures shrinks the image to the desired sizes.

seamcost = min(M(1,:)); % find the minimum of cumulative energy seamloc = find(M(1,:)==seamcost); seam(1) = seamloc(1); % locate the entry of first pixel of best seam for l = 2:R seam(l) = seam(l-1)+Seammat(l-1,seam(l-1)); % search for the best seam end

6

Function s_remove:

Figure 3.3 Example of Vertical and Horizontal Optimal Seams

Stretching Images

Stretching image needs some more efforts because simply duplicating the best seams will result

in the same seam being copied multiple times, making the image visually unpleasant, as Figure

3.4 shows.

function newimg = s_remove(img,S,alpha) % this function removes a seam from the image

sz = size(img); if alpha == 1 % vertical seam remove mask = true(sz(1:2)); % create a mask of true values mask(sub2ind(sz(1:2),1:sz(1),S)) = false; % set the seam pixels to

false newimg = zeros(sz-[0 1 0]); % initialize new image tempmask = mask'; for j = 1:3 temprgb = newimg(:,:,j)'; imgrgb = img(:,:,j)'; temprgb(:) = imgrgb(tempmask); newimg(:,:,j) = temprgb'; end elseif alpha == 0 % horizontal seam remove mask = true(sz(1:2)); mask(sub2ind(sz(1:2),S,1:sz(2))) = false; mask = repmat(mask,[1 1 3]); % replicate the mask to RGB layers newimg = zeros(sz-[1 0 0]); newimg(:) = img(mask); end

7

Figure 3.4 Same Seam Found and Inserted Multiple Times [1]

The general solution to this problem is to find all the seams to be removed first, and then

duplicate them all together. For the coding part, it takes some effort to find the exact location of

each seam in the original image, since all seam locations found are relative to the corresponding

image that have been shrunk. To keep track of the locations of seams, I create a vector named

seamvec to store the absolute location of each seam that has already been removed, and sort it in

ascending order. For each newly found seam, if there are m seams to the left of it, we just need to

add m to its relative location to get the absolute location. For implementation, assume the length

of sorted seamvec is p, then subtract vector [1:p] from this sorted vector, and calculate the

number of entries that is smaller than the newly found seam location. Adding this number back

to the seam location will give us the real location of the seam in the original image. This can be

illustrated in the following code fragments, in which matrix seamnew is the real seam matrix.

Segments of function Vertical:

After getting the absolute locations of seams, we need to duplicate these seams to expand the

image. For inserting seams, I implement a function called s_insert, which takes in the image, the

seam, and the direction as input parameters. My implementation of inserting seams is a little

different from removing seams. Here I need to create a new image matrix which is one column

larger than the original image, and set values before and at the seam to the corresponding pixel

values of the original image. For pixels after the seam, just make them equal to the pixels that are

one column before them.

seamsort = sort(seamvec); % sort the removed seams in ascending order p = length(seamsort); if p == 0 num(k) = 0; elseif p > 0 seamtemp = seamsort-(1:p); num(k) = sum(seamtemp < seam(k,1)); % check how many seams before the

current seam are reomved end seamnew(k,:) = seam(k,:)+num(k); % get the real location of current seam seamvec = [seamvec seamnew(k,1)]; % store it in seamvec

8

Function s_insert:

To successfully duplicate the seams for my algorithm, it is important to start inserting seams

from right to left. Therefore, before calling the s_insert function, we need to sort the seams in

descending order as the following code fragments show.

Object Removal/Protection

One important application of seam carving is object removal/protection. The basic idea is to

choose a region that needs to be removed, and assign pixels in this region with very high

negative/positive costs. To implement the idea in MATLAB, I included an argument

weightsmask to the ImageRetarget function. weightsmask is a matrix the size of the original

image that allows user to specify large negative or large positive weights for parts of the image

function newimg = s_insert(img,S,alpha)

% this function removes a seam from the image

sz = size(img);

if alpha == 1 % insert vertical seam

newimg = zeros(sz+[0 1 0]);

for k = 1:sz(1)

for l = 1:S(k)

newimg(k,l,:) = img(k,l,:);

end

for l = S(k)+1:sz(2)+1

newimg(k,l,:) = img(k,l-1,:);

end

end

elseif alpha == 0 % insert horizontal seam

newimg = zeros(sz+[1 0 0]);

for k = 1:sz(2)

for l = 1:S(k)

newimg(l,k,:) = img(l,k,:);

end

for l = S(k)+1:sz(1)+1

newimg(l,k,:) = img(l-1,k,:);

end

end

end

seam_h = seamnew(:,1); seam_s = sort(seam_h,'descend'); % sort all seams in descending order by

their first element for k = 1:abs(delta) % this loop just gets back the whole seams in the

order above sl = find(seam_h == seam_s(k)); seam_n(k,:) = seamnew(sl,:); end

9

that is desired to eliminate or preserve during seam carving. It will replace the image gradient in

all locations for which weightsmask is not equal to zero.

One important thing to notice is that as the image changes sizes during seam carving,

weightsmask also need to change sizes by deleting/inserting the same seam from/into the mask.

To this point, I implemented a function mask_resize to resize the weightsmask.

Function mask_resize (for shrink):

Function mask_resize (for stretch):

4. Results

All the results in this project are computed automatically, and for some images, the results are

quite satisfactory. Seam carving works fine on images that don’t have complex structures.

However, when interesting objects span the entire image, or when the image gets too small, seam

carving still deforms important image comtents [4].

e_weight = e.*(weightsmask == 0) + weightsmask; % cover mask on the

gradient M = e_weight; % matrix of cumulative energy M

[mr,mc] = size(weightsmask);

masktemp_v = zeros(mr,mc-1); for k = 1:mr if S(k) == 1 masktemp_v(k,:) = weightsmask(k,2:end); elseif S(k) == mc masktemp_v(k,:) = weightsmask(k,1:end-1); elseif S(k) > 1 && S(k) < mc masktemp_v(k,:) = [weightsmask(k,1:S(k)-1),weightsmask(k,S(k)

+1:end)]; end end mask_new = masktemp_v;

masktemp_v2 = zeros(mr,mc+1); for k = 1:mr masktemp_v2(k,:) = [weightsmask(k,1:S(k)),weightsmask(k,S(k):end)]; end mask_new = masktemp_v2;

10

Success

Original Image

Shrink Stretch

Original Image

11

Shrink Stretch

Original Image

Shrink Stretch

12

Original Image

Shrink Stretch

13

Original Image

Shrink Stretch

Original Image

14

Weight Mask

Shrink (without mask)

Shrink (with mask)

Failure

Original Image

15

Shrink Stretch

Original Image

16

Shrink Stretch

5. Discussion

As a conclusion, seam carving is a content-aware image editing method that can resize images

without distorting the major content of images. However, it also has certain limits. It works very

well on certain kinds of images with less complex structures, while performs poorly on images

with structures that span the entire image. Therefore, for the latter kind of images, a weighted

mask is necessary to protect the important structures.

However, above algorithm might not be optimal. Since I simply remove or insert horizontal

seams first and then vertical seams, the removal or insertion of previous horizontal seams may

affect the cost of later vertical seams and vice versa. Therefore, this raises the question of what is

the correct order of seam carving. Horizontal first? Or alternate between the two? To explore this

question, Avidan and Shamir (2007) proposed an algorithm to search for the optimal order by

using dynamic programming. The basic idea of this algorithm is to set a matrix T of size (r + 1) ×

(c + 1), where r and c are numbers of rows and columns of original image. T(i,j) means the

lowest possible cost to remove i horizontal seams and j vertical seams from the image.

Therefore, T(0,0) = 0, and our objective is to find T(r,c) which is the last entry of this matrix.

Also three other matrices of the same size of T, namely X, Y, and I are created. X(i,j) tells the

lowest cost to remove a horizontal seam from source image of dimension (r – i) × (c – j), and

similarly Y(i,j) tells the lowest cost to remove a vertical seam from source image of dimension (r

– i) × (c – j). I is used to trace back the optimal sequence of operation, i.e. if I(r,c) = 1, then

17

previous optimal operation is vertical one, and if I(r,c) = 0, then previous optimal operation is

horizontal one. Then we can use the following dynamic programming to find out T(r,c).

1) 0,0 0T

2) , min 1, 1, , , 1 , 1T r c T r c X r c T r c Y r c

By this dynamic programming, we store a simple matrix I which indicates the sequence of the

two options of operation, vertical or horizontal. We can backtrack from T(r,c) to T(0,0) and

apply corresponding removal operations to the image [1].

References

[1] S. Avidan, A. Shamir, Seam Carving for Content-Aware Image Resizing, ACM SIGGRAPH,

2007.

[2] Wikipedia, Seam Carving, http://en.wikipedia.org/wiki/Seam_carving.

[3] R. David, Seam Carving Implementation, 2007.

[4] D. Simakov, Y. Caspi, E. Shechtman, M. Irani, Summarizing Visual Data Bidirectional

Similarity, CVPR, 2008.