# seam carving

Post on 16-Jan-2016

62 views

Embed Size (px)

DESCRIPTION

Seam Carving. Project 1a due at midnight tonight. Project 1b goes out today. We’ll cover seam carving today. Image resizing. resampling. cropping. content-aware resizing. Seam carving: idea. Cropping removes pixels from the image boundary. We want to remove only “unimportant” pixels. - PowerPoint PPT PresentationTRANSCRIPT

Seam CarvingProject 1a due at midnight tonight.Project 1b goes out today.Well cover seam carving today.

Image resizingresamplingcroppingcontent-aware resizing

Seam carving: ideaCropping removes pixels from the image boundary.We want to remove only unimportant pixels.

How can we make sure the result is rectangular?

Seam carving: ideahttp://swieskowski.net/carve/

Something in between: remove seamsWhy not remove least important pixel from each row?

Pixel importanceMany possible measures of importanceOne simple one is gradient magnitudeor

Computing the optimal seamHow many vertical seams in an m-by-n image?

We can avoid trying all of them.Suppose this is the optimal seam:

What can we say about this one?

Computing the optimal seamM[x,y] = cost of minimum-energy vertical seam through rows 1 to yM[x,y]M[x+1,y-1]M[x,y-1]M[x-1,y-1]M[x,y] = min(M[x-1,y-1], M[x,y-1], M[x+1,y-1]) + E[x,y]

Computing the optimal seamFind pixel in bottom row with minimum seam cost.

Trace back optimal seam through image.

Remove seam pixels.M =

Seam carving algorithm

Compute energy at each pixelWhile image larger than m-by-n:Remove horizontal or vertical seam with minimum energy.

How do we choose between horizontal and vertical?

How might we enlarge an image?

Image SegmentationFrom Sandlot Science

From images to objectsWhat Defines an Object?Subjective problem, but has been well-studiedGestalt Laws seek to formalize thisproximity, similarity, continuation, closure, common fatesee notes by Steve Joordens, U. Toronto

Image histogramsHow many orange pixels are in this image?This type of question answered by looking at the histogramA histogram counts the number of occurrences of each colorGiven an image

The histogram isi.e., for each color value c (x-axis), plot # of pixels with that color (y-axis)

What do histograms look like?How Many Modes Are There?Easy to see, hard to compute

Histogram-based segmentationGoalBreak the image into K regions (segments)Solve this by reducing the number of colors to K and mapping each pixel to the closest color

Histogram-based segmentationGoalBreak the image into K regions (segments)Solve this by reducing the number of colors to K and mapping each pixel to the closest color Heres what it looks like if we use two colors

ClusteringHow to choose the representative colors?This is a clustering problem!ObjectiveEach point should be as close as possible to a cluster centerMinimize sum squared distance of each point to closest center

Break it down into subproblemsSuppose I tell you the cluster centers ciQ: how to determine which points to associate with each ci?A: for each point p, choose closest ciSuppose I tell you the points in each clusterQ: how to determine the cluster centers?A: choose ci to be the mean of all points in the cluster

K-means clusteringK-means clustering algorithmRandomly initialize the cluster centers, c1, ..., cKGiven cluster centers, determine points in each clusterFor each point p, find the closest ci. Put p into cluster iGiven points in each cluster, solve for ciSet ci to be the mean of points in cluster iIf ci have changed, repeat Step 2

Java demo: http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html

PropertiesWill always converge to some solutionCan be a local minimumdoes not always find the global minimum of objective function:

Probabilistic clusteringBasic questionswhats the probability that a point x is in cluster m?whats the shape of each cluster?K-means doesnt answer these questions

Basic ideainstead of treating the data as a bunch of points, assume that they are all generated by sampling a continuous functionThis function is called a generative model defined by a vector of parameters

Mixture of GaussiansOne generative model is a mixture of Gaussians (MOG)K Gaussian blobs with means b covariance matrices Vb, dimension dblob b defined by:

blob b is selected with probability the likelihood of observing x is a weighted mixture of Gaussians

where

Expectation maximization (EM)Goalfind blob parameters that maximize the likelihood function:

Approach:E step: given current guess of blobs, compute ownership of each pointM step: given ownership probabilities, update blobs to maximize likelihood functionrepeat until convergence

Grabcut [Rother et al., SIGGRAPH 2004]

Graph-based segmentation?

What if we look at relationships between pixels?

qImages as graphsFully-connected graphnode for every pixellink between every pair of pixels, p,qcost cpq for each linkcpq measures similaritysimilarity is inversely proportional to difference in color and positionpCpqc

Segmentation by Graph CutsBreak Graph into SegmentsDelete links that cross between segmentsEasiest to break links that have low cost (low similarity)similar pixels should be in the same segmentsdissimilar pixels should be in different segments

ABC

Cuts in a graphLink Cutset of links whose removal makes a graph disconnectedcost of a cut:ABFind minimum cutgives you a segmentationfast algorithms exist for doing this

But min cut is not always the best cut...

Cuts in a graphABNormalized Cuta cut penalizes large segmentsfix by normalizing for size of segments

volume(A) = sum of costs of all edges that touch A

Interpretation as a Dynamical SystemTreat the links as springs and shake the systemelasticity proportional to costvibration modes correspond to segmentscan compute these by solving an eigenvector problemfor more details, see J. Shi and J. Malik, Normalized Cuts and Image Segmentation, CVPR, 1997

Interpretation as a Dynamical System

Color Image Segmentation

Intelligent Scissors (demo)

Around n * 3^m seams (slightly fewer due to boundaries).The shorter seam is the optimal seam ending in that pixel**