seam carving

Download Seam Carving

Post on 16-Jan-2016




1 download

Embed Size (px)


Seam Carving. Project 1a due at midnight tonight. Project 1b goes out today. We’ll cover seam carving today. Image resizing. resampling. cropping. content-aware resizing. Seam carving: idea. Cropping removes pixels from the image boundary. We want to remove only “unimportant” pixels. - PowerPoint PPT Presentation


  • Seam CarvingProject 1a due at midnight tonight.Project 1b goes out today.Well cover seam carving today.

  • Image resizingresamplingcroppingcontent-aware resizing

  • Seam carving: ideaCropping removes pixels from the image boundary.We want to remove only unimportant pixels.

    How can we make sure the result is rectangular?

  • Seam carving: idea

    Something in between: remove seamsWhy not remove least important pixel from each row?

  • Pixel importanceMany possible measures of importanceOne simple one is gradient magnitudeor

  • Computing the optimal seamHow many vertical seams in an m-by-n image?

    We can avoid trying all of them.Suppose this is the optimal seam:

    What can we say about this one?

  • Computing the optimal seamM[x,y] = cost of minimum-energy vertical seam through rows 1 to yM[x,y]M[x+1,y-1]M[x,y-1]M[x-1,y-1]M[x,y] = min(M[x-1,y-1], M[x,y-1], M[x+1,y-1]) + E[x,y]

  • Computing the optimal seamFind pixel in bottom row with minimum seam cost.

    Trace back optimal seam through image.

    Remove seam pixels.M =

  • Seam carving algorithm

    Compute energy at each pixelWhile image larger than m-by-n:Remove horizontal or vertical seam with minimum energy.

    How do we choose between horizontal and vertical?

    How might we enlarge an image?

  • Image SegmentationFrom Sandlot Science

  • From images to objectsWhat Defines an Object?Subjective problem, but has been well-studiedGestalt Laws seek to formalize thisproximity, similarity, continuation, closure, common fatesee notes by Steve Joordens, U. Toronto

  • Image histogramsHow many orange pixels are in this image?This type of question answered by looking at the histogramA histogram counts the number of occurrences of each colorGiven an image

    The histogram isi.e., for each color value c (x-axis), plot # of pixels with that color (y-axis)

  • What do histograms look like?How Many Modes Are There?Easy to see, hard to compute

  • Histogram-based segmentationGoalBreak the image into K regions (segments)Solve this by reducing the number of colors to K and mapping each pixel to the closest color

  • Histogram-based segmentationGoalBreak the image into K regions (segments)Solve this by reducing the number of colors to K and mapping each pixel to the closest color Heres what it looks like if we use two colors

  • ClusteringHow to choose the representative colors?This is a clustering problem!ObjectiveEach point should be as close as possible to a cluster centerMinimize sum squared distance of each point to closest center

  • Break it down into subproblemsSuppose I tell you the cluster centers ciQ: how to determine which points to associate with each ci?A: for each point p, choose closest ciSuppose I tell you the points in each clusterQ: how to determine the cluster centers?A: choose ci to be the mean of all points in the cluster

  • K-means clusteringK-means clustering algorithmRandomly initialize the cluster centers, c1, ..., cKGiven cluster centers, determine points in each clusterFor each point p, find the closest ci. Put p into cluster iGiven points in each cluster, solve for ciSet ci to be the mean of points in cluster iIf ci have changed, repeat Step 2

    Java demo:

    PropertiesWill always converge to some solutionCan be a local minimumdoes not always find the global minimum of objective function:

  • Probabilistic clusteringBasic questionswhats the probability that a point x is in cluster m?whats the shape of each cluster?K-means doesnt answer these questions

    Basic ideainstead of treating the data as a bunch of points, assume that they are all generated by sampling a continuous functionThis function is called a generative model defined by a vector of parameters

  • Mixture of GaussiansOne generative model is a mixture of Gaussians (MOG)K Gaussian blobs with means b covariance matrices Vb, dimension dblob b defined by:

    blob b is selected with probability the likelihood of observing x is a weighted mixture of Gaussians


  • Expectation maximization (EM)Goalfind blob parameters that maximize the likelihood function:

    Approach:E step: given current guess of blobs, compute ownership of each pointM step: given ownership probabilities, update blobs to maximize likelihood functionrepeat until convergence

  • Grabcut [Rother et al., SIGGRAPH 2004]

  • Graph-based segmentation?

    What if we look at relationships between pixels?

  • qImages as graphsFully-connected graphnode for every pixellink between every pair of pixels, p,qcost cpq for each linkcpq measures similaritysimilarity is inversely proportional to difference in color and positionpCpqc

  • Segmentation by Graph CutsBreak Graph into SegmentsDelete links that cross between segmentsEasiest to break links that have low cost (low similarity)similar pixels should be in the same segmentsdissimilar pixels should be in different segments


  • Cuts in a graphLink Cutset of links whose removal makes a graph disconnectedcost of a cut:ABFind minimum cutgives you a segmentationfast algorithms exist for doing this

  • But min cut is not always the best cut...

  • Cuts in a graphABNormalized Cuta cut penalizes large segmentsfix by normalizing for size of segments

    volume(A) = sum of costs of all edges that touch A

  • Interpretation as a Dynamical SystemTreat the links as springs and shake the systemelasticity proportional to costvibration modes correspond to segmentscan compute these by solving an eigenvector problemfor more details, see J. Shi and J. Malik, Normalized Cuts and Image Segmentation, CVPR, 1997

  • Interpretation as a Dynamical System

  • Color Image Segmentation

  • Intelligent Scissors (demo)

    Around n * 3^m seams (slightly fewer due to boundaries).The shorter seam is the optimal seam ending in that pixel**


View more >