graph-based segmentation
DESCRIPTION
Graph-based Segmentation. Main Ideas. Convert image into a graph Vertices for the pixels Edges between the pixels Additional vertices and edges to encode other constraints Manipulate the graph to segment the image. Papers. - PowerPoint PPT PresentationTRANSCRIPT
Graph-based Graph-based SegmentationSegmentation
Main IdeasMain Ideas
Convert image into a graphConvert image into a graph Vertices for the pixelsVertices for the pixels Edges between the pixelsEdges between the pixels Additional vertices and edges to encode other Additional vertices and edges to encode other
constraintsconstraints
Manipulate the graph to segment the Manipulate the graph to segment the imageimage
PapersPapers
Interactive graph cuts for optimal boundary Interactive graph cuts for optimal boundary & region segmentation of& region segmentation ofobjects in N-D images objects in N-D images Boykov and JollyBoykov and Jolly Minimize an energy functionMinimize an energy function
Efficient Graph-based SegmentationEfficient Graph-based Segmentation Felzenszwalb and HuttenlocherFelzenszwalb and Huttenlocher Cluster the vertices based on edge weightCluster the vertices based on edge weight
Boykov and JollyBoykov and JollyBinary image segmentationBinary image segmentation Classify pixels as object or backgroundClassify pixels as object or background Their contribution is adding interactivityTheir contribution is adding interactivityMinimise an energy functionMinimise an energy function E(A) = B(A) + E(A) = B(A) + λλR(A)R(A)A: Segmentation (assign each pixels to object A: Segmentation (assign each pixels to object or background)or background)B(A): The cost of all the edges between object B(A): The cost of all the edges between object pixels and background pixelspixels and background pixelsR(A): The cost of deciding a pixel to be object or R(A): The cost of deciding a pixel to be object or backgroundbackground
Creating the GraphCreating the Graph
Each pixel has a corresponding vertexEach pixel has a corresponding vertexAdditionally, a source (“object”) and a sink Additionally, a source (“object”) and a sink (“background”)(“background”)Each pixel vertex has an edge to its Each pixel vertex has an edge to its neighbours (e.g. 8 adjacent neighbours in neighbours (e.g. 8 adjacent neighbours in 2D), an edge to the source, an edge to 2D), an edge to the source, an edge to the sinkthe sink
Edge Weights between pixelsEdge Weights between pixels
Weight of edges between pixel vertices Weight of edges between pixel vertices are determined by the B() functionare determined by the B() functionLow score when boundary is likely to pass Low score when boundary is likely to pass between the verticesbetween the verticeshigh score when vertices are probably part high score when vertices are probably part of the same elementof the same elementE.g. the difference in pixel intensities, the E.g. the difference in pixel intensities, the gradientgradient
Edges to Source/SinkEdges to Source/Sink
If pixel is known to be an object, use a high If pixel is known to be an object, use a high weight (K) to the source, zero weight to the sinkweight (K) to the source, zero weight to the sink K is chosen so that it will never be cutK is chosen so that it will never be cut
Conversely, if pixel is background, use weight K Conversely, if pixel is background, use weight K to the sink, zero weight to the sourceto the sink, zero weight to the sourceOtherwise, weigh edges to source and sink Otherwise, weigh edges to source and sink appropriately using the R() functionappropriately using the R() function Note that the edge to the source is the “likelihood” for Note that the edge to the source is the “likelihood” for
the pixel being the the pixel being the backgroundbackground – – we break this edge we break this edge when the pixel is assigned to the backgroundwhen the pixel is assigned to the background
ApplicationsApplications
Handles arbitrary number of dimensionsHandles arbitrary number of dimensionsFinds global minimum energyFinds global minimum energy
Needs “good” user input to work effectivelyNeeds “good” user input to work effectivelyNeed intelligent functionsNeed intelligent functionsNeed to select Need to select λλ
Felzenszwalb and HuttenlocherFelzenszwalb and Huttenlocher
Download the program from the webpage:Download the program from the webpage: http://people.cs.uchicago.edu/~pff/segment/http://people.cs.uchicago.edu/~pff/segment/
Minimal documentationMinimal documentation Short README fileShort README file PaperPaper
The programThe program
Comes as a tar.gz archive and .zip archiveComes as a tar.gz archive and .zip archiveProcessProcess Extract archiveExtract archive ‘‘make’ (Makefile supplied)make’ (Makefile supplied)
Program consists ofProgram consists of A .cpp “wrapper” file (only calls the functions)A .cpp “wrapper” file (only calls the functions) Actual algorithm functions are in .h filesActual algorithm functions are in .h files Basic portable C++ codeBasic portable C++ code
Program TestingProgram Testing
Built on Mac OS X, Linux, Windows cygwinBuilt on Mac OS X, Linux, Windows cygwin Gcc toolchain, but any C++ compiler should workGcc toolchain, but any C++ compiler should work Supplied basic MakefileSupplied basic Makefile
Results were basically the same between Results were basically the same between platformsplatforms Colors are chosen randomlyColors are chosen randomly
Results obtained are not the same as posted on Results obtained are not the same as posted on the websitethe website Image files on the website may be modified Image files on the website may be modified
(scaled/compressed/downsampled)(scaled/compressed/downsampled)
AlgorithmAlgorithm
Create a graphCreate a graph Each vertex corresponds to a vertexEach vertex corresponds to a vertex
Edges are between “neighbouring” verticesEdges are between “neighbouring” vertices Choose a small neighbourhood to reduce Choose a small neighbourhood to reduce
computation time (otherwise we have a complete computation time (otherwise we have a complete graph)graph)
Weight on the edge is the 5D distance between the Weight on the edge is the 5D distance between the points (for a 2D image)points (for a 2D image)
5D vector = x position, y position and 3 color components5D vector = x position, y position and 3 color components
ParametersParameters
σσ: Use this value and do Gaussian : Use this value and do Gaussian smoothing (preprocessing the image to smoothing (preprocessing the image to reduce noise)reduce noise)k: threshold value for doing the clusteringk: threshold value for doing the clusteringmin: “hack” parametermin: “hack” parameter the smallest cluster size must contain at least the smallest cluster size must contain at least
this many vertices – clusters that are too this many vertices – clusters that are too small will be merged with other clusters until small will be merged with other clusters until sufficiently largesufficiently large
ClusteringClustering
Put each vertex in a component Put each vertex in a component Sort edges by weightSort edges by weightTake each edge in turnTake each edge in turn If the edge is between vertices in two different If the edge is between vertices in two different
components A and B, we can merge if the edge components A and B, we can merge if the edge weight is lower enough than the threshold weight is lower enough than the threshold
Threshold is the minimum of the following value, Threshold is the minimum of the following value, computed on A and Bcomputed on A and B
(Lowest weight edge in minimum spanning tree of the (Lowest weight edge in minimum spanning tree of the component) + (k / size of component)component) + (k / size of component)
NotesNotes
Low Low edge weights between vertices that edge weights between vertices that are likely to be in the same clusterare likely to be in the same clusterAs a cluster gets larger, it becomes As a cluster gets larger, it becomes harder to add vertices to itharder to add vertices to itHeuristic – not really minimising a Heuristic – not really minimising a particular energy functionparticular energy function More similar to “region growing”More similar to “region growing”User has select “good” parameters to get User has select “good” parameters to get good resultsgood results
Effect of Effect of σσ
Increased smoothing results in removal of Increased smoothing results in removal of noisenoiseCan cause “bleeding” – the algorithm has Can cause “bleeding” – the algorithm has difficulty separating background from the difficulty separating background from the object if the boundaries are too smoothobject if the boundaries are too smooth
Reference ImagesReference Images
Increasing Increasing σσClouds are Clouds are recognised as recognised as one objectone object
Palm tree Palm tree gets confused gets confused with oceanwith ocean
GrainGrain
Increasing Increasing σσ introduces more blurring introduces more blurring (reduces the edge weight between pixels)(reduces the edge weight between pixels)
Vertebrae MRIVertebrae MRI
Gets rid of noise (bottom left, right hand side), Gets rid of noise (bottom left, right hand side), but purple vertebrate piece bleeds outbut purple vertebrate piece bleeds out
Increasing ThresholdIncreasing Threshold
Clusters Clusters more more aggresivelyaggresively
Palm tree is Palm tree is confused confused with ocean with ocean and cloudsand clouds
GrainGrain
Non-grain pixels are almost all clustered Non-grain pixels are almost all clustered togethertogetherMeasure of how “similar” all the pixels of an Measure of how “similar” all the pixels of an object should beobject should be
MRIMRI
Vertebrae and region next to vertebrae are Vertebrae and region next to vertebrae are very similar shade so are easily confusedvery similar shade so are easily confused
Increasing ‘min’ valueIncreasing ‘min’ value
Segmentation Segmentation is the same, is the same, but small but small components components are merged are merged with with neighboring neighboring onesones
GrainGrain
Easy to control change, gets rid of small Easy to control change, gets rid of small artifactsartifacts
MRIMRI
Not Not much much effect if effect if regions regions are are already already largelarge
Parameter TweakingParameter Tweaking
Need to manually Need to manually tune parameters to tune parameters to get a good imageget a good imageImage of MRI after Image of MRI after selectingselecting σσ = 0.6 = 0.6 k = 200k = 200 min = 60min = 60
PerformancePerformance
Need to tune parameters by handNeed to tune parameters by handVery fastVery fast Program usually takes a couple seconds on Program usually takes a couple seconds on
the test images providedthe test images providedOnly takes ppm images in RAW data Only takes ppm images in RAW data formatformatTheoretically, algorithm generalises to Theoretically, algorithm generalises to arbitrary number of dimensions and arbitrary number of dimensions and arbitrary number of features per pixelarbitrary number of features per pixel
Command-line toolsCommand-line toolsWorking with images involves opening up the results Working with images involves opening up the results in an image viewer – can get messyin an image viewer – can get messy