Object-Graphs for Context-Aware Category Discovery

Download Object-Graphs for Context-Aware Category Discovery

Post on 24-Feb-2016

26 views

Category:

Documents

0 download

Embed Size (px)

DESCRIPTION

Object-Graphs for Context-Aware Category Discovery. Yong Jae Lee and Kristen Grauman University of Texas at Austin. Motivation. 1) reveal structure in very large image collections 2) greatly reduce annotation time and effort 3) training data is not always available. Unlabeled Image Data. - PowerPoint PPT Presentation

TRANSCRIPT

<p>Object-Graphs for Context-Aware Category Discovery</p> <p>Object-Graphs for Context-Aware Category DiscoveryYong Jae Lee and Kristen GraumanUniversity of Texas at Austin</p> <p>11MotivationUnlabeled Image Data</p> <p>Discovered categories</p> <p>1) reveal structure in very large image collections2) greatly reduce annotation time and effort3) training data is not always available2Existing approachesPrevious work treats unsupervised visual discovery as an appearance-grouping problem.</p> <p>- Topic models e.g., pLSA, LDA.[Fergus et al. 2005], [Sivic et al. 2005], [Quelhas et al. 2005], [Fei-Fei &amp; Perona 2005], [Liu &amp; Chen 2007], [Russell et al. 2006]</p> <p>- Partitioning of the image data.[Grauman &amp; Darrell 2006], [Dueck &amp; Frey 2007], [Kim et al. 2008], [Lee &amp; Grauman 2008], [Lee &amp; Grauman 2009]</p> <p>33Existing approachesPrevious work treats unsupervised visual discovery as an appearance-grouping problem.41342Can you identify the recurring pattern?4How can seeing previously learned objects in novel images help to discover new categories?</p> <p>1342Our idea5Can you identify the recurring pattern?5Discover visual categories within unlabeled images by modeling interactions between the unfamiliar regions and familiar objects.</p> <p>Our idea13426Can you identify the recurring pattern?6</p> <p>drive-wayskyhouse?grassContext-aware visual discovery</p> <p>grassskytruckhouse?drive-waygrassskyhousedrive-wayfence?</p> <p>???7Context in supervised recognition:[Torralba 2003], [Hoiem et al. 2006], [He et al. 2004], [Shotton et al. 2006], [Heitz &amp; Koller 2008], [Rabinovich et al. 2007], [Galleguillos et al. 2008], [Tu 2008], [Parikh et al. 2008], [Gould et al. 2009], [Malisiewicz &amp; Efros 2009], [Lazebnik 2009]7</p> <p>Key IdeasContext-aware category discovery treating previously learned categories as object-level context.</p> <p>Object-Graph descriptor to encode surrounding object-level context.</p> <p>Note: Different from semi-supervised learning unlabeled data do not necessarily belong to categories of the labeled data.</p> <p>88Approach Overview9Learn category models for some classesDetect unknowns in unlabeled imagesDescribe object-level context via Object-GraphGroup regions to discover new categories9Learn Known CategoriesOffline: Train region-based classifiers for N known categories using labeled training data.</p> <p>skyroadbuildingtree10Detect UnknownsObject-level ContextDiscoveryLearn Models10Identifying Unknown Objects</p> <p>Input: unlabeled pool of novel images</p> <p>Compute multiple-segmentations for each unlabeled image11Detect UnknownsObject-level ContextDiscoveryLearn Modelse.g., [Hoiem et al. 2006], [Russell et al. 2006], [Rabinovich et al. 2007]11</p> <p>P(class | region)bldgtreeskyroadP(class | region)bldgtreeskyroadP(class | region)bldgtreeskyroadP(class | region)bldgtreeskyroadPrediction: knownPrediction: knownPrediction: knownHigh entropy Prediction:unknownFor all segments, use classifiers to compute posteriors for the N known categories. Deem each segment as known or unknown based on resulting entropy. 12Identifying Unknown ObjectsDetect UnknownsObject-level ContextDiscoveryLearn Models12</p> <p>Model the topology of category predictions relative to the unknown (unfamiliar) region.Incorporate uncertainty from classifiers.</p> <p>An unknown region within an image0</p> <p>13Object-GraphsDetect UnknownsObject-level ContextDiscoveryLearn Models13</p> <p>An unknown region within an image0Closest nodes in its object-graph 2a</p> <p>2b</p> <p>1b1a</p> <p>3a</p> <p>3bConsider spatially near regions above and below, record distributions for each known class.Sb t s r1aabove1bbelowH1(s)b t s rb t s rH0(s)0selfg(s) = [ , , , ] </p> <p>HR(s)b t s rb t s rRaaboveRbbelow1st nearest regionout to Rth nearestb t s r0selfObject-GraphsDetect UnknownsObject-level ContextDiscoveryLearn Models1414Object-Graphs</p> <p>Average across segmentations</p> <p>N posterior prob.s per pixel</p> <p>b t s rb t s rN posterior prob.s per superpixel</p> <p>b t s rb t s rObtain per-pixel measures of class posteriors on larger spatial extents.15Detect UnknownsObject-level ContextDiscoveryLearn Models15</p> <p>g(s1) = [ : , , : ]</p> <p>b t g rabovebelowHR(s)H1(s)abovebelowb t g rb t g rb t g rg(s2) = [ : , , : ]</p> <p>b t g rabovebelowHR(s)H1(s)abovebelowb t g rb t g rb t g rObject-graphs are very similar produces a strong matchKnown classesb: buildingt: treeg: grassr: road16Object-Graph matchingDetect UnknownsObject-level ContextDiscoveryLearn Modelsbuilding?roadbuilding / roadbuilding/ roadtree / roadbuilding?roadbuilding/ road16grass?g(s1) = [ : , , : ]</p> <p>b t g rabovebelowHR(s)H1(s)abovebelowb t g rb t g rb t g rg(s2) = [ : , , : ]</p> <p>b t g rabovebelowHR(s)H1(s)abovebelowb t g rb t g rb t g r</p> <p>Object-graphs are partially similar produces a fair matchKnown classesb: buildingt: treeg: grassr: road17</p> <p>Object-Graph matchingDetect UnknownsObject-level ContextDiscoveryLearn Modelsbuilding?roadbuilding / roadbuilding/ roadbuildingroadroad17Unknown Regions</p> <p>Clusters from region-region affinities</p> <p>18Detect UnknownsObject-level ContextDiscoveryLearn Models18Object Discovery AccuracyFour datasets</p> <p>Multiple splits for each dataset; varying categories and number of knowns/unknowns</p> <p>Train 40% (for known categories), Test 60% of data</p> <p>Textons, Color histograms, and pHOG Features</p> <p>MSRC-v2 PASCAL 2008 Corel</p> <p>MSRC-v0 191920</p> <p>MSRC-v2 PASCAL 2008 Corel</p> <p>MSRC-v0 Object Discovery Accuracy</p> <p>20Comparison with State-of-the-artRussell et al., 2006: Topic model (LDA) to discover categories among multiple segmentations using appearance only.Significant improvement over existing state-of-the-art.21</p> <p>MSRC-v2 21</p> <p>Example Object-Graphsbuildingskyroadunknown22</p> <p> Color in superpixel nodes indicate the predicted known category.</p> <p>Examples of Discovered Categories2323Collect-Cut (poster Thursday)24</p> <p>Best Bottom-up (with multi-segs)</p> <p>Collect-Cut(ours)Discovered Ensemble from Unlabeled Multi-Object Images</p> <p>Unlabeled ImagesUse discovered shared top-down cues to refine both the segments and discovered categories with an energy function that can be minimized with graph cuts.</p> <p>Unsupervised Segmentation Examples24ConclusionsDiscover new categories in the context of those that have already been directly taught.</p> <p>Substantial improvement over traditional unsupervised appearance-based methods.</p> <p>Future work: Continuously expand the object-level context for future discoveries.</p> <p>25Category Retrieval Results</p> <p>26</p> <p>27</p> <p>Impact of Known/Unknown DecisionsRed star denotes the cutoff (half of max possible entropy value).Regions considered for discovery are almost all true unknowns (and vice versa), at some expense of misclassification.</p> <p>Impact of Object-Graph Descriptor How does the object-graph descriptor compare to a simpler alternative that directly encodes the surrounding appearance features?28Appearance-level context</p> <p>Object-level context </p> <p>2829Perfect Known/Unknown SeparationPerformance attainable were we able to perfectly separate segments according to whether they are known or unknown.</p> <p>Random Splits of Known/Unknown30</p> <p>31Previous Work: [Scholkopf 2000], [Markou &amp; Singh 2003], [Weinshall et al. 2008]</p> <p>ImageGT known/unknown</p> <p>Multiple-Segmentation Entropy Maps</p> <p>unknownsbuildingtreeknownsskyroadIdentifying Unknown ObjectsDetect UnknownsObject-level ContextDiscoveryLearn Models31</p>