Putting Context into Vision

Download Putting Context into Vision

Post on 16-Jan-2016

69 views

Category:

Documents

0 download

DESCRIPTION

Putting Context into Vision. Derek Hoiem September 15, 2004. Questions to Answer. What is context? How is context used in human vision? How is context currently used in computer vision? Conclusions. Context. What is context?. - PowerPoint PPT Presentation

TRANSCRIPT

Putting Context into VisionDerek HoiemSeptember 15, 2004Questions to AnswerWhat is context?How is context used in human vision? How is context currently used in computer vision?ConclusionsWhat is context?Any data or meta-data not directly produced by the presence of an objectNearby image dataWhat is context?Any data or meta-data not directly produced by the presence of an objectNearby image dataScene informationContextContextWhat is context?Any data or meta-data not directly produced by the presence of an objectNearby image dataScene informationPresence, locations of other objectsTreeHow do we use context?AttentionAre there any live fish in this picture?Clues for FunctionWhat is this?Clues for FunctionWhat is this?Now can you tell?Low-Res ScenesWhat is this?Low-Res ScenesWhat is this?Now can you tell?More Low-ResWhat are these blobs?More Low-ResThe same pixels! (a car)Why is context useful?Objects defined at least partially by functionTrees grow in ground Birds can fly (usually)Door knobs help open doorsWhy is context useful?Objects defined at least partially by functionContext gives clues about functionNot rooted into the ground not treeObject in sky {cloud, bird, UFO, plane, superman} Door knobs always on doorsWhy is context useful?Objects defined at least partially by functionContext gives clues about functionObjects like some scenes better than othersToilets like bathroomsFish like waterWhy is context useful?Objects defined at least partially by functionContext gives clues about functionObjects like some scenes better than othersMany objects are used together and, thus, often appear togetherKettle and stoveKeyboard and monitorHow is context used in computer vision?Neighbor-based ContextMarkov Random Field (MRF) incorporates contextual constraintsBlobs and Words Carbonetto 2004Neighbor-based context (MRF) useful even when training data is not fully supervisedLearns models of objects given captioned imagesDiscriminative Random Fields Kumar 2003Using data surrounding the label site (not just at the label site) improves resultsBuildings vs. Non-BuildingsMulti-scale Conditional Random Field (mCRF) He 2004mCRFFinal decision based on Classification (local data-based) Local labels (what relation nearby objects have to each other Image-wide labels (captures coarse scene context)mCRF ResultsNeighbor-based Context ReferencesP. Carbonetto, N. Freitas and K. Barnard. A Statistical Model for General Contextual Object Recognition, ECCV, 2004 S. Kumar and M. Hebert, Discriminative Random Fields: A Discriminative Framework for Contextual Interaction in Classification, ICCV, 2003X. He, R. Zemel and M. Carreira-Perpin, Multiscale Conditional Random Fields for Image Labeling, CVPR, 2004Scene-based ContextAverage pictures containing heads at three scalesContext Priming Torralba 2001/2003PoseScaleLocationObject PresenceImage MeasurementsObject PrimingFocus of AttentionScale SelectionPose and Shape PrimingContext (generally ignored)Local evidence (what everyone uses)Getting the Gist of a SceneSimple representationSpectral characteristics (e.g., Gabor filters) with coarse description of spatial arrangementPCA reductionProbabilities modeled with mixture of Gaussians (2003) or logistic regression (Murphy 2003)Context Priming ResultsFocus of AttentionObject PresenceScale SelectionSmallLargeUsing the Forrest to See the Trees Murphy (2003)Adaboost Patch-Based Object DetectorGist of Scene(boosted regression)Detector Confidence at Location/ScaleExpected Location/Scale of ObjectCombine(logistic regression)Object Probability at Location/ScaleObject Detection + Scene Context ResultsOften doesnt help that muchMay be due to poor use of contextAssumes independence of context and local evidenceOnly uses expected location/scale from contextScene-based Context ReferencesE. Adelson, On Seeing Stuff: The Perception of Materials by Humans and Machines, SPIE, 2001 B. Bose and E. Grimson, Improving Object Classification in Far-Field Video, ECCV, 2004 K. Murphy, A. Torralba and W. Freeman, Using the Forrest to See the Trees: A Graphical Model Relating Features, Object, and Scenes, NIPS, 2003 U. Rutishauser, D. Walther, C. Koch, and P. Perona, Is bottom-up attention useful for object recognition?, CVPR, 2004A. Torralba, Contextual Priming for Object Detection, IJCV, 2003A. Torralba and P. Sinha, Statistical Context Priming for Object Detection, ICCV, 2001A. Torralba, K. Murphy, W. Freeman, and M. Rubin, Context-Based Vision System for Place and Object Recognition, ICCV, 2003 Object-based ContextMutual Boosting Fink (2003)Local WindowContextual WindowRaw ImageObject Likelihoods++FilterseyesfacesConfidenceMutual Boosting ResultsLearned FeaturesFirst-Stage Classifier (MIT+CMU)Contextual Models using BRFs Torralba 2004Template featuresBuild structure of CRF using boostingOther objects locations likelihoods propagate through networkLabeling a Street SceneLabeling an Office SceneF = localG = compatibilityObject-based Context ReferencesM. Fink and P. Perona, Mutual Boosting for Contextual Inference, NIPS, 2003A. Torralba, K. Murphy, and W. Freeman, Contextual Models for Object Detection using Boosted Random Fields, AI Memo 2004-013, 2004What else can be done?Scene StructureImprove understanding of scene structureFloor, walls, ceilingSky, ground, roads, buildingsSemantics vs. Low-levelLow-LevelSemanticsPutting it all togetherScene GistObjectDetectionBasic StructureIdentificationSceneRecognitionContext PrimingNeighbor-Based ContextObject-Based ContextScene-Based ContextSummaryNeighbor-based contextUsing nearby labels essential for complete labeling tasksUsing nearby labels useful even without completely supervised training data Using nearby labels and nearby data is better than just using nearby labelsLabels can be used to extract local and scene contextSummaryScene-based contextGist representation suitable for focusing attention or determining likelihood of object presenceScene structure would provide additional useful information (but difficult to extract)Scene label would provide additional useful informationSummaryObject-based contextEven simple methods of using other objects locations improve results (Fink)Using BRFs, systems can automatically learn to find easier objects first and to use those objects as context for other objectsConclusionsGeneralFew object detection researchers use contextContext, when used effectively, can improve results dramaticallyA more integrated approach to use of context and data could improve image understandingReferencesE. Adelson, On Seeing Stuff: The Perception of Materials by Humans and Machines, SPIE, 2001B. Bose and E. Grimson, Improving Object Classification in Far-Field Video, ECCV, 2004 P. Carbonetto, N. Freitas and K. Barnard. A Statistical Model for General Contextual Object Recognition, ECCV, 2004 M. Fink and P. Perona, Mutual Boosting for Contextual Inference, NIPS, 2003 X. He, R. Zemel and M. Carreira-Perpin, Multiscale Conditional Random Fields for Image Labeling, CVPR, 2004 S. Kumar and M. Hebert, Discriminative Random Fields: A Discriminative Framework for Contextual Interaction in Classification, ICCV, 2003 J. Lafferty, A. McCallum and F. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, ICML, 2001K. Murphy, A. Torralba and W. Freeman, Using the Forrest to See the Trees: A Graphical Model Relating Features, Object, and Scenes, NIPS, 2003U. Rutishauser, D. Walther, C. Koch, and P. Perona, Is bottom-up attention useful for object recognition?, CVPR, 2004 A. Torralba, Contextual Priming for Object Detection, IJCV, 2003A. Torralba and P. Sinha, Statistical Context Priming for Object Detection, ICCV, 2001 A. Torralba, K. Murphy, and W. Freeman, Contextual Models for Object Detection using Boosted Random Fields, AI Memo 2004-013, 2004 A. Torralba, K. Murphy, W. Freeman, and M. Rubin, Context-Based Vision System for Place and Object Recognition, ICCV, 2003