sensible visual search

Download Sensible Visual Search

If you can't read please download the document

Post on 11-Jan-2016




2 download

Embed Size (px)


Sensible Visual Search. Shih-Fu Chang Digital Video and Multimedia Lab Columbia University June 2008 (Joint Work with Eric Zavesky and Lyndon Kennedy). User Expectation for Web Search. Keyword search is still the primary search method - PowerPoint PPT Presentation


  • Sensible Visual SearchShih-Fu Chang

    Digital Video and Multimedia LabColumbia 2008

    (Joint Work with Eric Zavesky and Lyndon Kennedy)

    digital video | multimedia lab

    User Expectation for Web Searchtype in a few words at most, then expect the engine to bring back the perfect results. More than 95 percent of us never use the advanced search features most engines include, The Search, J. Battelle, 2003 Keyword search is still the primary search methodStraightforward extension to visual search

  • Keyword-based Visual Search Paradigm

    digital video | multimedia lab

    Web Image SearchText Query : Manhattan Cruise over Goggle ImageWhat are in the results?Why are these images returned?How to choose better search terms?

    digital video | multimedia lab

    Minor Changes in Keywords Big DifferenceText Query : Cruise around Manhattan

  • When metadata are unavailable:Automatic Image ClassificationAudio-visual features Geo, social featuresSVM or graph modelsContext fusion. . .Rich semantic description based on content analysisStatistical models

    digital video | multimedia lab

    A few good detectors for LSCOM conceptswaterfrontbridgecrowdexplosion fireUS flagMilitary personnelRemember there are many not so good detectors.

  • Keyword Search over Statistical Detector Objects, people, location, scenes, events, etcConcepts defined by expert analysts over news video

  • Query car crash snow over TRECVID video using LSCOM conceptsHow are keywords mapped to concepts?What classifiers work? What dont?How to improve the search terms?

  • Relevant DetectorscarCar crashsnowHow did individual detectors influence the search results?

  • Frustration of Uninformed Users of Keyword SearchDifficult to choose meaningful words/concepts without in-depth knowledge of entire vocabulary

  • Pains of Uninformed UsersForced to take one shot searches, iterating queries with a trial and error approach...

  • Challenge: user frustration in visual search

  • Proposal: Sensible SearchMake search experience more SensibleHelp users stay informedin selecting effective keywords/concepts in understanding the search resultsin manipulating the search criteria rapidly and flexibly Keep users engagedInstant feedback with minimal disruption as opposed to trial-and-error

  • A prototype CuZero: Zero-Latency Informed Search & Navigation

  • Informed User: Instant Informed Query Formulation

  • Informed User for Visual Search:Instant visual concept suggestionquery time concept mining Instant Concept Suggestion

  • Lexical mappingMapping keywords to concept definition, synonyms, sense context, etcLSCOM

  • Co-occurrent conceptsroadcarBasketball courts and the American won the Saint Denis on the Phoenix Suns because of the 50 point for 19 in their role within the National Association of BasketballGeorge Rizq led Hertha for the local basketball game the wisdom and sports championship of the presidentBaghdad to attend the game I see more goals and the players did not offer great that Beijing Games as the beginning of his brilliance Nayyouf 10 this atmosphere the culture of sports championshipimagestext

  • Query-Time Concept Mining

  • CuZero Real-Time Query Interface (demo)Instant Concept SuggestionAuto-complete speech transcripts

  • A prototype CuZero: Zero-Latency Informed Search & Navigation(Zavesky and Chang, MIR2008)

  • Informed User:Intuitive Exploration of Results

  • Informed User:Rapid Exploration of ResultsMedia Mill Rotor Browser

  • Revisit the user struggle Car detectorCar crash detectorSnow detectorQuery: {car, snow, car_crash}How did each concept influence the results?

  • CuZero:Real-Time Multi-Concept Navigation MapCreate a multi-concept gradient mapDirect user control: nearness = more influenceInstant display for each location, without new query

  • Achieve Breadth-Depth Flexibilityby Dual Space Navigation (demo)Breadth: Quick scan of many permutationsDepth: Instant exploration of results with fixed weightsmany query permutationsDeep exploration of single permutation

    dvmm - when a concept is deleted, its result list need to be erased also. Currently it is not done

  • Content planning to remove redundancyboatwaterscored result liststraditional list planningguaranteed uniquelist planning

  • execute query and download ranked concept list

    package results with scores

    transmit to client

    unpackage results at interface

    score images by concept weights; guarantee unique positions

    download images to interface in cached mode

    Latency Analysis: Workflow Pipelinetime to execute is disproportional!

  • Pipelined processing for low latencyConcept formulation (car) Overlap (concept formulation) with (map rendering) Hide rendering latency during user interaction Course-to-fine concept map planning/rendering Speed optimization on-going

  • Challenge: user frustration in visual searchsensible search: (1) query(2) visualize+(3) analyze

  • Frequent reuse and dissemination of media objects on Web 2.0Informed User: understand trend/history in search resultschanneltimeABCCNNMSNBCFOXCCTVTVBSLBCAl-JazeeraGoogle.cnGoogle.comYahoo news

    DVMM Lab, Columbia University

    Help Users Make Sense of Image TrendMany re-used content foundHow did it occur?What manipulations?What distribution path?Correlation with perspective change?Query: John Kennedy

    DVMM Lab, Columbia University

    Manipulation correlated with PerspectiveRaising the Flag on Iwo Jima Joe Rosenthal, 1945Anti-Vietnam War, Ronald and Karen Bowen, 1969

    DVMM Lab, Columbia University

    Reused Images Over Time

    digital video | multimedia lab

    Question for Sensible Search: Insights from Plain Search Results?

    digital video | multimedia lab

    Duplicate Clusters Reveal Image ProvenanceBiggest Clusters Contain Iconic ImagesSmallest Clusters Contain Marginal Images

    DVMM Lab, Columbia University

    A simple reranking applicationDetect duplicate pairs across video/image setJoin duplicate pairs such that all pairs are in the same clusterDisplay only one image or shot per cluster (others will be duplicates / redundant)Rank results by ordering clusters (various approaches to ranking clusters)

    DVMM Lab, Columbia University

    Example: Yahoo! Image Search?????

    DVMM Lab, Columbia University

    Example: Reranked ResultsMore Diverse and Relevant

    DVMM Lab, Columbia University

    Deeper Analysis of Search Results: Visual Migration Map (VMM)Duplicate ClusterVisual Migration Map(Kennedy and Chang, ACM Multimedia 2008)

    DVMM Lab, Columbia University

    Visual Migration Map (VMM)Most Divergent at the leaves Images Derived through Series of ManipulationsVMM uncovers history of image manipulation and plausible dissemination paths among content owners and users.

    DVMM Lab, Columbia University

    Ground truth VMM is hard to getHypothesisApproximation of history is feasible by visual analysis.Detect manipulation types between two imagesDerive large scale history among a large image set

    DVMM Lab, Columbia University

    Basic Image Manipulation OperatorsEach is observable by inspecting the pairEach implies direction (one image derived from other)Other possible manipulations: color correction, multiple compression, sharpening, blurring

    digital video | multimedia lab

    Detecting Near-DuplicatesDuplicate detection is very useful and relatively reliableRemaining challenges: scalability/speed; video duplicates; object (sub-image) (TRECVID08)Graph Matching [Zhang & Chang, 2004]Matching SIFT points [Lowe, 1999]

    DVMM Lab, Columbia University

    Automatic Manipulation DetectorsObjective: automatically detect various types of image manipulationsContext-free detectors: rely only on two imagescopy, scaling, grayscaleContext-dependent detectors: rely on cues from a plurality of imagescropping, insertion, overlay

    DVMM Lab, Columbia University

    Simple Copy DetectionExtract SIFT descriptors (local shape features)Detect correspondences between features between imagesCount total number of matched points

    DVMM Lab, Columbia University

    Scale DetectionDraw bounding box around matching points in each imageCompare heights/widths of each boxRelative difference in box size can be used to normalize scales

    DVMM Lab, Columbia University

    Color RemovalSimple case: image stored in single channel fileOther cases: image is grayscale, but stored in 3-channel fileExpect little difference in values in various channels within pixels

    DVMM Lab, Columbia University

    More Challenging:Overlay Detection?Given two images, we can observe that a region is different between the twoBut how do we know which is the original?

    DVMM Lab, Columbia University

    Cropping or Insertion?Can find differences in image areaBut is the smaller-area due to a crop or is the larger area due to an insertion?CroppingOriginalInsertion

    DVMM Lab, Columbia University

    Use Context from Many DuplicatesNormalize Scales and PositionsGet average value for each pixelComposite image

    DVMM Lab, Columbia University

    Cropping Detection w/ ContextIn cropping, we expect the content outside the crop area to be consistent with the composite imageImage AComposite AResidue A

    DVMM Lab, Columbia University

    Overlay Detection w/ ContextComparing images against composite image reveals portions that differ from typical contentImage with divergent content may have overlayImage AComposite AResidue A

    DVMM Lab, Columbia Un


View more >