visual information retrieval
DESCRIPTION
Visual Information Retrieval. Alberto Del Bimbo Dipartimento di Sistemi e Informatica Universita di Firenze Firenze, Italy. Chapter 1 Introduction. Visual Information Retrieval. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/1.jpg)
Visual Information Retrieval
Chapter 1
Introduction
Alberto Del BimboDipartimento di Sistemi e InformaticaUniversita di Firenze
Firenze, Italy
![Page 2: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/2.jpg)
Visual Information Retrieval
• Information retrieval, image/video analysis and processing, pattern recognition and computer vision, visual data modeling and representation, multimedia database organization, multidimensional indexing, psychological modeling of user behavior, man-machine interaction and data visualization
![Page 3: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/3.jpg)
Visual Information Retrieval
• Types of associated information– content-independent metadata (CIM)
• format, author's name, date
– content-dependent metadata (CDepM)• low-level features concerned with perceptual facts:
color, texture, shape, spatial relationship, motion
– content-descriptive metadata (CDesM)• high-level content semantics: cloud, good weather, 白雲蒼狗
![Page 4: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/4.jpg)
Visual Information Retrieval
• First-generation visual information retrieval systems– CIM by alphanumeric strings, CDepM and
CDesM by keywords or scripts
![Page 5: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/5.jpg)
![Page 6: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/6.jpg)
Visual Information Retrieval
– find images of paintings by Chagall with a blue background
• Select IMAGE# from PAINTINGS where PAINTER = "Chagall" and BACKGROUND = "blue"
– find images of paintings by Chagall with a girl in red dress and a blue background
• full text retrieval
![Page 7: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/7.jpg)
Visual Information Retrieval
– find images of paintings depicting similar figures in similar positions as in 收割景緻
• it is difficult for text to capture the perceptual saliency of some visual features
• text is not well suited for modeling perceptual similarity
• perception is mainly subjective, so is its text descriptions
![Page 8: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/8.jpg)
Visual Information Retrieval
• New-generation visual information retrieval systems– retrieval not only by concepts but also by
perception of visual contents• objective measurements of visual contents and
appropriate similarity models
• automatically extract features from raw data by image processing, pattern recognition, speech analysis and computer vision techniques
![Page 9: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/9.jpg)
Visual Information Retrieval
![Page 10: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/10.jpg)
![Page 11: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/11.jpg)
Visual Information Retrieval
• Image retrieval– by perceptual features
• for each image in the database, a set of features (model parameters) are precomputed
• to query the image database– express the query through visual examples
» authored by the user
» extracted from image samples
– select features and ranges of features
– choose a similarity measure
• compute similarity degrees, ranking, relevance feedback
![Page 12: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/12.jpg)
![Page 13: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/13.jpg)
![Page 14: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/14.jpg)
![Page 15: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/15.jpg)
Visual Information Retrieval
– system architecture• extraction of perceptual features (CDepM)
• extraction of high-level semantics (CDesM) from low-level features
• manual annotation of CIM and CDesM
• index structure
• graphical query tool
• retrieval engine
• visualization tool
• relevance feedback mechanism
![Page 16: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/16.jpg)
Visual Information Retrieval
• Video retrieval– special characteristics
• frames are linked together using editing effects
• color, texture, shape and position (camera or object) are changed in multiple frames
• richer semantics
• different types of video
![Page 17: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/17.jpg)
Visual Information Retrieval
– by structure• Figure 1.4
• frame: basic unit of information
• shot: elementary segment of video with perceptual continuity
• clip: set of frames with some semantic meaning
• scene: consecutive shots with simultaneous space, time and action
• episode: specific sequence of shot types such as a news episode
![Page 18: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/18.jpg)
![Page 19: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/19.jpg)
Visual Information Retrieval– by content
• perceptual properties, motion and type of an object• situations between objects• motion of camera• semantics of shots by color- or motion-induced sensations• semantics of scenes• stories• audio properties: dialogue, music or storytelling• textual information: caption or text recognized from video
![Page 20: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/20.jpg)
Visual Information Retrieval
– system architecture• extraction of shots and the associated semantics, key-
frames or mosaics
• extraction of scenes and stories
• manual annotation tool
• browsing/visualization tool– video summarization
• graphical query tool
• index structure
• retrieval engine
![Page 21: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/21.jpg)
Visual Information Retrieval
• 3D image and video retrieval
• WWW visual information searching– efficiency has to be emphasized due to limited
network bandwidth• operate in compressed domain
• visual summarization
• visualization at different levels of resolution
![Page 22: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/22.jpg)
Visual Information Retrieval
• Research directions– tools for automatic extraction of low-level
features– tools for automatic extraction of high-level
semantics– models for representing visual content– effective indexing– effective database models
![Page 23: Visual Information Retrieval](https://reader035.vdocuments.net/reader035/viewer/2022062421/56812ac2550346895d8e8c58/html5/thumbnails/23.jpg)
Visual Information Retrieval
– visual interfaces• allow querying and browsing
• allow querying by text and visual information
– similarity models• fit human similarity judgement
• psychological similarity models
– Web search– 3D image and video retrieval