IST DIVAS Presentation 1
Advanced search technologies for digital audio-visual content
Divas represents the combined efforts of eight companies and institutions to:
› design and develop a multimedia search engine › based on advanced direct video and audio
search algorithms applied directly on encoded (compressed) content.
IST DIVAS (FP6 IST-2-04582) was officially launched at the 1st of January 2007, with a duration of 24 months.
IST DIVAS Presentation 2
Availability of huge and ever expanding distributed repositories of media in various formats
how can a system efficiently and reliably identify
content fragments captured from various streams?
Only techniques for indexing and searching raw (uncompressed) content are available today (and text-based techniques)
IST DIVAS Presentation 3
› provides the capability to the user to locate captured video and audio feeds with missing additional context like title, filename, origin, location, service provider
etc.
› or in situation where metadata based queries are inapplicable
IST DIVAS Presentation 4
› Search of multimedia libraries using the techniques for uncompressed content is a heavy duty/ costly solution
because for the search each item has to be decompressed.
IST DIVAS Presentation 5
› Metadata annotation may be a heavy duty/costly solution to content owners.
› A complementary solution should therefore also be made available
IST DIVAS Presentation 6
Audio-visual signature/ fingerprint extraction directly from compressed resources
Extend Search TechniquesBy supporting content queries, DIVAS extends the state of the
art beyond nowadays pursued search techniques based on metadata.
Improve the reliability of audio-visual content detection
By its multimodal (video & audio content) approach, and by combining the query results obtained from both modalities.
IST DIVAS Presentation 7
IST DIVAS Presentation 8
DIVAS proposes characterization, feature extraction and direct search of compressed video › as opposed to cognitive-level metadata annotation from
uncompressed video streams.
“Video fingerprinting”, › as envisaged (but not extensively exploited) in the MPEG-7
standard is the term approximately fitting to our approach.
DIVAS will pursue:› Mpeg-2 compliant implementation
› a H.264 compliant implementation.
IST DIVAS Presentation 9
Already a relatively mature technology on uncompressed audio
Based on the extraction of fingerprints, which capture the characteristic features of an audio clip. › These fingerprints are then compared to the
fingerprint of a query (an audio clip to search for).
IST DIVAS Presentation 10
DIVAS system search techniques incorporate in parallel both audio and video based searching.
In terms of functional decomposition the system will address audio and video in a different way.
DIVAS system utilizes two different engines:
› a/generate unique indexes from each clip › b/search among the aforementioned identifiers,
providing a match/no match answer to the user.
IST DIVAS Presentation 11
Open architecture Future-proof design Scalability Interoperability Expandability Modularity
IST DIVAS Presentation 12
IST DIVAS Presentation 13
Functional OverviewFunctional Overview
IST DIVAS Presentation 14
Compressed Audio Signal
Direct conversion into the suitable time/frequency
domain
Feature Extraction
Speech Recognition Music Information Retrieval
Decoding
Conversion to suitable time/frequency domain
DIVASConventional
Tool “A”Content uploading
Content index
Tool “C”Administration Updating
Tool “B”Content search
Result of content search
Indexes Indexes (fingerprints)(fingerprints)
DB DB Writing
Reading
DIVASDIVASENGINEENGINE
Multiplexed
Content
Multiplexed
indexes
Contentdemultiplexer
Indexmultiplexer
Video features
extractionengine
Audio features
extractionengine
Engine of text/meta features
extraction
Video/audio/textcontent
Video/audio/textindexes
Plug-ins Plug-ins
See next slide
Video
content
Video Decoder
(Transcoder)
Featuresextractor
Video
index
Plug-ins Plug-insPlug-ins supporting video fingerprints
Scene change plug-in
Brightness change plug-in
Frame content plug-in
Plug-ins supporting video formats
MPEG2
MPEG4
AVC/H.264
VC1
etc.etc.
Query
content
CONTENT FEATURES
EXTRACTION ENGINE
COMPARISON ENGINE
Query
index
Search
result
Query
indexSearchresult
Index reader
Indexcomparer
Plug-ins
Plug-insIndexes Indexes (fingerprints)(fingerprints)
DBDB
Search
result
Sea
rche
d
inde
x
Read
Query
index
Plug-ins for statistics comparison
Plug-in for scene change statistics comparison
Plug-in for brightness statistics comparison
Plug-in for time characteristics statistics comparison
Plug-in for average brightness and contrast statistics
comparison
Query
index
Monitoring resultIndexcomparer
CONTENT FEATURES
EXTRACTION ENGINE
Content stream reader
Plug-insContent Content streamstream
Monitored
content
Inde
x
Read
Query
index
Plug-ins of content stream reading
Plug-ins of read stream from capture devices
Plug-ins of read stream from remote source