c14 fiatifta dubai 2013, the mpeg-7 audiovisual description profile standard for describing...

#FIATIFTADubai2013 Mike Matton, Peter Schallauer: “MPEG-7 AVDP”

“THE MPEG-7 AUDIOVISUAL DESCRIPTION PROFILE STANDARD FOR DESCRIBING RESULTS OF AUTOMATIC ANNOTATION SERVICES” Mike Matton, Werner Bailer, Masanori Sano, Alberto Messina, Jean-Pierre Évain, Peter Schallauer Vlaamse Radio- en Televisieomroeporganisatie, Belgium JOANNEUM RESEARCH, Austria Get in contact: [email protected]; [email protected]


Copyright © of this presenta1on is the property of the author(s). FIAT/IFTA is granted permission to reproduce copies of this work for purposes relevant to the above conference and future communica1on by FIAT/IFTA without limita1on, provided that the author(s), source and copyright no1ce are included in each copy. For other uses, including extended quota1on, please contact the author(s).


Contents

•  Introduction & motivation •  MPEG-7 AVDP •  Applications of AVDP •  Conclusion


MPEG-7 Audiovisual Description Profile

•  Motivated by activities of EBU MIM/SCAIE group –  Study and evaluation of Content analysis-based Automatic Information

Extraction tools in media production –  facilitate the introduction of new production tools based on automatic

extraction of information •  Relevant tools include

•  Speech Recognition •  Audiovisual Segmentation •  Personality Identification •  Semantic Detection •  Text Recognition •  Subject Classification

•  Format Detection"•  Named Entities Detection"•  Copy / Repetition Detection"•  Content Summarisation"•  Keyword Extraction"•  Automatic text Translation"


MPEG-7 Audiovisual Description Profile

•  Automatic analysis tools generate –  detailed time-based metadata –  low-level audio/video features (e.g., color descriptors)

•  Common metadata format is needed to –  represent results –  exchange between organisations –  perform automated evaluation


MPEG-7 AVDP Motivation (TOSCA-MP project)

•  Put automatic annotation in automated workflow •  Services need to communicate with one another

(e.g. Scene segmentation needs input from shot boundary detection) –  Even across vendors –  MPEG-7 AVDP specifically created for this purpose –  Different such services later in this presentation


MPEG-7

•  Multimedia Content Description Interface –  ISO/IEC 15938, version 1 (2001) and 2 (2004)

•  Metadata of multimedia content –  descriptors, description schemes –  definition based on an extension of XML Schema –  serialised in XML or binary format

•  Multimedia description schemes (part 5) –  content structure (decompositions, segments) –  descriptive and technical metadata

•  Visual and Audio (parts 3 and 4) –  descriptors for visual and audio low-level features

•  use of parts of MPEG-7 for TV Anytime, EBU Classification Schemes


MPEG-7 Pros and Cons

•  Support for describing arbitrary content structures –  time segments, regions, moving regions

•  Support for low-level feature descriptors •  Benefits of MPEG-7

–  designed as a data model, not just as an exchange format –  broad range of applications –  flexible, fine-grained description –  content structuring capabilities

•  Drawbacks –  complexity: generic types, deep hierarchical structures –  interoperability problems: several ways to describe the same, conformance to

standard can only be checked on syntactic level but not on a semantic level


Why profiles for MPEG-7?

•  Address issues of complexity and interoperability

•  Profiles are a proven concept – subsets of the standard for certain

functionalities and/or applications – cf. MPEG-2, MPEG-4


Why profiles for MPEG-7?

•  Definition of an MPEG-7 profile – Selection of description tools

•  which description schemes and descriptors may be used – Constraints on description tools

•  restrictions on cardinalities, child elements, attributes – Semantic constraints

•  semantics of the elements in the description •  specify use of the selected tools in the context of the

profile •  ensure interoperability

→ Profile XML Schema

→ Profile XML Schema

→ Textual description


Source used for defining the new profile

•  Detailed Audiovisual Profile (DAVP) –  single multimedia content entities –  comprehensive structural description of the content –  audio and visual low-level feature descriptors (parts 3 and 4) –  defines semantic constraints on structure and elements to address

interoperability issues •  NHK Metadata Production Framework (MPF)

–  based on the Core Description Profile (CDP) –  addresses the complexity and ambiguity problems of MPEG-7 –  audio and visual low-level feature descriptors (parts 3 and 4) –  semantic constraints for the structure of the description as well as several

syntactic and semantic constraints on different elements


Design goals for a new profile

•  Fulfill the requirements of EBU MIM/SCAIE •  Harmonise DAVP and MPF

–  both share many design principles –  some implementation choices are different

•  Simplify –  based on experience from using DAVP and MPF –  some included elements are never used –  some constructs can be simplified in terms of cardinality/flexibility

•  First profile based on MPEG-7 v2 (2004)


AudioVisual Description "Profile – AVDP •  An MPEG-7 AVDP file contains

–  0 or more descriptions of "audiovisual contents

–  0 or 1 summaries of the content(s) •  Describe audio, video or "

audiovisual content –  no other types of multimedia "

content –  no collections of content –  same top-level structure for audio, "

video or audiovisual content

Mpeg7

Description type=“ContentEntityType”

Figure 1 Top level structure

Mpeg7

Description type=“SummaryDescriptionType

Mpeg7






(a) A ContentEntityType for one audiovisual content.

(b) Several ContentEntityTypes for copy detection result.

(d) A SummaryDescription with several ContentEntityTypes.

(c) A SummaryDescription.

Mpeg7

Description type=“SummaryDescriptionType


AVDP content description structure


Constraint definition

•  Profile specification document defines –  list of included types –  constraints on elements and cardinalities –  semantic constraints on use of these types –  semantics of description structure

•  Profile XML schema –  included types –  excluded elements, restrictions on cardinality –  few of the semantic and structure constraints


Examples

•  Examples provided with AVDP specification –  General description structure –  Content segmentation –  Speech-to-text –  Copy detection –  Summarization

•  Annex –  example outputs from RAI and JRS content analysis tools –  MIM/MDN Knowledge base

•  http://workspace.ebu.ch/display/ecmmdn/Knowledge+base+-+contributions

•  EBU MIM/SCAIE is currently working on a guidelines document


Validation

•  Validation against MPEG-7 XML Schema or Profile XML Schema –  e.g., using validator of an XML editor, Xerces, MS XML, etc.

•  many constraints of a profile cannot be represented in XML schema –  e.g., the AudioVisual element, representing the entire content (root

a/v segment) is of type AudiovisualSegmentType –  the same type is used to represent many audiovisual segments at

other places in the description –  the specific constraints on the elements that need to be present on

the root a/v segment but not on others cannot be checked


AVDP Applications


VAMP: Validation of "MPEG-7 Profile Descriptions (including AVDP)

•  validate metadata documents –  conforming to complex metadata standards such as MPEG-7 –  semantics of temporal annotations –  beyond XML schema validation

•  prevent interoperability problems •  http://vamp.joanneum.at

•  Further MPEG-7 ressources •  http://mpeg7.joanneum.at/


VidiCert: AVDP for Video Essence QC Metadata •  Video Essence QC

•  Automatic Analysis + Interactice Verification

Request

Automatic Analysis Interactive Verification

Analysis Profile Setup (once) Video / Movie

QualityReport Verified

QualityReport

Storage

Movie, Video

AVDP AVDP


AVDP for "Automatic Video Essence Quality Analysis •  Metadata in VidiCert Analyser

–  Technical MD –  Structural Video MD –  Essence Defects MD

•  Video Breakup (major video disruptions) •  Tape Dropouts (e.g. Digital BETACAM™) •  Monochrome Frames •  Test Pattern •  Silence •  Dolby®E™

–  Essence Quality Measures •  Noise/Grain •  Blurriness

–  All metadata are AVDP compliant –  More at www.vidicert.com


AVDP for "Interactive Essence Quality Verification

•  Metadata in VidiCert Summary –  Timeline based summarisation

metadata views of automatic analysis results

–  Synchronised navigation in all metadata views

–  Efficient human quality judgement by defect severity based inspection

–  Overall rating metadata support for multi-stage QC

–  Optional metadata filtering (verified vs. non-verified) before final export

–  All metadata are AVDP compliant –  More at www.vidicert.com


AVDP for "Interactive Essence Quality Verification

22


AVDP for Web based metadata visualisation


NHK Metadata Production Framework

!

hJp://www.nhk.or.jp/strl/mpf/english/index.htm


Conclusion

•  AVDP new metadata standard for automated information extraction

•  AVDP enables metadata extraction service interoperability

•  Several tools working with AVDP are already available


Acknowledgements

hJp://david-‐preserva1on.eu

hJp://www.tosca-‐mp.eu


“THE MPEG-7 AUDIOVISUAL DESCRIPTION PROFILE STANDARD FOR DESCRIBING RESULTS OF AUTOMATIC ANNOTATION SERVICES” Mike Matton, Werner Bailer, Masanori Sano, Alberto Messina, Jean-Pierre Évain, Peter Schallauer Vlaamse Radio- en Televisieomroeporganisatie, Belgium JOANNEUM RESEARCH, Austria Get in contact: [email protected]; [email protected]


Copyright © of this presenta1on is the property of the author(s). FIAT/IFTA is granted permission to reproduce copies of this work for purposes relevant to the above conference and future communica1on by FIAT/IFTA without limita1on, provided that the author(s), source and copyright no1ce are included in each copy. For other uses, including extended quota1on, please contact the author(s).

c14 fiatifta dubai 2013, the mpeg-7 audiovisual description profile standard for describing...

Technology