cs257 modelling multimedia information lecture 2 dr lee gillam

CS257 Modelling Multimedia InformationCS257 Modelling Multimedia Information

LECTURE 2LECTURE 2

Dr Lee GillamDr Lee Gillam

Learning OutcomesLearning OutcomesAt the end of the module students should be able to: • 1. Apply a standard metadata set to describe media items in an archive• 2. Apply appropriate techniques for modelling text documents• 3. Compare and apply different kinds of metadata for image data• 4. Distinguish and apply appropriate data models for temporal media

(video/audio)• 5. Describe video content formally, in terms of objects, events and temporal

relationships between events• 6. Design and implement a synchronised multimedia presentation including a

variety of media types and temporal relationships• 7. Explain the theory behind hypermedia systems and assess how this is put into

practice in current hypermedia applications• 8. Explain, contrast and evaluate the modelling of multimedia information to

support media access, especially retrieval and browsing, in a variety of applications - these include personal media collections, organisation-wide media archives and web-based search engines

Overview of LECTURE 2Overview of LECTURE 2

• Part 1: Multimedia Data– How do we describe images?– Multimedia and the “information explosion”

• What multimedia encodings are there?

• Part 2: Metadata– Why do we need metadata standards?

• Interoperability between people and systems

– What are the issues and challenges in making and using metadata standards?

• Describing multimedia at different levels of abstraction; adding metadata at different stages of the media production process

• Manual creation (cost) / automation; formalisation / restrictiveness • Objectivity / subjectivity – metadata is context-sensitive, time-

variant, culturally variant

LECTURE 2:LECTURE 2:LEARNING OUTCOMESLEARNING OUTCOMES

• After the lecture and lab exercise, you should be able to:– Demonstrate an awareness of issues and challenges

for creating metadata– Explain why we need metadata standards, and

explain how different kinds of metadata interoperability are required in different applications

– Give a reasoned argument for choosing one multimedia data coding format over another, for a given application

Part 1: Multimedia dataPart 1: Multimedia data

Multimedia data generally produced by humans for humans.

• How do we describe images?– What do people doing with images?– Differences between seeing and understanding– What can we express, and how?– How much information is there - multimedia and the “information

explosion”?

Engaging?Engaging?

• Which languages do you speak?– English, French, German, Mandarin Chinese,

Portuguese, Arabic, Spanish, Urdu, Gujurati

• What country are you from?– England, Wales, Portugal, Saudi Arabia

• What countries have you visited?

Set Exercise 1-1Set Exercise 1-1

Guildford art gallery has digitised its collection of paintings and wants to make them available for searching online.

What metadata would you specify for:1) The general public to search for paintings?

1) Artist name [characters – subfields: surname, given name, other names], Gallery found in [city, country], Date/time [day, month, year, hours, minutes, seconds]

2) Art experts to search for paintings?1) Medium (e.g. canvas), Materials (oil, watercolours),

Genre (renaissance, impressionists)


1) The general public to search for paintings?

2) Art experts to search for paintings?


• Madonna with baby?

PaintedLeonardo da Vinci

Born near

FlorenceCountry

Italy

Painted

Mona Lisa


Related work: The CLEF Cross Language Image Retrieval Track (ImageCLEF)

28,133 images and descriptions from the photographic collection provided by St Andrews University Library



JV-.087636

London. St Margaret's Church [Westminster]. Raleigh Window looking west.

St Margaret's Church, London.

Church interior with recessed gothic arches, and stained glass window in end wall; bible on eagle lectern left, pulpit right.

Registered 1923

J Valentine & Co

London, England

JV-87636 pc/jf/mbTECH: Real Photograph.

[arches unclassified],[London all views],[stained glass unclassified],[churches & chapels],[buildings - stone],[architecture - Gothic],[Collection - J Valentine & Co]



<DOC>

<DOCNO>stand03_2065/stand03_22080.txt</DOCNO>

<HEADLINE>London. St Margaret's Church [Westminster]. Raleigh Window looking west.</HEADLINE>

<RECORD_ID>JV-.087636</RECORD_ID>

<DESCRIPTION>St Margaret's Church, London. Church interior with recessed gothic arches, and stained glass window in end wall; bible on eagle lectern left, pulpit right. Registered 1923 J Valentine and Co London, England JV-87636 pc/jf/mbTECH: Real Photograph.</DESCRIPTION>

<CATEGORIES>[arches unclassified],[London all views],[stained glass unclassified],[churches and chapels],[buildings - stone],[architecture - Gothic],[Collection - J Valentine and Co]</CATEGORIES>

<SMALL_IMG>stand03_2065/stand03_22080.jpg</SMALL_IMG>

<LARGE_IMG>stand03_2065/stand03_22080_big.jpg</LARGE_IMG>

</DOC>



Men and women processing fish

A baby in a pram

Picture postcard views of St Andrews

Seating inside a church

Woodland scenes

Scottish marching bands

Home guard on parade during World War II

Tea rooms by the seaside

Fishermen by the photographer Adamson

Ships on the river Clyde

Portraits of Mary Queen of Scots

North Street St Andrews

War memorials in the shape of a cross

Boats on Loch Lomond

Tay bridge rail disaster

Great Yarmouth beach

Metal railway bridges

Culross abbey

Road bridges

Animals by the photographer Lady Henrietta Gilmour

Ruined castles in England

London bridge

Damage due to war

Golf course bunkers

Portraits of Robert Burns


Golf course bunkersGolfplatz Bunker Bunkers de terrain de golfe Un bunker in un percorso di golf ; Bunkers in un

campo di golf Búnkers en un campo de golf ; Pista de golf Bunkers op een golfbaanA relevant image will show a picture of a golf course

in which a bunker can be clearly identified. The picture must be a photograph or a postcard, but not a drawing, e.g. a plan of the golf course. A bunker is a sandy hollow formed by wearing away of the turf, or nowadays an artificial sand-hole with a built-up face. An example relevant document is [stand03_1714/stand03_7020].


Automatic query translation? Golf course shelter

Bunkers of ground of gulf

A bunker in a distance of golf

bunkers in a golf course

B??nkers in a golf course

Track of golf

Bunkers on a wave job

<Description>XRay: next to the diaphysis of the right femur, there is a large soft tissue mass with an

oval ossification of 5 cm in length with multiple punctate calcifications (difficult to see an this digital radiograph). The cortex on the medial side of the distal part of the femoral diaphysis is of irregular outlines.

MRI: demonstrates a large tumour mass of the right thigh which involves the medial and distal portions of the anterior rectus muscle and the vastus medialis. The lesion is relatively well delineated from the subcutaneous soft tissues, but not from the muscle. It is in intimate contact with the cortex of the femoral diaphysis which is slightly thin in its distal portion (on T1, coronal a) The superficial femoral vessels are partially encased by tumour and displaced posteriorly (red arrow on axial view on T2). There is subcutaneous soft tissue oedema and thickening of the adjacent skin. …..

CT of the chest: There are nodules in both lungs and in the right subpleural space consistent with metastases (considering the clinical situation). There are bilateral pleural effusions, larger on the left than on the right, and partial left posterior atelectasis.

CT of the pelvis: demontrates a large metastatic mass consisting of large iliac adenopathies which encase the iliac vessels which however remain patent.

Macroscopy: firm, elastic, relatively homogeneous, whitish mass with an ill defined ossified centre of about 5cm in length (arrow on macroscopy 2).

Microscopy: there is proliferation of epitheloid cells with abundant cytoplasm and moderately atypical cyto-nuclear features (microscopy 3), separated by strands of collagen (stained blue on microscopy2). Small foci of tumour necrosis …..

</Description><Diagnosis>Epitheliod Fibrosarcoma</Diagnosis>….<ClinicalPresentation> ; <Commentary>; <KeyWords> ;<Anatomy> ; <Hospital> ;

<Department> ; <Date> ; <Language> ; <Birthdate> ; <Age>

<Creation> ; <DateTime> ; …

Describing imagesDescribing images

• In groups of 2 or 3, provide 4 keywords (single words, or phrases) to describe what you see in each of these signs:


• 40

• Circle

• Red

• White


• Circle

• Red

• White

• Measurements

• Triangles


• Red

• White

• Triangle

• Car

• Wavy lines


• Yellow

• Black

• Red

• Two arrows


• Rectangle


• What is common?– Colours: Red White Yellow Blue– Shapes: Triangle Circle Rectangle

• What differs?– “Content”

Understanding imagesUnderstanding images

• In groups of 2 or 3, provide 4 keywords (single words, or phrases) to describe what you understand about each of these signs:


• 40 miles per hour speed limit (UK): kilometres (France)


• Height restriction (imperial and metric)


• Slippery road warning (warning)


• The two right-hand lanes are closing within 800 yards, and you need to move into the two left hand lanes


• Motorway (M62) - information


• What is common?

• What differs?

Understanding images?Understanding images?

• Understanding vs. seeing - easier or more difficult?

• What did you agree on? Disagree on?• What classes exist?

– Types: Signs giving orders; Warning signs; Direction signs; Information signs; Road works signs

• “Why are you asking us to do all of this?”


• What’s on a DVD?– “moving picture”– subtitles– audio description– dubbed speech– scene indexing– bibliographic

information– …..

Image from: http://www.ee.columbia.edu/~winston/research/trecvid2003/


• What’s beyond the DVD?– catalogues– reviews & reviewers– recommenders– …..

Image from: http://www.ee.columbia.edu/~winston/research/trecvid2003/

Recall from Lecture 1:Recall from Lecture 1:

• It’s becoming easier and easier to create, distribute and ‘view’ multimedia data

BUT• As the amount of multimedia data

increases, it becomes harder to find what you want – “information explosion”

• Multimedia information only has value if it can be found (retrieved) and used

““information explosioninformation explosion””

• Data at the exascale - a few new exabytes (billion GB) of unique information / year – According to research at Berkeley, print, film, magnetic,

and optical storage media - about 5 exabytes of new information in 2002; equivalent to 37,000 libraries the size of the Library of Congress book collections (29+m books)

– New Media and ever-larger volumes of information:• Written: 24-hour news coverage published on news-wires and

more recently using Really Simple Syndication (RSS)• Spoken: digital radio and on-demand podcasts; continuous

televisual coverage through the variety of available digital television channels; broadband television efforts such as IET.tv, and on-demand programme schedules referred to as “timeshift TV”; CCTV


• Multimedia data coding formats:– Text: TXT, PDF, DOC, HTML ….– Image: .JPEG , GIF ….– Audio: MP3 , WAV, MIDI …. – Video: .MPEG , AVI ….

• Some formats emphasise compression• Some formats enable flexible editing /

interaction with the multimedia content


• Average digital photograph in 24-bit colour uncompressed requires about 6Mb of storage

• 1 second of digital video at 640x480 pixels uncompressed requires 27Mb; 1 hour requires 97Gb– multiply by 4 for full-screen video (1.23m pixels)!

[Dunckley 2003: p. 26; p. 37]

• HDTV ~ 2 million pixels (1920x1080)?


• Lossless Compression – a compression technique is lossless when no information is lost after compression-decompression

• Lossy Compression – some information is lost after compression-decompression: depends how much loss can be tolerated

[Dunckley 2003: p. 31]• Transcoding (moving between formats) may

introduce loss also• Format can impact on data handling


• The compression of image and video data can take advantage of the fact that some changes to the data (i.e. loss of detail) are either imperceptible to the human eye, or result in a loss of perceived quality that can be tolerated for a certain application

• There are trade-offs between:– Image quality and compression ratio– Codec processing time and compression ratio

• 4Mb before compression, 1Mb after = 4:1 compression ratio; • codec = compression-decompression

• Preferable to have compression techniques where users can set parameters for compression ratio / loss of quality / codec processing time


• Further detail on compression, e.g.:– Run-length encoding:2211333334333222 22 21 53 14 33 3216 characters becomes 12 characters– Huffman coding:12334123343212334 AABA17 characters becomes 4 characters plus ‘map’

12334 = A; 32 = B[Dunckley 2003: p. 32]


• Catalogues– 0 GENERALITIES

– 1 PHILOSOPHY. PSYCHOLOGY

– 2 RELIGION. THEOLOGY

– 3 SOCIAL SCIENCES

– 4 VACANT

– 5 NATURAL SCIENCES

– 6 TECHNOLOGY

– 7 THE ARTS

– 8 LANGUAGE. LINGUISTICS. LITERATURE

– 9 GEOGRAPHY. BIOGRAPHY. HISTORY

(Around 60k categories)

Catalogues elsewhere?

Set Exercise 1-2:Set Exercise 1-2:what is metadata?what is metadata?

Skim read* the following web pages and extract / summarise 2-3 definitions of what metadata is and 3-4 reasons why people say metadata is important.

http://dublincore.org/documents/usageguide/#whatismetadata

www.tasi.ac.uk/advice/delivering/metadata.htmlwww.openp2p.com/pub/a/p2p/2001/01/18/metadata.html

*You do NOT need to read these articles in detail

Part 2: MetadataPart 2: Metadata

Multimedia data coding formats represent the multimedia content to reproduce it, metadata represents information

about the content

• What are the issues and challenges in making and using metadata standards?– Describing multimedia at different levels of abstraction; adding

metadata at different stages of the media production process– Manual creation (cost) / automation; formalisation /

restrictiveness – Objectivity / subjectivity – metadata is context-sensitive, time-

variant, culturally variant

What is metadata?What is metadata?26

26 what?

26 days

Full days or working days?

• What assumptions have been made in this exchange?

Web content by language, 2001

English68%

Spanish2%

Russian2%

French3%

Chinese4%

German6%

Japanese6%

Italian2%

Portuguese1%

Korean1%

Other5%

English

Japanese

German

Chinese

French

Spanish

Russian

Italian

Portuguese

Korean

Other

Sources: Various

Why do we need Why do we need metadata metadata standardsstandards??

• Short answer – interoperability• Longer answer - interoperability between people,

services and systems, including the automatic composition of computer programs for undertaking novel tasks

“At least, Pat should be able to use Pat’s metadata. Slightly better, Chris should be able to use Pat’s metadata. Even better, Chris’s computer should be able to use Pat’s metadata. At best, Chris’s

computer and Chris should be able to use Pat’s and Pat’s computer’s metadata”

Adapted from Davis (1995) – Davis wrote ‘annotations’ rather than ‘metadata’.

Why do we need Why do we need metadata metadata standardsstandards??

• The Semantic Web: interoperability between people, services and systems. Increased coverage for XML content: XML meant for machines and machines like precision

• “Standardized metadata” – Enable convergence of formats, systems and collections to

improve reusability / interoperability– Systemic efforts in ISO

• Standards-conformant metadata can improve management of data and facilitate scaling up of research.

• Research results identified using finer-grained metadata become available for use beyond that originally intended, ethical considerations allowing.

• The Multilingual Internet? - September 2005, ICANN approved the first “ccTLD” for a particular human language and culture: ‘.cat’ (Catalan): not just ASCII?

What is metadata?What is metadata?<p>

<p>?

??

<para>

An unknown or unexpected error

has occurred.

Metadata in HTMLMetadata in HTML

• Keywords (search engines?):– <meta name="keywords" content=“Multimedia,

Metadata">

• Description:– <meta name="description" content=“Lectures on

Multimedia">

• Refresh every 10 seconds:– <meta http-equiv="refresh" content=“10">

• Retire your content– <meta http-equiv=“expires" content=“Tues, 23 Jan

2007 09:00:00 GMT">


• Simple standards for metadata are “incorporated” into HTML and used in HTTP headers (ISO 639 / ISO 3166 => IETF RFC 3066), used at document level.

• With hundreds and thousands of codes (to 400 x 250), and combinations, to choose from, users should assume nothing:– Naïve users, or lazy users, might assume that AF stands for “African”,

rather than Afrikaans. Or they may assume it is a two-letter country code for Afghanistan.

– Country code “CS”. Was Czechslovakia, has its own, heavily used, ccTLD that was deleted after 1993. Became ISO 3166-1 code for Serbia and Montenegro. S&M have now split from their union. How should users interpret information tagged with CS? How should retrieval systems be coded to cope with such infrequent but significant changes?

– If your destination is Genoa, Italy, beware of baggage handlers inferring from the GOA tag that your bags should be sent to India!


• Do search engines pay any attention?– searchenginewatch.com

• Does Google index more than first 101k of text?

• Is there a limit for other search engines?

• Is attention paid to important information (e.g. <h1>)

• If there are limits now, why bother?

Interoperability requirements of Interoperability requirements of different applicationsdifferent applications

Personal media collections– The metadata is made and used by the same

person, i.e. the owner of the media– But maybe wants friends and family to share


Online music / video-on-demand / online art gallery applications– The users searching for multimedia data are

different from the people making the metadata– The success of the business depends on

users finding (and buying) what they want– Businesses may want to trade media assets:

these days metadata is becoming as much (if not more) of an asset as the rights to the multimedia data itself

Dublin Core Metadata InitiativeDublin Core Metadata Initiative

• A prominent attempt at providing a common framework for multimedia metadata

• Aim is for simplicity and extensibility, in order to ensure general applicability

“The Dublin Core Metadata Initiative is an open forum engaged in the development of interoperable online metadata standards that support a broad range of purposes and business models. DCMI's activities include consensus-driven working groups, global workshops, conferences, standards liaison, and educational efforts to promote widespread acceptance of metadata standards and practices.”

http://dublincore.org

Dublin CoreDublin Core15 metadata elements15 metadata elements

Element: TitleA name given to the resource. Typically, a Title will be a name

by which the resource is formally known.

Element: CreatorAn entity primarily responsible for making the content of the

resource. Examples of a Creator include a person, an organisation, or a service.

Element: SubjectThe topic of the content of the resource. Typically, a Subject

will be expressed as keywords, key phrases or classification codes that describe a topic of the resource. Recommended best practice is to select a value from a controlled vocabulary or formal classification scheme.

Dublin CoreDublin Core15 metadata elements15 metadata elements

Element: DescriptionAn account of the content of the resource. Description may

include but is not limited to: an abstract, table of contents, reference to a graphical representation of content or a free-text account of the content.

Element: PublisherAn entity responsible for making the resource available.

Examples of a Publisher include a person, an organisation, or a service.

Element: Contributor; Element: Date; Element: Type; Element: Format; Element: Identifier; Element: Source; Element: Language; Element: Relation; Element: Coverage; Element: Rights

Use of Dublin Core MetadataUse of Dublin Core Metadata

• Dublin Core can be applied to any kind of media object, including text and images: it has been applied in a wide variety of online digital library projects – see http://dublincore.org/projects/

• It can also be used as a metadata set for building your own digital library in using freeware such as the Greenstone digital library software:

http://www.greenstone.org/cgi-bin/library

Beyond DCBeyond DC

• ISO 639: Language Codes

• ISO 3166: Country Codes

• ISO 8061: Date and Time formats

• ISO 11179: Metadata Registries

• ISO 12620: Terminology Data Categories

• Related: RDF, RDFS, OWL, TEI, …

Beyond DCBeyond DC

Beyond DCBeyond DC

Image from Google MapsWGS 84

51.243566-0.589576

What are the issues and challenges in making and

using metadata standards?


Automatic Media Content Production– Applications are being developed to (semi-)automate

the production of multimedia content, e.g. automatically authoring a PowerPoint presentation on a given topic

– Such applications will rely on existing metadata to find relevant content for the applications

– NB. the efforts of the Semantic Web community to enable applications to find/reuse media content on the Internet – “metadata harvesting”: cost of manually creating metadata means that machines will have an increasingly important role in automating the process

• Multimedia information needs to be described and dealt with at different levels of abstraction…

Levels of abstractionLevels of abstraction

Meanings… ScenesActions Movements People / ObjectsRegions (colour, shape, texture, motion) PixelsElectronic bits

STOPPED HERE

Metadata and the process of Metadata and the process of media productionmedia production

• The metadata problem may be reduced if the creation of metadata is integrated into the media production process– Digital cameras that automatically add

time/date/location information to images (e.g. Kodak); many users do not set date/time

– MPEG-7 cameras that add shot details to video data – Media asset managers who are part of the media

production team in organisations like the BBC

Humans vs MachinesHumans vs Machines

• The manual creation of metadata is expensive, e.g. 45 minutes to index one painting in an online art gallery– However it may still be economically viable, especially as trained

humans can currently produce better metadata than machines – For many scenarios, especially multimedia data on the Internet,

automatically created metadata is becoming increasingly important

• If machines are to be metadata users then the metadata descriptions have to be more formalised than for human searchers – this formalisation can restrict what can be captured by the description

Objectivity / SubjectivityObjectivity / Subjectivity

Metadata is:

context-sensitive;

time-variant;

culturally-variant.

Generic or Specific Generic or Specific approaches to Metadata?approaches to Metadata?

• Standards such as the Dublin Core allow the same set of metadata elements to be used to describe / index different kinds of multimedia data potentially metadata about texts, images,

videos, etc. stored in one Table in a relational database

is this sufficient?

Images and MetadataImages and Metadata

• Related work in the Department – NHS images– Scene of Crime Images: SOCIS– Crime CCTV Images: REVEAL– Languages and metadata: SALT, LIRICS

Choosing a Coding Format: Choosing a Coding Format: for a documentfor a document

• Imagine you were digitising pages of an old book to make a library’s content available on-line (cf. recent Google project): you could…– Take a photo / scan each page as an image. It would

reproduce the page perfectly to look at.

– You could type in the words of each page and save as a .txt file. Now users can easily search for character strings (i.e. words) that there are interested in

– You could use HTML to add formatting (bold / font size) and to link different parts of the book to one another. Now users can browse between related sections.

Coding Formats Coding Formats for a documentfor a document

• This is NOT to say that any of the three options is better than the others: rather that it is important to consider the intended functionality / user requirements when choosing.

• If you were digitising old parchments it might be preferable to photograph them to keep the original appearance – or ideally combine the image with a TXT or HTML file

• What metadata would be useful?

Coding Formats Coding Formats for musicfor music• The sampled waveform probably captures more

subtlety of a real-life musical performance than a .midi file ever could (a little bit like an image of an old parchment versus a text file containing the same words)

• However, doing music retrieval based on melodies would probably be much easier on .midi files

• (There is a demand for ‘query-by-humming’ applications. The Shazam system - www.shazam.com - appears to do exact-match retrieval very well based on sampled waveforms, but searching for ‘similar’ tunes may be harder).

• Music retrieval via musical scores / notation?

Optional ReadingOptional Reading

For text-book coverage of multimedia data and metadata, available in the Library:

Dunckley (2003), Multimedia Databases: an object-relational approach. Chapter 7 – especially Sections 7.1, 7.4.2 and 7.4.3, and a skim-read of Chapter 2

For a recent article published in industry/research journal:Van Ossenbruggen, Nack and Hardman (2004), ‘That Obscure Object

of Desire: Multimedia Metadata on the Web, Part 1’, IEEE Multimedia, Oct-Dec 2004, pp. 38-48.

The IEEE Multimedia journal can be accessed online via the Library’s eJournals service

cs257 modelling multimedia information lecture 2 dr lee gillam

Documents

metadata standards

multimedia encodings

standard metadata

variant slide

multimedia data coding

media items

media access

different applications