using audio description text for shot-by-shot indexing of films  faculté des arts et des...

Using audio description textfor shot-by-shot indexing of films

QuickTime™ et undécompresseur TIFF (non compressé)

sont requis pour visionner cette image.

Faculté des arts et des sciencesÉcole de bibliothéconomie et des sciences de l’information

James M TurnerUniversité de Montréal

Suzanne MathieuVille de Montréal

Toronto 2007.06.30

Outline

• Context• The E-Inclusion Research Network• Project 3.1• Types of information in audio description• Audio description and image description• Why this is good indexing• Future work

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Context

• General research problem: indexing shots by recycling text produced for other reasons, e.g.• subtitles• audio description• production scripts• editors’ logs• camera reports• many others

• Perspective: information science

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Other aspects

IndexingmovingimagesW3C

standards

open access

multiplelanguages

webenvironment

webtools

minimalhuman

intervention

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Previous work

• Determining the subject of still & moving images (PhD thesis, 1994)

• Comparing user-assigned terms with indexers’ terms for the same shots (1995)

• Audio description as a tool for indexing moving images (1998)

• Using shooting scripts for indexing moving images (2005)

• Using ancillary text to index web objects (2006)

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

The E-InclusionResearch Network

• http://e-inclusion.crim.ca• Funded mainly by Canadian Heritage• Partnerships with ÉTS, McGill, UdeM, Laval• Also CNIB, NFB, AudioVision, a dozen others• 2005-2007:

• access to audiovisual material for deaf/hearing loss

• access to audiovisual material for blind/vision loss

• Funding newly obtained for 2007-2009

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Mostly low-level approaches

QuickTime™ et undécompresseur TIFF (non compressé)

sont requis pour visionner cette image.

• shot detection• face recognition• voice recognition• voice synthesis

• High-level work: text manipulation

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Project 3.1

• Audio description for films and television• Background: new CRTC regulations for

digital channels• As a pilot project, the NFB describing about

200 films (many already online)

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Objectives

• Validate typology of information elements• Compare types of information in description

to user needs• Compare English text with French• Recommendations/guidelines for describers

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Method

• Analysis of 11 productions using method developed in previous projects• identify individual shots• transcribe audio description text• relate it to corresponding shots• write shot descriptions before viewing with

sound

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Productions analysed

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Shots and audio description

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Types of information

• Typology worked out in first audio description project

• Refined in second project (minimal changes)

• Refined again this project (minimal changes)

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Typology with examplesTypologie Turner & Colinet 2005 Description Exemples

Action Action Action, en mouvement, qui agit Amélie – « Elle fait chanter un verre en cristal. » (0.02:16)

Attitude Information about attitude of characters

Information sur l’attitude des personnages, qui reflète un état d’esprit

Amélie – « Cligne souvent de l'œil sous ses lunettes sévères. » (0.04:15)

Décor Decor L‘ensemble des éléments qui représentent les lieux où se passe une action (GDT)

La vie – « Dans la cuisine. » (00.59:46)

Éclairage Lighting Information sur la lumière, l’éclairage naturel ou artificiel dans le plan

Amélie – « Sous un soleil jaune d’or… » (00.00:48)

Espace Spatial relationships between characters

Relation spatiale entre les personnages Amélie – « Tous deux peignent côte à côte. » (00.26:00)

Expression Facial and corporal expressions

Expression faciale et/ou corporelle des personnages, signes physiques apparents

La vie – « Josette sidérée remet son voile. » (00.24:50)

Habillement Clothing Ensemble de tous les vêtements et accessoires qui couvrent, protègent et ornent le corps des personnages (GDT)

La vie – « Elle porte un bonnet de laine blanc, un anorak et des gants verts. » (01.13:03)

Météo Weather Indication météorologique Sécurité – « Il fait soleil dans le quartier agréable. » (00.00:22)

Mouvement Movement of the characters

Mouvement, déplacement des personnages La vie – « Le mari se lève. » (00.06:05)

Physique Physical description of the characters

Description physique des personnages, énumération des caractéristiques physiques

Amélie – « Un homme barbu. » (00.47:36)

Proportion Indicators of proportions Indication de proportions, de dimensions N/A

Rôle Occupation, roles of the characters

Occupation, rôle des personnages Amélie – « La vendeuse. » (00.44:49)

Scène Setting Mise en scène, organisation matérielle du plan, emplacement

Amélie – « À table chez son père. » (00.53:29)

Son Description of sound Description du son, des bruits N/A

Temps Temporal indicators Indication temporelle, courte ou longue étendue

Amélie – « Au réveil, il est 4 heures. » (00.55:41)

Texte Textual information included in the image

Information textuelle dans l’image Amélie – « Elle lit: Perdue - sacoche - photos et un numéro de téléphone. » (00.54:58)

Titre Appearance of titles Apparition, ajout de titres, d’étiquettes, etc. Hen Hop – « L’Off ice national du film présente. » (10.00:45)

Générique Credits Information relative au générique Chaise – « Un film de Norman McLaren & Claude Jutra. Sur une musique de Ravi Shankar & Chatur Lal, 1957. » (10.00:26)

Audiovision Audiodescription Information relative à l’audiovision dans le film

Amélie – « L’Association Valentin Haüy a produit l’audiovision de ce film. » (01.53:38)

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Types of information given• Action• Information about

attitudes of characters• Decor• Lighting• Spatial relationships

between characters• Facial and corporal

expressions• Clothing• Weather• Movement of characters

• Physical description of characters

• Indicators of proportions• Occupation, roles of

characters• Setting• Description of sound• Temporal indicators• Text information in image• Appearance of titles• Credits• Audio description

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Types of info in this projectTous les films - Typologie

0,00%

5,00%

10,00%

15,00%

20,00%

25,00%

30,00%

35,00%

40,00%

45,00%

50,00%

ActionAttitudeDécorÉclairageEspaceExpressionHabillementMétéoMouvementPhysiqueProportionRôleScèneSonTempsTexteTitreGénériqueAudiovision

Typologie

PourcentageAnimation

Documentaires

Longs métrages

Tous les films

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Audio description &image description

• Here we want to compare:• keywords found in the audio description text

with• keywords in a written description of each shot

• Objective: find out if one type of text is more fruitful than the other for generating indexing terms

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Orange keywords appear in both

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Analysis for 6 chapters

• Total 238 shots• 50 shots (21%) have no keywords in

common• For the 188 remaining shots, 463 keywords

appear in both types of text

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Observations

• Essential keywords appear in both, other useful keywords in one or the other

• For indexing: one or the other text source ok, but both better

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Improve this performance

• If queries got filtered through a thesaurus, synonyms could also be searched in the text

• This should improve performance• However, performance is already quite

good

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Why this is good indexing

• Interindexer consistency studies: the success rate is only about 50%

• Our own studies: art pictures require special knowledge to index, but everyday pictures do not

• User indexing, social tagging à la flickr, YouTube, MySpace is widespread

• Some information science studies on this

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Other aspects

• Since indexing by humans is so expensive, shot-level indexing will only happen if it is automated

• Exceptions: stockshot libraries, tv newsrooms

• Where we need to invest: automatically identify indexing terms, tag them, attach them to shots

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

Future work

• E-Inclusion funding renewed for 2007-2009• Identify levels of description, types of

information with XML tags, so users can choose

• Assess to what degree audio description could be automated using existing text, voice recognition, gesture analysis, facial expressions, and so on

Context • E-Inclusion • Project 3.1 • Types of info • Types of text • Good indexing • Future

The Endjames.turner@umontreal.ca