semantic mastering: content adaptation in the creative drama production workflow

Download Semantic Mastering: content adaptation in the creative drama production workflow

Post on 26-Aug-2016




3 download

Embed Size (px)


  • Multimed Tools Appl (2012) 59:307340DOI 10.1007/s11042-010-0710-0

    Semantic Mastering: content adaptation in the creativedrama production workflow

    Dieter Van Rijsselbergen Chris Poppe Maarten Verwaest Erik Mannens Rik Van de Walle

    Published online: 8 January 2011 Springer Science+Business Media, LLC 2011

    Abstract In order to provide audiences with a proper universal multimedia expe-rience, all classes of media consumption devices, from high definition displays tomobile media players, must receive a product that is not only adapted to their capa-bilities and usage environments, but also conveys the semantics and cinematographybehind the narrative in an optimal way. This paper introduces a semantic videoadaptation system that incorporates the media adaptation process in the center ofthe drama production process. Producers, directors and other creative staff instructthe semantic adaptation system using common cinematographic terminology and vo-cabulary, thereby seamlessly extending the drama production process into the realmof content adaptation. The multitude of production metadata obtained from varioussteps in the production process provides a valuable context of narrative semanticsthat is exploited by the adaptation process. As such, high definition imagery can beintelligently adapted to smaller resolutions while optimally fulfilling the filmmakersdramatic intentions with respect to the original narrative and obeying various rulesof cinematographic grammar.

    Keywords Semantic adaptation UMA Universal multimedia experiences Cinematography Drama production

    1 Introduction

    Viewers today have access to a multitude of platforms for the consumption of anever increasing supply of broadcast audiovisual media: from movie theaters and

    D. Van Rijsselbergen (B) C. Poppe E. Mannens R. Van de WalleDepartment of Electronics and Information Systems (ELIS)Multimedia Lab,Ghent UniversityIBBT, Gaston Crommenlaan 8/201, 9050 Ghent, Belgiume-mail:

    M. VerwaestVRT-medialab, Gaston Crommenlaan 10/101, 9050 Ghent, Belgium

  • 308 Multimed Tools Appl (2012) 59:307340

    high definition-enabled or standard definition television screens at home, to smartcellular phones on the road, and online using the web. These platforms vary widelyin capabilities and specifications. Content producers and providers are challenged toprovide this entire range of different devices with proper access to large quantities ofaudiovisual products. In practice, this implies that content must be altered to meetthe limitations of a users terminal and network, essentially realizing the promise ofUniversal Multimedia Access (UMA) [37].

    Considering the increasing prevalence of high definition presentation devices,we expect content creators to start actively focusing more on wide cinematic-likeframing. However, this presents issues for mobile device viewers. Due to significantlysmaller display surfaces, they can not reasonably be provided with identical, butsimply down-scaled versions of material originally framed with high definitionpresentation in mind. Studies have shown that in order to provide a comfortable userviewing experience, adapted images should be constrained to contain image regionsthat are most meaningful in terms of the content they represent [17]. Adaptationsshould be guided by notions of the semantics and narrative that the content beingadapted conveys. Such semantic adaptations would, for example, crop the pictureto include only a particular story character; or would, more elaborately, pan andscan from one character to another within the imagery recorded for a single originalwide camera shot. Hence, an additional effort is required to present users with aworthwhile multimedia experience, despite possibly limited terminal capabilities,thereby turning plain UMA into true Universal Multimedia Experiences [28].

    Unfortunately, we have found that content adaptation systems described inliterature today perform adaptation processes almost as an afterthought, whenregular media production has finished. Many algorithms have been developed thatanalyze video signals in an attempt to automatically infer possible semanticallyinteresting regions, of which some can be manually annotated by human operators.While many of these systems can operate in a context-agnostic fashion, none of theoriginal production information concerning the semantics associated with the sourcematerial is reused. A rich set of production metadata, whether in paper or electronicform or implicit in the heads of production people, is left unconsidered. Havingthis production metadata available can help us reduce the semantic gap betweenaudiovisual signals and their original narrative and aesthetic intentions. This reducesthe need for computational aesthetics algorithms [14] that can produce incorrectconclusions, and provides a semantically rich context in which content adaptationcan be performed.

    One particular class of media production where semantics significantly influencethe structure and imagery of the final product is drama production, which includessoap operas, prime time quality fiction and motion pictures. In fact, the semanticsof the story drive the entire production process. Narrative and creative decisionsare taken to emphasize specific aspects of this story. We have built an adaptationsystem that includes notions of the adaptation process from early in the productionprocess. This allows directors, producers and other creative staff to decide whichinteresting objects in the video frames should be retained for smaller displays. Thecreative staff is in a better position to make these decisions than automated systemsor operators beyond the production chain would be. After all, cinematography canbe considered a work of art and must be handled carefully when being adapted forvarious output channels. Because drama production involves many creative planningdecisions anyway [40], we let the drama crew define adaptation parameters them-

  • Multimed Tools Appl (2012) 59:307340 309

    selves. However, we are aware that the additional burden placed on the productioncrew should be limited, and such, we have balanced their required efforts and theamount of automated algorithms used to drive the actual adaptation. Essentially,we have constructed a system that is interactive and provides essential elements ofintelligence.

    In the following section, we define the functionality of our semantic adaptationprocess and describe how it can be incorporated into the existing drama mediaproduction workflow. An overview of the related work concerning spatial videoadaptation is presented in Section 3. Section 4 explains the concepts behind oursemantic adaptation system and how it is impacted by cinematographic vocabularyand grammar. In Section 5 we explain how we implemented our semantic adaptationsystem, after which we provide an evaluation in Section 6. We also list a number ofsuggestions of future research, and conclude this paper with Section 7.

    2 Semantic Mastering and the drama production process

    We have seamlessly integrated the semantic adaptation process into an existingdrama production workflow. In this section, we provide an overview of this extensivedrama production workflow and explain how semantic adaptation was included. Theworkflow we describe in this paper represents the typical production process forvarious drama productions, as we have observed from research in the field, as wellas in literature [40]. Our assumption about the implementation of this workflow ina file-based production facility is quickly becoming a reality as most broadcastersand production houses are transitioning away from legacy tape-based systems [19].The tight integration of all components and processes, connected by extensive andelectronic metadata streams is not yet realistic in practice, although proof of conceptsystems do exist, one of which our adaptation system is based on [12]. The processesand workflow metadata that flows between them have been mapped out in Fig. 1.

    2.1 Script writing and 3-D previsualization

    In conventional drama production workflows, the production process typically startswith the definition of a story synopsis, which is later extended into a complete scriptor screenplay during the script writing process. Once approved, the screenplay is thenelaborated by the director into a shooting script. This document defines which aural


    Semantic Adaptation

    Semantic Mastering


    Script Writing &Shooting Scripting

    Analysis &Quality Assurance



    Semantic OOIDefinitions

    OoI Projections,Mastering


    Definitions Semantic OoIDefinitions


    AV Essence,Continuity & Logging Info.

    AV Essence,Edit Decisions

    Fig. 1 The drama production workflow with semantic adaptation processes

  • 310 Multimed Tools Appl (2012) 59:307340

    and visual points of view of the scene must be realized and serves as the templateaccording to which cast and crew performances will be coordinated. In some cases,the functionality of these processes can be combined into a single previsualizationstep where scenes are set up in a virtual 3-D environment, in which characters areplaced, dialogue is written and virtual cameras are parameterized and animated [6].

    Right from the beginning of the production workflow, objects of semantic interest(OoI) are defined. The narrative described by screenplay documents involves anumber of characters that actively participate in the story (e.g., as protagonist orantagonist), and prop objects which serve a more illustrative function but can besemantically relevant nonetheless. A scenes narrative progresses by