Semantic Mastering: content adaptation in the creative drama production workflow

Download Semantic Mastering: content adaptation in the creative drama production workflow

Post on 26-Aug-2016

217 views

Category:

Documents

3 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>Multimed Tools Appl (2012) 59:307340DOI 10.1007/s11042-010-0710-0</p><p>Semantic Mastering: content adaptation in the creativedrama production workflow</p><p>Dieter Van Rijsselbergen Chris Poppe Maarten Verwaest Erik Mannens Rik Van de Walle</p><p>Published online: 8 January 2011 Springer Science+Business Media, LLC 2011</p><p>Abstract In order to provide audiences with a proper universal multimedia expe-rience, all classes of media consumption devices, from high definition displays tomobile media players, must receive a product that is not only adapted to their capa-bilities and usage environments, but also conveys the semantics and cinematographybehind the narrative in an optimal way. This paper introduces a semantic videoadaptation system that incorporates the media adaptation process in the center ofthe drama production process. Producers, directors and other creative staff instructthe semantic adaptation system using common cinematographic terminology and vo-cabulary, thereby seamlessly extending the drama production process into the realmof content adaptation. The multitude of production metadata obtained from varioussteps in the production process provides a valuable context of narrative semanticsthat is exploited by the adaptation process. As such, high definition imagery can beintelligently adapted to smaller resolutions while optimally fulfilling the filmmakersdramatic intentions with respect to the original narrative and obeying various rulesof cinematographic grammar.</p><p>Keywords Semantic adaptation UMA Universal multimedia experiences Cinematography Drama production</p><p>1 Introduction</p><p>Viewers today have access to a multitude of platforms for the consumption of anever increasing supply of broadcast audiovisual media: from movie theaters and</p><p>D. Van Rijsselbergen (B) C. Poppe E. Mannens R. Van de WalleDepartment of Electronics and Information Systems (ELIS)Multimedia Lab,Ghent UniversityIBBT, Gaston Crommenlaan 8/201, 9050 Ghent, Belgiume-mail: Dieter.VanRijsselbergen@UGent.be</p><p>M. VerwaestVRT-medialab, Gaston Crommenlaan 10/101, 9050 Ghent, Belgium</p></li><li><p>308 Multimed Tools Appl (2012) 59:307340</p><p>high definition-enabled or standard definition television screens at home, to smartcellular phones on the road, and online using the web. These platforms vary widelyin capabilities and specifications. Content producers and providers are challenged toprovide this entire range of different devices with proper access to large quantities ofaudiovisual products. In practice, this implies that content must be altered to meetthe limitations of a users terminal and network, essentially realizing the promise ofUniversal Multimedia Access (UMA) [37].</p><p>Considering the increasing prevalence of high definition presentation devices,we expect content creators to start actively focusing more on wide cinematic-likeframing. However, this presents issues for mobile device viewers. Due to significantlysmaller display surfaces, they can not reasonably be provided with identical, butsimply down-scaled versions of material originally framed with high definitionpresentation in mind. Studies have shown that in order to provide a comfortable userviewing experience, adapted images should be constrained to contain image regionsthat are most meaningful in terms of the content they represent [17]. Adaptationsshould be guided by notions of the semantics and narrative that the content beingadapted conveys. Such semantic adaptations would, for example, crop the pictureto include only a particular story character; or would, more elaborately, pan andscan from one character to another within the imagery recorded for a single originalwide camera shot. Hence, an additional effort is required to present users with aworthwhile multimedia experience, despite possibly limited terminal capabilities,thereby turning plain UMA into true Universal Multimedia Experiences [28].</p><p>Unfortunately, we have found that content adaptation systems described inliterature today perform adaptation processes almost as an afterthought, whenregular media production has finished. Many algorithms have been developed thatanalyze video signals in an attempt to automatically infer possible semanticallyinteresting regions, of which some can be manually annotated by human operators.While many of these systems can operate in a context-agnostic fashion, none of theoriginal production information concerning the semantics associated with the sourcematerial is reused. A rich set of production metadata, whether in paper or electronicform or implicit in the heads of production people, is left unconsidered. Havingthis production metadata available can help us reduce the semantic gap betweenaudiovisual signals and their original narrative and aesthetic intentions. This reducesthe need for computational aesthetics algorithms [14] that can produce incorrectconclusions, and provides a semantically rich context in which content adaptationcan be performed.</p><p>One particular class of media production where semantics significantly influencethe structure and imagery of the final product is drama production, which includessoap operas, prime time quality fiction and motion pictures. In fact, the semanticsof the story drive the entire production process. Narrative and creative decisionsare taken to emphasize specific aspects of this story. We have built an adaptationsystem that includes notions of the adaptation process from early in the productionprocess. This allows directors, producers and other creative staff to decide whichinteresting objects in the video frames should be retained for smaller displays. Thecreative staff is in a better position to make these decisions than automated systemsor operators beyond the production chain would be. After all, cinematography canbe considered a work of art and must be handled carefully when being adapted forvarious output channels. Because drama production involves many creative planningdecisions anyway [40], we let the drama crew define adaptation parameters them-</p></li><li><p>Multimed Tools Appl (2012) 59:307340 309</p><p>selves. However, we are aware that the additional burden placed on the productioncrew should be limited, and such, we have balanced their required efforts and theamount of automated algorithms used to drive the actual adaptation. Essentially,we have constructed a system that is interactive and provides essential elements ofintelligence.</p><p>In the following section, we define the functionality of our semantic adaptationprocess and describe how it can be incorporated into the existing drama mediaproduction workflow. An overview of the related work concerning spatial videoadaptation is presented in Section 3. Section 4 explains the concepts behind oursemantic adaptation system and how it is impacted by cinematographic vocabularyand grammar. In Section 5 we explain how we implemented our semantic adaptationsystem, after which we provide an evaluation in Section 6. We also list a number ofsuggestions of future research, and conclude this paper with Section 7.</p><p>2 Semantic Mastering and the drama production process</p><p>We have seamlessly integrated the semantic adaptation process into an existingdrama production workflow. In this section, we provide an overview of this extensivedrama production workflow and explain how semantic adaptation was included. Theworkflow we describe in this paper represents the typical production process forvarious drama productions, as we have observed from research in the field, as wellas in literature [40]. Our assumption about the implementation of this workflow ina file-based production facility is quickly becoming a reality as most broadcastersand production houses are transitioning away from legacy tape-based systems [19].The tight integration of all components and processes, connected by extensive andelectronic metadata streams is not yet realistic in practice, although proof of conceptsystems do exist, one of which our adaptation system is based on [12]. The processesand workflow metadata that flows between them have been mapped out in Fig. 1.</p><p>2.1 Script writing and 3-D previsualization</p><p>In conventional drama production workflows, the production process typically startswith the definition of a story synopsis, which is later extended into a complete scriptor screenplay during the script writing process. Once approved, the screenplay is thenelaborated by the director into a shooting script. This document defines which aural</p><p>Editing</p><p>Semantic Adaptation</p><p> Semantic Mastering</p><p>Acquisition</p><p>Script Writing &amp;Shooting Scripting</p><p>Analysis &amp;Quality Assurance</p><p>HDDistribution</p><p>MobileDistribution</p><p>Semantic OOIDefinitions</p><p>OoI Projections,Mastering</p><p>ParametersOoIProjectionsShooting</p><p>Definitions Semantic OoIDefinitions</p><p>ShootingDefinitions,Screenplay</p><p>AV Essence,Continuity &amp; Logging Info.</p><p>AV Essence,Edit Decisions</p><p>Fig. 1 The drama production workflow with semantic adaptation processes</p></li><li><p>310 Multimed Tools Appl (2012) 59:307340</p><p>and visual points of view of the scene must be realized and serves as the templateaccording to which cast and crew performances will be coordinated. In some cases,the functionality of these processes can be combined into a single previsualizationstep where scenes are set up in a virtual 3-D environment, in which characters areplaced, dialogue is written and virtual cameras are parameterized and animated [6].</p><p>Right from the beginning of the production workflow, objects of semantic interest(OoI) are defined. The narrative described by screenplay documents involves anumber of characters that actively participate in the story (e.g., as protagonist orantagonist), and prop objects which serve a more illustrative function but can besemantically relevant nonetheless. A scenes narrative progresses by means of aseries of events that impact the scenes OoIs. Typically, this encompasses dialogueexchanged between characters, but it can also be actions performed by characters orother OoIs.</p><p>2.2 Acquisition</p><p>The Acquisition process, where real-life audiovisual essence is recorded on soundstages or location, is typically the most prominent drama production process.During acquisition, takes or realizations are recorded for the duration of a singleperformanceusually a single scenefrom the perspective of one or more acquisi-tion devices such as cameras and microphones. Because this process is driven by thedirectors shooting script, the acquired file-based essence can be directly associatedwith the semantics of the screenplay [40].</p><p>2.3 Analysis and quality assurance</p><p>After acquisition a combination of Quality Assurance and Analysis steps determinethe fitness of acquired audiovisual material. Quality Assurance and Analysis donot directly contribute to the production of audiovisual media assets, but they areimportant processes that help boost the efficiency of the production process furtheralong the workflow. Some material will be deemed unusable for further use duringeditorial operations while other material will be kept but only as a last resort or forother purposes than to end up in the final product (e.g., as part of a gag reel). DuringQuality Assurance logging and continuity information gathered during acquisition iscollected, conformed, and streamlined to accompany audiovisual media assets downthe production pipeline.</p><p>The generic Analysis label represents any combination of processes that generatesnew information based on the audiovisual characteristics of the recorded material.This includes color information and feature extraction, shot-cut detection and theselection of representative key frames. In our system, it is primarily focused on thedetection of OoIs in audiovisual media assets, such that they can be reasoned withby the semantic adaptation processes.</p><p>2.4 Semantic Mastering and adaptation</p><p>In professional audio production, mastering denotes the process where sound op-erators prepare and optimize the dynamic range of a sound track for a givenoutput medium. Similarly, we reuse this term for our implementation of a semantic</p></li><li><p>Multimed Tools Appl (2012) 59:307340 311</p><p>adaptation system. The process where filmmakers decide, expressed using semanticelements, how the video repurposing should be performed is called Semantic Mas-tering. We also define a counterpart process, Semantic Adaptation, that performs theactual essence transformations using the mastering specifications as input.</p><p>In order to let production people feel quickly at home with the Semantic Master-ing process, it should naturally extend existing drama production practices. This isaccomplished by exposing mastering and adaptation to users using proper cinematicterminologywhich includes shot types such as close-ups, long shots and verbs suchas panning, tilting, zoomingand by incorporating notions of the semantic elementsof the production. In particular, the scene OoIs are actively used by directors todetermine picture framing, e.g., a close-up of a character. The crew can take creativedecisions on how a virtual camera, implemented within the imagery of the originalfootage, should behave for different mastering outputs. Our Semantic Masteringconcept is depicted in Fig. 2. Two soap opera characters, Jenny and Rosa, wereoriginally shot at either side of the frame. For display area-constrained products,Jenny was deemed more relevant, so the focus is shifted there in the adapted image(indicated by dashed rectangle). The Semantic Mastering parameters specified are:show a waist-shot of Jenny at the left of frame. Additional adaptation parameterscould be used to shift the image back and forth between characters as the dialogueprogresses. Instructions are given to the adaptation system in a similar fashion asthey are conveyed to the camera crew on the set. By implementing the SemanticMastering process at this point in the workflow, the adaptation parameters can bedefined for use with the originally acquired media assets or dailies without themhaving been edited, cut and intertwined with other takes or camera perspectives.This simplifies the mastering process as it does not have to work around irreversibleshot cut boundaries introduced during editing operations. As a result, masteringparameters are specified over the length of an entire take of continuously shotfootage, reducing the amount of work required during mastering, and increasing thecontinuity and fluidity of the executed semantic adaptations.</p><p>2.5 Editing</p><p>In the meantime, craft editing is performed on the acquired high definition essence.Editing decisions are imported into the production workflow so that they can be</p><p>Fig. 2 An example ofSemantic Mastering</p></li><li><p>312 Multimed Tools Appl (2012) 59:307340</p><p>re-applied for other output profiles. Although temporal adaptations have not been atopic of research in this work, different edits can be explicitly made for differentoutput profiles. Despite the fact that, in principle, the Semantic Mastering andEditing processes are executed independent of one another, any available resultsfrom the editorial processes can be previewed during Semantic Mastering in...</p></li></ul>

Recommended

View more >