integrating semantic analysis and scalable video coding for...
TRANSCRIPT
Integrating semantic analysis and scalable video
coding for efficient content-based adaptation*
Luis Herranz
Grupo de Tratamiento de Imágenes, Escuela Politécnica Superior
Universidad Autónoma de Madrid, Madrid, Spain
Abstract Scalable video coding has become a keytechnology to deploy systems where the adaptation ofcontent to diverse constrained usage environments (suchas PDAs, mobile phones and networks) is carried out ina simple and efficient way. Content-based adaptationand summarization are fields that aim for providingimproved adaptation to the user, trying to optimize thesemantic coverage in the adapted/summarized version.This paper proposes the integration of content analysiswith scalable video adaptation paradigm. They must befitted in such a way that the efficiency of scalableadaptation is not damaged. An integrated framework isproposed for semantic video adaptation, as well as anadaptive skimming scheme that can use the results ofsemantic analysis. They are described using the MPEG-21 DIA tools to provide the adaptation in a standardframework. Particularly, the case of activity analysis isdescribed to illustrate the integration of semanticanalysis in the framework, and its use for online contentsummarization and adaptation. Overall efficiency isachieved by means of computing activity usingcompressed domain analysis with several metricsevaluated as measures of activity.
1 Introduction
In the current multimedia scenario, many possible waysof access to the content are available, as different andheterogeneous terminals and networks can be used bythe potential users. Adaptation[1] is a key issue to bringthe content from the service providers to the actualusers, each one using their own terminals and networks,with their own capabilities and constraints, enabling theso called Universal Multimedia Access (UMA)[2].
Especially important is the case of mobile devices, suchas PDAs and mobile phones, where other issues aslimited computational resources and low powerconsumption requirements become very important andlow-complexity codecs and decoders are necessary.MPEG-21 is a standarization effort to develop ainteroperable multimedia framework. This framework isbuilt around the concept of Digital Item (DI). The DI isthe basic unit of transaction in MPEG-21, including astandard representation, identification and metadata.Digital Item Adaptation (DIA) is the part of the standardthat tackles the problem of adaptation of DI toheterogeneous usage contexts. DIA specifies adaptationrelated metadata, including those related to the usageenvironment such as terminal capabilities, network anduser characteristics.
Scalable coding is an elegant solution toadaptation[3]. A scalable video stream containsembedded versions of the source content that can bedecoded at different resolutions, frame rates andqualities, simply selecting the required parts of thebitstream. Thus, scalable video coding allows a verysimple, fast and flexible adaptation framework to avariety of terminals and networks, with differentcapabilities and characteristics. The numerousadvantages of this coding paradigm have motivated anintense research activity with the development of newwavelet based video codecs[4] and the forthcomingMPEG-4 SVC standard[5].
Adaptation techniques, such as scalable videoadaptation, transcoding or transmoding, lead to amodified version that usually will result in a reductionof the fidelity of the content. This information loss isdistributed uniformly in the whole content, reducing theresolution, the frame rate or the quality. However, this
*Work partially supported by the European Commission under the 6th Framework Program (FP6-001765 - aceMedia Project).This work is also supported by the Ministerio de Ciencia y Tecnología of the Spanish Government under project TIN2004-07860(MEDUSA) and by the Comunidad de Madrid under project P-TIC-0223-0505 (PROMULTIDIS). The final publication is available at Springer via http://dx.doi.org/10.1007/s00530-007-0090-0
approach is blind to the content itself, as it does notknow anything about what is actually happening. As thefinal user will prefer to know what is happening in thecontent, blind adaptation sometimes can lead to someloss or severe degradation of important information forthe user. It would be better to try to preserve those partsmore “semantically” relevant. Low level semanticsextracted from the analysis of the content can help toperform a better adaptation and personalization[6].
There are many applications where such informationloss is desirable at some extent. In these applications,the user wants to have a shorter or smaller, but stillinformative version. A short trailer of the video or acollection of relevant frames can be very helpful for theuser to have in mind what happens in the content in lesstime than playing the whole content, and in a moreuseful way than reading the title or an abstract. This setof keyframes or segments of video need to be selectedfrom the source content according to some semanticcriterion, as they should be representative of as much ofthe whole content as possible. Semantics enable a betterretrieval and browsing of the multimedia content,providing to the user ways to access and browse largemultimedia repositories more effectively
To extract the semantics from the content some kindof agent or system needs to analyze the content. Onesolution is manual annotation but it is very timeconsuming, hard, expensive and unfeasible for mostapplications. Automatic analysis is very important toextract semantics avoiding human participation. Highlevel analysis and understanding is still very difficultand complex for automatic systems (the so calledsemantic gap), but low level analysis is much easier andvery helpful for semantic adaptation. Specially,compressed domain analysis gives the efficiencyrequired to allow fast solutions, avoiding full decodingand taking advantage of the compressed domain data asusable features.
The need to effectively browse and manage the hugeamount of video content available in personal andcommercial repositories, and the possibility of accessfrom diverse terminals and networks, makes essentialthe use of systems combining summarization techniquesand adaptation for effective delivery to the user. Thispaper contributes exploring the use of semantic analysisin the adaptation process of a scalable video stream, inan integrated framework. It also discusses how theanalysis can take advantage of the compressed domainand the scalability, in order to keep efficient the wholeadaptation engine. The second contribution is a specificvideo skimming scheme for scalable video, developedwithin this framework, and the way that it is describedusing MPEG-21 DIA tools. In this scheme, the results ofsemantic analysis guide the temporal adaptation of thecontent, but keeping the advantages of scalable videoadaptation.
The semantic analysis plays an important role in thequality of the resulting summaries/skims, but thesemantic analysis itself is not in the scope of the paper,which is focused on the discussion and development ofan integrated adaptation framework. The framework isgeneric enough to use a variety of semantic analysisapproaches. However, the framework and the skimmingmethod are illustrated with a simple low level analysisbased on computing the activity using compresseddomain data.
The rest of the paper is organized as follows. Section2 briefly discusses the related work. Then, Section 3describes a generic framework for semantic adaptationof scalable video which is used in Section 4 to propose ascheme for content based skimming. In Section 5, thescheme is integrated in the MPEG-21 DIA adaptationtools. In Section 6 and 7 experimental tests arepresented. Section 8 discusses the proposed frameworkand related approaches, and finally Section concludesthe paper.
2 Related work
2.1 Analysis for video adaptation and
summarization
Conventional video adaptation (e.g. transcoding) doesnot take into account that the sequence is actuallytransmitting information with some meaning for theuser. Any level of understanding of the structure or thesemantics in the sequence can be useful to takeadaptation decisions that improve the quality andmeaninfulness of the sequence perceived by the user[1,6, 7]. Dynamic frame dropping is an example ofadaptation technique using information fromanalysis[8].
Summarization can be considered as a specific typeof semantic/structural adaptation where the mainobjective is the reduction of the information removingsemantic redundancy for efficient browsing andpresentation of the content. Static storyboards are widelyused in video browsing and retrieval to represent thecontent in few keyframes[9-11]. Several representativeframes are selected according to some analysisalgorithm in a way that the semantic coverage of thecontent is optimized and represented in few specificframes.
At a higher semantic level, the sequence can bestructured as a collection of shots, in which the relevantframes are selected using motion information or featureclustering. Complex representations have beeninvestigated to model the relationships between shots inmore abstract units (scenes), in order to obtain moremeaningful summaries for retrieval[[12]. Sometimes theselected frames are not presented as a collection of
images, but in a shorter video sequence, resulting in afast preview of the content [13].
Video skims are another modality of videoabstraction, similar to summaries, but usually builtextracting significant segments from the sourcesequence rather than individual frames. A number ofapproaches have been used to generate video skims,including visual attention[14], rate-distortion[15], graphmodelling[16] and image and audio analysis[17].
Efficient video analysis for adaptation andsummarization has been also explored in MPEGcompressed domain. The analysis in compressed domainis constrained by the data available in the bitstream,such as DCT coefficients, motion vectors andmacroblocks modes in MPEG-1/2 video codecs.Although the analysis is more limited than inuncompressed domain, the main advantage is the highefficiency. Activity and motion features have beenwidely used[13, 18] as criteria to select or drop frames.Camera motion can be also extracted from thecompressed data. The specific coding structure can bealso also exploited to drop motion compensatedframes[19, 20]. [18] uses MPEG-21 DIA tools toefficient drop frames of a H.264 coded bitstream usingsimple frame dropping operations.
2.2 Fully scalable video
Scalable coding of media has received an increasinginterest in recent times. The advantage of a scalablebitstream is that it can be encoded once, but multipleversions can be decoded, each of them being suitable fordifferent practical usage environments. Besides, the costof building these adapted versions is extremely lowcompared with conventional adaptation techniques, suchas transcoding, due to the embedded structure of thescalable bitstream.
A fully scalable video bitstream is organizedaccording to a three dimensional spatio-temporal-qualitystructure where the parts (or atoms) are arranged asblocks in a cube where each coordinate (s, t, q) in thisspace represents an adapted sequence, and determineswhich blocks should be included in the adaptedbitstream (see Figure 1). The increment of thecoordinate in one of the dimensions corresponds to anadditional layer that enhances the adapted sequence inthat dimension.
The scalable bitstream is processed in a GOP (Groupof pictures) basis (each GOP is the adaptation unit), anddepending on the constraints in the usage environment,the three independent coordinates (s, t, q) should bedetermined, in order to obtain the desired adaptation.Note that several properties of the GOP such as bitrateor PSNR are functions of these three coordinates.
Figure 1. Representation of the first two GOPs of afully scalable video bitstream.
There are a number of techniques developed toachieve spatial and temporal scalabilities. Interframewavelet codecs[4, 21, 22] use hierarchical subbanddecompositions to provide an embedded representationfor spatial scalability. Temporal scalability is achievedusing a hierarchical subband decomposition of frameswithin a GOP. This decomposition is combined withmotion compensation (Motion Compensated TemporalFiltering-MCTF[23]) to improve the coding efficiency.
The other major category of scalable video codecsare the H264 extensions, most notably MPEG-4SVC[5]. It uses H264/AVC as base layer and the rest ofthe spatial enhancement layers are coded exploitinginter-layer prediction. For temporal scalability,hierarchical prediction structures with motioncompensation are used.
3 Adaptation framework
This section discusses the design of a generic frameworkcombining efficient scalable video adaptation forapplications that need some level of understanding ofwhat happens in the content, in order to improve theutility of the adaptation to the user. It also enables morecomplex adaptation modalities using this low level ofunderstanding of the content. In order to try to fit thesetwo aspects into an efficient integrated framework, anumber of requirements are discussed.
The adaptation engine should be aware of the usageenvironment. This first requirement is essential, as noadaptation is possible if no information about thecontext to adapt is available. MPEG-21 DIA[2] toolsinclude the Usage Environment Description (UED) toolto detail the characteristics of networks and terminals.Scalable video coding can use this context informationand perform this context-aware adaptation in a veryefficient way.
The adaptation engine needs a mechanism to addsemantic information, in order to guide the adaptationprocess and enable semantic-based applications. Suchcontent-aware adaptation is required for content-basedsummarization. This semantic information of the contentmust be available to the adaptation engine, either aspreviously stored metadata or either obtained from ananalysis stage.
A desirable third requirement is online processing.This requirement is not always necessary, but it is
imposed to allow applications in scenarios where futurecontent is not available. Online processing refers to theprocessing of the content as it arrives, with no need offuture content (usually because it is not available). Livebroadcasting, videoconference and surveillance areexamples of scenarios where online processing is useful,but other scenarios can also benefit from the onlineprocessing. The main advantage of online processing isthe low delay in the delivery of the adapted content.Another advantage is the low processing and storagerequirements that usually need these approaches.
However, there is an inherent problem in onlineprocessing for video summarization and indexing, andin general, for all the techniques that need a temporalstructure of the content into meaningful segments.Online processing is causal, and temporal condensationtechniques that try to optimize the coverage usingsemantics are of necessity non-causal, as they require allthe content analyzed in order to build a meaningfulsummary. Even for human users it is not possible tosummarize a movie without having seen it completely.Buffering parts of the content is useful to relax thatproblem, but there is always a trade off betweencoverage and delay in the delivery. So this requirementwill lead to non optimal summaries, but still can beuseful enough to build fast summaries and previews.
Finally, it is always highly desirable to keep thewhole framework efficient. Effciency is the mainmotivation to develop the framework proposed. Usually,most of the applications requiring online processing willalso require efficiency, in order to be able to process allthe content at it arrives.
Although the importance of semantic analysis iscritical for good video abstraction, the paper is notfocused on any specific analysis algorithm, but in aflexible framework that could easily include otheranalysis methods. In order to take advantage of thescalability and the compressed domain, some ideas arediscussed to obtain meaningful data directly from thebitstream. These points are shown later with a specificactivity based analysis.
3.1 Semantic adaptation engines
An adaptation engine should be able to generatesequences that satisfy a given set of constraints of theusage environment. Then, the terminal will be able todecode the adapted sequence. Using scalable video, theadaptation engine is a simple bitstream extractor thattruncates the bitstream selecting only the parts requiredaccording to the constraints (see Figure 2a). Thus, theadaptation process is very fast and performed online. Inthis case, the adaptation is completely blind to thecontent itself.
A more complex adaptation engine can use thesemantics in the content to guide and improve theadaptation process. Content-based summarization andskimming are also possible as part of the adaptation
engine. The semantic data could be generated previouslyand stored as metadata along with the content (seeFigure 2b). Standards such as MPEG-7 provide standarddescription tools for specifying low level semantics thatcan be used in summarization[24]). In this architecture,the adaptation process is driven by metadata[25], andthe computational burden due to the analysis is avoidedat time of adaptation and shifted to a previous stage.There are no constraints in the efficiency andcomplexity of the analysis algorithms. It is suitable forscenarios where the content is created and stored beforethe consumption, such as retrieval of video frompreviously stored content repositories. The adaptation isstill very efficient and online, with knowledge of thesemantics of the whole content. When content metadatais available it should be used as it can help to improvethe adaptation and summarization without a significantoverload of the system.
(a)
(b)
Figure 2. Adaptation engines: (a) Content independent;(b) Content aware (metadata-driven)
However, metadata is not always available. Besides,in many scenarios the content should be delivered andadapted online just after its creation with low delay, andthe analysis and adaptation are very constrained. In thiscase, the analysis should be included in the adaptationengine. The architecture of such content basedadaptation engine includes an analysis stage that willguide the bitstream extractor (see Figure 3).
Figure 3. Proposed content aware adaptation engine,with online analysis.
Note that if the engine is required to process onlinethe content, the analysis will be required to be onlinetoo. Thus, most of the semantic summarizationalgorithms cannot be used, as they need to have all or atleast certain amount of the content preprocessed (shot,or scene) to extract a summary.
In this last architecture, it is important to be careful,because one of the key advantages of scalability is thatadaptation engines are extremely efficient. The semanticadaptation should be also very efficient in order to keepthe advantages of the scalable video framework. Tosatisfy this efficiency requirement, the analysis shouldbe very efficient as well. For this reason, it is verydesirable that the analysis stage work directly over thecompressed domain parameters present in the codedbitstream, in order to avoid decoding. The rest of thepaper is focused on this architecture using compresseddomain analysis to provide efficient analysis andkeeping the whole adaptation process efficient.
3.2 Compressed domain analysis in scalable video
Working directly on the compressed domain for efficientanalysis has been extensively explored for codingstandards like MPEG-1/2/4, based on hybrid videocoding using spatial DCT together with motioncompensation. Compressed domain analysis avoidsmost of the decoding stages, but more important,dramatically reduces the amount of data to process(e.g.DC images[26]) and provides meaningful parametersthat can be used to extract useful features foranalysis[27]. There is a wide variety of experimentalfully scalable codecs, most of them sharing commonapproaches to the scalability that can be used to extractuseful parameters. There are two features that can beextracted easily from most of these codecs in a genericway: Low resolution images. Discarding all the spatial
enhancement layers. Only the base layer is requiredto be decoded. Discarding temporal layers canfurther reduce the decoding time, as no inversetemporal steps are required. These low resolutionframes carry almost the same information as the fullresolution source frames, and can be used as input
of many analysis algorithms without significantdegradation of the results.
Motion fields. Temporal interframe redundancy isusually exploited using a motion compensatedapproach, either MCTF, or either hierarchicalprediction structures. In both approaches there aredifferent sets of motion vectors coded in headers,giving a coarse representation of the motion of theobjects in the frames.
These basic features can be used by the analysisalgorithms to obtain higher level semantics, such as shotchange detection[28], activity analysis[29], cameramotion[30] and coarse spatial segmentation[31], amongothers, in the same way as they are widely used inMPEG-1/2 working with motion vectors and DCimages. Section 6 describes some experiments wherecompressed domain data from a scalable video codecare used to perform activity analysis, using the basicfeatures discussed previously.
4 Video skimming of scalable video
Previous section has discussed the abstract model toprovide semantic adaptation integrated with scalablevideo adaptation. This section and the rest of the paperfocuses on a specific adaptation scheme that uses theprevious model, taking advantage of the temporalscalability to provide semantic selection of frames(skimming). In this scheme, the adaptation engine canvary the frame rate of the adapted sequence dependingalso on the semantics of the content.
Figure 4. Proposed scheme for content based skimmingof scalable video.
The skimming scheme is depicted in Figure 4. Thesemantic information is linked to the bitstreamextraction using a skimming curve, indicating thenumber of frames that should be selected for each GOP.This skimming curve is the result of the analysis stage.Due to temporal scalability, each GOP will have severalpossible temporal versions. Depending on the value ofthe skimming curve, the most suitable temporal version
of the GOP will be selected from the input bitstream,and included in the adapted bitstream.
The skimming curve can be obtained by means ofsemantic analysis of the content or from previouslystored metadata (following the metadata-drivenapproach of Figure 2). The quality of thesummary/skim will depend on the capability of thesemantic analysis to extract the relevant frames of thesequence for a given application (e.g. skimming,summarization, fast browsing) and context. However,complex analysis algorithms are out of the scope of thispaper, instead a simple method is described to illustratethe use of low level semantics in the framework. Anextended approach is the use of an activity or motionintensity index that can easily be converted to askimming curve, and also can be extracted online.
4.1 Activity based skimming
Activity has been proposed as a measure ofsummarizability of a video sequence[12], where staticparts of the video sequence can be represented with fewframes, and high activity parts due to motion need moreframes to cover the events of the frames. Following thisreasoning, activity analysis is used to obtain theskimming curve. An activity measure is given to eachGOP in the source sequence using information presentin the coded bitstream. Using the skimming schemeproposed (see Figure 4), the number of frames availablefor each GOP will depend on the scalability framework,and will be selected according to its activity.
Let T be the number of temporal decompositions(there will be T+1 levels, from 0-lowest frame rate- to T-highest frame rate-). Assuming that level 0 has 1 frameper GOP the length of the GOP will be
TL 2 0and the number of frames per GOP at level t
TtL tt ,,2,1,02) 0In the scheme, the activity is assumed to vary
linearly with the measured activity. The skimming curverk is then related with the activity ak of the GOP k with aconstant K:
kk aKr · 0Due to the dyadic structure used in temporal
scalability, each additional temporal layer doubles thenumber of frames in the GOP. If M is the number ofdesired skimming levels in the skimmed sequence, thepossible number of frames that the GOPs of the adaptedsequence will have is in the set
02,,2 1 MSK TMT
0The temporal level to be selected will be the result of
quantizing the skimming curve rk in the SK set, andgives the number of frames per GOP in the skimmedsequence
SKkk rm 0The previous skimming curve could be used to guide
the bitstream extraction process. However, suddenvariations in the curve lead to too many fast andcontinuous accelerations and disaccelerations that areundesirable in the skimmed sequence. In order to keep amore natural behaviour in the skimming, a simplesmoothing filter has been used.
SKkkk rrm 13.07.0 0It is convenient to use parameters directly related
with the temporal scalability in terms or temporal levelsmore than frames per GOP. The parts that will remain inthe bitstream are those required to decode each GOPwith at a given number of frames km , corresponding
to the temporal level:
kk mp 2log 0Note that M can be higher than the number of
available temporal levels, and the number of frames perGOP can be lower than 1 frame. (0) is only valid when
km is 1 frame per GOP or greater. When the number
of frames per GOP is below 1 frame, only is kept thelowest frame rate version (temporal level 0) of the GOP,and a number of the subsequent GOPs will be skipped,according to the values of the skimming curve, in orderto achieve the desired frame rate. This skippingprocedure is signalled to the adaptation stage with anextra variable skipk that is true when the GOP is skipped.
The bitstream extractor will use the values of pk andskipk to achieve the desired temporal adaptation. Notethat, regarding to spatial and quality adaptation, thebehaviour of the adaptation engine is non content-based,adapting to the constraints specified in the usageenvironment. Only those parts required for decoding thegiven frame size and quality will be kept in the adaptedbitstream.
4.2 Application to dynamic frame dropping and
fast browsing
This scheme enables the use of the scalable videoadaptation for applications using content basedskimming of frames.
One of such applications is fast browsing[13] ofvideo sequences. The input sequence is compacted in anadapted sequence with fewer frames. Played at fullframe rate (or the highest frame rate allowed by theterminal), the adapted sequence can be much shorter, butstill with most of the coverage of the content. It can beuseful when the user wants a fast glance of the videocontent (a quick video summary), but trying to keep thesemantics in terms of ‘pace’ in the sequence, skippingfaster low activity parts and focusing on those partswhere more activity is detected. In the case of using theproposed adaptation scheme for fast browsing, theadapted sequence is played with constant frame rate, butwith variable GOP rate, as the number of frames of eachGOP is selected dynamically.
If the same set of selected frames is played atconstant GOP rate, instead of playing at constant framerate, the length of the adapted sequence will be thesame. The adapted sequence will be presented in theterminal as the same sequence but with fewer frames instatic parts and more frames in more active ones. Suchadaptation can be seen as an adaptation engine usingdynamic frame dropping[20] in scalable video.
5 Adaptation to the usage environment
At this point, the usage environment has not beingconsidered yet. The problem of adapting bitstreams to agiven usage context can be formulated using some toolsof MPEG-21 DIA, designed to describe genericoptimization problems and the constraints involved inthe adaptation process[32].
The decision problem in DIA is formulated using thedescription tools Universal Constraints Description(UCD) and Adaptation Quality of Service (AQoS). TheUCD tool specifies directly the constraints involved inthe problem (e.g. maximum width, maximum bitrate)using numeric expressions that relate the constrainingvalue with the variable that is being constrained. TheAQoS tool specifies the dependencies between thevariables (termed as IOPins in MPEG-21 DIA) involvedin the optimization problem. Some of these variables areindependent, while the rest depend on the othervariables. The decision-taking engine processes theinput bitstream selecting the best adaptation variablesusing the information available in AQoS and UCD for agiven adaptation unit. Then, the bitstream adaptationengine performs the actual extraction and modificationof the bitstream according to the adaptation variables.
A third adaptation tool that is usually used togetherwith UCD and AQoS is the UED, where a description of
the usage context can be described. As the objective isthe adaptation to a given constrained usage context,most of the constraints specified in UCD are related tothe capabilities described in UED (e.g. terminal size,network bitrate).
5.1 Scalable video adaptation in MPEG-21 DIA
In this work, the MPEG-21 DIA approach is followed,using the same tools and concepts as in the case ofconventional scalable video adaptation, and extending itto include the semantic adaptation as new elements inthe optimization problem. The problem of adapting fullyscalable video consists of selecting the adaptationcoordinates (s, t, q) of each GOP (the adaptation unit)that gives the best adapted version for a given set ofconstraints.
In the case of fully scalable video, usually theoptimization problem is to maximize a measure of thequality, given the constraints of the usage environment(network and terminal). In this application, the metricselected to be maximized is an ad-hoc metric(PSNRGOP) based on the peak signal to noise ratio(PSNR) from the mean square error of the full GOP ofthe frames upsampled to the highest frame rate. For aGOP with spatial level s, temporal level t and qualitylevel q is computed as
kqts
kQS
qtsk
ggSTUMSEPSNRGOP
,
255log10
),,),
2
10),,
0where MSE is the mean squared error, STUS,Q) is thespatiotemporal upsampling operator to the highest
temporal and spatial levels, and ),, qts
kg is the
reconstructed GOP. Figure 5 shows the PSNRGOPcomputed for different temporal levels and differentquality layers. However, a proper objective perceptualmetric is desirable in order to measure better the realdistortion perceived by the user due to the combinedscaling in spatial, temporal and quality axis.
0 20 40 60 80 100 120-30
-20
-10
0
10
20
30
40
50
GOP
PSN
R[d
B]
Q layers = 1
0 20 40 60 80 100 120-30
-20
-10
0
10
20
30
40
50
GOP
PSN
R[d
B]
Q layers = 2
0 20 40 60 80 100 120-40
-20
0
20
40
60
80
GOP
PSN
R[d
B]
Q layers = 3
0 20 40 60 80 100 120-40
-20
0
20
40
60
80
GOP
PSN
R[d
B]
Q layers = 4
temporal level 0temporal level 1temporal level 2
temporal level 0temporal level 1temporal level 2temporal level 3
temporal level 0temporal level 1temporal level 2temporal level 3
temporal level 0temporal level 1temporal level 2temporal level 3
0 20 40 60 80 100 120-30
-20
-10
0
10
20
30
40
50
GOP
PSN
R[d
B]
Q layers = 1
0 20 40 60 80 100 120-30
-20
-10
0
10
20
30
40
50
GOP
PSN
R[d
B]
Q layers = 2
0 20 40 60 80 100 120-40
-20
0
20
40
60
80
GOP
PSN
R[d
B]
Q layers = 3
0 20 40 60 80 100 120-40
-20
0
20
40
60
80
GOP
PSN
R[d
B]
Q layers = 4
temporal level 0temporal level 1temporal level 2
temporal level 0temporal level 1temporal level 2temporal level 3
temporal level 0temporal level 1temporal level 2temporal level 3
temporal level 0temporal level 1temporal level 2temporal level 3
Figure 5. Quality measure PSNRGOP for 2 and 4temporal layers.
The optimization problem involved in thisapplication can be expressed as:
maximize PSNRGOP=f(NTEMP, NSPATIAL, NQUAL; GOP)subject to
FRAMEWIDTH ≤ display_widthFRAMEHEIGHT ≤ display_heightFRAMERATE ≤ display_refresh_rateBITRATE ≤ max_bitrateThe constraints are imposed by the usage
environment (described in the UED tool). Theindependent variables to be optimized are NSPATIAL,NTEMP and NQUAL (equivalent to the (s,t,q)adaptation coordinates), indicating the number ofspatial, temporal and quality layers in the adapted GOP.The rest of variables are not independent, and can beobtained as functions of NSPATIAL, NTEMP andNQUAL, and the number of the GOP, declared in theAQoS tool.
5.2 Adding semantic constraints
The scheme of activity based skimming of scalablevideo described previously can be included in theMPEG-21 DIA framework modifying the optimizationproblem in order to include the results of analysis asnew constraints and variables. The skimming level pk
limits the temporal coordinate t (represented byNTEMP). The GOP skipping can be included as a newvariable INCLUDE_GOP signaling if the GOP shouldbe included or not according to the value of skipk. Thenew optimization problem with these semanticconstraints is:
maximize PSNRGOP=f(NTEMP, NSPATIAL, NQUAL, INCLUDE_GOP; GOP)
subject toFRAMEWIDTH ≤ display_widthFRAMEHEIGHT ≤ display_heightFRAMERATE ≤ display_refresh_rateBITRATE ≤ max_bitrateNTEMP ≤ (analysis_temporal_level = mk)INCLUDEGOP = NOT(skipk)These two new constraints are imposed by the
activity analysis stage in the previously describedframework. Note that bitrate should take into account, ifnecessary, the fact of the speeding up of the sequenceand consequently the GOPs as adaptation units. Whenthe presentation of the adapted sequence includes timereduction (e.g. fast browsing), each GOP can havedifferent number of frames but presented in a constantframe rate. Thus, GOPs with fewer frames should bedelivered faster than those with more frames.
The value of analysis_temporal_level can also becomputed previously and stored along with the contentand the rest of adaptation metadata, usually in the sameAQoS descriptor, in a fully metadata-driven architecture
(see Figure 2b). However, this work addresses thearchitecture guided by compressed domain analysis,where the activity is computed online as the contentarrives (see Figure 3). A hybrid architecture withcaching can take advantage of the compressed domainanalysis the first time that the content is requested andstore the measure of activity or the temporal level in anadditional AQoS descriptor, and then follow a fullymetadata-driven approach for the next adaptation.
5.3 Personalization aspects
Each user could have different preferences on theamount of skimming for fast browsing. In the proposedsystem, it depends on the minimum activity threshold Kand the number of levels M in the skimming curve, andit will lead to different skimmed versions. These twoparameters could be specified by the user or predefinedin a personalization profile. However, they cannot beincluded directly in as preferences in the summarization.
A more intuitive parameter could be the minimumskimming ratio, expressed as
)1(2 M
SEQUENCEINFRAMES
SUMMARYINFRAMESratio
0where the number of levels M can be easily derived. Asat least 1 frame each 2M-1 frames, this ratio allows theuser to express a minimum, in the case of staticsequences. In practice, more frames are selected, asusually the videos have activity, and the smoothing filterwill also affect the final number of frames per GOP.
From the user’s point of view, it is more useful toquantize the parameter K in some levels and include it inthe profile in terms of predefined values linked to high,medium or low amount of skimming.
6 Experiments on Activity Analysis
The activity measure should reflect as much as possiblethe actual activity present in the content. Theperformance of the adaptation process will rely on it.The activity should be computed in a GOP basis, as it isthe basic unit in the adaptation engine.
However, in order to keep an efficient adaptationengine even when there is no previous activity data, themeasure should be rapidly computed from thecompressed domain parameters present in the bitstream,avoiding decoding or partial decoding as much aspossible.
6.1 Data extraction for analysis
As it was pointed previously, the extraction of usefulfeatures from the coded bitstream is of necessitydependent on the codec itself, its format andimplementation. In this section we will focus on waveletbased video coding and the specific implementationused for this paper. Wavelet coding enables a naturalmultiresolution framework for highly scalable videocoding. Most works on this subject are based on twotransforms: a wavelet transform in the temporal axis,combined with motion compensation (MCTF), andanother 2D spatial discrete wavelet transform (2DDWT). There are many codecs that use the framework,known as t+2D framework, where the temporaltransform is performed before the spatial transform. Inthis paper we consider the specific codec described in[22, 33]. It uses MCTF with hierarchical variable sizeblock matching (HVSBM)[34] and forward motioncompensation.
To extract low resolution images we can takeadvantage of the spatial scalability to avoid almost allthe decoding. In the coded bitstream, the lowest framesize version embedded in the stream is used. Discardingall the spatial subbands except the lowest one, theamount of data is highly reduced, all the inverse 2DDWT are avoided and inverse MCTF is performed overa number of pixels much lower. Thus, the decoding timeis dramatically reduced, as most of the decoding is notnecessary. In the case of the studies presented in thissection, only a frame per GOP is required. We can takeadvantage of the temporal scalability to further reducethe decoding steps, noting that the first temporal layeralready is the low pass temporal subband containing thesmall resolution image, so there is no need of inverseMCTF (see Figure 6). Only texture or entropy decodingis required.
Figure 6. Extraction of compressed domain data foranalysis from a wavelet scalable video codec.
Due to the hierarchical motion structure of thedyadic decomposition using MCTF, different sets ofmotion vectors for each temporal level are available inthe GOP. Depending on the analysis algorithm, one set
can be more useful than others. Motion vectors areusually coded in the header and can be easily extractedwithout further decoding. However this feature is morecodec dependent, as it also depends on the blockmatching algorithm used.
6.2 Motion Activity
Motion activity is defined as a measure of the “intensityof motion” or “pace” of a sequence as it is perceived bya human. MPEG-7[13] includes a descriptor of theintensity of motion activity. It is based on the standarddeviation of the motion vectors quantized in five levels.For this work, the quantization step is not used, in orderto have a continuous motion activity measure. For thecase of a scalable video codec, previousexperiments[35] showed that it is enough to use only theset of motion vectors from the lower temporal level toestimate the motion activity of a GOP. Using this set ofmotion vectors (see Figure 6) from the lowest temporallevel and including the area of the blocks, the intensityof motion activity descriptor is then computed as:
1
0
2)()(
1 N
ikkk
MVk mimia
Aa
0where N is the number of motion compensation blocksand )(im
is the motion vector at block i. The
parameters a(i), A and )(im
are respectively the areaof the block i, the total area, both in pixels, and the meanmotion vector. The use of the area is to cope withveriable block size motion compensation. Figure 7shows the motion activity computed using theexpression as was proposed in (0).
0 100 200 300 400 500 600 700 800 9000
10
20
30
40
50
60
70
80
Frames
Activ
ity
akiyo foreman stefan
Figure 7. Activity computed using the motion vectors.
An important drawback of the use of motion vectorsto perform compressed domain analysis is the presenceof a large amount of spurious vectors, which do notrepresent the real motion in the scene, and will dependon the specific implementation of the encoder and itsconfiguration (usually, search algorithm and window).
6.3 PCA Space Euclidean Metric
Li et al[15] argue that motion activity is still veryexpensive to compute, and a metric based on Principal
Component Analysis (PCA) is proposed. Each frame isdownsampled to a very low frame size WxH, such as8x6, and projected through PCA to a linear subspacepreserving most information in a reduced dimension.The activity is computed as the weighted euclideandistance between the first frames of two consecutiveGOPs. Only the six most significant of the 48coefficients are considered in our experiments:
kxPCAPCA
k FSTY 68 0
5
0
2
1ˆˆ
ii
PCAk
PCAk
PCAk iYiYa
0In (0) and (0), PCAT denotes the PCA transform,
68xS is the scaling operation to 8x6 size arranged in a
vector of 48 values, PCAkY and PCA
kY are the PCA
feature vector of the frame Fk and the truncated featurevector to the first 6 values, where I6x48 is the identitymatrix truncated in rows to 6x48, i are the weights,
and PCAka denotes the activity curve in the GOP k.
Thus, the activity of the GOP can be computed veryfast with this technique, avoiding decoding and using avery low amount of values. In our experiments, the PCAsubspace is computed for the whole sequence, so it hasthe drawback of having to process all the sequence firstto obtain the basis, and then process it again to obtainthe features. With this approach, it is not possible toadapt on-line the sequence, or avoiding buffering ofblocks of GOPs.
6.4 DCT Space Euclidean Metric
A possible variation of the previous metric is the use ofthe Discrete Cosine Transform (DCT). In this case, theprojection subspace is suboptimal, though very close tothe PCA subspace when the source is a Markov processof 1st order[36]. In this case, the feature vector is:
kxDCTDCT
k FSTY 68 0
where DCTT is the 1-D DCT transform. Thecorresponding activity will be
5
0
2
1ˆˆ
ii
DCTk
DCTk
DCTk iYiYa
0
The activity curves obtained using the DCT-basedmetric are very close to those obtained with the PCA-based one (see Figure 8). Besides, DCT can becomputed faster and the basis does not need to becomputed for each sequence, so it could be used inonline adaptation.
Figure 8 also shows that shot changes have anextraordinary high value of activity. Another advantageof using these metrics in a GOP basis, is the possibilityof the detection of shot changes by hard thresholding,with a GOP precision, which is enough in many cases. Itcan be detected both cuts and gradual transitions withlengths up to a half GOP (in the experiments GOP is 8frames, so up to 4 frames long). This is not possible withthe motion activity curve, as there are no motion vectorsrelating adjacent GOPs.
0 100 200 300 400 500 600 700 800 9000
100
200
300
400
500
Frames
Activ
ity
akiyo foreman stefan
PCA spaceDCT space
Figure 8. Activity computed in PCA and DCT spaces.
7 Experimental Results
The proposed framework and skimming algorithm weretested with a sequence built chaining three standardsequences: akiyo (low activity), foreman and stefan(medium-high activity). Each of these sequences has300 frames with CIF resolution at 30 frames per second.This sequence was encoded with the QMUL’s waveletSVC codec][22], using 3 temporal decompositions (4possible reconstructed frame rates), 3 spatialdecompositions (4 possible reconstructed frame sizes)and 4 quality levels, with a GOP of 8 frames. Thesequence was adapted to a terminal with a display sizeof 176x144 and 30 frames per second using theproposed semantic adaptation engine. The only semanticfeature used to guide the frame selection in this test isthe activity.
0 20 40 60 80 100 1200
50
100
150
GoP
Activ
ity
(a)
0 20 40 60 80 100 1200
2
4
6
8
GoP
Num
ber o
f fra
mes
Non filteredFiltered
(b)Figure 9. Adaptation of the test sequence. (a) Activityand thresholds; (b) Number of frames per GOP in theadapted sequence before and after smoothing filter.
The metric used to measure the activity was the DCTbased. Error: Reference source not founda shows theactivity curve and the thresholds used, setting the valueof the minimum activity threshold K to 0.6, and thenumber of skimming levels M to 8. In Error: Referencesource not foundb it is shown the number of frames perGOP before and after the smoothing filter. Comparingboth adapted sequences generated from the non filteredand filtered curves, the second has much less variationsand looks more natural. The adapted sequence is shownin Figure 10. The frames have been selected accordingto the activity. A number of 99 frames were selected outof 900. A lot of significance is given to the panning inforeman sequence, because the DCT measure (as PCAdoes as well) gives a high activity measure to the partswith camera motion. If these camera effects are notdesirable in the summarized sequence, or desirable to berepresented with fewer frames, a proper metric focusingmore on object motion should be used.
Figure 10. Example of adapted sequence.
Another consequence of discarding temporal layersis the reduction of the decoding time in the terminal, asseveral MCTF steps are avoided (see Figure 11). Thisfact can be very useful to reduce the complexity at theterminal side. For the implementation used in theexperiments, the decoding time depends more heavilyon the quality level, than on the temporal level, due toentropy decoding.
0 1 2 30
1
2
3
4
5 x 107
Temporal level
Byte
s
Bytes
0 1 2 30
0.2
0.4
0.6
0.8
1Time per GoP
Temporal level
Nor
mal
ized
tim
e
quality level 0quality level 1quality level 2quality level 3
quality level 0quality level 1quality level 2quality level 3
Figure 11. Decoding time (normalized) for full spatialresolution.
Figure 12a shows the results for different values ofthe parameters K in the test sequence. A higher valuewill lead to shorter summaries. This summary can beplayed much faster giving an effective fast overview ofthe content. If this result is used in adaptation withcontent based frame dropping the decoding time ishighly reduced (see Figure 12b). Note that skipping aGOP takes almost no time. This second application canbe very useful to adapt to complexity constrainedenvironment, due to reduced processing capability, ordue to limited battery power which is usually preferredto be maximized.
0 1 2 3 4 50
200
400
600
800
1000
Minimum activity threshold K
Num
ber o
f fra
mes
temporal level = 0temporal level = 1temporal level = 2temporal level = 3Adapted
(a)
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
0.2
0.4
0.6
0.8
1
Minimum activity threshold K
Deco
ding
tim
e (n
orm
aliz
ed) temporal level = 0
temporal level = 1temporal level = 2temporal level = 3Adapted
(b)Figure 12. Results of adaptation for diffent values of K.(a) Number of frames in the adapted sequence; (b)Decoding time reduction.
A second test was done using a higher resolutionsequence (4CIF, i.e. 704x576), built with other threestandard sequences: mobile (low activity), harbour(low-medium activity) and basket (high activity). Theresulting curve of activity and the number of frames perGoP are shown in Figure 13, for the same set ofthresholds as in the previous test. As it was expected,basket has a higher number of frames in the adaptedsequence (see Figure 14) than mobile and harbour.
0 100 200 300 400 500 600 7000
50
100
150
200
250
300
350
Activ
ity
GoP
mobile harbour basket
(a)
0 10 20 30 40 50 60 70 80 900
2
4
6
8
Num
ber o
f fra
mes
GoP
(b)Figure 13. Results for the second test sequence. (a)Activity; (b) Number of frames per GOP in the adaptedsequence.
Figure 14: Example of adapted skim for the second testsequence
Finally, in order to show the differences due to thecontent semantics (activity), the algorithm was testedwith the six basic sequences used in the previous testsequences. Figure 15 shows the number of framesselected (in percent) for each of the sequences. Due tothe online processing, the number of frames in theadapted sequence depends on the activity of the content,and it cannot be determined prior to the adaptation itself.For a given threshold, more frames are selected insequences with more activity (foreman, basket) whilefew frames are selected in static ones (akiyo).
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
10
20
30
40
50
60
70
80
90
100
Minimum activity threshold K
Num
ber o
f fra
mes
(%)
akiyoforemanstefanmobileharbourbasket
Figure 15. Comparison of the number of framesselected.
8 Comparison and relation with other
approaches
The proposed framework and skimming schemeintegrate two of the stages necessary to deliver anadapted skim or summary to a user in a constrainedenvironment. Existing approaches focus either onsummarization[14-17] or either in adaptation[1, 3, 37].In summarization works, adaptation usually is notconsidered and, if necessary, it is added as an additionalindependent stage, leading to a much less efficientsystem.
From the adaptation point of view, scalable videocoding and transcoding are the main technologies forcontent-blind adaptation. Transcoding[37] is the mostused approach for adaptation of video summaries,usually only encoding from uncompressed frames tovideo. Compressed domain transcoding helps toimprove the efficiency. However, scalable video codingis the most efficient approach to adaptation of the videosignal, in contrast to transcoding. Scalable video can beused to extract non content based skims, just selecting alower temporal version (see Figure 2a). Compared withthe proposed approach, it has the drawback of not usingthe semantic relevance, leading to less usefulsummaries. Figure 16 shows the skim resulting from theselection of the second temporal level of the bitstream ofthe second test sequence. Activity is not considered inthe adaptation. Although it has a similar number offrames compared to that resulting from the proposedsystem (see Figure 14), static and very active parts havethe same proportion of frames.
Figure 16: Example of skim generated using a noncontent based approach
Some approaches for non scalable video adaptationuse low level semantics to help the adaptation. [13] usesmotion activity for adaptive fast playback of videosequences. [18] uses a similar method to drop framesdynamically in H.264, according to a measure of theperceived motion. In this case, B frames are dropped inparts with low perceived motion. The proposed methodtakes advantage of the temporal scalability to obtain asimilar behaviour.
Metadata-driven approaches[25] to semanticadaptation separate the analysis from the adaptation.They have the advantage of enabling semanticadaptation, as semantic metadata is available before theadaptation, but they need to have parsed and analyzedpreviously the content (sometimes manually). [25]describes the use of metadata in image transcoding.Figure 2b is the equivalent version of the metadatadriven approach for scalable video. The proposedskimming system follows this architecture if theskimming curve is stored previously as metadata andincluded as semantic constraints as shown in Section 7.It has the advantage of using scalable video and MPEG-21 DIA, so the adaptation is very efficient.
From the semantic point of view, it is difficult tocompare the system with other works, as the role of theanalysis method in the paper is only illustrative of howsemantic analysis can be included into the frameworkand in the skimming method. The activity analysis is anapproach widely used in the literature[13, 18], and theresults of the system are comparable to those usingactivity analysis. It has the drawback of being online(causal) and not optimum, as no future content data isconsidered and it is not expected to obtain better resultsthan those using sophisticated and high level semanticanalysis[14, 16], usually with offline analysis, Howeverthe framework is flexible enough to use most of thesealgorithms, so the semantic quality of the summaries orskims would improve. Besides the proposed method has
the advantage of high efficiency and the adaptationintegrated with the summarization.
9 Conclusions and future work
This paper discusses the use of semantic analysis withscalable video to enable content based applicationsusing the particularities of the adaptation model,focusing on the efficiency as the main objective. Theproposed solution is a framework combining bothcompressed domain analysis and bitstream extraction. Atemporal adaptation scheme has been also proposed,taking advantage of the organization of the temporallayers in the bitstream, and using a simple activityanalysis to illustrate the integration of semantic analysis.
The use of the system for fast browsing and dynamicframe dropping of video sequences, with activity assemantic clue, has been studied in the MPEG-21 DIAstandard framework, including new semantic constraintsinto the optimization problem of adaptation. The resultis a highly efficient approach, even in the analysis, asonly few data extracted directly from the compresseddomain are necessary to compute an activity measure.
Experimental results show that, even using a simpleactivity-based keyframe selection algorithm, the adaptedvideo sequences are useful for applications such as fastbrowsing of video search results, and are also useful toreduce the decoding requirements in complexityconstrained environments, as mobile devices.
However, many aspects can be improved. A keyissue to achieve a good summary is the skimmingalgorithm itself and future work will focus on improvingthe skimming algorithm including more and bettersemantic clues, and hopefully achieving a moremeaningful summary. Another issue that is expected tobe improved is the metric that measures the perceivedquality, in order to improve the overall user’s experiencein the adaptation.
10 Acknowledgments
The author would like to acknowledge Nikola Šprljan,Marta Mrak and Ebroul Izquierdo for their help andsupport with their scalable video codec, and José M.Martínez for his suggestions and comments.
11 References
1. S.-F. Chang and A. Vetro: Video adaptation: concepts,technologies, and open issues, Proceedings of the IEEE,93 (1), 148-158 (2005)
2. A. Vetro: MPEG-21 digital item adaptation: Enablinguniversal multimedia access, IEEE Multimedia, 11 (1),84-87 (2004)
3. J. R. Ohm: Advances in scalable video coding,Proceedings of the IEEE, 93 (1), 42-56 (2005)
4. J. R. Ohm, M. van der Schaar, and J. W. Woods:Interframe wavelet coding motion picture representationfor universal scalability, Signal Processing: ImageCommunication, 19 (9), 877-908 (2004)
5. H. Schwarz, D. Marpe, and T. Wiegand: Overview of thescalable H.264/MPEG4-AVC extension. In: ImageProcessing. Proceedings of International Conference on(2006)
6. F. Pereira, P. Van Beek, A. C. Kot, and J. Ostermann:Special issue on analysis and understanding for videoadaptation, IEEE Transactions on Circuits and Systemsfor Video Technology, 15 (10), 1197-1199 (2005)
7. N. Dimitrova, H.-J. Zhang, B. Shahraray, I. Sezan, T.Huang, and A. Zakhor: Applications of video-contentanalysis and retrieval, Multimedia, IEEE, 9 (3), 42--55(2002)
8. M. Furini and V. Ghini: A video frame droppingmechanism based on audio perception. In: IEEE GlobalTelecommunications Conference Workshops, 2004, pp.211-216 (2004)
9. M. M. Yeung and B.-L. Yeo: Video visualization forcompact presentation and fast browsing of pictorialcontent, Circuits and Systems for Video Technology,IEEE Transactions on, 7 (5), 771-785 (1997)
10. H. S. Chang, S. Sull, and S. U. Lee: Efficient videoindexing scheme for content-based retrieval, Circuits andSystems for Video Technology, IEEE Transactions on, 9(8), 1269-1279 (1999)
11. S. Pfeiffer, R. Lienhart, S. Fischer, and W. Effelsberg:Abstracting digital movies automatically, Journal OfVisual Communication And Image Representation, 7 (4),345--353 (1996)
12. X. Zhu, A. K. Elmagarmid, X. Xue, L. Wu, and A. C.Catlin: InsightVideo: toward hierarchical video contentorganization for efficient browsing, summarization andretrieval, Multimedia, IEEE Transactions on, 7 (4), 648-666 (2005)
13. K. A. Peker, A. Divakaran, and H. Sun: Constant paceskimming and temporal sub-sampling of video usingmotion activity. In: Image Processing. Proceedings ofInternational Conference on, pp. 414-417 (2001)
14. Y.-F. Ma, X.-S. Hua, L. Lu, and H.-J. Zhang: A genericframework of user attention model and its application invideo summarization, Multimedia, IEEE Transactionson, 7 (5), 907-919 (2005)
15. Z. Li, G. M. Schuster, A. K. Katsaggelos, and B. Gandhi:Rate-distortion optimal video summary generation, IEEETransactions on Image Processing, 14 (10), 1550-1560(2005)
16. C.-W. Ngo, Y.-F. Ma, and H.-J. Zhang: Automatic videosummarization by graph modeling. In: Computer Vision,
2003. Proceedings. Ninth IEEE International Conferenceon, pp. 104-109 (2003)
17. M. A. Smith and T. Kanade: Video skimming andcharacterization through the combination of image andlanguage understanding. In: Content-Based Access ofImage and Video Database, 1998. Proceedings., 1998IEEE International Workshop on, pp. 61-70 (1998)
18. Z. Gang, L. T. Chia, and Y. Zongkai: MPEG-21 digitalitem adaptation by applying perceived motion energy toH.264 video. In: Image Processing, 2004. InternationalConference on, pp. 2777-2780 (2004)
19. W. Lai, X. D. Gu, R. H. Wang, L. R. Dai, and H. J.Zhang: Perceptual video streaming by adaptive spatial-temporal scalability, In: Advances in MultimediaInformation Processing - PCM 2004, Lecture Notes inComputer Science (3332), Springer-Verlag Berlin, pp.431-438 (2004)
20. H. J. Cha, J. H. Oh, and R. Ha: Dynamic frame droppingfor bandwidth control in MPEG streaming system,Multimedia Tools and Applications, 19 (2), 155-178(2003)
21. S. T. Hsiang and J. W. Woods: Embedded video codingusing invertible motion compensated 3-Dsubband/wavelet filter bank, Signal Processing: ImageCommunication, 16 (8), 705-724 (2001)
22. N. Sprljan, M. Mrak, G. C. K. Abhayaratne, and E.Izquierdo: A scalable coding framework for efficientvideo adaptation. In: Proc. Int. Work. on Image Analysisfor Multimedia Interactive Services (WIAMIS) (2005)
23. J. R. Ohm: Three-dimensional subband coding withmotion compensation, Image Processing, IEEETransactions on, 3 (5), 559--571 (1994)
24. P. M. Fonseca and F. Pereira: Automatic videosummarization based on MPEG-7 descriptions, SignalProcessing: Image Communication, 19 (8), 685-699(2004)
25. P. van Beek, J. R. Smith, T. Ebrahimi, T. Suzuki, and J.Askelof: Metadata-driven multimedia access, SignalProcessing Magazine, IEEE, 20 (2), 40--52 (2003)
26. K. Shen and E. J. Delp: A fast algorithm for videoparsing using MPEG compressed sequences. In: ImageProcessing, 1995. Proceedings., International Conferenceon, pp. 252-255 (1995)
27. H. L. Wang, A. Divakaran, A. Vetro, S. F. Chang, and H.F. Sun: Survey of compressed-domain features used inaudio-visual indexing and analysis, Journal of VisualCommunication and Image Representation, 14 (2), 150-183 (2003)
28. J. Bescos: Real-time shot change detection over onlineMPEG-2 video, Circuits and Systems for VideoTechnology, IEEE Transactions on, 14 (4), 475-484(2004)
29. S. Jeannin and A. Divakaran: MPEG-7 visual motiondescriptors, IEEE Transactions on Circuits and Systemsfor Video Technology, 11 (6), 720-724 (2001)
30. Y.-P. Tan, D. D. Saur, S. R. Kulkami, and P. J. Ramadge:Rapid estimation of camera motion from compressedvideo with application to video annotation, Circuits andSystems for Video Technology, IEEE Transactions on, 10(1), 133--146 (2000)
31. R. V. Babu, K. R. Ramakrishnan, and S. H. Srinivasan:Video object segmentation: a compressed domainapproach, Circuits and Systems for Video Technology,IEEE Transactions on, 14 (4), 462--474 (2004)
32. D. Mukherjee, E. Delfosse, J. G. Kim, and Y. Wang:Optimal adaptation decision-taking for terminal andnetwork quality-of-service, IEEE Transactions onMultimedia, 7 (3), 454-462 (2005)
33. T. Zgaljic, N. Sprljan, and E. Izquierdo: Bitstream syntaxdescription based adaptation of scalable video. In:Integration of Knowledge, Semantics and Digital MediaTechnology, 2005. EWIMT 2005. The 2nd EuropeanWorkshop on the (Ref. No. 2005/11099), pp. 173-178(2005)
34. M. H. Chan, Y. B. Yu, and A. G. Constantinides: Variablesize block matching motion compensation withapplications to video coding. In: Communications,Speech and Vision, IEE Proceedings I, pp. 205-212(1990)
35. L. Herranz, F. Tiburzi, and J. Bescós: Extraction ofMotion Activity from Scalable-Coded Video Sequences,In: Semantic Multimedia, Lecture Notes in ComputerScience (4306), Springer-Verlag Berlin, pp. 148-158(2006)
36. M. Hamidi and J. Pearl: Comparison of the cosine andFourier transforms of Markov-1 signals, IEEETransactions on Signal Processing on Acoustics, Speechand Signal Processing, 24 (5), 428-429 (1976)
37. I. Ahmad, X. Wei, Y. Sun, and Y.-Q. Zhang: Videotranscoding: an overview of various techniques andresearch issues, IEEE Transactions on Multimedia, 7 (5),793-804 (2005)