Signal Processing: Image Communication 21 (2006) 550561

A new method for the chao

o A

ica de

form

A great deal of the audiovisual content used for broadcasting and automatic video indexing comes from lm. This lm

A great deal of the audiovisual content used for rate of 30 fps in the case of the NTSC system and1

ARTICLE IN PRESS

www.elsevier.com/locate/image

of Valencia interdisciplinary project 5607-2004 and the Cicyt

project TIC 2002-02469.Corresponding author. Tel.: +34 963877746; 1NTSC is the TV standard used in America and Japan, PAL is0923-5965/$ - see front matter r 2006 Elsevier B.V. All rights reserved.

doi:10.1016/j.image.2006.03.006

fax: +34963877309.

E-mail address: vnaranjo@dcom.upv.es (V. Naranjo).

the standard used in the rest of the world, as in Britain, Germany,

Spain, etc.broadcasting and automatic video indexing comesfrom lm. These contents range from material shot,in the early days of lm production, using camerasof 12; 18; . . . frames per second (fps) up to lmsrecorded at 24 fps, the frame rate that has been usedin lm production since the introduction of soundtracks [1].

25 fps for the PAL system. This conversion isusually carried out by a telecine using the inter-polation method of eld repetition [1,2,5]. Fig. 1depicts the telecine process in the NTSC system.This process is known as 3:2 Pull-down and consistsof obtaining ve video frames from four lmframes, thus converting the 24 fps frame rate to30 fps. As shown in the gure, each frame of the setis separated into its two elds (the odd eld,containing the odd frame lines, and the even, with$This work has been supported by the Polytechnic Universityor 30 fps which are the frame rates used in TV broadcasting. This conversion is carried out by a telecine and consists of a

periodic repetition of elds. In many applications the need arises to reverse this conversion (inverse telecine). This paper

presents a statistical method for automatic removal of the repeated elds present in the video sequence. The method

consists of two steps: the rst is to determine, using an energy detector, if the sequence has been telecined. Then, the two

parameters necessary for removing the telecine effect are determined: the system for which the lm has been telecined (for

NTSC or PAL system) and the location of one repeated eld.

r 2006 Elsevier B.V. All rights reserved.

Keywords: Video analysis; PAL and NTSC; Telecine removal; w-square distribution

1. Introduction However, for broadcasting, these materials mustbe converted to video-format, which has a framecontent has to undergo a conversion of its frame rate, from 24 fps, the characteristic frame rate of lm recording, to 25 fpseffect in vide

Valery Naranjo, Luis Vergara, Antoni

Departamento de Comunicaciones, Universidad Politecn

Received 15 July 2005; received in revised

Abstractracterization of telecinesequences$

lcaraz, Jose M. Mossi, Antonio Albiol

Valencia, Camino de Vera s/n, 46022 Valencia, Spain

2 March 2006; accepted 3 March 2006

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561 551the even frame lines). Then, in each set of fourframes, two elds are repeated (one frame in total),to be precise, the rst eld of the second lm frame(2A), which is placed after the second eld of thesecond frame (2B), and the second eld of thefourth frame (4B). Finally, the ve video framesobtained from a set of four lm frames, are made upas shown in Fig. 1. Similarly, for the PAL system,the method used for converting the frame rate from24 to 25 fps is by repeating two elds in a set of 24lm frames, the rst eld of the frame 12 (12A) andthe second eld of 24th frame (24B), as Fig. 2illustrates.Another feature of the method is the eld

inversion after the repetition of the rst eld (2A,in NTSC and 12A in PAL). From this point on, theB elds of a lm frame are rst in time in the videoframe before the A elds (13B-13A-14B-14Ay24B-24A in PAL and 3B-3A-4B-4A in NTSC). Thisinversion is required in order to guarantee the eldorder in the sequence (ABABABy.). Therefore, the

Fig. 1. Conversion from lm frames tovideo frames composed after the repetition of the Aeld will consist of elds which belong to differentlm frames.The rate conversion procedure, implemented as

we have explained above, causes artifacts in thosevideo frames which are made up of elds belongingto different lm frames, such as video frames 3 and4 shown in Fig. 1 and video frames 13 to 24 depictedin Fig. 2. The artifacts which appear in these videoframes are often referred to as jaggies [2] because theedges of moving objects appear staggered or jagged.This effect can be observed in the video framesshown in Figs. 3 and 4, where a correctly convertedvideo frame and a defective frame are presentedfrom two different lms.

1.1. Problem definition

In many applications, such as video indexing orold lm restoration, the need to reverse the telecineconversion arises. A great deal of these applications

video frames in the NTSC system.

ARTICLE IN PRESS

Fig. 2. Conversion from lm frames to video frames in the PAL system.

Fig. 3. Video frames from the sequence EggandUss from the Open Video Project, a shared digital video collection available in http://

www.open-video.org: (a) video frame composed of two elds from the same lm frame; (b) video frame composed of two elds from two

different lm frames.

V. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561552

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561 553need to implement a temporal segmentation of thesequence, and if the input of this process is atelecined sequence, the process will probably fail. Inparticular, the cut detection process [3] (detection ofabrupt scene changes) will fail in a situation like theone depicted in Fig. 5, where the two elds whichcompose the video frame belong to different shots.Therefore, before carrying out the cut detection, wehave to perform an inverse telecine process. Toreverse telecine automatically, rst the telecinesystem used in the original process must be

Fig. 4. Video frames of the sequenceMiss Taronja provided by the Film

the same lm frame; (b) video frame composed of two elds from two

Fig. 5. Video frame from the sequence Taronja. The frame are composed

occurs.determined (PAL or NTSC) and then, we have tosearch out a repeated eld. If the eld found is of Atype, it is removed and the following elds arereordered (inverted), up to the repeated B eld,which is also removed. This process is repeatedthroughout the whole sequence, with a period equalto 5 for the NTSC system and 25 for PAL.In addition, the inverse telecine process provides

another advantage in video indexing and storage:the elimination of the redundant informationintroduced by the eld repetition.

Archive of Valencia: (a) video frame composed of two elds from

different lm frames.

of two elds from two different lm frames where a scene change

telecined, the inverse telecine process has to be

of t

elds.

slow variation, wn the stationary white Gaussianno

ofFoth

3.

th

from a telecined sequence. The rst step of the

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561554This paper is organized as follows. Section 2describes the statistical model of the differencesignal. In Section 3 the telecine presence detector isexplained. Section 4 describes the method used todetermine the essential parameters to reverse thetelecine process, T and n0. Finally, some experi-mental results are given in Section 5 and ourconclusions are summarized in Section 6.

2. Signal modelling

Before describing the method of reversing thetelecine process, we have to obtain the most suitablesignal for our purpose. As we have seen above, theframes whose elds belong to the same originalframe do not present any artifacts. This is becausethe elds belonging to the same original framecontain fewer differences than those which belongcomto dhe difference signal. This signal is obtained byputing the differences between the video frameThecarried out. For that, two parameters have to beestimated: the period of eld repetition, T(T 5 for NTSC and T 25 for PAL) andthe frame number containing the rst repeatedeld, n0 0pn0pT.

detection method is based on a statistical studySpecically, we have developed this method as arst stage in old lm restoration. So, we assume thatthe audio-visual material to be analyzed has notbeen edited and, therefore, each lm has only onekind of telecine parameters (or PAL or NTSC or itis not telecined).In the literature, there are few references to the

subject of inverse telecine. Atkinson [4] proposes amethod for implementing the inverse 3:2 pulldown,i.e. specic for the NTSC system, based on multiplecomparisons of elds in order to nd thoserepeated. A similar method is presented by Acker-man [1]. In this paper we present a method that notonly carries out the inverse 3:2 pulldown, but alsothe inverse telecine for the PAL system. It alsoautomatically determines the telecine system and ifthe sequence has been telecined or not. The processis divided into two stages:

(1) Detection stage: to detect if the sequence hasbeen telecined.

(2) Estimation stage: if the sequence has beenifferent original frames. This suggests that theinverse telecine consists of detecting whether thesequence has been telecined or not. We propose thatthis previous decision be implemented by an energydetector, as described below.The rst step consists of obtaining the difference

between the elds of each frame. Fig. 6 shows anexample of the difference signal for a telecinedsequence, and in Fig. 7 an example of the differencesignal for a non-telecined sequence is depicted.reqCoIn Section 4 we propose a method for reversinge telecine process. As we will see, this methoduires the input to be a difference signal obtainedise and rn the non-Gaussian impulsive noise.In the following sections the two different stagesthe telecine reversion process will be explained.r both stages, the input signal is that described inis section and modelled in Eq. (1).

Detection stagemappropriate signal for the input of the inversetelecine processor is the difference between elds ofthe video (telecined) frame.An example illustrating the appearance of this

difference signal is displayed in Fig. 6, where thedifference between the elds of the video frames ispresented. This sequence came from a lm whichhad been PAL-telecined. The signal, as we canobserve in Fig. 6, presents various components:

Amplitude modulated periodic component: aperiodic train of pulses whose amplitude variesslowly.

Noise components: there are two noise compo-nents: Random noise, which causes non-impulsiverandom variations in the signal. We willassume this noise to be zero mean stationarywhite Gaussian.

Impulsive noise, which provokes high impul-sive peaks in the signal.

Thus, a mathematical representation of the elddifference signal is depicted in Eq. (1).

xn Xk

akY n n0 kT

T=2

wn rn, (1)

where ak is the random amplitude of the pulses, withparing the signals, we see that the main

ARTICLE IN PRESS

0 50 100 150 200 250 300 350 400 450 5000

5

10

15

n

x[n]

Fig. 7. Field differences xn vs. frame number n.

0 200 400 600 800 1000 1200 1400 1600 18000

10

20

30

40

50

60

70

80

90

n

x[n]

Fig. 6. Field differences xn vs. frame number n.

V. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561 555

difference is the absence of a periodic component inthe signal from a non-telecined sequence, beingcomposed only of the random components. There-fore, if both signals are normalized in variance, theenergy of the signals from the telecined sequenceswill be higher than the energy of the signals from thenon-telecined sequences.

3.1. Description of the detector

According to the assumption mentioned above,previous to the detection we have to estimate thenoise variance in order to normalize the detectorinput, i.e. the difference signal. To do this, almost10 000 frames, from non-telecined sequences, wereanalyzed. The sequences were decoded and severalregisters of 1000 frames have been extracted fromea

ENx is compared with a threshold which isselected to achieve the desired false alarmprobability. In order to select the threshold, westudy the dependency of this threshold on thefalse alarm probability for difference signals fromnon-telecined sequences. In this case, the sampleenergy (of a Gaussian noise) follows a w-squaredistribution with as many degrees of freedom assignal samples [8]. In a w-square distribution,the probability of a false alarm depends on thethreshold in the form presented in Fig. 9. If thesample energy exceeds the threshold, the se-quence will be detected as telecined. Otherwise,the sequence will be labelled as non-telecined.

4. Estimation stage

Once the non-telecined sequences have been

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 5505615564 2 0 2 4 6 80

50

100

150

200depicted in Fig. 8. The shape of this histogramsuggests that the difference signal (the mean hasbeen subtracted) from a non-telecined sequence canbe modelled as a Gaussian noise. Finally, we alsoestimated the noise variance, obtaining an averagevalue of 2.5.Once the signal from the non-telecined sequence

has been modelled, the detector consists of thefollowing steps:

From the sequence (whether telecined or not), thedifference signal is obtained.

250

300

350

400

450

500reghisch one. The difference signal of a part of oneister is shown in Fig. 7 and the estimatedtogram, using the whole set of registers, isFig. 8. Histogram of signal xn x. diswhere xn is the difference signal, x is the samplemean of xn and 2.5 the estimated value of thenoise variance.The sample energy of the normalized signal iscalculated as:

ENx XNn1

x2N n.This signal is normalized in variance accor-ding to:

xN n xn x

2:5p ,800 850 900 950 1000 1050 1100 1150 1200 1250 13000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1PFA/threshold

PFA

threshold

Fig. 9. Probability of false alarm versus threshold for a w-squaredistribution with 1000 degrees of freedom.carded in the detection stage, the next step in

the reversion process is the estimation stage. Theaim of this estimation stage is to determine theperiod T and the initial phase n0 of the pulsetrain for the signal given in Eq. (1). The perioddepends on the type of television system in whichthe lm has been telecined: in PAL/SECAMsystems T 25 and in the NTSC system T 5. Itwill therefore only be necessary to decide betweentwo different periods. After the knowledge of T andn0, reversing the rate conversion becomes straight-forward, as we explained in Section 1.The maximum likelihood (ML) estimator of T

and n0 maximizes the conditional probability

density function of xn : Pfxg=n0;T, where x isthe observation vector x x1; . . . ; xNT .To tackle this maximization problem directly on

x is very complicated even considering the model ofEq. (1). However, starting from this model, beforemaximizing Px=n0;T , a more simplied signalxpn is obtained from xn using the processdescribed in the block diagram of Fig. 10. Thisprocess consists of two steps:

Gradient: This block obtains the differencesbetween consecutive samples of signal xn. Bymeans of this operation, the inuence of ak isrelaxed, and the pulses are turned into impulses;as a difference lter is in essence a high-passlter. Examples of the input and output signals ofthis block, for the rst 1000 samples of Fig. 6, areshown in Fig. 11.

Tophat: We are interested in the positive differ-ence impulses with duration 1, because we want

ARTICLE IN PRESS

0 100 200 300 400 5000

20

40

60

80

100

n

x[n]

5

100

Fig. 10. Block diagram of the preprocessor.

V. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561 5570 100 200 300 400100

50

0

50

grad

(x[n])Fig. 11. Field differences xn and it00 600 700 800 900 1000ns gr600 700 800 900 1000adient vs. frame number n.

to know the transition between correctly anduncorrectly ordered elds. To extract thesedifference impulses, the tophat operator [6,7] isapplied, using a structuring element of size 2. Thetophat is a morphological operator which ex-tracts the positive impulses of a signal with awidth smaller than the structuring element size.Finally, we build a binary signal which will be 1in those positions where the result of the gradientfunction has a positive impulse of width 1, and 0otherwise.

As a result of the simplication process, we obtain asignal like the one shown in Fig. 12. This is a binarysignal which ideally should be equal to 1 at n n0 kT and 0 at nan0 kT . However, in general,

P )

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561558we can also nd some wrong values: 0 at somen n0 kT and 1 at some nan0 kT .To approach the problem in a statistical manner,

we consider that the mathematical expression forthe signal xpn is:xpn

Xk

bkdn n0 kT X

man0kTcmdnm,

where fbkg are independently and identically dis-tributed random variables with probability

pbb P00db 1 1 P00db, (2)P00 being the probability of having 1 in a correctposition n n0 kT.Similarly, fcmg are independently and identically

distributed random variables with probability

pcc P0dc 1 P0dc 1, (3)

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

1

n

x p[n]Fig. 12. Simplied signal xpn vs. frame number n.This probability must be maximized to obtain theML estimators of T and n0.Using Eqs. (2) and (3) in Eq. (4), results in

Pxp=n0;T Y

nn0kTP00dxpn 1

1 P00dxpn

Ynan0kT

P0dxpn

1 P0dxn 1and nally the conditional probability can beexpressed as

Pxp=n0;T P00L0 1 P00M

0 P0L1 P0M(5)

where L0 and M 0 are the number of xpn sampleswhich are 1 and 0, respectively, in timesn n0 kT . So, L0 will be the number of correctdetections of frame change (i.e., instant where thesequence changes from well ordered frame tobadly ordered frame) and M 0 the number ofmisdetections. On the other hand, L and M are thenumber of xpn samples which are 1 and 0,respectively, in times nan0 kT . Thus, L is thenumber of false indications of frame change and Mthe number of correct indications of no frame change.Considering that P0o0:5 and that P0040:5, a

hypothesis which can be corroborated in practice,the ML estimator implies maximizing the expressionof the conditional probability described in Eq. (5),so that we must search for the pair n0;T which:

Produces the maximum number of 1s at thepositions n n0 kT , that is, maximizes L0.

Produces the minimum number of 1s at thepositions nan0 kT , that is, minimizes L.

4.1. Practical estimator implementation

Instead of tackling the minimization of Eq. (5)

dirxp=n0;T YNn1

Pxpn=n0;T. (4P0 being the probability of having 1 where itshould be a 0 (in nan0 kT).With the above hypotheses, the samples of xpn

are independent. Thus, the conditional probabilitydensity function of the observations of xpn,represented by vector xp, are:ectly, testing all possible pairs of n0;T and

computing L0 and L until the best couple is found,we propose a more efcient method describedbelow.A sequence xT ;mn is built, in the following way:

xT ;mn Xk

dnm kT ; 1pnpN

after that, the difference d xp xT ;m is obtained.The objective of the estimator is to nd the pairT ;m which minimizes:kdk2 kxp xT ;mk2 xp xT ;mTxp xT ;m

xTpxp xTT ;mxT ;m 2xTpxT ;mwhere

The term xTpxp does not depend on T and m. xTT ;mxT ;m is the sample energy of signal xT ;mn. And xTpxT ;m is the cross-correlation between xpnand xT ;mn Rxp ;xT m.

Therefore, the minimization of kdk2 is equivalent to

x5n, (period of the NTSC telecined signal). For eachsignal, the sample energy will be calculated, as well asthe maximum of the cross-correlation between eachsignal and the pre-processed signal xpn. The selectedperiod will be the period of the signal with theminimum difference between its sample energy andthe maximum of the cross-correlation. Once theperiod of the signal is decided, the system of thetelecined signal has been determined. Next, n0 isobtained as the arg maxfRxT ;xp mg.The different results obtained during the process,

in a particular case, are illustrated in Figs. 14 and15. In this example, the input signal xpn, shownin Fig. 12, corresponds to the eld differencesof a PAL telecined sequence. We can observe thetwo constructed sequences x25 and x5 in Figs. 14 (a)and (b), respectively. Finally, Figs. 15(a) and (b)represent the signals yNTSCn0 and yPALn0. Theminimum appears in the PAL branch and is markedin Fig. 15 (b) with a *.

5.

applied to a set of real sequences from different

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561 559minimizing the difference between the energy ofsignal xT ;m and the cross-correlation between xp andxT ;m. Both operations have little computationalcost, which makes the method very efcient.The block diagram is shown in Fig. 13. The

procedure consists of constructing two impulse trains,one whose period is 25, x25n, (period of the PALtelecined signal) and the other with a period of 5,Fig. 13. Block diagramsources:

Sequences with NTSC telecine: from the OpenVideo Project, a shared digital video collectionTofExperimental results

he method described in this paper has beenthe estimator.

AseqofAhiswhacenor10

ARTICLE IN PRESSx

[n]

(b

Fig

Tsys

100

V. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 5505615601(a 0 10 20 30 40 50 60 70 80 90 1000

n)0.2

0.4

0.6

0.8

1

x 5[n]available in http://www.open-video.org: EggAn-dUss1952.mpg, UGS04.mpg, SeeingLo1920.mpeg,ManofAct1955.mpeg, LivingSt1958.mpeg, Gold-enGa1936.mpeg, Children1940.mpeg.Sequences with PAL telecine: from the Film Archiveof Valencia, the lm La Senorita Naranja.Sequences without telecine from commercial lmsas TheMask of Zorro, Shrek or Malvaloca.

total of 100 000 frames have been analyzed. Theuences have been decoded and several registers1000 frames have been extracted from each one.set of these frames has been used to compute thetogram of the non-telecined signal (see Fig. 8),ich has allowed us to select the threshold tohieve the desired false alarm probability in theergy detector which was nally set at 1100, inder to achieve a false alarm probability of 22 (according to Fig. 9).

0 10 20 30 40 50 60 70 80 90 1000

0.2

0.4

0.6

0.8

n

25

)

. 14. Signals in the estimation process: (a) signal of period

5. NTSC system, x5n; (b) signal of period T 25. PALtem, x25n.0 5 10 15 20 250

50

m

[m] 40

60

80(a)150

200

250

300

350

400

E N(x 5

)Rx p

,x 5

[m]100 sets of 1000 frames per set have been analyzedobtaining only one case where the system failed tolabel a register as telecined when it came from anon-telecined sequence. The estimated false alarmprobability in the energy detector is therefore 0.01.The telecine system detector had no false alarms andthe pair T ; n0 were always correctly estimated.

6. Conclusions

In this paper, we present a method of reversingthe telecine process, which consists of returning thesequences from video frame rate (25 or 30 fps) tocinema frame rate 24 fps, eliminating the repeatedelds inserted by the telecine.Our method uses the difference signal between the

two elds of a video frame and automaticallyestimates the parameters needed to accomplish the

E N(x 2

5)R

x p,x 2

5

0 5 10 15 20 2560

40

20

0

20

m(b)

Fig. 15. Signals in the estimation process: (a) estimator result in

the branch of the NTSC system; (b) estimator result in the branch

of the PAL system. The minimum detected in (*).

inverse telecine process, previously discarding thedifference signals from non-telecined sequences.The algorithm is based on the hypothesis of a

model for the signal difference of both telecined andnon-telecined sequences. In order to corroborate themodel and to tune the algorithm parameters, a setof real sequences (test set) has been analyzed.The method has been tested using a great deal of

sequences from different types of lm, including oldand modern, telecined in NTSC or PAL and alsonon-telecined sequences. In all cases, the perfor-mance of the algorithm has been highly satisfactory,as shown by the results presented in Section 5.

Acknowledgements

We would like to thank the RD i LinguisticAssistance Ofce at the Universidad Politecnica ofValencia and Mike Hardinge for their help inrevising this paper.

References

[1] S. Ackerman, Film sequence detection and removal in dtv

format and standards conversion, White Paper, Teranex, Inc.,

http://www.teranex.com/support/whitepapers.htm.

[2] Advanced video scaling, Educational guide, A Communica-

tions Specialties, Inc., http:www.roscor.com/resources-white

papers/ed-adv-video-scaling.pdf.

[3] A. Albiol, V. Naranjo, J. Angulo, Low complexity cut detection

in the presence of icker, in: IEEE (Ed.), Proceedings of the

International Conference of Image Processing, 2000.

[4] K. Atkinson, A better method of telecine removal, Draft,

Kevin at atkinsons dhs org, http://kevin.atkinson.dhs.org/tel/

method.pdf.

[5] R. Roberts, Television Engineering I and II, Pentech Press,

London, 1987.

[6] J. Serra, Image Analysis and Mathematical Morphology,

Academic Press, London, 1982.

[7] J. Serra, Image Analysis and Mathematical Morphology,

vol. II: Theoretical advances, Academic Press, London,

1988.

[8] K. Shanmugan, A.M. Breipohl, Random Signals, Detec-

tion, Estimation and Data Analysis, Wiley, New York,

1988.

ARTICLE IN PRESSV. Naranjo et al. / Signal Processing: Image Communication 21 (2006) 550561 561

A new method for the characterization of telecine effect in video sequencesIntroductionProblem definition

Signal modellingDetection stageDescription of the detector

Estimation stagePractical estimator implementation

Experimental resultsConclusionsAcknowledgementsReferences