Animation of Synthetic Faces in MPEG-4

Jiirn OstermannAT&T Labs - Research,

Room 3-23 1, 100 Schultz Dr., Red Bank, NJ, 07701,email: ostermam@-esearch.att.com

AbstractMPEG-4 is the first international standard that stan-

dardizes true multimedia communication - includingnatural and synthetic audio, natural and synthetic video,as well as 30 graphics. Integrated into this standard isthe capability to define and animate virtual humans con-sisting of synthetic heads and bodies. For the head, morethan 70 model-independent animation parameters de$n-ing low-level actions like move left mouth corner up tohigh-level parameters like facial expressions and visemesare standardized. In a communication application, theencoder can dejne the face model using MPEG-4 BInalyFormat for Scenes (BIFS) and transmit it to the decoder.Alternatively, the encoder can rely on a face model avail-able at the decoder. The animation parameters are quan-tized, predictively encoded using an arithmetic encoder ora DCT. The decoder receives the model and the animationparameters in order to animate the model. Since MPEG-4defines the minimum MPEG-4 terminal capabilities inpro$les and levels, the encoder knows the quality of theanimation at the decoder.

1. IntroductionThe goal of MPEG4 is to provide a new kind of stan-

dardization that responds to the evolution of technology,when it does not always make sense to specify a rigidstandard addressing just one application. MPEG-4 willallow the user to configure and build systems for manyapplications by allowing flexibility in the system configu-rations, by providing various levels of interactivity withaudio-visual content of a scene, and by integrating asmany as possible audio visual data types like natural andsynthetic audio, video and graphics [1][2]. MPEG4 willbecome an International Standard in spring 1999, just intime for the new faster and more powerful media proces-sors and in time for using the upcoming narrow- andbroadband wired and wireless networks for audio-visualapplications like database browsing, information retrievaland interactive communications.

As far as synthetic multimedia contents are concerned,MPEG4 will provide synthetic audio like structuredaudio and a text-to-speech interface (TTSI). For syntheticvisual contents, MPEG4 allows to build 2D and 3D ob-

jects composed of primitives like rectangles, spheres, in-dexed face sets and arbitrarily shaped 2D objects, The3D-object description is based on a subset of VRMLnodes [3] and extended to enable seamless integration of2D and 3D objects. Objects can be composed into 2D and3D scenes using the Binary Format for Scenes (BIFS).BIFS also allows to animate objects and their properties.

Special 3D objects are human faces and bodies.MPEG-4 allows using decoder resident proprietary mod-els as well as to transmit 3D models to the decoder suchthat the encoder can predict the quality of the presentationat the decoder. This paper focuses on the tools MPEG4provides to describe and animate 3D face models.

In Sections 2, we explain how MPEG-4 defines thespecification of a face model and its animation using fa-cial animation parameters (FAP). Section 3 provides de-tails on how to efficiently encode FAPs. The integrationof face animation into the overall architecture of anMPEG-4 terminal with text-to-speech capabilities isshown in Section 4. The capabilities of an MPEG-4 ter-minal are defined in profiles specifying a predefined setof tools and performance parameters for these tools asdiscussed in Section 5.

2. Specification and Animation of FacesMPEG4 specifies a set of face animation parameters

(FAPs), each corresponding to a particular facial actiondeforming a face model in its neutral state. The FAPvalue for a particular FAP indicates the magnitude of thecorresponding action, e.g., a big versus a small smile. Aparticular facial action sequence is generated by deform-ing the face model in its neutral state according to thespecified FAP values for the corresponding time instant.Then the model is rendered onto the screen.

The head in its neutral state is defined as follows(Figure 1): Gaze is in direction of Z axis; all face musclesare relaxed; eyelids are tangent to the iris; the pupil is onethird of IRISDO; lips are in contact; the line of the lips ishorizontal and at the same height of lip comers; the mouthis closed and the upper teeth touch the lower ones; thetongue is flat, horizontal with the tip of tongue touchingthe boundary between upper and lower teeth.

491087-4844198 $10.00 0 1998 IEEE

For the renderer to interpret the FAP values using itsface model, the renderer has to have predefined modelspecific animation rules to produce the facial action corre-sponding to each FAP. Since the FAPs are required toanimate faces of different sizes and proportions, the FAPvalues are defined in face animation parameter units(FAPU). FAPU are defined as fractions of distances be-tween key facial features (Figure 1). These features likeeye separation, eye-nose separation, mouth nose separa-tion, and mouth width, are defined for the face in its neu-tral state. They allow interpretation of the FAPs on anyfacial model in a consistent way, producing reasonable

results in terms of expression andspeech pronunciation.

MPEG4 defines the animationrule for each FAP by specifyingfeature points and their direction ofmovement. The renderer can eitheruse its own animation rules for hisproprietary model or download theface model and the FaceDefTablesthat define the animation rules forthe model.

Figure 1: FAPUs [2].

In the following sections, we describe first howMPEG4 defines the shape of a generic face model in itsneutral state using feature points. Then we explain thefacial animation parameters for this generic model. Fi-nally we show how to define an MPEG4 compliant facemodel that can be transmitted from the encoder to thedecoder for animation.

2.1. Face Feature PointsIn order to define face animation parameters for arbi-

trary face models, MPEG-4 specifies 84 feature pointslocated in a face according to Figure 2 in order to provide areference for defining facial animation parameters. Somefeature points like the ones along the hairline are not af-fected by FAPs. They are required for defining the shapeof a proprietary face model using feature points (Section5). Feature points are arranged in groups like cheeks, eyes,and mouth (Table 1). The location of these feature pointshas to be known for any MPEG4 compliant face model.

2.2. Face Animation ParametersThe FAPs are based on the study of minimal percepti-

ble actions and are closely related to muscle action [4].The 68 parameters are categorized into 10 groups relatedto parts of the face (Table 1). FAPs represent a completeset of basic facial actions including head motion, tongue,eye, and mouth control. They allow the representation ofnatural facial expressions. They can also be used to definefacial action units [5]. Exaggerated values permit the

definition of actions that are normally not possible forhumans, but are desirable for cartoon-like characters.The FAP set contains the two high-level parameters vise-mes and expressions (FAP group 1). A viseme is a visualcorrelate to a phoneme. Only 14 static visemes that areclearly distinguished are included in the standard set(Table 2). In order to allow for coarticulation of speechand mouth movement [6], transitions from one viseme tothe next are defined by blending the two visemes with aweighting factor. Similarly, the expression parameter de-fines 6 high level facial expressions like joy and sadness(Figure 3). In contrast to visemes, facial expressions areanimated with a value defining the excitation of the ex-pression. Two facial expressions can be blended with aweighting factor. Since expressions are high-level anima-tion parameters, they allow animating unknown modelswith high subjective quality.

Table 1: FAP groups.

2.3. Face Model SpecificationMPEG4 allows the encoder to completely specify theface mode1 the decoder has to animate. This involves de-fining the static geometry of the face mode1 in its neutralstate using a scene graph and defining the animation rulesusing FaceDefTables that specify how this model getsdeformed by the facial animation parameters [7].

2.3.1. Static Geometry using a Scene Graph

The static geometry of the head model is defined witha scene graph specified using MPEG4 BIFS [I]. For thepurpose of defining a head model, BIFS provides thesame nodes as VRML. VRML and BIFS describe geo-metrical scenes with objects as a collection of nodes, ar-ranged in a scene graph. Three types of nodes are of par-ticular interest for the definition of a static head model. AGroup node is a container for collecting child objects: itallows for building hierarchical models. For objects tomove together as a group, they need to be in the sameTransform group. The Transform node defines geometric

50

3

3.10

Right eye3.9

Left e y e g6T9 97

’ Feature points affected by FAPs

0 Other feature pants

Figure 2: Feature points may be used to de-fine the shape of a proprietary face model.FAPs are defined by motion of featurepoints [2].

Figure 3: Primary facial expressions.affine 3D transformations like scaling, rotation and trans-lation that are performed on its children. When Transformnodes contain other Transforms, their transformation set-tings have a cumulative effect. Nested Transform nodescan be used to build a transformation hierarchy. An In-

dexedFaceSet node defines the geometry (3D mesh) andsurface attributes (color, texture) of a polygonal object.Texture maps are coded with the wavelet coder of theMPEG texture coder. Since the face model is specifiedwith a scene graph, this face model can be easily extendedto a head and shoulder model.

Table 2: Visemes and related phonemes.

2.3.2. Animation Rules using FaceDefT ables

A FaceDefTable defines how a model is deformed as afunction of the amplitude of the FAP. It specifies, for aFAP, which Transform nodes and which vertices of anIndexedFaceSet node are animated by it and how.FaceDefTables are considered to be part of the facemodel.

Animation Definition for a Transform Node: If aFAP causes solely a transformation like rotation, transla-tion or scale, a Transform node can describe this anima-tion. By means of a FaceDeffransform node, theFaceDefTable specifies the type of transformation and aneutral factor for the chosen transformation. During ani-mation, the received value for the FAP and the neutralfactor determine the actual value.

Animation Definition for an IndexedFaceSet Node:If a FAP like smile causes flexible deformations of theface model, the animation results in updating vertex posi-tions of the affected IndexedFaceSet nodes. The affectedvertices move along piece-wise linear trajectories thatapproximate flexible deformations of a face. A vertexmoves along its trajectory as the amplitude of the FAPvaries. By means of a FaceDefMesh node, the FaceDef-Table defines for each affected vertex its own piece-wiselinear trajectory by specifying intervals of the FAP am-plitude and 3D displacements for each interval (Table 3).

If Pm is the position of the mth vertex in the Indexed-FaceSet in neutral state (FAP = 0), P’, the position of thesame vertex after animation with the given FAP and D,,,,kthe 3D displacement in the kth Interval, following algo-rithm must be applied to determine the new position P’,:1. Determine, in which of the intervals listed in the table

the received FAP is lying.

51

2. If the received FAP is lying in the jth interval [Ij, Ij+,]and O=Ik 5 Ij, the new vertex position P’, of the mthvertex of the IndexedFaceSet is given by:p’rn = FAPU * ((lk+,-0) * &,,k + (lk+Tlk+,) * D,,,, k+l+ . ..(lj - 1j.l) * Dm,j_, + (FAP-1,) * Dm,j) + Pm.

3. If FAP > I,,, then P’, is calculated by using theequation given in 2 and setting the index j = max- 1.

4. If the received FAP is lying in the jth interval [Ij, I,+{]and Ij+i I Ik=O, the new vertex position P’,,, is givenby:P’m = FAPU * (( Ii+, - FAP) * Dm,j + (lj+z- Ii+,) *Dm,j+f + . . . ((k-1 - (k-2) * &,/c-2 + (0 - (k-1) * &a,&-7) +

pm.

5. If FAP < Ii, then P’, is calculated by using the equa-tion 4 in step 4 and setting the index j = 1.

6. If for a given FAP and ‘IndexedFaceSet’ the tablecontains only one interval, the motion is strictly lin-ear:P:, = FAPU * FAP * D,,,f + P,,,.

Table 3: Simplified example of twoFaceDefTables.

#FaceDeff ableFAP 6 (stretch left corner lip)

IndexedFaceSet: Faceinterval borders: -1000, 0,500, 1000displacements:

vertex 50 1 0 0,0.9 0 0, 1.5 0 4vertex 51 0.8 0 0,0.7 0 0,2 0 0

MFaceDetTableFAF’ 23 (yaw left eye ball)

Transform: LeftEyeXrotation neutral value: 0 -1 0 (axis) 1 (factor)

Example for a FaceDefTable: In Table 3, two FAPsare defined by FaceDeff ables: FAP 6, which stretches theleft comer lip, and FAP 23, which manipulates the hori-zontal orientation of the left eyeball.

FAP 6 deforms the IndexedFaceSet Face. For thepiecewise-linear motion function three intervals are de-fined: [-1000, 01, [0,500] and [500, lOOO]. Displacementsare given for the vertices with indices 50 and 5 1. The dis-placements for vertex 50 are: (1 0 0), (0.9 0 0) and (1.5 04), the displacements for vertex 51 are (0.8 0 0), (0.7 0 0)and (2 0 0). Given a FAP Value of 600, the resulting dis-placement for vertex 50 would be:

P’s0 = Pg)+500*(0.9 0 o)‘+ 100 * (1.5 0 4)T= P5,+(600 0 400)T.

FAP 23 updates the rotation field of the Transformnode LeftEyeX The rotation axis is (0, -1, 0), and themultiplication factor for the angle is 1. The FAP valuedetermines the rotation angle.

Figure 4 shows 2 phases of an left eye blink (plus theneutral phase) which have been generated using a simpleanimation architecture [7].

Figure 4: Neutral state of the left eye (left)and two deformed animation phases for theeye blink (FAP 19). The FAP definition de-fines the motion of the eyelid in negative y-direction; the FaceDefTable defines the mo-tion of the vertices of the eyelid in x, y and zdirection. For FAP 19, positive FAP valuesmove the vertices downwards.

3. Coding of Face Animation ParametersFor coding of facial animation parameters, MPEG-4

provides two tools. Coding of quantized and temporallypredicted FAPs using an arithmetic coder allows for cod-ing of FAPs introducing small delay only. Using a dis-crete cosine transform (DCT) for coding a sequence ofFAPs introduces significant delay but achieves highercoding efficiency.

3.1. Arithmetic Coding of FAPsFigure 5 shows the block diagram for encoding FAPs.

The first set of FAP values FAP(i)O at time instant 0 iscoded in intra mode. The value of an FAP at time instantk FAP(i)l, is predicted using the previously decoded valueFAP(i)k_l. The prediction error e is quantized using aquantization stepsize that is specified for each FAP multi-plied by a quantization parameter FAP_QUANT withO<FAP QUANT<9. FAP QUANT is identical for allFAP values of one time in$nt k. Using the FAP depend-ent quantization stepsize and FAP_QUANT assures thatquantization errors are subjectively evenly distributedbetween different FAPs. The quantized prediction error e ’is arithmetically encoded using a separate adaptive prob-ability model for each FAP. Since the encoding of thecurrent FAP value depends only on one previously codedFAP value, this coding scheme allows for low-delaycommunications. At the decoder, the received data is ar-ithmetically decoded, dequantized and added to the previ-ously decoded value in order to recover the encoded FAPvalue.

In order to avoid transmitting all FAPs for everyframe, the encoder can transmit a mask indicating forwhich groups (Table 1) FAP values are transmitted. Theencoder can also specify for which FAPs within a groupvalues will be transmitted. This allows the encoder tosend incomplete sets of FAPs to the decoder.

52

Quantizere’ Arithmetic

+ C o d e r -+

+

FAP’k.1 FrameDelay

Figure 5: Block diazf the encoder forFAPs.

The decoder can extrapolate values of unspecifiedFAPs, in order to create a more complete set of FAPs.The standard is vague in specifying how the decoder issupposed to extrapolate FAP values. Examples are that ifonly FAPs for the left half of a face are transmitted, thecorresponding FAPs of the right side have to be set suchthat the face moves symmetrically. If the encoder onlyspecifies motion of the inner lip (FAP group 2), the mo-tion of the outer lip (FAP group 8) has to be extrapolated.Letting the decoder extrapolate FAP values may createunexpected results unless FAP interpolation functions aredefined (Section 3.3).

3.2. DCT Coding of FAPsThe second coding tool that is provided for coding

FAPs is the discrete cosine transform applied to 16 con-secutive FAP values (Figure 6). This introduces a signifi-cant delay into the coding and decoding process. Hence,this coding method is mainly useful for application whereanimation parameter streams are retrieved from a data-base. This coder replaces the coder shown in Figure 5.After computing the DCT of 16 consecutive values of oneFAP, DC and AC coefficients are coded differently.Whereas the DC value is coded predictively using theprevious DC coefficient as prediction, the AC coefficientis directly coded. The AC coefficient and the predictionerror of the DC coefficient are linearly quantized.Whereas the quantizer stepsize can be controlled, the ratiobetween the quantizer stepsize of the DC coefficients andthe AC coefficients is set to !4. The quantized AC coeffi-cients are encoded with one variable length code word(VLC) defining the number of zero-coefficients prior tothe next non-zero coefficient and one VLC for the ampli-tude of this non-zero coefficient. The handling of the de-coded FAPs is not changed (see Section 3.1).

3.3. FAP Interpolation TablesAs mentioned in Section 3.1, the encoder may allow

the decoder to extrapolate the values of some FAPs fromthe transmitted FAPs. Alternatively, the decoder canspecify the interpolation rules using FAP interpolationtables (FIT). A FIT allows a smaller set of FAPs to besent during a facial animation. This small set can then beused to determine the values of other FAPs, using a ra-tional polynomial mapping between parameters. For ex-ample, the top inner lip FAPs can be sent and then used to

determine the top outer lip FAPs. The inner lip FAPswould be mapped to the outer lip FAPs using: a rationalpolynomial f&&ion that is specif;ed in the FIT.

Figure 6: Block diagram of the FAP encoderusing DCT. DC coefficients are predictivelycoded. AC coefficients are directly coded.To make the scheme general, sets of FAPs are speci-

fied, along with a FAP Interpolation Graph (FIG) betweenthe sets that specifies which sets are used to determinewhich other sets (Figure 7). The FIG is a graph with di-rected links. Each node contains a set of FAPs. Each linkfrom a parent node to a child node indicates that the FAPsin child node can be interpolated from parent node.

Figure 7: A FIG example for interpolatingunspecified FAP values of the lip. If only theexpression is defined, the FAPs get inter-polated from the expression. If all inner lipFAPs are specified, they are used to inter-polate the outer lip FAPs.In FIG, a FAP may appear in several nodes, and a node

may have multiple parents. For a node that has multipleparent nodes, the parent nodes are ordered as 1” parentnode, 2”d parent node, etc. During the interpolation proc-ess, if this child node needs to be interpolated, it is firstinterpolated from 1”’ parent node if all FAPs in that parentnode are available. Otherwise, it is interpolated from Zndparent node, and so on. An example of FIG is shown inFigure 7. Each node has an ID. The numerical label oneach incoming link indicates the order of these links.

Each directed link in a FIG is a set of interpolationfunctions. Suppose F,, Fl, . . ., F, are the FAPs in a parent

53

set and fi, f2, . . ., f, are the FAPs in a child set. Then,there are m interpolation functions denoted as: fi = Ii(F,,F2, . . ., F,), f2 = 12F1, F2, . . ., Fn), f,n = MFI, F2, . . . . F,).

Each interpolation function Ik () is in a rational polyno-mial form

K-l IIz(F,,F,,...,F,)=,~~(ci IIF,ivI

j=l

where K and P are the numbers of polynomial products,ci and bi are the coefficient of the ith product. IV and mii

are the power ofFj in the ith product The encoder should

send an interpolation function table which contains allK , P , ci , bi , l, , mij to the decoder for each link in the

FIG.

4. Integration of Face Animation into anMPEG-4 Terminal

MPEG-4 defines a user terminal that allows decoding,composing and presenting multiple audio-visual objects.These A/V objects can be music, speech, synthesizedspeech from a text-to-speech (TTS) synthesizer, syntheticaudio, video sequences, arbitrarily shaped moving videoobjects, images, 3D computer animated models or syn-thetic face models. MPEG-4 arranges and renders theseobjects into an audio-visual scene according to a scenedescription. This scene description allows defining vari-able dependencies of objects. As an example, it is notonly possible to map an image as a texture onto a 3Dmode1 but also map video onto this object and align theposition of the visual object and the position of the relatedsound source. Synchronization between the different me-dia is achieved using the timing information of the indi-vidual media. The scene description allows also for inter-activity inside the MPEG-4 player as well as for feedbackto the encoder using a return channel.

Of particular interest to face animation is the MPEG4capability to map images as texture maps onto a facemodel, the synchronization of facial animation usingFAPs and related audio, and the integration with a text-to-speech synthesizer. Synchronization with audio andspeech streams with the FAP stream is achieved by evalu-ating the timing information that these streams have. Syn-chronization of an FAP stream with TTS synthesizers iscurrently only possible, if the encoder sends prosody andtiming information. This is due to the fact that a conven-tional TTS system driven by text only behaves as anasynchronous source where the encoder does not knowthe exact timing behavior. The following section dis-cusses the current integration of face animation and TTSand the ongoing work in this area.

4.1. Text-to-Speech Interface(TTS1)MPEG4 foresees that talking heads will serve an im-

portant role in future customer service applications.Therefore, MPEG-4 provides interfaces to proprietarytext-to-speech (TTS) synthesizer that allows driving atalking head from text (Figure 8) [8]. A TTS stream con-tains text or prosody in binary form. The decoder decodesthe text and prosody information according to the inter-face defined for the TTS synthesizer. The synthesizercreates speech samples that are handed to the compositor.The compositor presents audio and if required video tothe user. The second output interface of the synthesizersends the phonemes of the synthesized speech as well asstart time and duration information for each phoneme tothe Phoneme/Bookmark-to-FAP-Converter. The convertertranslates the phonemes and timing information into faceanimation parameters that the face renderer uses in orderto animate the face model. The precise method of how theconverter derives visemes from phonemes is not specifiedby MPEG and left to the implementation of the decoder

In the current MPEG4 standard, the encoder is ex-pected to send a FAP stream containing FAP number andvalue for every frame, to enable the receiver to producedesired facial actions. Since the TTS synthesizer can be-have like an asynchronous source, synchronization ofspeech parameters with facial expressions of the FAPstream is usually not exact - unless the encoder transmitstiming information for the synthesizer. An on-goingworking item in MPEG-4 is to provide a more efficientmeans for overlaying facial expression, by insertingbookmarks in the TTS stream. In addition to the pho-nemes, the synthesizer identifies bookmarks in the textthat convey non-speech related facial expressions to theface renderer. The timing information of the bookmarks isderived from their position in the synthesized speech. Theprecise method of how the converter derives a continuousstream of FAPs from bookmarks is not specified byMPEG and left to the implementation of the decoder.

honeme/Bookmarkto FAP Converter

FAP Stream

izE.B

.Figure 8: Block diaxowing the i

Audio-b

Video-+

Ite-gration of a proprietary Text-to-Speech Syn-thesizer into an MPEG-4 face animationsystem.

54

5. ProfilesMPEG-4 foresees that applications for face animation

fall in three scenarios and defines tools for these scenariosin three corresponding profiles.

Simple Profile: The decoder has its own proprietarymodel that is animated by a coded FAP stream (Sections3.1 and 3.2.)

Calibration Profile: This profile includes the simpleprofile. The encoder transmits to the decoder calibrationdata for some or all of the predefined feature points(Figure 2). The decoder adapts its proprietary face modelsuch that it aligns with the position of these feature points.This allows for customization of the model although theresult is not predictable since the standard does not definea minimum quality for a decoder face model and the ad-aptation process of an arbitrary face model to these fea-ture points is not specified. The decoder is required tounderstand FITS (Section 3.3). Hence this profile allowsfor a higher coding efficiency than the simple profile.

Predictable Profile: This profile includes the calibra-tion profile. MPEG-4 also provides a mechanism fordownloading a model to the decoder according to Section2.3 and animating this model. This gives the encodercontrol over the presentation at the receiver and allows forsensitive applications like web-based customer service.

In order to provide guaranteed levels of performance atthe decoder, MPEG4 defines for each profile levels thatspecify minimum requirements in terms of FAP decodingspeed and render speed. Only terminals that pass confor-mance tests for defined profiles and levels are MPEG4compliant.

6. ConclusionsMPEG-4 integrates animation of synthetic talking

faces into audio-visual multimedia communications. Aface model is a representation of the human face that isstructured for portraying the visual manifestations ofspeech and facial expressions adequate to achieve visualspeech intelligibility and the recognition of the mood ofthe speaker. A face model is defined as a static 3D modeland related animation rules that define how the modeldeforms if it is animated with FAPs. The model is definedusing a scene graph. Therefore, a customized model withhead and shoulders can be defined for games or web-based customer service applications. MPEG-4 defines acomplete set of animation parameters tailored towardsanimation of the human face. However, face animationparameters are defined independent of the proportions ofthe animated face model. Therefore, a face animation pa-rameter stream can be used to animate different models.Successful animations of humans, animals and cartooncharacters have been demonstrated.

In order to enable animation of a face model over lowbitrate communication channels, for point to point as wellas multi-point connections, MPEG-4 encodes the FAPsusing temporal prediction, quantization and coding of theprediction error. For low-delay applications, the predic-tion error is coded using an adaptive arithmetic coder, forother applications a discrete cosine transform is applied toa sequence of each facial animation parameter. In order toavoid coding of the entire set of more than 70 animationparameters for each frame, undefined animation parame-ters can be interpolated from the coded parameters. Theencoder can specify these interpolation rules using ra-tional polynomials. Face models can be animated with adata rate of 300 - 2000 bits/s

For talking head applications, MPEG4 defines appli-cation program interfaces for TTS synthesizer. Usingthese interfaces, the synthesizer provides phonemes andrelated timing information to the face model enablingsimple talking head applications.

In order to use facial animation for entertainment andbusiness applications, the performance of the MPEG4player has to be known to the content creator. Therefore,MPEG defined 3 profiles and conformance points forfacial animation that allow different levels of contigura-tion of the decoder. This will make MPEG4 face anima-tion an attractive platform for many applications.

Acknowledgements: The author would like to thankYao Wang and Ariel Fischer for the review of this paper.

7.[II

VI

[31

[41

[51

C61

171

181

ReferencesISOiIEC JTCliWGl 1 N1901, Text for CD 14496-l Sys-tems, Fribourg meeting, November 1997.ISO/IEC JTCl/WGl 1 N1902, Text for CD 14496-2 Vis-ual, Fribourg meeting, November 1997.J. Hartman, J. Wemecke, The VRML handbook, AddisonWessley, 1996.Kalra P., Mangili A, Magnenat-Thalmann N, ThalmannD. “Simulation of Facial Muscle Actions Based on Ra-tional Free Form Deformations”, Proc. Eurographics 92,pp. 59-69, 1992.P. Ekman, W.V. Friesen, Manual for the facial actioncoding system, Consulting Psychologist Press, Inc. PaloAlto, CA, 1978.M. M. Cohen and D. W. Massaro, Modeling Coarticula-tion in Synthetic Visual Speech, In M. Thalmann & D.Thalmann (Eds.) Computer Animation “93, Tokyo:Springer-Verlag.J. Ostermann, E. Haratsch, “An animation definition inter-face: Rapid design of MPEC-4 compliant animated facesand bodies”, International Workshop on synthetic - naturalhybrid coding and three dimensional imaging, pp. 216-2 19, Rhodes, Greece, September 5-9, 1997.K. Waters, T. Levergood, “An automatic lip-synchronization algorithm for synthetic faces”, Proceed-ings of the Multimedia Conference, ACM, pages 149- 156,San Francisco, California, September 1994.

55