image morphing: a surveywolberg/pub/vc98.pdf · image morphing. we refrain from discussing relat-ed...

13
The Visual Computer (1998) 14:360–372 Springer-Verlag 1998 360 Image morphing: a survey George Wolberg Department of Computer Science, City College of New York, New York, NY 10031, USA E-mail: [email protected] Image morphing has received much atten- tion in recent years. It has proven to be a powerful tool for visual effects in film and television, enabling the fluid transfor- mation of one digital image into another. This paper surveys the growth of this field and describes recent advances in image morphing in terms of feature specification, warp generation methods, and transition control. These areas relate to the ease of use and quality of results. We describe the role of radial basis functions, thin plate splines, energy minimization, and multilev- el free-form deformations in advancing the state-of-the-art in image morphing. Recent work on a generalized framework for morphing among multiple images is de- scribed. Key words: Image morphing – Feature specification – Warp generation – Transi- tion control 1 Introduction Image metamorphosis has proven to be a powerful tool for visual effects. There are now many breath- taking examples in film and television of the fluid transformation of one digital image into another. This process, commonly known as morphing, is re- alized by coupling image warping with color inter- polation. Image warping applies 2D geometric transformations to the images to retain geometric alignment between their features, while color inter- polation blends their colors. Image metamorphosis between two images begins with an animator establishing their correspondence with pairs of feature primitives, e.g., mesh nodes, line segments, curves, or points. Each primitive specifies an image feature or landmark. The fea- ture correspondence is then used to compute map- ping functions that define the spatial relationship between all points in both images. Since mapping functions are central to warping, we shall refer to them as warp functions in this paper. They will be used to interpolate the positions of the features across the morph sequence. Once both images have been warped into alignment for intermediate feature positions, ordinary color interpolation (i.e., cross-dissolve) generates inbetween images. Feature specification is the most tedious aspect of morphing. Although the choice of allowable prim- itives may vary, all morphing approaches require careful attention to the precise placement of prim- itives. Given feature correspondence constraints between both images, a warp function over the whole image plane must be derived. This process, which we refer to as warp generation, is essential- ly an interpolation problem. Another interesting problem in image morphing is transition control. If transition rates are allowed to vary locally across inbetween images, more interesting animations are possible. The explosive growth of image morphing is due to the compelling and aesthetically pleasing effects made possible by warping and color blending. The extent to which artists and animators can ef- fectively use morphing tools is directly tied to so- lutions to the following three problems: feature specification, warp generation, and transition con- trol. Together, they influence the ease and effec- tiveness in generating high-quality metamorphosis sequences. This paper surveys recent advances in image morphing in terms of their roles in address- ing these three problems. We compare various morphing techniques, including those based on

Upload: others

Post on 19-Aug-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

The Visual Computer (1998) 14:360±372� Springer-Verlag 1998360

Image morphing:a survey

George Wolberg

Department of Computer Science,City College of New York, New York, NY 10031,USAE-mail: [email protected]

Image morphing has received much atten-tion in recent years. It has proven to be apowerful tool for visual effects in filmand television, enabling the fluid transfor-mation of one digital image into another.This paper surveys the growth of this fieldand describes recent advances in imagemorphing in terms of feature specification,warp generation methods, and transitioncontrol. These areas relate to the ease ofuse and quality of results. We describethe role of radial basis functions, thin platesplines, energy minimization, and multilev-el free-form deformations in advancing thestate-of-the-art in image morphing. Recentwork on a generalized framework formorphing among multiple images is de-scribed.

Key words: Image morphing ± Featurespecification ± Warp generation ± Transi-tion control

1 Introduction

Image metamorphosis has proven to be a powerfultool for visual effects. There are now many breath-taking examples in film and television of the fluidtransformation of one digital image into another.This process, commonly known as morphing, is re-alized by coupling image warping with color inter-polation. Image warping applies 2D geometrictransformations to the images to retain geometricalignment between their features, while color inter-polation blends their colors.Image metamorphosis between two images beginswith an animator establishing their correspondencewith pairs of feature primitives, e.g., mesh nodes,line segments, curves, or points. Each primitivespecifies an image feature or landmark. The fea-ture correspondence is then used to compute map-ping functions that define the spatial relationshipbetween all points in both images. Since mappingfunctions are central to warping, we shall refer tothem as warp functions in this paper. They willbe used to interpolate the positions of the featuresacross the morph sequence. Once both imageshave been warped into alignment for intermediatefeature positions, ordinary color interpolation (i.e.,cross-dissolve) generates inbetween images.Feature specification is the most tedious aspect ofmorphing. Although the choice of allowable prim-itives may vary, all morphing approaches requirecareful attention to the precise placement of prim-itives. Given feature correspondence constraintsbetween both images, a warp function over thewhole image plane must be derived. This process,which we refer to as warp generation, is essential-ly an interpolation problem. Another interestingproblem in image morphing is transition control.If transition rates are allowed to vary locally acrossinbetween images, more interesting animations arepossible.The explosive growth of image morphing is due tothe compelling and aesthetically pleasing effectsmade possible by warping and color blending.The extent to which artists and animators can ef-fectively use morphing tools is directly tied to so-lutions to the following three problems: featurespecification, warp generation, and transition con-trol. Together, they influence the ease and effec-tiveness in generating high-quality metamorphosissequences. This paper surveys recent advances inimage morphing in terms of their roles in address-ing these three problems. We compare variousmorphing techniques, including those based on

Page 2: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

361

mesh warping (Wolberg 1990), field morphing(Beier and Neely 1992), radial basis functions(Arad et al. 1994), thin plate splines (Lee et al.1994; Litwinowicz and Williams 1994), energyminimization (Lee et al. 1996), and multilevelfree-form deformations (Lee et al.1995).A tradeoff exists between the complexity of fea-ture specification and warp generation. As featurespecification becomes more convenient, warp gen-eration becomes more formidable. The recent in-troduction of spline curves to feature specificationraises a challenge to the warp generation process,making it the most critical component of morph-ing. It influences the smoothness of the transfor-mation and dominates the computational cost ofthe morphing process. We comment on thesetradeoffs and describe their role in influencing re-cent progress in this field.The scope of this paper is limited to traditional 2Dimage morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al.1995), view interpolation (Chen and Williams1993), or view morphing (Seitz and Dyer 1996).With the exception of 3D volume morphing, whichis a direct extension of 2D image morphing, thelatter two areas apply morphing to compute newviews of a scene, given a set of prestored images.

2 Morphing algorithms

Before the development of morphing, image tran-sitions were generally achieved through the useof cross-dissolves, e.g., linear interpolation to fadefrom one image to another. Figure 1 depicts thisprocess applied over five frames. The result ispoor, owing to the double exposure effect apparentin misaligned regions. This problem is particularlyapparent in the middle frame, where both input im-ages contribute equally to the output. Morphingachieves a fluid transformation by incorporatingwarping to maintain geometric alignment through-out the cross-dissolve process.In this section, we review several morphing algo-rithms, including those based on mesh warping,field morphing, radial basis functions, thin platesplines, energy minimization, and multilevelfree-form deformations. This review is intendedto motivate the discussion of progress in featurespecification, warp generation, and transitioncontrol.

2.1 Mesh warping

Mesh warping was pioneered at Industrial Light &Magic (ILM) by D. Smythe for use in the movieWillow in 1988 (Smythe 1990). It has been suc-cessfully used in many subsequent motion pic-tures. To illustrate the two-pass mesh warping al-gorithm, consider the image sequence shown inFig. 2. The five frames in the middle row representa metamorphosis (or morph) between the two facesat both ends of the row. We refer to these two im-ages as IS and IT, the source and the target images,respectively. The source image is associated withmesh MS. It specifies the coordinates of controlpoints, or landmarks. A second mesh MT specifiestheir corresponding positions in the target image.Meshes MS and MT are respectively shown over-laid on IS and IT in the upper left and lower rightimages of the figure. Notice that landmarks suchas the eyes, nose, and lips lie below the corre-sponding grid lines in both meshes. Together, MSand MT are used to define the spatial transforma-tion that maps all points in IS onto IT. The meshesare constrained to be topologically equivalent, i.e.,no folding or discontinuities are permitted. There-fore, the nodes in MT may wander as far from MSas necessary, as long as they do not cause self-in-tersection. Furthermore, for simplicity, the meshesare constrained to have frozen borders.All intermediate frames in the morph sequence arethe product of a four-step process:

for each frame f doLinearly interpolate mesh M, between MS

and MTWarp IS to I1, using meshes MS and MWarp IT to I2, using meshes MT and MLinearly interpolate image If, between I1

and I2end

Figure 2 depicts this process. In the top row of thefigure, mesh MS is shown deforming to mesh MT,producing an intermediate mesh M for each framef. These meshes are used to warp IS into increas-ingly deformed images, thereby deforming IS fromits original state to those defined by the intermedi-ate meshes. The identical process is shown in re-verse order in the bottom row of the figure, whereIT is shown deforming from its original state. Thepurpose of this procedure is to maintain the align-

Page 3: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

362

1

2

3

Fig. 1. Cross-dissolve

Fig. 2. Mesh warping

Fig. 3. Multilevel free-formdeformation (MFFD) basedmorphing

Page 4: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

363

ment of landmarks between IS and IT as they bothdeform to some intermediate state, producing thepairs of I1 and I2 images shown in the top and bot-tom rows, respectively. Only after this alignment ismaintained does a cross-dissolve between succes-sive pairs of I1 and I2 become meaningful, asshown in the morph sequence in the middle row.This sequence was produced by applying theweights [1, 0.75, 0.5, 0.25, 0] and [0, 0.25, 0.5,0.75, 1] to the five images in the top and bottomrows, respectively, and adding the two sets togeth-er. This process demonstrates that morphing issimply a cross-dissolve applied to warped imagery.The important role that warping plays here isreadily apparent in the comparison of the morphsequence in Fig. 2 with the cross-dissolve resultin Fig. 1.The use of meshes for feature specification facili-tates a straightforward solution for warp genera-tion: bicubic spline interpolation. The examplehere employed Catmull-Rom spline interpolationto determine the correspondence of all pixels.Fant©s algorithm was used to resample the imagein a separable implementation (Fant 1986; Wol-berg 1990).

2.2 Field morphing

While meshes appear to be a convenient mannerof specifying pairs of feature points, they aresometimes cumbersome to use. The field morph-ing algorithm developed by Beier and Neely(1992) at Pacific Data Images grew out of the de-sire to simplify the user interface to handle corre-spondence by means of line pairs. A pair of corre-sponding lines in the source and target images de-fines a coordinate mapping between the two imag-es. In addition to the straightforward correspon-dence provided for all points along the lines, themapping of points in the vicinity of the line canbe determined by their distance from the line.Since multiple line pairs are usually given, the dis-placement of a point in the source image is actual-ly a weighted sum of the mappings due to eachline pair, with the weights attributed to distanceand line length.This approach has the benefit of being more ex-pressive than mesh warping. For example, ratherthan requiring all the correspondence points ofFig. 2 to lie on a mesh, line pairs can be drawn

along the mouth, nose, eyes, and cheeks of thesource and target images. Therefore, only key fea-ture points need be given.Although this approach simplifies the specifica-tion of feature correspondence, it complicateswarp generation. This is due to the fact that all linepairs must be considered before the mapping ofeach source point is known. This global algorithmis slower than mesh warping, which uses bicubicinterpolation to determine the mapping of allpoints not lying on the mesh. An optimizationbased on a piecewise linear approximation is of-fered by Lee et al. (1998b) to accelerate the pro-cess. A more serious difficulty, though, is that un-expected displacements may be generated afterthe influence of all line pairs are considered at asingle point. Additional line pairs must sometimesbe supplied to counter the ill effects of a previousset. In the hands of talented animators, though,both the mesh warping and field morphing algo-rithms have been used to produce startling visualeffects.

2.3 Radial basis functions/thinplate splines

The most general form of feature specification per-mits the feature primitives to consist of points,lines, and curves. Since lines and curves can bepoint sampled, it is sufficient to consider the fea-tures on an image to be specified by a set of points.In that case, the x and y components of a warp canbe derived by constructing the surfaces that inter-polate scattered points. Consider, for example, Mfeature points labeled (uk, vk) in the source imageand (xk, yk) in the target image, where 1 £k £ M.Deriving warp functions that map points from thetarget image to the source image is equivalent todetermining two smooth surfaces: one that passesthrough points (xk, yk, uk) and the other that passesthrough (xk, yk, vk) for 1£ k £ M.This formulation permits us to draw upon a largebody of work on scattered data interpolation to ad-dress the warp generation problem. All subsequentmorphing algorithms have facilitated general fea-ture specification by appealing to scattered data in-terpolation.Warp generation by this approach is extensivelysurveyed by Ruprecht and Müller (1995) andWolberg (1990). Lee et al. (1994) and Litwinowicz

Page 5: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

364

and Williams (1994) independently propose twosimilar methods that employ the thin plate surfacemodel. Hassanien and Nakajima (1998) describe arelated Navier spline technique, which uses Navierpartial differential equations to construct the warpfunction. Arad et al. (1994) describe another meth-od using radial basis functions. These techniquesgenerate smooth warps that exactly reflect the fea-ture correspondence. Furthermore, they offer themost general form of feature specification, sinceany primitive (e.g., spline curves) may be sampledinto a set of points. Elastic Reality, a commer-cial morphing package from Avid Technology(www.avid.com), uses curves to enhance featurespecification. Their warp generation method, how-ever, is unpublished.

2.4 Energy minimization

All of the methods just described do not guaranteethe one-to-one property of the generated warpfunctions. When a warp is applied to an image,the one-to-one property prevents the warped imagefrom folding back upon itself. Lee et al. (1996)propose an energy minimization method for deriv-ing one-to-one warp functions. Their method al-lows extensive feature specification primitivessuch as points, polylines, and curves. Internally,all primitives are sampled and reduced to a collec-tion of points. These points are then used to gener-ate a warp that is interpreted as a 2D deformationof a rectangular plate. A deformation technique isprovided to derive C1-continuous and one-to-onewarps from the positional constraints. The require-ments for a warp are represented by energy termsand satisfied by minimizing their sum. The tech-nique generates natural warps since it is based onphysically meaningful energy terms. The perfor-mance of the method, however, is hampered byits high computational cost.

2.5 Multilevel free-form deformation

Lee et al. (1995) present a new warp generationmethod that is much simpler and faster than the re-lated energy minimization method (Lee et al.1996). Large performance gains are achieved byapplying multilevel free-form deformation(MFFD) across a hierarchy of control lattices to

generate one-to-one and C2-continuous warp func-tion. In particular, warps are derived from posi-tional constraints by introducing the MFFD as anextension to free-form deformation (FFD) (Seder-berg and Parry 1986). In their paper, the bivariatecubic B-spline tensor product is used to define theFFD function. A new direct manipulation tech-nique for FFDs, based on 2D B-spline approxima-tion, is applied to a hierarchy of control lattices toexactly satisfy the positional constraints. To guar-antee the one-to-one property of a warp, a suffi-cient condition for a 2D cubic B-spline surface tobe one-to-one is presented. The MFFD generatesC2-continuous and one-to-one warps that yield flu-id image distortions. The MFFD algorithm is com-bined with the energy minimization method (Leeet al. 1996) in a hybrid approach.An example of MFFD-based morphing is givenin Fig. 3. Notice that the morph sequence shownin the middle row of the figure is virtually iden-tical to that produced by mesh warping in Fig. 2.The benefit of this approach, however, is that fea-ture specification is more expressive and lesscumbersome. Rather than editing a mesh, a smallset of feature primitives are specified. To furtherassist the user, snakes are introduced to reducethe burden of feature specification. Snakes areenergy-minimizing splines that move under theinfluence of image and constraint forces. Theywere first adopted in computer vision as an activecontour model (Kass et al. 1988). Snakes stream-line feature specification because primitives mustonly be positioned near the features. Image forc-es push snakes toward salient edges, thereby re-fining their final positions and making it possibleto capture the exact position of a feature easilyand precisely.

2.6 Work minimization

Given that two images are sufficiently similar, im-age registration techniques (Brown 1992) offer apotential solution for automatically determiningthe correspondence among all points across thesource and target images. Optic flow techniques,for instance, are regularly used to track theframe-to-frame motion of video frames for usein compression and digital video processing(Tekalp 1995). Such methods exploit the fact thatthe two frames represent the same scene imaged

Page 6: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

365

only moments apart, and therefore only small dis-placements must be recovered. This is generallynot true in morphing applications, where thesource and target images may be vastly different.Nevertheless, it is reasonable to consider the ex-tent to which the feature specification problemcan be automated for similar input images. Thisis precisely the problem considered by Gao andSederberg (1998).Gao and Sederberg (1998) present a new algorithmfor determining the warp function between twosimilar images with little or no human interaction.This approach is consistent with the trend towardsminimizing the amount of user interaction in spec-ifying feature correspondence. The algorithm isbased on a work minimization approach that de-rives its cost directly from the image intensities,and not from user-specified constraints. Motivatedfrom related work conducted by the authors forsolving the polygon shape blending problem (Se-derberg and Greenwood 1992), they attempt tominimize the cost associated with warping and re-coloring the image. They apply a hierarchical warpusing a grid of bilinear B-spline patches and asso-ciate a cost function to it. Even after the warp isapplied, there will be differences in the intensityvalues and a cost function is associated with recol-oration. The morph generation process is then re-duced to that of finding the global minimum costfunction among all possible warps. The authors of-fer a method that is simple and fast, albeit it doesnot guarantee a global minimum.The premise of the work minimization algorithmis that the most visually pleasing morphs are asso-ciated with those images that satisfy the minimalwork constraints. This assumption is sometimesfalse, and therefore some user interaction is neces-sary to specify features and initialize a warp. Inspite of any initial warp that may need to be spec-ified, the rationale behind this approach is that us-er interaction can be significantly reduced by ex-ploiting work minimization constraints. Althoughthis work is still in its early stages, it has shownpromising results for morphing among similar im-ages.

2.7 Discussion

The progression of morphing algorithms has beenmarked by more expressive and less cumbersome

tools for feature specification. A significant stepbeyond meshes was made possible by the specifi-cation of line pairs in field morphing. The compli-cations that this brought to warp generation, how-ever, sometimes undermined the usefulness of theapproach. For instance, the method sometimesproduced undesirable artifacts, referred to asghosts, due to the computed warp function (Beierand Neely 1992). To counter these problems, theuser was required to specify additional line pairsbeyond the minimal set that would otherwise bewarranted. All subsequent algorithms, includingthose based on radial basis functions, thin platesplines, and energy minimization, formulatedwarp generation as a scattered data interpolationproblem and sought to improve the quality(smoothness) of the computed warp function.They did so at a relatively high computationalcost. The newest approach, based on the MFFD al-gorithm, significantly improves matters by accel-erating warp generation. The use of snakes furtherassists the user in reducing the burden of featurespecification. The ultimate goal of eliminating us-er interaction from feature specification remainselusive, but the work by Gao and Sederberg(1998) demonstrates the feasibility of an algo-rithm approaching this goal when the source andtarget images are similar.

3 Transition control

Transition control determines the rate of warpingand color blending across the morph sequence. Iftransition rates differ from part to part in inbe-tween images, more interesting animations arepossible. Such nonuniform transition functionscan dramatically improve the visual content. Notethat the examples shown thus far all use a uniformtransition function, whereby the positions of thesource features steadily move to their correspond-ing target positions at a constant rate.Figures 4 and 5 show examples of the use of uni-form and nonuniform transition functions, respec-tively. The upper left and lower right images ofFig. 4 are the source and target images, respective-ly. The features used to define the warp functionsare shown overlaid on the two images. The top andbottom rows depict a uniform transition rate ap-plied to the warping of the source and target imag-es, respectively. Notice, for instance, that all points

Page 7: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

366

4

5Fig. 4. Uniform metamorphosis

Fig. 5. Nonuniform metamorphosis

Page 8: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

367

in the source and target images are moving at auniform rate to their final positions. Those tworows of warped imagery are attenuated by thesame transition functions and added together toyield the middle row of inbetween images. Notethat geometric alignment is maintained amongthe two sets of warped inbetween images beforecolor blending merges them into the final morphsequence.The example in Fig. 5 demonstrates the effects of anonuniform transition function applied to the samesource and target images. In this example, a transi-tion function has been defined that accelerates thedeformation of the nose of the source image, whileleaving the shape of the head intact for the firsthalf of the sequence. The deformation of the headbegins in the middle of the sequence and continueslinearly to the end. The same transition function isused for the bottom row. Notice that this use ofnonuniform transition functions is responsible forthe dramatic improvement in the morph sequence.Transition control in mesh-based techniques isachieved by assigning a transition curve to eachmesh node. This proves tedious when complicatedmeshes are used to specify the features. Nishita etal. (1993) mention that the transition behavior canbe controlled by a BØzier function defined on themesh.In the energy minimization method, transitionfunctions are obtained by selecting a set of pointson a given image and specifying a transition curvefor each point. Although earlier morphing algo-rithms generally couple the feature specificationand transition control primitives, this method per-mits them to be decoupled. That is, the locationof transition control primitives does not necessari-ly coincide with those of the features. The transi-tion curves determine the transition behavior ofthe selected points over time. For a given time,

transition functions must have the values assignedby the transition curves at the selected points. Con-sidering a transition rate as the vertical distancefrom a plane, transition functions are reduced tosmooth surfaces that interpolate a set of scatteredpoints. The thin plate surface model (Terzopoulos1983) is employed to obtain C1-continuous surfac-es for transition functions.In the MFFD-based approach, the MFFD tech-nique for warp generation is simplified and appliedto efficiently generate a C2-continuous surface forderiving transition functions. The examples inFigs. 4 and 5 were generated with the MFFD-basedmorphing algorithm.Transition curves can be replaced with proceduraltransition functions (Lee et al.1995, 1996). An ex-ample is depicted in Fig. 6, where a linear functionvarying in the vertical direction is applied to twoinput images. The result is a convincing transfor-mation in which one input image varies into theother from top to bottom.

4 Generalized morphing framework

The traditional formulation for image morphingconsiders only two input images at a time, i.e.,the source and target images. In this case, morph-ing among multiple images is understood to meana series of transformations from one image to an-other. This limits any morphed image to take onthe features and colors blended from just two inputimages. Given the success of morphing using thisparadigm, it is reasonable to consider the benefitspossible from a blend of more than two imagesat a time. For instance, consider the generation ofa facial image that is to have its eyes, ears, nose,and profile derived from four different input imag-es. In this case, morphing among multiple images

Fig. 6. Proceduraltransformation

Page 9: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

368

is understood to mean a seamless blend of severalimages at once. We refer to this process as poly-morph.Rowland and Perrett (1995) consider a specialcase of polymorph to obtain a prototype face, interms of gender and age, from several tens of sam-ple faces. They superimpose feature points on in-put images to specify the various positions of fea-tures in sample faces. The shape of a prototypeface is determined by averaging the specified fea-ture positions. A prototype face is generated byapplying image warping to each input image andthen performing a cross-dissolve operation amongthe warped images. The shape and color differ-ences between prototypes from different gendersand ages are used to manipulate a facial imageto perform predictive gender and age transforma-tions.Despite the explosive growth of morphing in re-cent years, the subject of morphing among multi-ple images has been neglected. In recent work con-ducted by the author and his colleagues, a generalframework is developed that extends the tradition-al image morphing paradigm applied to two imag-es (Lee et al.1998a). We formulate each input im-age to be a vertex of a regular convex polyhedronin (n�1)-dimensional space, where n is the numberof input images. An inbetween (morphed) image isconsidered to be a point in the convex polyhedron.The barycentric coordinates of the point determinethe weights used to blend the input images into theinbetween image.In morphing among two images, nonuniformblending was introduced to derive an inbetweenimage in which blending rates differ across the im-age (Lee et al.1995, 1996). This allows us to gen-erate more interesting animations such as transfor-mation of the source image to the target from topto bottom. Nonuniform blending has also beenconsidered in volume metamorphosis to controlblending schedules (Hughes 1992; Lerios etal.1995). In (Lee et al. 1998a), the framework forpolymorph includes nonuniform blending of fea-tures in several input images. For instance, a facialimage can be generated to have its eyes, nose,mouth, and ears derived from four different inputfaces.An example is shown in Fig. 7. The top row ofFig. 7 shows input images I0, I1, and I2. We select-ed three groups of features in these images and as-signed them unity blending values to generate in-

between images. The feature groups consist ofthe hair, eyes and nose, and mouth and chin. Eachfeature group was selected in a different input im-age. For instance, Fig. 8 shows the feature groupsselected to generate the leftmost inbetween imagein the middle row of Fig. 7. The lower rightmostimage in the figure shows their projections ontoa single image. The middle and bottom rows ofFig. 7 show the inbetween images resulting fromall possible combinations in selecting those featuregroups.Polymorph is ideally suited for image compositionapplications in which elements are seamlesslyblended from two or more images. A compositeimage is treated as a metamorphosis of selected re-gions in several input images. The regions seam-lessly blend together with respect to geometryand color. In future work, we will determine theextent to which the technique produces high-qual-ity composites with considerably less effort thanconventional image composition techniques. Inthis regard, the technique can bring to image com-position what image warping has brought to cross-dissolve in deriving morphing: a richer and moresophisticated class of visual effects that areachieved with intuitive and minimal user interac-tion.

5 Applications

Image morphing has traditionally been associatedwith visual effects for entertainment. Visuallycompelling fluid transformations are created bysynthesizing intermediate images between sup-plied image pairs. The basis for these results is at-tributed to the geometric alignment that is main-tained throughout the image sequence. This sameresult applies to other domains where image inter-polation can benefit from supplied geometric cor-respondence. One such application includes medi-cal visualization.In medical imaging, CT or MRI scans are availableat a fixed interslice resolution. Although interme-diate slices can be computed with conventional lin-ear, cubic, or higher-degree interpolation func-tions, this traditional approach does not considerthe underlying structure of the imaged organs. Su-perior results are made possible by establishinggeometric correspondence of features among suc-cessive pairs of scans. Features may include organ

Page 10: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

369

7

8

Fig. 7. Input images (top row) from which combinations of the hair, eyes-and-nose, and mouth-and-chin (middle and bottom rows) have been taken

Fig. 8. Selected parts on input images and their projections onto the centralimage

7

8

Page 11: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

370

boundaries and anatomical landmarks. The com-puted morphed images constitute the intermediateimages that conform to the geometric correspon-dence provided by the user. Ruprecht and Müller(1994) describe work in this area. Automatic fea-ture selection remains an active area of research,and the feature selection process remains largelymanual.Recent work in face recognition has also drawnupon morphing. Bichsel (1996) presents a newmethod for generating an optimum mapping ofone image into another. The mapping is calculatedby maximizing the probability of the morph fieldin a Bayesian framework. The approach uses an it-erative approximation where, in a neighborhoodaround the current morph field solution, the prob-ability distribution is approximated by a Gaussianprobability distribution for which the most likelysolution is evaluated by linear algebra techniques.This method has been used to interpolate betweendifferent views of a single face and between imag-es of different persons. The technique was alsodemonstrated on face recognition under varyingview and illumination conditions.An automated technique to recover facial featuresfor automatic image morphing has been exploredby Covell (1996). That work has been demonstrat-ed for use in lip syncing (Bregler et al. 1997). In anapplication that the authors refer to as video re-write, the system automatically synthesizes faceswith proper lip sync. It is a facial animation systemthat is driven by audio input, and it can be used fordubbing movies, teleconferencing, and special ef-fects.In related work on facial analysis and synthesis,Ezzat and Poggio (1996) explore an image-basedfacial modeling approach, in which they model fa-cial movements using example images. They ex-ploit the viability of a view interpolation approachto image synthesis. A learning network is trainedon example images, each of which is associatedwith a position in a high-level, multidimensionalparameter space denoting pose and expression.The authors demonstrate their analysis-by-synthe-sis technique by estimating pose, eye, and mouthmovements, as well as head translations, rotations,and scales.

6 Conclusions

This paper has reviewed the growth of imagemorphing and described recent advances in thefield. Morphing algorithms all share the followingcomponents: feature specification, warp genera-tion, and transition control. The ease with whichan artist can effectively use morphing tools is de-termined by the manner in which these compo-nents are addressed. We surveyed various morph-ing techniques, including those based on meshwarping, field morphing, radial basis functions,thin plate splines, energy minimization, and multi-level free-form deformations.The earliest morphing approach was based onmesh warping. It was motivated by a reasonablystraightforward interface requiring meshes to markfeatures and bicubic spline interpolation to com-pute warp functions. The field morphing approachattempted to simplify feature specification with theuse of line pairs to select landmarks. This addedbenefit demanded a more computationally expen-sive warp generation stage. Subsequent morphingalgorithms have sought to maintain the use ofcurves and polylines to select features. Warp gen-eration has consequently become formulated as ascattered data interpolation problem. Standard so-lutions such as radial basis functions and thin platesplines have been presented.The newest approach based on multilevel free-form deformations has further accelerated warpgeneration. This approach uses snakes to assistthe user in placing feature primitives, thereby re-ducing the burden in feature specification. Snakesare particularly useful when features lie alonglarge intensity gradients.Transition control determines the rate of warpingand color blending across the morph sequence. Iftransition rates differ from part to part in inbe-tween images, more interesting animations arepossible. The techniques used to compute warpfunctions can also be applied for transition controlfunctions, thereby propagating that information ev-erywhere across the image. Although early morph-ing algorithms generally coupled the feature spec-ification and transition control primitives, more re-cent algorithms have permitted them to be decou-pled.For those cases where the input images are suffi-ciently similar, feature specification can poten-

Page 12: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

371

tially be automated. The algorithm proposed byGao and Sederberg (1998) addresses this prob-lem and offers some promising results. The at-tempt to attain feature specification with no userinteraction will likely remain elusive for morph-ing applications. Aside from the standard regis-tration problem, a key difficulty with fully auto-mating feature correspondence is the aestheticissue that must be considered in generating a vi-sually pleasing morph. High-level understandingof the input images is necessary to specify thecorrespondence of points in areas where featureinformation is either partially occluded or miss-ing.Future morphing work includes further work inmorphing among multiple images, and improvingautomatic morphing among a limited class of im-ages and video sequences. In the limited, but com-mon, class of facial images, researchers have ex-plored the use of computer vision techniques to au-tomatically register features between two images.As the examples given in Figs. 2 and 3 show, facialimages require feature primitives to be specifiedalong the eyes, nose, mouth, hair, and profile.Model-based vision has been investigated to ex-ploit knowledge about the relative position of thesefeatures and automatically locate them for featurespecification (Brunelli and Poggio 1993). Current-ly, this is an active area of research, particularly forcompression schemes designed for videoconfer-ence applications. The same automation appliesto morphing among two video sequences, wheretime-varying features must be tracked. Progressin this area is tied to that of the motion estimationresearch community.

Acknowledgements. This work was supported in part by aNASA Faculty Award for Research grant (NAG-7129) and aPSC-CUNY grant (RF-668373). The author thanks S. Lee forvaluable discussions and help with some of the figures.

References

Arad N, Dyn N, Reisfeld D, Yeshurun Y (1994) Image warpingby radial basis functions: applications to facial expressions.CVGIP: Graph Models Image Processing 56:161±172

Beier T, Neely S (1992) Feature-based image metamorphosis.(Proceedings of SIGGRAPH'92) Comput Graph 26:35±42

Bichsel M (1996) Automatic interpolation and recognition offace images by morphing. The 2nd International on Auto-matic Face and Gesture Recognition. Killington, Vermont,IEEE Computer Society Press, Los Alamitos, CA, pp 128±135

Bregler C, Covell M, Slaney M (1997) Video rewrite: drivingvisual speech with audio. (Proceedings of SIGGRAPH'97)Comput Graph, pp 353±360

Brown LG (1992) A survey of image registration techniques.ACM Comput Surveys 24:325±376

Brunelli R, Poggio T (1993) Face recognition: Features versustemplates. IEEE Trans Patt Anal Machine Intell 15:1042±1052

Chen SE, Williams L (1993) View interpolation for image syn-thesis. (Proceedings of SIGGRAPH'93) Comput Graph pp279±288

Covell M (1996) Eigen-points: control-point location using prin-cipal component analysis. The 2nd International on Auto-matic Face and Gesture Recognition, Killington, IEEE Com-puter Society Press, Vermont, pp 122±127

Ezzat T, Poggio T (1996) Facial analysis and synthesis using im-age-based models. The 2nd International on Automatic Faceand Gesture Recognition, Killington, IEEE Computer Soci-ety Press, Los Alamitos, CA, pp 116±121

Fant KM (1986) A nonaliasing, real-time spatial transform tech-nique. IEEE Comput Graph Appl 6:71±80

Gao P, Sederberg TW (1998) A work minimization approach toimage morphing. Visual Comput (this volume)

Hassanien AE, Nakajima M (1998) Image morphing of the fa-cial image transformation based on Navier elastic bodysplines. Proceedings of Computer Animation'98, Geneva,Switzerland, IEEE Computer Society Press, Los Alamitos,CA, pp 119±125

Hughes JF (1992) Scheduled fourier volume morphing. (Pro-ceedings of SIGGRAPH'92) Comput Graph 26:43±46

Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contourmodels. Int J Comput Vision 1(4):321±331

Lee S, Chwa K-Y, Hahn J, Shin SY (1994) Image morphing us-ing deformable surfaces. Proceedings of Computer Anima-tion'94, IEEE Computer Society Press, Geneva, Switzer-land, pp 31±39

Lee S, Chwa K-Y, Shin SY, Wolberg G (1995) Image metamor-phosis using snakes and free-form deformations. (Proceed-ings of SIGGRAPH'95) Comput Graph, pp 439±448

Lee S, Chwa K-Y, Hahn J, Shin SY (1996) Image morphing us-ing deformation techniques. J Visualization Comput Anim7:3±23

Lee S, Wolberg G, Shin SY (1998a) Polymorph: morphingamong multiple images. IEEE Comput Graph Appl 18:58±71

Lee T, Lin Y-C, Lin L, Sun Y (1998b) Fast feature-based meta-morphosis and operator design. (Proceedings of Eurograph-ics'98), Computer Graphics Forum, 17(3):15±22

Lerios A, Garfinkle CD, Levoy M (1995) Feature-based volumemetamorphosis. (Proceedings of SIGGRAPH'95) ComputGraph, pp 449±456

Litwinowicz P, Williams L (1994) Animating images withdrawings. (Proceedings of SIGGRAPH'94) Comput Graph,pp 409±412

Nishita T, Fujii T, Nakamae E (1993) Metamorphosis using BØ-zier clipping. Proceedings of the 1st Pacific Conference onComputer Graphics and Applications, Seoul, Korea, WorldScientific, Singapore, pp 162±173

Rowland DA, Perrett DI (1995) Manipulating facial appearancethrough shape and color. IEEE Comput Graph Appl 15:70±76

Ruprecht D, Müller H (1994) Deformed cross-dissolves for in-terpolation in scientific visualization. J Visualization Com-put Anim 5:167±181

Page 13: Image morphing: a surveywolberg/pub/vc98.pdf · image morphing. We refrain from discussing relat-ed work in 3D volume morphing (Lerios et al. 1995), view interpolation (Chen and Williams

372

Ruprecht D, Müller H (1995) Image warping with scattered datainterpolation. IEEE Comput Graph Appl 15:37±43

Sederberg TW, Greenwood E (1992) A physically based ap-proach to 2D shape blending. (Proceedings of SIG-GRAPH'92) Comput Graph 26:25±34

Sederberg TW, Parry SR (1986) Free-form deformation of solidgeometric models. (Proceedings of SIGGRAPH'86) ComputGraph 20:151±160

Seitz SM, Dyer CR (1996) View morphing. (Proceedings ofSIGGRAPH'96) Comput Graph, pp 21±30

Smythe DB (1990) A two-pass mesh warping alogrithm for ob-ject transformation and image interpolation. Technical Re-port 1030, ILM Computer Graphics Department, Lucasfilm,San Rafael, Calif

Tekalp AM (1995) Digital video processing. Prentice Hall PTR,Upper Saddle River, NJ

Terzopoulos D (1983) Multilevel computational processes forvisual surface reconstruction. Comput Vision Graph ImageProcessing 24:52±96

Wolberg G (1990) Digital image warping. IEEE Computer So-ciety Press, Los Alamitos, Calif

GEORGE WOLBERG is a FullProfessor of Computer Scienceat the City College of New York(CCNY). He received his BSand MS degrees in ElectricalEngineering from Cooper Unionin 1985, and his PhD degree inComputer Science from Colum-bia University in 1990. His re-search interests include imageprocessing, computer graphics,and computer vision. Prof. Wol-berg is the recipient of a 1991National Science Foundation(NSF) Presidential Young In-vestigator Award and the 1997

CCNY Outstanding Teaching Award. He is the author of DigitalImage Warping, the first comprehensive monograph on warpingand morphing.