Image Deformation Based on Cubic Splines and Moving Least Squares

Hua Shungang, Li Shaoshuai, Li Xiaoxiao Key Laboratory for Precision and Non-traditional Machining Technology of Ministry of Education

Dalian University of Technology Dalian , China

hsgang02@dlut.edu.cn

Abstract—Based on cubic splines, an image deformation method using Moving Least Squares is proposed in this paper. According to shape information or deformation requirement, key points are set to create control curves. The curves are moved to new positions to guide image warping. The transformation functions are deduced to compute affine, similarity and rigid deformations, by which we can obtain various image deformation results. Experimental results show that our method can be used to describe outline and gain realistic deformation for complex image warping.

Keywords-image deformation; moving least squares; cubic splines; transformation function

I. INTRODUCTION Image deformation is one kind of popular image processing

technologies and is widely used in animation, morphing, medical imaging, etc. Generally, image deformation can be implemented by using control handles, such as points, lines, curves, grids, and so on, which can produce continuous, smooth and realistic deformed image.

So far, various image deformation algorithms have been presented to create realistic image deformation. Sederberg [1] and Lee et al. [2] provided grid-based free-form deformation techniques, which parameterize the image using bivariate cubic splines to create deformations. Typically, these methods require aligning grid lines corresponding to the control points of the splines with features of the image. Beier and Neely [3] improved upon these grid-based techniques, which allows user to specify the deformation using sets of lines. This technique can give the illusion that the photographed or computer generated subjects are transforming in a fluid, surrealistic, and often dramatic way. To avoid the shrinkage in morphing sequences, Meng et al. [4] defined the connection graph of feature specification to deal with the rotations.

An interactive system by Igarashi et al. [5] allows the user to manipulate a shape without using a skeleton or free-form deformation. Weng et al. [6] used non-linear least squares to generate realistic deformations. Guo et al. [7] provided a detailed analysis of the 2D deformation algorithm based on non-linear least squares optimization, and proved that different mesh structure is of critical importance to deforming result. Fang and Hart [8] proposed an image editing system that

decouples feature position from pixel color generation, by resynthesizing texture from the source image to preserve its detail and orientation around a new feature curve location. Schaefer et al. [9] proposed an image deformation algorithm using Moving Least Squares that deforms images based on control points or control lines. They researched the affine, similarity and rigid transformation. Their method is sophisticated for image deformation, but it doesn’t consider shape topology relations of image, and when the outline for warping is irregular, it is tedious to describe the shape.

In this paper, based on sets of cubic splines and moving Least Squares, we research and provide a new method for irregular outline and complex warping. Our method can be used to describe outline curves of the image, and separate desired deformation regions effectively, which produce realistic resulting deformation.

II. IMAGE DEFORMATION USING MOVING LEAST SQUARES The deformation can be viewed as a function f that maps

points in original image to deformed image. Applying the function to each pixel point v in the original image creates the deformed image. Let s , d be sets of specifying control points in original, deformed image respectively. According to the Moving Least Squares theory model [10], given a point v in the image, it can solve for the best deformation function f that minimizes

2)( iii

i dsfw −∑ . (1)

Where is and id are coordinates of the thi corresponding control points that are expressed by row vectors; the weights

iw have the form α21 vsw ii −= , where α is the parameter to adjust deformation results and its value is 1 in our paper. When v takes the value of is , let ii dsf =)( . Furthermore, if

ii ds = , f is the identity transformation iii dssf ==)( . In such sort of image deformation, deformation function )(xf consists of two parts: a linear transformation matrix M

and a translation transformation T ,

TxMxf +=)( . (2)

978-1-4244-4507-3/09/$25.00 ©2009 IEEE

Equation (1) is quadratic in T . We can get the minimum where the derivatives with respect to each of the free variables in f are zero, T can be solved in term of the matrix M ,

MwswwdwTi

ii

iii

ii

ii ⎟⎠

⎞⎜⎝

⎛−= ∑∑∑∑ . (3)

With this observation we can substitute T into (2) and rewrite )(xf in terms of the matrix M ,

∑∑∑∑ +−=

ii

iii

ii

iii wdwMwswxxf )()( . (4)

The least squares problem of (1) can be rewritten as

2ˆˆ iii

i dMsw −∑ . (5)

Where ∑∑−=i

ii

iii wswss and ∑∑−=i

ii

iii wdwdd .

Notice that matrix M can be considered as a general transformation including components of scaling, shearing and rotating. Constraining these components, we can obtain the affine, similarity and rigid transformation functions. For details, the reader is referred to literature [9].

We implement the image deformation based on the same sets of control points as shown in Fig.1. We can get different deformed image, and it is obvious that rigid transformation gives user the impression of manipulating real-world objects.

(a) (b) (c) (d)

Figure 1. Image deformation using moving least squares: (a) Original image; (b)Affine transformation;(c)Similarity transformation;(d) Rigid transformation

III. IMAGE DEFORMATION BASED ON CONTROL CURVES We deform the character’s face outline by method above

based on control points, as shown in Fig.2. We make her face thin and deform the mouth shape. Fig.2(b) shows the rigid resulting deformation. We note that the outline of character’s face and mouth undergo undesirable and unrealistic deformation. So we consider to research and provide a method controlled by curves to realize realistic warping.

(a) Original image (b)Rigid transformation

Figure 2. An example of image deformation based on control points

To control the path of curve, we adopt cubic spline method to generate control curve. Cubic splines can accurately pass all

the control points and achieve second-order continuity. We set control points in the image by the mouse. Given n+1 control points ja ( nj 2,1,0= ) to generate the control curve, the cubic splines are generated according to free end-point condition and consist of n segments. )(tsi ( 1,2i n= ) indicates the thi segment of the control curve before deformation and

)(tdi indicates the deformed thi segment corresponding to )(tsi , where assume )1,0(∈t . Based on (1) we can conclude

dttdTMtstwi

iii∑ ∫ −+1

0

2)()()( . (6)

Where )(twi has the formα2)(

)()(

vtsts

twi

ii

−

′= . We can

get the minimum where the derivatives with respect to each of the free variables in f are zero.

MsdT ** −= . (7) Where *s and *d are

∑ ∫

∑ ∫=

ii

iii

dttw

dttstws 1

0

1

0

*)(

)()(

∑ ∫

∑ ∫=

ii

iii

dttw

dttdtwd 1

0

1

0

*)(

)()( , (8)

Equation (6) can be rewritten as

dttdMtstwi

iii∑ ∫ −1

0

2)(ˆ)(ˆ)( . (9)

Where *)()(ˆ ststs ii −= and *)()(ˆ dtdtd ii −= . )(tsi can be expressed as the matrix

( ) ( )1

3 2

1

2 2 1 13 3 2 1( ) ( ) ( ) 1 0 0 1 0

1 0 0 0

i

ii ix iy

i

i

aas t s t s t t t t aa

−

−

− ⎛ ⎞⎛ ⎞⎜ ⎟⎜ ⎟− − −= = ⋅ ⋅ ⎜ ⎟⎜ ⎟ ′⎜ ⎟⎜ ⎟ ⎜ ⎟′⎝ ⎠ ⎝ ⎠

. (10)

In the original image, 1ia − and ia are respectively start-point and end-point coordinates of the thi segment in control curves; 1ia −′ and ia′ are respectively the derivatives of the start-point and end-point. Similarly, in the deformed image,

1ib − and ib denote coordinates; 1ib −′ and ib′ denote the derivatives.

According to (10), the parameter equation of cubic splines can be rewritten as

( ) ( ) ( )3 21 2 3 4( ) ( ) ( ) 1 T

i ix iy i i i is t s t s t t t t m m m m= = ⋅ . (11)

Where 1 1 1

2 1 1

3 1

4 1

2 23 3 2

i i i i i

i i i i i

i i

i i

m a a a am a a a am am a

− −

− −

−

−

′ ′= − + +⎧⎪ ′ ′= − + − −⎨ ′=⎪ =⎩

. (12)

Similarly,

( ) ( ) ( )3 21 2 3 4( ) ( ) ( ) 1 T

i ix iy i i i id t d t d t t t t n n n n= = ⋅ . (13)

Where 1 1 1

2 1 1

3 1

4 1

2 23 3 2

i i i i i

i i i i i

i i

i i

n b b b bn b b b bn bn b

− −

− −

−

−

′ ′= − + +⎧⎪ ′ ′= − + − −⎨ ′=⎪ =⎩

. (14)

Then, we deduce the affine, similarity and rigid transformation functions based on control curves.

A. Affine Transformation Function According to (11) and (13), we can represent )(ˆ tsi and

)(ˆ tdi ,

( )1

3 2 2

3

4

ˆˆˆ ( ) 1 ˆˆ

i

ii

i

i

mms t t t t mm

⎛ ⎞⎜ ⎟

= ⋅ ⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

( )1

3 2 2

3

4

ˆˆˆ ( ) 1 ˆˆ

i

ii

i

i

nnd t t t t nn

⎛ ⎞⎜ ⎟

= ⋅ ⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

. (15)

Substituting )(ˆ tsi and )(ˆ tdi into (9), we can rewrite it in the form

( ) dt

nnnn

M

mmmm

ttttwi

i

i

i

i

i

i

i

i

i

2

1

0

4

3

2

1

4

3

2

1

23

ˆˆˆˆ

ˆˆˆˆ

1)(∑ ∫⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

−

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⋅ . (16)

Based on (16), we can get the minimum where the derivatives are zero, and then educe the transformation matrix M :

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

= ∑∑

−

j

j

j

j

jj

T

j

j

j

j

i

i

i

i

i

i

T

i

i

i

i

nnnn

W

mmmm

mmmm

W

mmmm

M

4

3

2

1

4

3

2

1

1

4

3

2

1

4

3

2

1

ˆˆˆˆ

ˆˆˆˆ

ˆˆˆˆ

ˆˆˆˆ

. (17)

Where weight matrix iW is symmetry matrix and given by 00 01 02 03

10 11 12 13

20 21 22 23

30 31 32 33

i i i i

i i i ii

i i i i

i i i i

W

φ φ φ φφ φ φ φφ φ φ φφ φ φ φ

⎡ ⎤⎢ ⎥

= ⎢ ⎥⎢ ⎥⎣ ⎦

, (18)

and iφ are integrals of the weight function )(twi multiplied by the different polynomials

⎪⎪⎪⎪

⎩

⎪⎪⎪⎪

⎨

⎧

=

=

=

=

∫∫∫∫

dttw

dtttw

dtttw

dtttw

ii

ii

ii

ii

1

0

33

21

0

22

41

0

11

61

0

00

)(

)(

)(

)(

φ

φ

φ

φ

,

⎪⎪

⎩

⎪⎪

⎨

⎧

=

=

=

∫∫∫

dtttw

dtttw

dtttw

ii

ii

ii

31

0

03

41

0

02

51

0

01

)(

)(

)(

φ

φ

φ,

⎪⎪

⎩

⎪⎪

⎨

⎧

=

=

=

∫∫∫

tdttw

dtttw

dtttw

ii

ii

ii

1

0

23

21

0

13

31

0

12

)(

)(

)(

φ

φ

φ

(19)

Substituting M into (4), we can rewrite it as

( ) *

4

3

2

1

4

3

2

1

1

4

3

2

1

4

3

2

1

*

ˆˆˆˆ

ˆˆˆˆ

ˆˆˆˆ

ˆˆˆˆ

)( d

nnnn

W

mmmm

mmmm

W

mmmm

svvfj

j

j

j

j

j

T

j

j

j

j

i

i

i

i

i

i

T

i

i

i

i

ca +

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⋅−= ∑∑

−

. (20)

B. Similarity Transformation Function We constrain the matrix M to have the property that that

IMM T 2λ= for some scalar λ . We can find the column vector 1M and 2M to have the form ( )21 MMM = , restricting M to be a similarity transform requires that

22211 λ== MMMM TT and 021 =MM T . This implies

that ⊥= 12 MM , where ⊥ is an operator on 2D vectors such

that ⊥−= ),(),( xyyx . So from (16), we can get 2

1

1

2 13 21

2 213 20 3 3

3 4

4

4

ˆˆ

ˆ ˆˆ ˆ0 0 0 1 0( ) ˆ ˆ0 0 0 0 1ˆ ˆ

ˆˆ

i

iT

i iT

i ii T

ii iT

i i

i

i

mm

m nm nt t tw t M dtmt t t nm n

mm

⊥

⊥

⊥

⊥

⎛ ⎞⎛ ⎞⎜ ⎟⎜ ⎟−⎜ ⎟⎜ ⎟ ⎛ ⎞⎜ ⎟⎜ ⎟ ⎜ ⎟⎜ ⎟⎜ ⎟⎛ ⎞ − ⎜ ⎟⋅ −⎜ ⎟⎜ ⎟⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎜ ⎟⎜ ⎟ ⎜ ⎟−⎜ ⎟⎜ ⎟ ⎝ ⎠⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟−⎝ ⎠⎝ ⎠

∑ ∫

. (21)

Above equation has a unique minimum, which yields the optimal transformation matrix M ,

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎝

⎛

−

−

−

−

=

⊥

⊥

⊥

⊥

⊥

⊥

⊥

⊥

∑T

iT

i

Ti

Ti

Ti

Ti

Ti

Ti

T

i

i

i

i

i

i

i

i

s

nnnnnnnn

W

mm

mm

mm

mm

kM

44

33

22

11

ii

4

4

3

3

2

2

1

1

ˆˆˆˆˆˆˆˆ

ˆˆ

ˆˆ

ˆˆ

ˆˆ

1 . (22)

Where the symmetric matrix iW is given by

⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

=

33323130

33323130

23222120

23222120

13121110

13121110

03020100

03020100

00000000

00000000

00000000

00000000

iiii

iiii

iiii

iiii

iiii

iiii

iiii

iiii

iW

φφφφφφφφ

φφφφφφφφ

φφφφφφφφ

φφφφφφφφ

, (23)

and iφ is from (19), sk has the form

dttststwki

Tiiis ∑ ∫=

1

0)(ˆ)(ˆ)( . (24)

It can be expresses by iφ , 00 11 22 33

1 1 2 2 3 3 4 401 02 03

1 2 1 3 1 412 13 23

2 3 2 4 3 4

ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆˆ ˆ ˆ ˆ ˆ ˆ2 2 2ˆ ˆ ˆ ˆ ˆ ˆ2 2 2

T T T Ti i i i i i i i i i i i

T T Ts i i i i i i i i i

T T Tii i i i i i i i i

m m m m m m m mk m m m m m m

m m m m m m

φ φ φ φφ φ φφ φ φ

⎧ + + +⎪= + + +⎨⎪+ + +⎩

∑ (25)

We can write )(vf cs as

( ) *4321 )1(ˆˆˆˆ()( dAk

nnnnvfi

is

iiiic

s += ∑ . (26)

Where

1

1

2

*2

3 *

3

4

4

ˆˆ

ˆˆ

ˆ ( )ˆ

ˆˆ

i

i

i Ti

i ii

i

i

i

mm

mv smA W m v s

mmm

⊥

⊥

⊥

⊥

⊥

⎛ ⎞⎜ ⎟−⎜ ⎟⎜ ⎟ −− ⎛ ⎞= ⎜ ⎟ ⎜ ⎟− −⎝ ⎠⎜ ⎟

−⎜ ⎟⎜ ⎟⎜ ⎟−⎝ ⎠

(27)

C. Rigid Transformation Function For the rigid transformation, the transformation matrix M

is the same as (22) except that we choose a different scaling constant rk .

( )1

21 1 2 2 3 3 4 4

3

4

ˆˆˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆˆˆ

Ti

TT T T T T T T T i

r i i i i i i i i i Ti i

Ti

nnk m m m m m m m m Wnn

⎛ ⎞⎜ ⎟⎜ ⎟= − − − −⎜ ⎟⎜ ⎟⎝ ⎠

∑ . (28)

Rigid transformation function )(vf cr has the form

* *( )( )( )

cc r

rc

r

f vf v v s df v

→

→= − +

(29)

Where

( ) ii

iiiic

r Annnnvf ∑=→

4321 ˆˆˆˆ)( , (30)

and iA is the same as (27).

IV. DEFORMATION EXPERIMENTS Based on deduced transformation function above, we have

done the image deformation experiments and compared the resulting images.

Fig.3 shows an example of the image deformation based on control curves. With the same sets of points, we implement affine, similarity and rigid transformation experiments to deform the girl’s face and mouth outline. Comparing with Fig.2, our method using cubic splines can express the character’s outline and make her face and mouth deformation smooth and realistic.

(a) (b) (c) (d)

Figure 3. Image deformation based on control curves: (a) Original image ;(b) Affine transformation; (c) Similarity transformation; (d) Rigid transformation

Fig.4 shows another example of the image deformation.

The rigid deformation based on control points is shown in Fig.4(b). The roof has rotated, and the cloud undergoes distortion, which is not our desirable deformation. Fig.4(d) shows the resulting deformation based on sets of control curves, which can divide deformation region. Note that the roof has rotated and nearby region deforms smoothly and realistically.

Figure 4. Comparison between two deformation methods: (a) Original image with control points; (b) Rigid transformation based on control points; (c) Original image with control curves; (d) Rigid transformation based on control curves

V. CONCLUSION The curve can be used for expressing shape and complex

outline. In this paper, based on cubic splines, we study and realize image deformation using Moving Least Squares. The affine, similarity and rigid transformation functions are deduced. We achieve accelerating warping by image grid partition and OpenGL texture mapping. Because of considering shape and outline of image, our method can be used widely and conveniently. Experimental results show that our method can obtain satisfactory resulting deformation.

REFERENCES [1] T. W. Sederberg, and S. R. Parry, Free-form deformation of solid

geometric models, In Proceedings of ACM SIGGRAPH 1986, ACM Press, 1986:151-160.

[2] S. Y. Lee, K. Y. Chwa, and S. Y. Shin, Image metamorphosis using snakes and free-form deformations, In SIGRAPH’ 95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY. USA, 1995: 439-448.

[3] T. Beier, and S. Neely, Feature-based image metamorphosis, In SIGGRAPH’92: Proceedings of the 19th annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY. USA, 1995: 35-42.

[4] M. Meng, and L. G. Liu, Sketching Image Morphing using Moving Least Squares, The 3rd Korea-China Joint Conference on Geometric and Visual Computing, Seoul, Korea, 2007, pp.190-197.

[5] T. Igarashi, T. Moscovich, and J. F. Hughes, As-rigid-as-possible shape manipulation, ACM Trans., Graph., 2005, 24(3): 1134-1141.

[6] Y. L. Weng, W. W. Xu, Y. C. Wu, K. Zhou, and B. Guo, 2D Shape Deformation Using Nonlinear Least Squares Optimization, The Visual Computer, 2006, 22(9): 653-660.

[7] H. Guo, X. Y. Fu, and F. Chen, As-rigid-as-possible shape deformation and interpolation, Journal of Visual Communication & Image Representation, 2008,19(4):245–255.

[8] H. Fang, and J. C. Hart, Detail Preserving Shape Deformation in Image Editing, ACM Trans., Graph., 2007, 26(3): 12-16.

[9] S. Schaefer, T. McPhail, and J. Warren, Image Deformation Using Moving Least Squares, ACM SIGGRAPH, 2006: 533-540.

[10] D. Levin, The approximation power of moving least squares, Mathematics of Computation, 1998, 67(224):1517-1531.

(a) (b)

(c) (d)