MOTION FLOW ESTIMATION FROM IMAGE SEQUENCES WITH APPLICATIONS TOBIOLOGICAL GROWTH AND MOTILITY

Gang Dong, Tobias I. BaskinDepartment of Biology

University of MassachusettsAmherst, Massachusetts 01003 USA

ABSTRACT

In this paper, a new method for motion flow estimation thatconsiders errors in all the derivative measurements ispresented. Based on the total least squares (TLS) model, weaccurately estimate the motion flow in the general noisecase by combining noise model (in form of covariancematrix) with a parametric motion model. The proposedalgorithm is tested on two different types of biologicalmotion, a growing plant root and a gastrulating embryo,with sequences obtained microscopically. The local,instantaneous velocity field estimated by the algorithmreveals the behavior of the underlying cellular elements.

Index Terms- image motion analysis, velocitymeasurement, biological cells.

1. INTRODUCTION

Motion estimation is one of the fundamental problems in theimage processing field. It is a process to extract a field ofvelocity measurement of patterns from an image sequence.This velocity field is often called the optical flow.

Numerous studies, empirical and theoretical, have beenperformed on the optical flow estimation and its informationcontent (see e.g., [1] and references therein). Performanceevaluations of some of the most popular algorithms can befound in [2], [3]. Many optical flow formulations assumebrightness consistency, i.e., the image brightness g(x(t), t) issupposed to be stationary with respect to the time variable tand change only due to motion. Thus we have g(x(t), t) = c,for some constant c, where x = [x, y]T denotes spatialcoordinates. This assumption leads to the well-knownopticalflow constraint by taking the temporal derivative

dgdt gXu+gyv+gt

= 0,

where Vg [gx, gy, gt]T is a vector representing the spatio-temporal derivatives, and v = [u, v]T is a vector representingthe optical flow field to be estimated.

Because the single equation of (1) is not sufficient touniquely solve the unknowns, a fact referred to as the

Kannappan PalaniappanDepartment of Computer ScienceUniversity of Missouri-ColumbiaColumbia, Missouri 65211 USA

aperture problem, additional constraints are needed. Oneapproach often used is to assume that the velocity flow v islocally constant in each small spatial neighborhood Q [4].The flow estimate can then be estimated by minimizing thefollowing least squares (LS) cost function

(2)JLS(v) = (gXu + gyv + gt)XGQ

which is referred to as the Lucas-Kanade method [4]. Thecost function can be interpreted as a measure of discrepancyhow much the signal within that region fluctuates in thedirection given by v = [u, v]T.

The use of classical LS method to estimate the opticalflow implicitly assumes that the spatial derivatives are error-free and the errors are confined to the temporal derivativemeasurements. However, this noise model assumption isinconsistent with the fact that the observed image generallycontains noise and all the spatio-temporal derivativemeasurements are computed numerically, which leads us touse the total least squares (TLS) method [5] or errors-in-variables (EIV) method [6] as known.

In this paper, as an extension work of [1], we proposean optical flow estimation algorithm with the followingcharacteristics: 1) the errors in the spatial derivativemeasurements are taken into account based on TLSframework; 2) instead of the constant velocity assumption in[1], we use more flexible parametric model to describe themotion of each neighborhood and use all the tensors in thatregion to compute the motion parameters; 3) we formulatethe problem in a close-form cost function that can beefficiently solved by suitable approximation schemes.

This work is motivated by the need to measuremovements of biological objects. The growth anddevelopment of organisms is one of the most fundamentalproblems in biology. Automated and accurate measurementof growth is needed to evaluate the underlying mechanismfor the size and form changes of biological objects, and toprovide a non-intrusive tool for the morphogenesis analysis.In our work, sequences of two types of biological object, theroot of the model plant Arabidopsis thaliana and theembryo of the frog Xenopus laevis, are used to illustrate theproposed algorithm.

1-4244-0481-9/06/$20.00 C2006 IEEE 1245 ICIP 2006

2. TLS MOTION ESTIMATION

The LS method in (2) admits analytic solutions.However, the underlying assumption of LS is not realistic inapplications since it implicitly assumes that the two spatialderivatives g, and gy are obtained without error. In fact, theimage sequences are contaminated by noise generally. Thus,the use of numerical differentiation methods to compute thederivatives makes that the errors maybe present in all of thecomputed spatio-temporal derivatives components.Therefore, when information about the derivativemeasurement noise is available, it is desirable that it beincorporated into the estimation process.

Let A be the covariance matrix of Vg characterizingthe uncertainty of the gradient components. We assume thatthe measurement noise has zero mean and the samecovariance matrix at all pixels. We further assume that x-,y- and t- components of the noise are independent. Thecovariance matrix thus takes the form

FT2 0Cx ° OA= 0 (T2 0

(T2~~~~~~2]~

(3)

Note that we have Tx = Ty2 due to the fact that same kernelis used to compute both spatial gradients.

Letting v = [vx, vy, vj]" as a 3D flow vector inspatiotemporal space, the solution of TLS estimation can beobtained by minimizing

JTLS (V) = [(Vg(x)) v]xQn v A

vTTvvTAv

whereT = [Vg(x)Vg(X)T (5)

is often referred to as the structure tensor [7]. This methodcan be deemed as a spatiotemporal variant of Lucas-Kanademethod.

Because the matrix A is non-singular, the minimizer ofJTLS(V), VT-LS, is an eigenvector of A-'T corresponding to thesmallest eigenvalue. In practice, the singular-valuedecomposition (SVD) of A-'T is carried out to determineVTLS Normalizing the third component of v to one yieldsoptical flow u = vlv, and v = vylv, as the first twocomponents of v.

3. TLS MOTION ESTIMATION WITH AFFINEMOTION MODEL

In this work, we introduce a constraint for the motionfield v within each localized neighborhood by assuming thatv can be described by an affine motion model. Often used asa simplified version of the perspective projection motionmodel, the affine motion model can be expressed as

vx x y 1 0 0 00 0 01FPov vy O O x y 1 0 0 0 .

vt 0 0 0 0 0 0 x y 1 P8Bp . (6)

Although affine model is generally unsuitable for accuratelydescribing 3D motion over a significant area of an image, ithas been shown to offer a good inter-frame motion modelfor small localized image regions [8].

Substituting (6) into (4) shows that affine parametersPITLS can be solved by finding the minimum of the costfunction

JTLS(V(PT))ZE B(X)T Vg(x)Vg(x)T B(x)p5GQ 15T'B(X)TAB(x)15z pS(X)pxE Pi A (x)i

whereS(x) = B(x)TVg(x)Vg(x) B(x),

(7)

(8)and

(9)The matrix A,(x) denotes the covariance matrix for thedifferent affine parameters at pixel x. Mathematically, it canbe proved that PTLS, JTLS-based estimate of p, is equivalentto the maximum likelihood estimate given Gaussian noisemodel with covariance A .

A similar work is presented by Liu et al. [9], where thesummation of S within a local neighborhood is calledparametric structure tensor. It is shown that the leastsquares estimation of motion parameters p5 can be recoveredby solving a generalized eigenvalue problem (see [9] fordetails).

Now we are interested in finding the minimum of theTLS cost function in (7). The cost function is a nonlinearequation and is infeasible to admit solutions in closed form.However, a tractable approximation can be derived byemploying a suitable numerical scheme. A commonlyadopted method to minimize a function involving fractionalexpressions is proposed by Sampson [10]. We applySampson's method to minimize JTLS. Let

. sp(x)p (10)xXG Pk Api(X)ik (0

for a specific Pk 'We have

JTLS = P TMk P 11Note that each function JTLS is now in the quadratic formof p5. Assuming that PTLS lies close to pLS' the numericalSampson's scheme is implemented as follows

1. Find LS estimation PLS and initialize 15o2. Given 15-1' calculate the matrix Mk 1.

PLS

1246

A. (x) = B(X)T AB(x) ,

3. Compute the eigenvector of Mk l corresponding tothe smallest (but non-negative) eigenvalue bySVD, and choose this eigenvector as pPk

4. Steps 2 and 3 are then repeated using the newlyestimated p5k until convergence is reached. Thecondition for convergence is that p, is sufficientlyclose to Pk-l1

In our case the convergence remains fast, no more than4-5 iterations being needed.

4. APPLICATIONS AND EXPERIMENTAL RESULTS

Our method has been tested on the Yosemite synthetic testsequence where the ground truth velocity field is known andon two real biological sequences: A. thaliana root and theXlaevis embryo sequences. In this section, we describe thesetests and provide the experimental results. There is currentlyno ground truth data for the real test sequences; however,these sequences themselves provide strong visual cues aboutthe flow directions so that we can evaluate the correctnessof the results.

Evaluation with the synthetic test sequence. The Yosemitesequence is of the observer 'flying' through a syntheticallygenerated landscape of Yosemite mountain. This sequenceis mostly well textured, except that the sky part is much lesstextured and loosely maintains its intensity profile. Theevaluation is performed by computing the angular errormeasure suggested by Barron et al. [1]. This error measureis given by cos '(vestVtrue) where vest denotes the estimatedflow and Vtme denotes the true flow.

For all the tests, we use a spatiotemporal Gaussian filterwith a standard deviation of 1.5 pixels-frames to smooth thesequences before further processing. The spatialneighborhood Q is 15 xi5 pixels, wherein a Gaussianweighting function with a standard deviation of 3.5 pixels isutilized. The weighting function is used to give moreconstraints to the pixels at the center of the neighborhoodthan those at the periphery. Additionally, we use (9) tocalculate each covariance matrix A .

The evaluation results for the Yosemite sequence aresummarized in Fig. 1 and in Table 1. In Fig. 1, we presentoriginal frame, the estimated velocity magnitude, andvelocity in the form of vector field. The numerical resultsfor the sequences with and without sky region are tabulatedin Table 1 where the average error, the standard deviationand the percentage of motion vectors whose angular error isless than certain thresholds are given. By comparing withpreviously published results [1], it shows that our methodachieves excellent accuracy in terms of angular error.

Application to A. thaliana root sequences. The plantspecies A. thaliana has been selected as an experimentaltool among biologists for intense study due to its favorable

genetics. The objective here is to identify the pattern ofgrowth velocity distribution to understand the mechanism ofcell elongation and its relation to division within thegrowing region of the root [ 1]. In this experiment, the rootis imaged in a series of overlapping stacks, each of whichincludes nine frames captured in the root's growth zone.

For the purpose of studying root elongation, thecomponent of the velocity field parallel to the root midlineis of interest. The root midlines are determined manually toimprove precision. Fig. 2 shows a representative sample of avelocity profile measured from multiple stacks by theproposed algorithm. The trends in the spatial distribution ofvelocity can be observed clearly in Fig. 2.

Application to X. laevis gastrulation sequence. The thirdexperiment is performed on the Xenopus laevis sequence toinvestigate the morphogenetic movements involved ingastrulation and neurulation of the frog embryo duringembryogenesis [12]. The spatial and temporal patterns ofcell motility are visually noticeable in the example imagesshown in Fig. 3(a) and (b). The estimated optical flowmagnitude profile and optical flow as vector field aredepicted in Figs. 3(c) and (d), respectively. The recoveredflow fields from our approach agree well with visualobservation.

5. CONCLUSION AND FUTURE WORK

We present a motion flow estimation algorithm, withapplication to recovery of biological growth velocity inparticular. By incorporating a comprehensive TLS model,our method takes into account the errors in all of themeasurements. The estimation accuracy is further improvedusing an affine parametric motion model. The biologicalvelocity profile is traditionally estimated by markinggrowing organ regions using markers such as ink dots orgraphite particles, and tracking the markers over time.Although this type of approach is potentially limited by theinvasiveness of marking and by the small number of marksthat can be applied, it provides a way to obtain ground truthdata for quantitative algorithm evaluation and algorithmcomparison between different optical flow algorithms and inwhich our work will be extended further.

AcknowledgementWe thank Prof. Ray Keller at University of Virginia for thegastrulation sequence. This work has been supported by theNIH Award EB00573-OIA1.

References:[1] K. Palaniappan, H. Jiang, and T. I. Baskin, "Non-rigid

motion estimation using the robust tensor method," in Proc.CVPR W, vol. 1, pp. 25, 2004.

[2] J. L. Barron, D. J. Fleet, and S. Beauchemin, "Performanceof optical flow techniques," Int. J. Comput. Vis., vol. 12, pp.43-77, 1994.

1247

[3] B. Galvin, B. McCane, K. Novins, D. Mason, and S. Mills,"Recovering motion fields: An analysis of eight optical flowalgorithms," in Proc. British Machine Vision Conference,Southampton, England, 1998.

[4] B. Lucas and T. Kanade, "An iterative image registrationtechnique with an application to stereo vision," in Proc.Seventh International Joint Conference on ArtificialIntelligence, pp. 674-679, 1981.

[5] S. Van Huffel and J. Vandewalle, The Total Least SquaresProblem. SIAM, Philadelphia, 1991.

[6] B. Matei and P. Meer, "A general method for errors-in-variables problems in computer vision," in Proc. ComputerVision Pattern Recognition, vol. 2, pp. 18-25, 2000.

[7] B. Jahne, H. Hassecker, H. Spies, D. Schmundt, and U.Schurr, "Study of dynamical processes with tensor-basedspatiotemporal image processing techniques," in Proc.ECCV, vol. 2, pp. 322-336, 1998.

[8] J. Giaccone, "Experiments in Recovering Affine OpticalFlow," School of Computer Science, Kingstone University,1996.

[9] H. Liu, R. Chellappa, and A. Rosenfeld, "Accurate denseoptical flow estimation using adaptive structure tensors and aparametric model," IEEE Trans. Image Processing, vol. 12,pp. 1170-1180, 2003.

[10] P. D. Sampson, "Fitting conics sections to 'very scattered'data: An iterative refinement of the Bookstein algorithm,"Computer Graphics and Image Processing, vol. 18, pp. 97-108, 1982.

[11] G. T. S. Beemster and T. I. Baskin, "Analysis of cell divisionand elongation underlying the developmental acceleration ofroot growth in Arabidopsis thaliana," Plant Physiology, vol.116, pp. 1515-1526, 1998.

[12] R. Keller, L. A. Davidson, and D. R. Shook, "How we areshaped: The biomechanics of gastrulation," Differentiation,vol. 71, pp. 171-205, 2003.

tDisance t[mihe quisecenitcenter (m)

Figure 2. Velocity profile for A. thaliana root beneath a mosaic ofthe root, approximately to scale.

(c)(a)

_~' / / / 1

_z,-< (c)

Figure 1. (a) Frame 8 from the Yosemite sequence with skyregion, (b) estimated velocity magnitude profile, (c) estimatedoptical flow as vector field (subsampled).

R / / __ _s * f of /E \; l / f /J _>__, uz J - S _ _- \

_ ___ v^ *s,, oosx / / wf-,,v,, B / / _ o-_ _ -r Sr _ / x \ > \ _ Sz r / _ //. v E

z / ,r r tS \ _ \w_ >., r ! / \ x>_., / t: fi \ \1 5 x fi . .

(d)

Figure 3. (a) Frame 1 from the Xenopus laevis sequence, (b) ditto,frame 9, (c) estimated optical flow magnitude profile, (d)estimated optical flow as vector field (subsampled).

Table 1. Flow estimation performance for Yosemite sequence.

With sky Withoutregion sky region

Average angular error 5.260 1.790Standard deviation of error 10.770 3.650

10 32.3% 54.8%Estimated motion 20 64.7% 80.6%vectors with error lessthan 50 89.2% 95.9%

100 93.9% 99.1%

1248