Generating Calligraphic Trajectories with Model Predictive ControlDaniel Berio∗

Goldsmiths, University of LondonSylvain Calinon†

Idiap Research InstituteFrederic Fol Leymarie‡

Goldsmiths, University of London

ABSTRACT

We describe a methodology for the interactive definition of curvesand motion paths using a stochastic formulation of optimal control.We demonstrate how the same optimization framework can be usedin different ways to generate curves and traces that are geometricallyand dynamically similar to the ones that can be seen in art forms suchas calligraphy or graffiti art. The method provides a probabilisticdescription of trajectories that can be edited similarly to the controlpolygon typically used in the popular spline based methods. Fur-thermore, it also encapsulates movement kinematics, deformationsand variability. The user is then provided with a simple interactiveinterface that can generate multiple movements and traces at once,by visually defining a distribution of trajectories rather than a singleone. The input to our method is a sparse sequence of targets definedas multivariate Gaussians. The output is a dynamical system gener-ating curves that are natural looking and reflect the kinematics of amovement, similar to that produced by human drawing or writing.

Index Terms: Computer Graphics [I.3.3]: Picture/ImageGeneration—Line and curve generation; Computer Graphics [I.3.6]:Methodology and Techniques—Interaction techniques; ComputerApplications [J.5]: Arts and Humanities—Fine arts

1 INTRODUCTION

The hand drawn curves that can be seen in certain art forms suchas calligraphy and graffiti art are often the result of skillful and ex-pressive movements that require years to master [29]. The resultingtraces often possess subtle qualitative features which are difficult toreproduce through traditional computer graphics methods such asB-splines or Bezier curves. We adopt the hypothesis that some ofthe aesthetic qualities observable in hand drawn traces are closelyassociated to the way such traces were created, and in particular withrespect to their dynamic properties (i.e. velocity, acceleration, and as-sociated curving behaviour). Such an hypothesis is commonly heldamongst visual artists, and is further supported by academic work intheories of art history [17, 34], as well as by studies stemming fromthe fields of psychology and neuroscience. Such studies indicatethat the visual perception of marks made by a drawing hand triggeractivity in the motor areas of the brain [18, 26], and further inducean approximate mental recovery of the movements and gesturesunderlying the artistic production [19, 32].

In computer graphics applications requiring the simulation ofhand drawn traces, it then looks advantageous to follow a movementcentric approach, in which a shape is defined by the movementunderlying its production rather than by an explicit definition ofits geometry. We argue that doing so simplifies the process ofcomputationally capturing the inherent qualities of human madetraces, such as smoothness and variability, which stem from theproperties of the motor system and from the type of movementsused when drawing. In this paper, we explore this approach with

∗e-mail: d.berio@gold.ac.uk†e-mail: sylvain.calinon@idiap.ch‡e-mail: ffl@gold.ac.uk

Figure 1: Left: Interactively editing of a trajectory through the manip-ulation of Gaussians. Right: interactive editing and visualisation ofthe trajectory distribution (gray).

a trajectory generation methodology based on stochastic optimalcontrol.

Optimal control consists of a family of methods aimed at deter-mining the trajectories of a dynamical system that minimise a givencost or performance criterion [10]. In our study we solve the optimalcontrol problem numerically with a stochastic formulation of ModelPredictive Control (MPC), a technique that is popular in roboticsand industrial related applications. While MPC is typically used in acontrol system setting, we demonstrate how it can be used as a flexi-ble curve generation tool for computer graphics applications. Thecontrol system produces trajectories that have desirable smoothnessproperties, while sharing common ground with conventional curvegeneration methods [13].

We formulate the optimisation objective as a discrete sequenceof targets that are defined probabilistically in the form of multivari-ate Gaussian distributions. The centres of each Gaussian definea coarse control-polygon which permits interactive manipulation(Fig. 1), behaving similarly to traditional spline/polynomial inter-polation methods. In addition, the covariances explicitly describethe variability of different parts of the movement, as well as to de-scribe curvilinear features that take into account human movementcoordination [8]. The optimisation process results in smooth curvesand motion paths that are, by design, dynamically and kinematicallysimilar to the ones that would result from a human movement.

The contributions of this paper are two-fold. First, we describea flexible and interactive trajectory generation method that can beused to synthesise curves for computer graphics applications, whilealso being able to generate motion paths for computer animationor robotics applications. Second, we show that this methodologyis particularly useful for applications requiring the simulation ofhuman made artistic traces. We focus on examples involving theproduction of letter forms as the ones that can be seen in traditionalcalligraphy as well as contemporary graffiti (street) art. We showthat our method is well suited to capture the visual features of suchtraces, with the additional benefits of facilitating a realistic animationof the trajectory evolution, and providing rich dynamic informationthat can be exploited to facilitate expressive rendering methods.

The rest of the paper is organised as follows. The next sectiongives a brief background on related work and further explains theconcepts that form the basis for this study. In Sect. 3 we describethe optimisation framework used for the trajectory generation meth-

ods, which are then outlined in Sect. 4. These include a stochasticdefinition of trajectories Sect. 4.1, which can also be reformulated todescribe interpolating trajectories (Sect. 4.2), as well as to approxi-mate Bezier curves (Sect. 4.3). Finally, in Sect. 5 we describe howour system can be used in an interactive setting for the purposed goalof generating instances of synthetic calligraphy. Additional mathe-matical and implementation details are given in the Appendices.

2 RELATED WORK AND BACKGROUND

While movement synthesis has not been commonly used for thesynthesis of artistic imagery (e.g. see [24]), a few examples exist thathave proposed approaches that are related to our method. Haeberliimplemented DynaDraw, a computer program that allows the user tointeractively generate strokes evocative of calligraphy by simulatinga mass attached to the mouse position [20]. House et al. [22] aswell as Liu et al. [25] generate sketch based renderings by using aProportional Integral Derivative (PID) controller, which controlsthe evolution of a 2nd order system describing the trajectory of a pen.Shinoda et al. minimize the jerk (time derivative of acceleration)of calligraphic traces defined as B-Splines in order to mimic theeffect of running-style Japanese calligraphy [36]. AlMeraj et al. [1],use the minimum-jerk model of human reaching movements [16] tomimic the visual qualities of hand drawn lines. With a motivationsimilar to ours, Berio and Leymarie [3] apply the Sigma Lognormalhandwriting model [33] to interactively generate variations of graffitilike traces. Similarly to our method, letter forms can be definedinteractively by defining a control polygon made of a sparse numberof targets. In this communication, we propose a method basedon stochastic optimal control that, in its different formulations, isapplicable to the same type of problem domains as the ones tackledin the previously mentioned studies. The main advantages of ourmethod is that: (i) it encompasses variability and coordination at thedescriptive level; and (ii) its generality and flexibility allow a userto experiment with different types of trajectory generation methodsand dynamical systems with a single framework.

Typically, hand drawn curves are interactively specified by using asketch based interface, in which a user traces a curve with a trackpad,mouse or tablet, and the trace is then processed to remove discontinu-ities and imperfections caused by the digitising device. This processis referred to as “beautification”, “neatening” or “fairing”, and hasbeen implemented in a number of methods, e.g. [27,28,41]. We pro-pose an alternative method to the definition of hand-drawn curves, inwhich a user defines a coarse series of targets with a point-and-clickprocedure and a control system defines a trajectory that tracks thetargets, resembling traces produced by a human movement.

In this study, we rely on principles that have been observed inthe movement science and handwriting analysis/synthesis domains.The velocity profile of rapid and straight reaching motions are char-acterised by “bell shaped” velocity profiles [30]. Such bell shapedvelocity profiles have been modeled with a variety of techniques,including sinusoidal functions [31], Beta functions [5], optimisa-tion methods [16], and lognormals [33]. Many studies propose thatsmooth arm motions can be described as the space time superposi-tion of a number of “ballistic” movement primitives [40], which arecommonly referred to as strokes. Each stroke can be representedwith the characteristic bell-shaped speed profile. Complex move-ments tend to show an inverse power relationship between absolutecurvature and speed, where curvature extrema usually correspond tominima in the speed profile [44] . Various experiments have exposedthe tendency of humans to keep the time of movements relatively in-dependent across different size ranges, aka isochrony [44]. Isochronycan be global, i.e. for movements and trajectories as a whole, orlocal, for parts of a movement [23]. Complex hand and arm motionstend to be executed in a smooth manner with trajectories that seemto minimise a cost or performance objective [15]. Our method sharesrelations with a number of models that formulate this objective with

the minimisation of the square magnitude of higher order derivativesof position, such as the minimum jerk (3rd order) [16], minimumsnap (4th order) [12] and minimum crackle (5th order) [11] mod-els. Humans movements show inherent variability [4], where thevariability tends to be higher in parts of a movement that are notcritical to the required precision of a task, which is known as theminimal intervention principle [42]. Our method is consistent withthis principle, and allows a user to explicitly define the required taskprecision in different parts of a movement through the manipulationof covariances.

We can identify in the literature two dominant representationsthat are used to describe the spatial evolution of hand-movementpaths: virtual targets and via-points. Virtual targets imply a ballisticstroke representation of movement, and describe the “imaginary”loci at which each stroke is aimed. As a result, these positions donot directly lie along the corresponding motion path. Such a repre-sentation has been adopted in a variety of models of handwriting,e.g. [5, 33]. As the name implies, via-points represent landmarkpositions along the trajectory, which are the basis for a number ofoptimisation-based models of movement (e.g. [12, 16]. In this paperwe describe a representation that functions as a “hybrid” betweenvia-points and virtual-targets, in which the evolution of a trajectoryis encoded in terms of multivariate Gaussians. This hybrid repre-sentation is capable of describing ballistic virtual target positions,via-points, as well as more complex spatial constraints, such asforcing a movement to pass through a narrow region of space.

The proposed method also shares relations with interpolatingand smoothing splines. Egerstedt and Martin [13] discussed theequivalence between several forms of splines and control theoreticformulations of dynamical systems. The authors cover the case ofinterpolating splines in which the curve passes through user-definedkeypoints, as well as smoothing splines in which a trade-off isfound between curve smoothness and curve fitting. It is shown thatsmoothing splines correspond to the output of a controller found byminimizing quadratic cost functions similar to the ones used in ourmethod. Our method extends this principle to a more generic case,in which each covariance matrix encodes the precision as well ascoordination patterns in the movement.

3 OPTIMAL CONTROL FRAMEWORK

Model Predictive Control (MPC) encompasses a series of numericalmethods used to predict the behaviour of a dynamical system, andcompute a series of optimal feedback or feedforward commands thatwill minimise a given cost function and constraints over a given timehorizon. In a typical control setting, only the first optimal commandis fed to the system, and the optimisation process is repeated iter-atively by shifting the time horizon forward to the next time step.As a result, MPC is also commonly referred to as receding horizoncontrol. Because our application is focused on curve/trajectory gen-eration rather than control, we perform only a single optimisationstep with a time horizon that corresponds to the duration of thetrajectory as a whole. From a control perspective, this correspondsto the assumption of a perfect reproduction of trajectory and noexternal disturbance. As we shall show, this assumption allows us toexploit MPC as a flexible motion and curve synthesis tool.

For the task at hand, we use the simplest form of MPC, such thatthe objective is unconstrained, the system is linear and the cost func-tion is quadratic. In this case the problem is equivalent to the controlproblems known as discrete Linear Quadratic Tracking (dLQT).While MPC is based on a well understood control-theoretical back-ground, in our application we focus on a step by step description,which follows and is complemented by mathematical and imple-mentation details in the Appendices. Thanks to the availability ofpowerful linear algebra libraries and software packages, the methodcan be implemented in a straightforward manner by following theequations described in the text. For more detailed theory and deriva-

tions, the interested reader is referred to [7].

3.1 Dynamical systemWe model the movement of a pen trajectory by optimising the evo-lution of a discrete linear time invariant (dLTI) system of order n ,which results in a discrete sequence of positions xt ∈ RD , wherethe index t denotes the t-th time step. The dynamical system isdescribed with the state space representation

ξt+1 = Aξt +But, (1)

where the system state

ξt =[x>t , x

>t , . . . ,

(n−2)x >t ,

(n−1)x >t

]>∈ RnD, (2)

is given by the position concatenated with its derivatives up to theorder n − 1. The dynamics of the system are determined by thematrices A ∈ RnD×nD and B ∈ RnD×D , which fully describethe response of the system to an input command ut. The methodsdescribed in this paper function for higher dimensions, but for thescope of this study we focus on planar trajectories, so we assumeD = 2. While the optimisation framework we will describe canfunction with arbitrary linear systems, in the examples demonstratedhere, we use a chain of n integrators that is controlled by its nthorder derivative. For example, a system of order 2 will correspondto a system controlled with acceleration commands. The reader isreferred to Appendix A for further mathematical details.

3.2 Quadratic costAn optimal trajectory is computed by minimising a quadratic costfunction that, for each time step, tries to reduce deviations from areference state sequence while keeping the amplitude of the controlcommands low. For a trajectory of N time steps, the cost is given by

J =

N∑t=1

(ξt − ξt

)>Qt

(ξt − ξt

)+

N−1∑t=1

u>tRtut, (3)

where for each time step, ξt is the desired state (position and option-ally consecutive order derivatives) andQt ∈nD×nD andRt ∈D×D

are positive semidefinite weight matrices. The weight matrices re-spectively define the tradeoff between tracking accuracy (state cost)and limiting the amplitude of the control commands (control cost).In our application, we keep the control cost fixed with a diagonalregularization term Rt = rI , where larger values of r producesmoother trajectories.

The optimisation problem can be solved either: (i) in a batchform, by solving a large regularized least squares problem; or (ii)iteratively, by using dynamic programming and resulting in the aseries of time-varying gains. While the latter method is faster, thebatch approach directly provides a probabilistic interpretation of theresult, which we exploit for stochastic sampling. For implementationdetails of both methods, the reader is referred to Appendix B.

4 TRAJECTORY GENERATION

In the following paragraphs we describe different trajectory gener-ation methods that can be achieved with MPC. All the proposedmethods run at interactive rates and produce trajectories that aresmooth and are kinematically similar to the ones that can be seenin movements made by a human when drawing or writing. Thecontrol term of the cost function enforces a smooth and continuoustrajectory, regardless of continuity of the desired state sequence. Asa result, it is possible to define the reference as a sparse sequenceof states. This results in a concise and easily manipulable repre-sentation of the trajectory geometry that can be exploited in a userinteraction scenario.

4.1 Minimal intervention trajectoriesIt has been demonstrated that the combination of MPC with a proba-bilistic representation can be exploited to capture the variability ofmultiple human demonstrations in a human-robot interaction sce-nario [9]. Here, we show how a similar approach can be exploitedfor the task of trajectory and curve generation.

1 2 3

stepwise reference

Control amplitude

Speed

Figure 2: 4th order trajectory generated with Gaussian targets, and thecorresponding speed and command (snap) magnitude profiles. Below,schematic visualisation of the state vector for a stepwise reference.The duration of each state is color coded with the correspondingGaussian.

We describe a trajectory with an ordered sequence of m states,where each state is defined with a multivariate Gaussian distributionND (µi,Σi). This representation can be seen as a ballistic decom-position of the movement in m− 1 strokes, where each stroke i isaimed at the center of the (i+ 1)th Gaussian. With an assumption ofperfect local isochrony (i.e. a fixed duration for each state), we definea state vector s ∈ ZN in which each consecutive state is indexedN/m times in a stepwise manner (e.g. s = {1, 1, 2, 2, 2, 3, . . . ,m},see Fig. 2). The reference state sequence and tracking weights forthe whole optimisation horizon are then given by

ξt = µst and Qt = C>Σ−1st C, (4)

where the C is the sensor matrix, the block entries of which de-termine what components of the state should be considered in theoptimisation objective. For this use case, we only consider theposition components, which is described with a sensor matrix

C = [I,0, . . . ,0] ∈ RD×nD. (5)

This produces zero entries ofQt for the state derivative terms, whichare consequently ignored in the cost function1. For the last timestep, we set the matrix QN identically to Equation 5, but we thenadd a large constant diagonal value to the derivative terms of matrixthat corresponds to a high precision and low variance (we used1× 1010 in our examples, which for our use case provided consistentresults across different system orders without numerical issues).This enforces a zero boundary condition on the state derivatives andbrings the movement to a smooth stop.

With this formulation the tracking weights are defined in terms ofrequired precisions, and the penalty of deviating from a given state isgiven by the Mahalanobis distance to the centre of the correspondingGaussian. The resulting trajectory formation method is thereforeconsistent with the minimal intervention principle [42]. Each Gaus-sian functions as a stochastic target: as the variance of the Gaussianapproaches zero, it increasingly constrains the trajectory to pass

1Note that a sensor matrix with all block entries set to I would corre-spond with an optimisation objective that takes all state derivatives intoconsideration.

(a) (b)

Figure 3: (a), smoothing effect of increasing the variance of a Gaussian. (b), manipulating the trajectory evolution with full covariances. Beloweach trajectory, its corresponding speed profile.

through its location, effectively behaving like a via-point. A highervariance produces a smoothing effect (similar to smoothing splines),resulting in a behavior that is similar to a virtual target (Fig. 3a).In addition, using full covariances can be exploited to mimic cal-ligraphic effects by allowing the definition of coordinations anddirectional trends in the trajectory Fig. 3b).

4.1.1 Stochastic sampling

If we consider a Bayesian formulation of the batch version of theminimisation problem, we can interpret the generated trajectory asthe center of a trajectory distribution [8]. We can then stochasti-cally sample this distribution in order to generate a possibly infinitenumber of variations over the mean trajectory (Fig. 4, mathematicaldetails in Appendix C). While variations could also be achieved byrandomly perturbing the means and covariances of each state, thismethod allows the user to define the spatial evolution, as well as thevariability of the trajectory within a single compact representation.This property is useful for the intended goal of generating calli-graphic traces, since it mimics the variability that can be observedin human handwriting, calligraphy and drawing.

Figure 4: Stochastic sampling of the trajectory distribution for a letter“N”. Top-left: the mean trajectory (black), random samples from thetrajectory distribution (gray) and the corresponding Gaussians. Top-right: The corresponding speed profiles. Bottom: Selected randomsamples from the trajectory distribution.

4.2 Interpolation with MPCWith a slightly different formulation of the same optimisation frame-work, we can generate various types of interpolating trajectories.For this task, we define a series of key points2 {vi}mi=1 and the cor-responding time steps {ti}mi=1. While in the previously describedformulation, we have specified the tracking costs in a stepwise fash-ion for the purpose of interpolation, we use here a sparse reference(Fig. 5, top). Intuitively, this corresponds to an optimisation objec-tive that prioritises trajectory smoothness and only enforces trackingthe given states at the given time steps. To achieve this objective,

2We adopt the term key-points rather than via-points here, because thelatter term does not describe the initial and final point of the trajectory.

(a)

(b)

1 2 3

Figure 5: Interpolation with MPC. a, comparison of MPC (black) with aclosed form solution of a minimum jerk trajectory with 1 via-point (red).On top, a color coded schematic of the corresponding state vector. b,examples of interpolating trajectories of increasing order (2, 3, 4).

we set all desired states and weights to zero except for the timeoccurrence of each key-point with

ξt =

{C>vti if ti = k

0, otherwise., (6)

and

Qt =

C>IC, if ti = k

I, if t = N

0, otherwise.(7)

This expresses the penalty of deviating from key-point positionsat a given time step k with an identity covariance, which reducesto a state cost given by the Euclidean norm to the key-point posi-tion. In order to minimize the command amplitude across the wholemovement, we then set Rt = λI , where λ is a very small regu-larisation term (1× 10−15 for the examples given in this section).Note that the matrix C is again used to determine the number ofderivatives that influence the cost of each via-point. The last entryof Qt is always set to a full identity matrix, which again enforcesa zero boundary condition for the whole state of the system. Thisformulation allows us to use MPC to produce close numerical ap-proximations of a number of trajectory generation methods, such aspolynomial interpolation/splines [14] as well as “minimum squarederivative” (MSD) methods such as the minimum jerk model [38].As a comparative example, we demonstrate that our method canclosely approximate curved minimum jerk trajectories with 1 via-point (Fig. 5a), the closed form solution for which is defined byFlash and Hogan in the form of a quintic polynomial [16]. It shouldbe noted that while the MSD methods predict a time differencebetween consecutive via-points that is approximately equal acrossa movement, the exact time occurrences of each via-point are pre-dicted by the models as part of the optimisation. This will result inthe via-point occurring in the proximity of a curvature extrema of the

trajectory. In the example shown in Fig. 5a, we compute the exacttime occurrence as predicted by the minimum jerk model, whichcan be done by numerically finding the real roots of a polynomial ofdegree 9 [16]. For more complex trajectories, we currently assumea uniform spacing in time between via-points (Fig. 5b), leaving theoptimization of this timing information for further work.

4.3 Mimicking Bezier curvesIn the following paragraphs we describe how the same optimisationframework can be used to mimic the shape and behavior of (cu-bic) Bezier curves. The resulting trajectory/curve generation methodprovides means of interaction almost identical to its parametric coun-terpart. At the same time, it provides the flexibility of MPC (suchas the ability to easily adjust the trajectory smoothness) and alsoguarantees trajectory smoothness regardless of the configuration ofcontrol points. This is particularly useful for calligraphy generation,where the desired trajectories are per se smooth.

It has been shown that cubic Bezier curves [14] and splines [13]can be interpreted as the trajectories of a 2nd order dynamical systemwhich minimise acceleration commands. Indeed, we can see thatwith the previously described key-point formulation, it is possible toclosely approximate a Bezier curve. This can be done by setting alsothe first order derivative entry of the sensor matrixC to I , and thenaugmenting the key-points with the derivative computed accordingto the cubic Bezier formulation (Fig. 6a). This method allows toclosely approximate a Bezier curve, and can be used with the sameconstraints and a higher order systems, which results in a higherdegree of continuity (Fig. 6b). However, the method has limitationsin the definition of piecewise curves, which for example requiresetting the same velocity where consecutive curve segments connect.

(a)

(b)

Figure 6: Approximating Bezier curves with a sparse reference andstates augmented with velocity. a, 2nd order system, which results inan identical curve and speed profile (as shown in [14]). b, the samestates with a 3rd order system. This produces a slightly different curvewith a higher degree of continuity.

We observe that we can also mimic the behavior and shape of aBezier curve by using a stepwise tracking reference. This can bedone by placing isotropic covariance Gaussians centered at eachcontrol point of the curve, and then adjusting the influence of inter-mediate control points on the trajectory by uniformly increasing thevariance of each corresponding Gaussian (Fig. 7). The variancesare currently set empirically, but we plan to explore methods forautomatising this process in further iterations of the study.

At the cost of a less precise approximation, we obtain a curvegeneration method that produces similar shapes to Bezier curveswith a similar representation, and with the additional flexibility ofthe Gaussian representation and the benefit of always maintainingsmooth and physiologically plausible kinematics. The utility of thisproperty in our application is emphasized if we randomly perturbthe control points of a letter form and compare the result with theone produced with a Bezier curve (Fig. 8).

While the Bezier curve becomes discontinuous due to the dif-ferently oriented tangents at the places where the curve segments

MPCBézier

Figure 7: Mimicking Bezier curves (red) with MPC (black) using astepwise reference, isotropic Gaussians and a 4th order system. Be-low, the corresponding speed profiles normalised and superposed forcomparison.

Figure 8: Effect of randomly displacing control point positions withBezier curves (red) and MPC (black).

meet, the MPC formulation tends to maintain a smooth trajectoryregardless of the positions of the control points. This can be ex-ploited as an additional method to generate synthetic variations ofa handwriting or calligraphy trajectory, which can be interactivelyedited with a traditional control point and tangent interface. Thesame smoothness property can be used to concatenate multiple letterforms with ligatures that evoke a smooth and natural motion, whichcan be easily achieved by treating the control points of the letters asa single trajectory (Fig. 9).

Figure 9: Automatic ligature generation by concatenating the controlpoints of two letters. On the right, a comparative example using Beziercurves.

5 INTERACTIVE TRAJECTORY SPECIFICATION AND REN-DERING

The MPC based methods described above are well suited for theinteractive definition of trajectories. It is in fact trivial to drag thekey-points (Sect. 4.2) or control points (Sect. 4.3) with a typicalpoint-and-click procedure, and it is also simple to interactively ma-nipulate the Gaussians (Sect. 4.1) for the probabilistic case (Fig.1). Each Gaussian can be edited interactively by manipulating anellipsoid, where the centre of the ellipsoid defines the mean µi,and the axes are used to manipulate the covariance Σi through itseigendecomposition. The latter can be described with

Σi = ΘiS2i Θ>i , (8)

where Θi corresponds to an orthogonal (rotation) matrix, and Si isa scaling matrix. Here, we describe the 2D case in which the rotation

Figure 10: Top left, a graffiti script (tag) made with a marker by LosAngeles artist “Trixter”. Top right, a user defined “motor plan” forthe tag, generated by placing targets near salient positions alongthe original trace and then adjusting the covariances to follow theoriginal trajectory. Second row, the reconstructed trajectory (left)and one variation made by increasing the regularisation parameter r.Bottom row, two random samples from the corresponding trajectorydistribution.

and scaling matrices are given by

Θi =

[cosθ −sinθsinθ cosθ

], θ = tan−1 a2

a1, Si =

[ ‖a‖2

0

0 ‖b‖2

],

(9)where a and b are the major and minor axes of an ellipse, which canbe interactively dragged to manipulate the shape of the distribution.Isotropic Gaussians influence the trajectory evolution, in a similarmanner to a virtual target or control point, where a small variancewill force the trajectory to pass close to the centre. Thinner Gaus-sians influence the curvilinear evolution of the trajectory, forcing itto follow the direction of the major axis of the ellipse. While the 3Dcase is currently not implemented, it would be straightforward toadapt the described technique to an arc-ball interface [37], in orderto manipulate the 3D rotation components.

In most of our examples we settle with a dynamical of order4, which we evaluate to give the best balance between trajectorysmoothness and a precise control on the trajectory evolution. Inorder to achieve approximately equal tracking performance acrossdifferent system orders, we express the control cost in terms of a max-imum displacement d, and compute R based on the low frequencygain of the system with

R =1

(ωnd)2I and ω = 2π

T

m− 1, (10)

with ω empirically chosen, corresponding to the average duration ofa stroke. Higher values of d will produce sharper trajectories, whilelower values will result in smoother trajectories. On the other hand,because the cost function is defined as a tradeoff between accuratetracking and smooth control, it is possible to keep the value of dfixed depending on the size of the working area, and then interactwith the trajectory by only manipulating the Gaussians.

By utilising the stochastic sampling technique described inSect. 4.1.1, we can interactively visualise the variability of the gener-ated trajectories, while manipulating the Gaussians. This results in a

Figure 11: Comparison of performance for the batch and iterativeapproach.

novel form of interactive trajectory editing that, due to its generativenature, directly describes a family of trajectories rather than a singleone. As an example, the same interface can be used to rapidly re-construct, and generate variations of, an existing instance of humanmade calligraphy (Fig. 10). In this use case, the user first defines acoarse sequence of Gaussians over salient positions along the inputtrace (approximately in correspondence with curvature extrema),and then adjusts the covariances to modify the trajectory and mimicthe curvature and smoothness of the original trace. Variations of theinput can then be generated by stochastic sampling.

The proposed methods generate trajectories with smooth deriva-tives up to the order of the system used in the optimisation. Thiscan be exploited to facilitate painterly and expressive renderings ofthe trajectory. In this study, we limit ourselves to a simple brushmodel (see Fig. 1 and 10), which assumes that the amount of paintdeposited is inversely proportional to the speed of the pen. To mimicthis effect, we sweep a pre-selected texture image along the tra-jectory with a size that varies as an inverse power function of theinstantaneous speed. While this is obviously not an accurate modelof a brush or pen, it results in a trajectory rendering that accentuatesthe perceived dynamics of the trace (see also accompanying video).

Furthermore, the distances between consecutive points along thegenerated trajectory reflect the smooth kinematics generated by themodel. As a result, it is possible to easily generate realistic andnatural looking animations, by incrementally sampling points alongthe trajectory with a fixed time-step. This approach can be used toeither generate stroke animations, or to guide the hand motion of avirtual character or robotic arm.

5.1 PerformanceWe have tested our method on a 2,5 GHz Intel Core i7 machineand used OpenGL for hardware accelerated rendering; We haveimplemented the optimisation code in Python, using the NumPy[43] linear algebra package, as well as in C, using the Armadillolibrary [35]. Both the batch and the iterative approaches run atinteractive rates up to a limit of time steps that depends on the orderof the system used in the optimisation. For the examples given inthis paper we use an optimisation horizon of approximately 200 timesteps, which results in trajectories that are perceived as smooth, andfor which the batch and iterative solutions take in average (acrossorders) 70 ms and 3 ms respectively. The batch approach requires theinversion of a matrix with a dimension which is directly proportionalto the number of time steps, and obviously this results in a rapidperformance drop as the latter increases (Fig. 11, left). This problemis overcome with the iterative solution, the complexity of which isapproximately linear to the number of time steps (Fig. 11, right).On the other hand, the batch solution is more compact and allowsthe intuitive formulation of probabilistic interpretation of the outputand stochastic sampling of the trajectory distribution.

6 CONCLUSION

We have proposed the use of model predictive control (MPC) asa curve generation tool, which can be used in a manner similar toconventional interfaces for polynomial curve generation. Our main

contribution is an application to computer graphics of a probabilisticformulation of MPC that explicitly describes intra-movement vari-ability and coordination. We have shown that the same frameworkcan also be used to compute numerical approximations for polyno-mial interpolation methods such as Bezier curves, or well knowntrajectory formation methods such as the minimum jerk model. Thisresults in a flexible and general trajectory generation method, that weconsider particularly useful for applications that necessitate the sim-ulation of traces such as the ones made by a human when drawingor writing.

While in this paper we focused on the generation of 2D tra-jectories, the proposed methodology can be generalised to higherdimensions. This opens up the possibility to extend the method to3D trajectories, as well as taking in consideration the evolution ofadditional variables, such as the drawing tool orientation or pressure.

In this paper, we have focused on the application of MPC for thegeneration of movements and traces that mimic the visual qualitiesof human made graffiti and calligraphy. At this stage, we have reliedon the qualitative evaluation by a number of experienced artists anddesigners (n = 5), who have characterised the output of our systemas valid instantiations of the targeted hand-styles. In future work,we intend to perform a series of controlled user studies in orderto further evaluate the quality of the generated traces and motions.It should be noted that evaluating the aesthetic quality of a visualmark or trace is not a well defined problem, and the notion of “stylesimilarity” can be subjective to the viewer and depends on factorssuch as cultural and artistic background. On the other hand, wepropose, in line with others before us such as Hertzmann [21] orStacey [39], that the computational study of style is worthwhile,and that a procedure that generates patterns that are perceived assimilar to a given artistic style, provides the grounds to establish apotential and computational theory of art that is both generative anddescriptive [21].

7 ACKNOWLEDGMENTS

We would like to thank Trixter and David Chang for kindly providingcalligraphy samples. This work has been partly supported by UK’sEPSRC Centre for Doctoral Training in Intelligent Games and GameIntelligence (IGGI; grant EP/L015846/1).

A DYNAMICAL SYSTEM

The continuous-time system matrices for the chain of integrators aregiven by

A =

0 I 0 · · · 00 0 I · · · 0...

......

. . ....

0 0 0 · · · I0 0 0 · · · 0

, B =

00...0I

. (11)

The discrete time state matrices A and B used in Equation 1 arethen computed using a Zero Order Hold (ZOH) or forward Eulerdiscretisation of A and A with a sampling period ∆t, where for theEuler case we simply have

A = ∆tA+ I and B = ∆tB. (12)

B COST FUNCTION SOLUTIONS

B.1 Batch solutionFor the batch solution, we express the cost function with

J =(ξ − ξ

)>Q(ξ − ξ

)+ u>Ru, (13)

where Q = blockdiag (Q1,Q2, ...,QN ) ∈ RnDN×nDN , R =blockdiag (R1,R2, ...,RN−1) ∈ RDN×DN , and the desired state,

current state and control commands are respectively stacked in inlarge column vectors ξ, ξ and u. We then express all future states asa function of the initial state ξ1, which can be compactly representedin matrix form as

ξ = Sξξ1 + Suu, (14)

with

Sξ =

IAA2

...AN

and Su =

0 0 . . . 0B 0 . . . 0AB B . . . 0

......

. . ....

AN−1B AN−2B . . . B

,(15)

where Sξ ∈ RnND×D and Su ∈ RnND×(N−1)D . SubstitutingEquation 14 into Equation 13, differentiating with respect to u andsetting to zero results in a regularized least squares estimate of theoptimal command sequence, given by

u =(S>uQSu +R

)−1S>uQ

(ξ − Sξξ1

), (16)

where the control weight matrix R effectively acts as a Tikhonovregularization term, see e.g. [6]. The command sequence u is thensubstituted back into (14), resulting in the optimal trajectory ξ.

We note that with increasing system orders the amplitude of thecommands will grow exponentially, which can result in a badlyscaled matrix in the inverse term of Equation 16. To overcomethis problem, we first scale the desired states ξ by a factor, and thestacked weight matricesQ by the same factor squared. This value ischosen in order to limit the standard deviation of the components ofξ below a maximum range. In the examples given in this paper wechoose a factor that keeps the standard deviation below 1× 10−6,which gives numerically stable results up to a system order of 5.

B.2 Iterative solutionA more efficient solution to the optimisation problem can be derivedusing dynamic programming or an extension of variational calculusknown as Pontryagin’s Maximum Principle. We refer the interestedreader to the work of Bryson [7] for the details of the derivations.It follows that the optimal solution is given in the form of a feed-back controller with time varying weighting matrix Kt, and thecommands are for each time step t given by

ut = −(B>PtB +Rt

)−1

B>PA︸ ︷︷ ︸Kt

ξt, (17)

where

Pt = Qt−A>(Pt+1B (B>Pt+1B +Rt)

−1B>Pt+1−Pt+1

)A

(18)is a Riccati difference equation, which can be solved backwards intime by setting a terminal condition PN = QN . In Equation 17 andEquation 18, we introduce the symbols ξt, Qt, A and B. Theserespectively denote an augmented state vector and tracking weight

ξt =[ξ>t , 1

]>and Qt =

[Q−1

t +ξtξ>t ξt

ξ>t 1

]−1

, (19)

and augmented system matrices

A =

[A 00 1

]and B =

[B0

]. (20)

This permit us to treat the tracking problem more compactly andefficiently as a regulation problem, resulting in a formulation that isequivalent to a Linear Quadratic Regulator (LQR).

C STOCHASTIC SAMPLING

The optimal trajectory resulting from Equation 14 in the batch solu-tion can be interpreted probabilistically as a trajectory distribution

ξ ∼ N (µξ,Σξ) (21)

with

µξ = Sξξ1 + Suu and Σξ = σSu(S>uQSu +R

)−1S>u,(22)

where σ is a scaling factor proportional to the mean squared errorof the least square estimate in Equation 16, which can be computedautomatically in Matlab with the lscov command. Additional detailson the derivations are given in [8].

This permits the generation of an infinite number of trajectoriesthrough stochastic sampling of the distribution, which can be donewith the eigendecomposition Σξ = VξΛξV

>ξ , where Vξ is a matrix

of eigenvectors of the symmetric matrix Σξ, and Λξ is a matrixwith the respective eigenvalues along the diagonal. We can thenstochastically generate variations around the mean trajectory with

ξ ∼ µξ + VξΛ12ξ N (0, I) . (23)

In practice, the covariance matrix Σξ will be of high dimension,which will result in slow computation if all eigencomponents areevaluated. It is sufficient here to utilise a reduced subset of eigen-components with the largest eigenvalues (between 5 and 9 in theprovided examples). This can be done at an interactive rate by usingthe Arnoldi iteration technique [2], which is readily implementedin commonly used linear algebra packages (Matlab®, NumPy inPython, and ARPACK in C).

REFERENCES

[1] Z. AlMeraj, B. Wyvill, T. Isenberg, A. A. Gooch, and R. Guy. Auto-matically mimicking unique hand-drawn pencil lines. Computers &Graphics, 33(4):496–508, 2009.

[2] W. E. Arnoldi. The principle of min. iterations in the soln. of the matrixeigenvalue problem. Quat. of App. Maths, 9(1):17–29, 1951.

[3] D. Berio and F. F. Leymarie. Computational Models for the Analysisand Synthesis of Graffiti Tag Strokes. In P. Rosin, ed., ComputationalAesthetics (CAe), pp. 35–47. Eurographics Association, June 2015.

[4] N. A. Bernstein, M. L. Latash, and M. Turvey. Dexterity and itsdevelopment. Taylor & Francis, 1996.

[5] H. Bezine, A. M. Alimi, and N. Sherkat. Generation and analysis ofhandwriting script with the beta-elliptic model. Int’l Workshop onFrontiers in Handwriting Recognition (IWFHR), 8(2):515–20, 2004.

[6] C. M. Bishop. Pattern Recognition and Machine Learning (InformationScience and Statistics). Springer, 2006.

[7] A. E. Bryson. Dynamic optimization. Addison Wesley, 1999.[8] S. Calinon. Stochastic learning and control in multiple coordinate

systems. In Intl Workshop on Human-Friendly Robotics, 2016.[9] S. Calinon. A tutorial on task-parameterized movement learning and

retrieval. Intelligent Service Robotics, 9(1):1–29, 2016.[10] M. J. T. Da Silva. Pre-computation for controlling character behavior

in interactive physical simulations. PhD thesis, Citeseer, 2010.[11] J. B. Dingwell, C. D. Mah, and F. A. Mussa-Ivaldi. Experimentally

confirmed mathematical model for human control of a non-rigid object.Journal of Neurophysiology, 91(3):1158–1170, 2004.

[12] S. Edelman and T. Flash. A model of handwriting. Biological cyber-netics, 57(1-2):25–36, 1987.

[13] M. Egerstedt and C. Martin. Control Theoretic Splines: OptimalControl, Statistics, and Path Planning. Princeton Univ. Press, 2009.

[14] M. B. Egerstedt, C. F. Martin, et al. A note on the connection be-tween Bezier curves and linear optimal control. IEEE Transactions onAutomatic Control, 49(10):1728–31, 2004.

[15] S. E. Engelbrecht. Minimum principles in motor control. Journal ofMathematical Psychology, 45(3):497–542, 2001.

[16] T. Flash and N. Hogan. The coordination of arm movements. Journalof Neuroscience, 5(7):1688–1703, 1985.

[17] W. C. Fong. Why Chinese painting is history. The Art Bulletin,85(2):258–80, 2003.

[18] D. Freedberg and V. Gallese. Motion, emotion and empathy in estheticexperience. Trends in cognitive sciences, 11(5):197–203, 2007.

[19] J. J. Freyd. Representing the dynamics of a static form. Memory &cognition, 11(4):342–346, 1983.

[20] P. Haeberli. Dynadraw: A dynamic drawing technique, 1989.http://www.graficaobscura.com/dyna/.

[21] A. Hertzmann. Non-photorealistic rendering and the science of art.In 8th Int’l Symp. on Non-Photorealistic Animation and Rendering(NPAR), pp. 147–57. ACM, 2010.

[22] D. H. House and M. Singh. Line drawing as a dynamic process.In Computer Graphics and Applications, 2007. PG’07. 15th PacificConference on, pp. 351–360. IEEE, 2007.

[23] M. I. Jordan and D. M. Wolpert. Computational motor control. InM. Gazzaniga, ed., The Cognitive Neurosciences. MIT Press, 2nd ed.,1999.

[24] J. E. Kyprianidis, J. Collomosse, T. Wang, and T. Isenberg. State ofthe ”art”: A taxonomy of artistic stylization techniques for images andvideo. IEEE Trans. on Vis. & C.G., 19(5):866–85, 2013.

[25] J. Liu, H. Fu, and C.-L. Tai. Dynamic sketching: Simulating theprocess of observational drawing. In Proceedings of the Workshop onComputational Aesthetics, pp. 15–22. ACM, 2014.

[26] M. Longcamp, J. L. Anton, M. Roth, and J. L. Velay. Visual Presenta-tion of Single Letters Activates a Premotor Area Involved in Writing.NeuroImage, 19(4):1492–1500, 2003.

[27] J. Lu, F. Yu, A. Finkelstein, and S. DiVerdi. Helpinghand: Example-based stroke stylization. ACM Trans. on Graphics, 31(4):46, 2012.

[28] J. McCrae and K. Singh. Sketching piecewise clothoid curves. Com-puters and Graphics (Pergamon), 33(4):452–461, 2009.

[29] C. Mediavilla. Calligraphy: From Calligraphy to Abstract Painting.Scirpus, 1996.

[30] P. Morasso. Spatial control of arm movements. Experimental BrainResearch, 42(2):223–7, 1981.

[31] P. Morasso and F. Mussa Ivaldi. Trajectory formation and handwriting:a computational model. Biological cybernetics, 45(2):131–142, 1982.

[32] A. Pignocchi. How the Intentions of the Draftsman Shape Perceptionof a Drawing. Consciousness and Cognition, 19(4):887–898, 2010.

[33] R. Plamondon et al. Recent developments in the study of rapid humanmovements with the kinematic theory. Pattern Recognition Letters,35:225–35, 2014.

[34] D. Rosand. Drawing acts: Studies in graphic expression and represen-tation. Cambridge University Press, 2002.

[35] C. Sanderson and R. Curtin. Armadillo: A template-based C++ libraryfor linear algebra. Journal of Open Source Software, 1(2), 2016.

[36] H. Shinoda et al. Generation of cursive characters using minimum jerkmodel. In IEEE Proc. SICE, vol. 1, pp. 730–3, 2003.

[37] K. Shoemake. ARCBALL: A user interface for specifying three-dimensional orientation using a mouse. In Graphics Interface, pp.151–6, 1992.

[38] G. Simmons and Y. Demiris. Optimal robot arm control using theminimum variance model. Journal of Robotic Systems, 22(11):677–690, 2005.

[39] M. Stacey. Psychological challenges for the analysis of style. ArtificialIntelligence for Engineering Design, Analysis and Manufacturing,20(3):167–84, 2006.

[40] H.-L. Teulings and L. Schomaker. Invariant properties between strokefeatures in handwriting. Acta psychologica, 82(1):69–88, 1993.

[41] Y. Thiel, K. Singh, and R. Balakrishnan. Elasticurves: Exploitingstroke dynamics and inertia for the real-time neatening of sketched2D curves. In Proc. 24th ACM Symp. on User Interface Software &Technology (UIST), pp. 383–92, 2011.

[42] E. Todorov and M. I. Jordan. Optimal feedback control as a theory ofmotor coordination. Nature neuroscience, 5(11):1226–1235, 2002.

[43] S. Van Der Walt, S. C. Colbert, and G. Varoquaux. The NumPy array:A structure for efficient numerical computation. Computing in Science& Engineering, 13(2):22–30, 2011.

[44] P. Viviani and G. Mccollum. The relation between linear extent andvelocity in drawing movements. Neuroscience, 10(1):211–8, 1983.