Optimization-based analysis of artistic expressive motions for roboticaction art

Alexander Schubert1 and Katja Mombaur1

Abstract— The role of an artist’s motion during the creationof a painting is especially interesting when looking at actionart, also called gestural abstraction – an art style primarilycharacterized by the dynamic painting movements carried outby the artists. In this short paper, we introduce a frameworkthat allows mathematical analysis of these types of movementsbased on inverse optimal control. We further demonstrate howmathematical properties extracted from motion-capture datacan be transferred to a painting robot in order to create newaction paintings. Finally, we performed a web-based perceptionstudy to evaluate how contemplators evaluate robot-generatedpaintings that arise from different mathematical componentsgiven to the robot platform.

I. INTRODUCTION

When looking at art works, people often report a sense ofbodily empathy. An often-cited example is Michelangelo’sprisoners, where contemplators in many cases claim to feelan activation of the muscles that appear to be activated withinthe person depicted by the sculpture. In 2007, David Freed-berg and Vittorio Gallese predicted similar results [FG07]to occur when the stimulus is chosen to be from art worksthat are characterized by the particular gestural traces of theartist, as in Fontana and Pollock. In recent years, there hasbeen increasing evidence for this claim. Several studies foundthat embodied simulation of motor resonance does indeedplay a role in the process of aesthetic experience. However,a quantitative analysis of the relation between an artist’smotion, the resulting work of art and the aesthetic experienceevoked in an observer has not been performed yet.

This short paper summarizes the work that has beenperformed in [AS17]. We present a framework in which weperform optimal control based motion analysis and motiongeneration to investigate this relation with a focus on thepainting style known as “action painting”. We recorded,reconstructed and analyzed the painting motion of an artistwhile creating action paintings and transferred that motionto a robotic arm.

II. HUMAN DATA ACQUISITION AND ROBOT ARTIST

To obtain data on artistic motion, we performed motioncapture experiments with a collaborating artist. We used thexsens MTw system containing three inertia sensor units thatwere placed on the artists hand, forearm and upper armrespectively. The artist was instructed to perform severalshort action-painting motions to apply paint to a canvas on

1Both authors are with Institute of Computer Engineer-ing, Heidelberg University, 69120 Heidelberg, Germanyalexander.schubert@ziti.uni-heidelberg.dekatja.mombaur@ziti.uni-heidelberg.de

Fig. 1. Robot artist JacksonBot and one of its action paintings

the floor. Before and after each motion, the artist positionedher arm in a ”zero position” for calibration. After eachmotion recorded, the artist was asked to describe the motionthat has been performed with her own words. Based on theseverbal descriptions, we separated the set of motions intotwo categories, to wit: The dynamic/aggressive category andthe steady/calm category. For motion analysis, we createda model of our collaborating artists upper body. The modelconsists of 3 segments of the torso and 4 segments for eacharm resulting in 7 actuated joints with 17 degrees of freedomand data adapted from [dL96] and [DCV07] to the artistsheight and body weight. Our action painting robot has beenbuilt specifically for this work. It has 3 segments and 5degrees of freedom, all of which are actuated by servo motors[FF15]. Both models have been set up using RBDL [RBDL].

III. OPTIMIZATION-BASED MOTION ANALYSIS OFHUMAN ARTISTIC MOVEMENT

We performed motion reconstruction using an optimalcontrol problem (OCP) formulation with a least-squares ob-jective, thereby obtaining the best model-fit to the data fromour motion capture experiments, using the direct multipleshooting method MUSCOD [Boc81, LBS+03]. With this wesuccessfully reconstructed a set of 32 motions, 16 belongingto the dynamic/aggressive category and the 16 belonging tothe steady/calm category.

For motion analysis, we performed an inverse optimalcontrol (IOC) analysis on the reconstructed motions. IOCequals to finding the underlying objective function that led tothe observed and reconstructed motion [MC17]. To do so, wefirst formulate a set of reasonable base functions that span upthe space in which we search for our objective function. Wecombine these base functions as a weighted sum to an overallobjective function with a priori unknown weights. The goalof IOC analysis is to obtain a (unique) set of weightscharacterizing a particular motion. The choice of these ob-jective functions is critical. There are many mathematicallyintuitive ways of choosing base functions, for our application,we choose base functions that are physically meaningfulexpert guesses motivated from current research in humanmotor control as well as by statements of the collaboratingartist. They were minimization of energy, minimization ofjerk, minimization of motion time and minimization of endeffector acceleration. The inverse optimal control problemcan be formulated as a bilevel problem:

On the upper level of the problem, we solve a param-eter estimation problem for the weights. The lower levelsolves and optimal control problem, minimizing the objectivefunction with the current set of weights. We have inves-tigated different solution methods for the inverse optimalcontrol problem, that either keep the bilevel structure (withMUSCOD for the solution of the optimal control problemand evaluating different derivative-free methods BOBYQA,COBYLA and SBPLX,as implemented in NLOPT [NLOPT],for the upper level), or with an all-at-once approach, thatresolves the bilevel structure [HSB12]. The inverse optimalcontrol analysis of the reconstructed motions resulted insignificantly different identified weights for the two sets ofmotions that were recorded. This es true independent ofthe numerical method that was applied to solve the inverseoptimal control problem. Fig. 2 shows the obtained weightsfor the two sets of motions. Note, that the base function forminimization of energy was fixed to 1, so the three otherweights depend on this normalization.

IV. MOTION GENERATION FOR THE ROBOT ARTIST

Using the objective functions obtained from IOC, we cannow generate motion trajectories for the robot painting arm.Therefore, we formulate a standard optimal control problemusing the robot model to calculate the right-hand-side andobjective functions that have been identified as characterizingeither extremely ”calm” or extremely ”dynamic” motions.

Fig. 2. Visualization of obtained weights for the ”dynamic/aggressive”motions (red) and ”steady/calm” motions (blue). Different icons representthe different methods used for weight identification.

We successfully used the described approach to generate”robot action paintings”. To make sure that there paintingscan later be used as stimuli in a web-based perception study,we ensured that the paintings do not differ in variableslike color distribution, paint consistency or overall canvascoverage. An example painting is shown in Fig. 1 bottom.

Fig. 3. Heatmap and dendrograms showing the result of our onlineperception survey. Paintings in the same cluster have been rated as beingsignificantly more similar to each other than to those from another cluster.

V. PERCEPTION STUDY AND RESULTS

Using several paintings created by our robot platform(both based on identified objectives and random motion ascontrol items) together with some original paintings fromJackson Pollock, we asked volunteers to participate in anonline survey to see if contemplators can distinguish robotpaintings that belong to different identified objective func-tions – in this case ”aggressive/dynamic” and ”steady/calm”.The participants were asked to perform a standard similarityrating test as well as a free sorting study with the givenimages. Fig.4 shows, that participants were clearly able todistinguish between original Pollock paintings and robotpaintings but also were clearly able to distinguish betweenrobot paintings that arose from different objective functions.

ACKNOWLEDGMENT

This work was supported by the Heidelberg GraduateSchool for Mathematical and Computational Methods for theSciences.

REFERENCES

[1] Kathrin Hatz, Johannes P.Schloder, and Hans Georg Bock. Estimatingparameters in optimal control problems. SIAM Journal on ScientificComputing, 34(3):A1707A1728, 2012.

[2] David Freedberg and Vittorio Gallese. Motion, emotion and empathyin esthetic experience. Trends in Cognitive Sciences, 11(5):197203,2007.

[3] Alexander Schubert. An optimal control based analysis of humanaction painting motions. Dissertation, University of Heidelberg. 2017

[4] Paolo de Leva. Adjustments to Zatsiorsky-Seluyanovs segment inertiaparameters. Journal of Biomechanics, 29 (9):12231230, 1996.

[5] R. Dumas, L. Cheze, and J.-P. Verriest. Adjustments to McConvilleet al. and Young et al. body segment inertial parameters. Journal ofBiome- chanics, 40(3):543 553, 2007.

[6] Fabian Finkeldey. Design and implementation of a new action paintingrobot. Bachelor thesis, Universitat Heidelberg.2015

[7] Felis, Martin L. ”RBDL: an efficient rigid-body dynamics library usingrecursive algorithms.” Autonomous Robots 41.2 (2017): 495-511.

[8] H. G. Bock. Numerical treatment of inverse problems in chemicalreac- tion kinetics. In Klaus H. Ebert, Peter Deuflhard, and Willi Jager,editors, Modelling of Chemical Reaction Systems: Proceedings of anInternational Workshop, Heidelberg, Fed. Rep. of Germany, September15, 1980, pages 102125. Springer Berlin Heidelberg, Berlin, Heidel-berg, 1981.

[9] D.B. Leineweber, I. Bauer, A.A.S. Schafer, H.G. Bock, and J.P.Schloder. An efficient multiple shooting based reduced SQP strategyfor large-scale dynamic process optimization (Parts I and II). Com-puters and Chemical Engineering, 27:157174, 2003.

[10] Steven G. Johnson, The NLopt nonlinear-optimization package,http://ab-initio.mit.edu/nlopt

[11] Mombaur, Katja, and Debora Clever. ”Inverse optimal control asa tool to understand human movement.” Geometric and NumericalFoundations of Movements. Springer, Cham, 2017. 163-186.