Bigger (Gesture) Isn’t Always Better

David Novick(✉), Ivan Gris, Adriana Camacho, Alex Rayon, and Timothy Gonzalez

Department of Computer Science, The University of Texas at El Paso,500 West University Avenue, El Paso, TX 79968-0518, USA

novick@utep.edu, ivangris4@gmail.com, caro4874@gmail.com,{amrayon2,trgonzalez}@miners.utep.edu

Abstract. The literature suggests that familiarity and rapport are enhanced bylarger, more extraverted gestures. However, the sizes of the increases in amplitudehave not been reported. We sought to determine whether this relationship holdstrue for interaction between humans and embodied conversational agents. To thisend, we conducted an experiment in which we increased gesture amplitude, withquantification of the gesture sizes. We hypothesized that rapport would beincreased in the larger-gesture condition. However, our results were exactly theopposite: Rapport fell significantly in the larger-gesture condition. This meansthat larger may not always be better for building human-agent rapport. Our unex‐pected results may be because our agent’s gestures were simply too big, odd,awkward, or strange, or because of a statistical anomaly.

Keywords: Embodied conversational agent · Gesture amplitude · Rapport

1 Introduction

The development of embodied conversational agents (ECAs) [7] holds the promise ofproviding important support for their human partners in fields ranging from educationto care-giving (see, e.g., [3, 11]). The creation and maintenance of human-agent rapport[12, 13, 21] will be a correspondingly important factor for agents serving in these criticalroles. Humans typically signal increased familiarity by, among other things, increasingthe amplitude of gestures [6, 19]. However, the reported research on the effects ofincreased gesture amplitude on human-agent rapport is thin and unsettled. In the workreported here, we test the claim that increasing an agent’s gesture amplitude will leadhumans to report greater rapport with the agent, and we do so with quantification of theagent’s gesture amplitudes. Although our hypotheses in initiating this research were thatthe larger-amplitude gestures associated with extraversion would produce greater feel‐ings of rapport, our results suggest that this is not true, at least for the gestures andamplitudes that our agent used.

In this paper, then, we briefly review the literature relating gesture amplitude withrapport, describe a 55-subject empirical study of this relationship, report the study’sresults, and discuss the implications of these results.

© Springer International Publishing AG 2017M. Kurosu (Ed.): HCI 2017, Part I, LNCS 10271, pp. 609–619, 2017.DOI: 10.1007/978-3-319-58071-5_46

2 Background

An ECA is a form of human-computer interaction that involves an intelligent virtual char‐acter that can communicate by using speech, facial expressions, and gestures [7]. ECAs canvary graphically in appearance depending on the desired virtual environment the ECA livesin and the role assigned to them. A help-desk ECA may only have its upper body visible,while a museum’s tour-guide ECA may need a full body to convey more lifelike gesturesand behavior [22]. ECAs that appear more human-like are easier for humans to interact withand develop rapport [9]. Their combination of gesture, speech, and facial expressions factornot only into believability and rapport but also into the perceived personality of the agent[20]. ECAs are designed to be used in conversational settings. They should be able to handlethe discourse within a conversation and respond in humanlike ways to input [4]. ECAs thatdo not behave in the same humanlike manner that is expected from them may ultimatelylose respect and rapport from the user.

To meet the high expectations of users, several features must be considered whendesigning an ECA, based on its application. Extraversion, agreeableness, and other BigFive personality traits are important to an ECA’s design [8]. Extraversion is being talk‐ative, outgoing, and enjoying social interactions [18]. Users show higher levels ofrapport when interacting with extraverted agents even if the users themselves are notextraverted [5, 6]. This may be because extraverts can be perceived as seeking thecompany of others, and exhibit positive emotions in their behavior [16, 17]. Introvertsare characterized by the opposite: they like keeping to themselves, making decisions byreflecting on internal conversations, and avoiding social interactions. These personalitytraits can be expressed not just through an agent’s speech, but also in conjunction withnon-verbal behavior [19, 20]).

Because humans use larger gestures to signal increased familiarity, it seems plausiblethat an agent’s use of larger gestures would lead their human conversational partners toperceive greater rapport in their interaction. Neff et al. [19] extensively reviewed therelationship of gesture and extraversion. They summarized the relevant literature, inpart, as finding that people express extraversion through gestures that are broad and widerather than narrow. In their study of human perception of agents’ verbal and nonverbalbehaviors, they parameterized the relative amplitudes of introverted and extravertedgestures, for the x, y, and z axes of motion respectively, as x*.5, y*.6, z*.8 (introverted)and x*1.4, y*1.2, z*1.1 (extraverted). Their results indicated that, as expected, subjectsperceived the larger-amplitude gestures as extraverted.

A subsequent study conducted by Clausen-Bruun, Ek, and Haake [9] examined theeffect of gesture amplitude on subjects’ uptake of a short narrative but did not providedetails on the relative sizes of the lower- and higher-amplitude gestures. The authorscreated the high-amplitude gestures by manually extending and fine-tuning the low-amplitude gestures on a case-by-case basis. They found that increased amplitude led tosignificantly improved comprehension. They also asked subjects to provide a scaledanswer to the prompt “I like the character” but did not report results relating amplitudeto this emotional preference.

Hu et al. [15] found that users notice if an agent is extraverted, based on the amplitudeof its gesture. This study used a storytelling scenario between two agents to see if users

610 D. Novick et al.

perceived the personality of each agent. One agent would gesture with large amplitudewhile the other used gestures that were the same but smaller in size. Users noticed thedifference between the two agents and correctly perceived that the larger amplitude agentwas the extraverted agent.

Novick and Gris [21] looked explicitly at the relationship between gesture amplitude andhuman’s perception of rapport. However, this study had only 20 subjects, and its resultswere only weakly suggestive of a positive relationship between gesture amplitude andrapport.

The research to date suggests that agents who perform gestures with high amplitude andfrequency do appear extraverted to users. Many of these studies, though, the lack a scale todefine the amplitude of the gestures and omit clear full-body measurements of the agent inrelation to its gestures. The gestures themselves are given a range between extraverted andintroverted on a 3D plane, but there is no measurement of the initial point and apex of agesture. There could be a limit: if a gesture is too large then it may seem unnatural. The samelogic applies to gestures that are too small in amplitude. Also the type of gestures used byextraverts may greatly differ than those used by introverts. If that is the case, then agents whoare designed to be extraverted should use these specific gestures more frequently than intro‐verted ones. Introverts tend to perform gestures that are closer to their bodies [2], but inmost studies done with ECAs an introverted gesture is the same gesture used by the extro‐vert, just smaller. We are aware of no study that has looked for a diversity of gesturesbetween these personality traits. So it is possible that users may classify an agent as intro‐verted not only on the size of its gesture but on the different types of gesture it uses.

Accordingly, in the present study we wanted to determine whether there is, in fact, apositive relationship between gesture amplitude and rapport. Moreover, in our study wewere able to quantify the gesture sizes absolutely. We hypothesized that rapport would beincreased in the larger-gesture condition.

3 Methodology

We studied the relationship of quantified gesture to rapport using the ECA application ofGris et al. [14], “Escape from the Castle of the Vampire King.” This is an adventure game,inspired by text-based games such as Zork [1] and Colossal Cave [10], where the user triesto escape from the castle of an evil vampire king. The game had a graphical interface witha full-sized ECA that served as the game’s narrator, and the player controlled the gamethrough speech commands. The game comprised 26 different rooms, each with its ownsecret passages, exits, items and clues. The game included 3D scenery, recorded speech,agent movement based on motion-capture, and the quick-reference commands and incre‐mental map display. Players’ interactions occurred in 30-minute sessions on two differentdays, for a total of approximately 60 min per participant. Figure 1 shows a user interactingin a typical scene in the game, with the narrator agent, one of the rooms in the castle, and amap of the castle’s rooms explored to that point in the adventure.

Bigger (Gesture) Isn’t Always Better 611

Fig. 1. User interacting in typical scene in “Escape from the Castle of the Vampire King”adventure. The tripod for a video camera, partially hidden in an artificial tree, for recording theuser’s interaction is visible at the lower left of the image.

The game dialogs spoken by the agent were scripted in such a way so that, even withlimited commands, users could answer with natural language constrained by the specificcontext. Table 1 presents an excerpt of a player’s interaction, showing the agent’sscripted utterance and the player’s verbal production.

Table 1. An interaction transcript from the first session of the “Vampire King” game.

Agent You enter the courtyard. A fountain rests in the room’s center. A chapel lies in thedistant west and the lobby to the south

Player Examine the fountainAgent The water is green with algae and unsafe to drink. You notice a sign that reads: No

magic allowedPlayer Use the magic potionAgent You dump the magic potion into the overgrown fountain. The vines within the

fountain begin to smoke and hiss, finally burning away until you can see a dirty redgem lying over the drain in the fountain’s base

Player Pick up the red gemAgent You pick up a dirty red gem that looks like an eyePlayer Go to the chapel

The experiment used a between-subjects design. In both the experimental and controlconditions, subjects played the first half of the adventure in a 30-minute session with theagent using gestures with introverted amplitude. The subjects then returned a day or twolater to play the second half of the adventure, again in a 30-minute session. In the secondsession, subjects in the control condition continued with the agent using gestures with

612 D. Novick et al.

introverted amplitude, and subjects in the experimental condition continued with the agentusing gestures with extraverted amplitude. The sessions were completed by 30 subjects inthe control condition and by 25 subjects in the experimental condition.

The agent’s gestures were generated via motion capture. Five separate sets of sixgestures each were recorded for the high-amplitude and low-amplitude conditions, in aneffort to avoid the problems of simple mechanical amplification described by Clausen-Bruun, Ek, and Haake [9]. The gestures were separated into five categories. Three of thecategories (A, B, C) were gestures captured from human-human conversation that increasedin amplitude. The other two categories (I, E) acted as a control group. The control-groupgestures were animated from a previous experiment, with the E animations acting as modi‐fied versions of the I gestures but with a larger amplitude. In total there were 30 anima‐tions, six from each category. An example of a gesture from each category can be seen inFig. 2.

Fig. 2. Gesture from each category from left to right: A, B, C, I, and E. A has the smallestamplitude and E and C have the largest.

Figure 3 presents sequences of agent poses that illustrate of one of the extraverted andone of the introverted gestures. The animation gestures were a variation of hand gestureswhere the agent appears to be explaining or simply speaking. The first sequence of imagesrepresents an extraverted animation gesture where the agent lifts her hands and moves theright hand in circles. The second series of images represent the equivalent introvertedanimation gestures of the first image sequence. As can be noted from the images, the intro‐verted animation does not have the hands lifted as high as the extroverted one.

We measured the absolute displacements for the six gestures from both versions of the“Vampire King” agent. To do so, we projected the agent in its actual experimental settingand physically measured the x, y, and z displacements of the agent’s right and left hands. (Wemeasured the displacements in the actual physical space of the experiment so that the resultswould be more accurate than if we measured from, say, a desktop display). For each hand,we calculated the hand’s displacement vector (in inches) and summed the right and leftdisplacement vectors to produce an overall measure of gesture size. Figure 4 shows one ofthe authors preparing to measure the x and y displacements for one of the agent’s gestures.

Bigger (Gesture) Isn’t Always Better 613

Table 2 reports the absolute displacements of representative high- and low-amplitudegestures, Table 3 reports the maximum, minimum, mean, and standard deviation for thegestures, and Fig. 5 compares the mean amplitudes of the stimuli gestures. Overall thegesture stimuli ranged in amplitude from the A gestures as the smallest, through the B, I, andC gestures, to the E gestures, which were the largest.

Fig. 4. One of the authors prepares to measure the x and y displacements of one of the agent’sgestures.

Fig. 3. Image sequences of introverted gesture (left) and extraverted gesture (right). In additionto the hands being higher in the extraverted version than the introverted, the length of time betweeneach image is longer for the introverted animation movements than for the extraverted movements.

614 D. Novick et al.

Table 2. Absolute displacements of representative high- and low-amplitude gestures

Gesture R vector L vector Sum vectorsHigh amplitude (“Extraverted”)E1 37.01 4.12 41.14E2 31.19 31.10 62.29E3 39.48 38.27 77.75E4 21.37 31.95 53.32E5 30.48 28.72 59.21E6 36.12 6.08 42.21Low amplitude (“Introverted”)I1 25.42 4.03 29.45I2 16.58 18.97 35.56I3 16.36 19.29 35.65I4 10.11 11.92 22.03I5 16.07 13.67 29.74I6 32.41 5.00 37.41

Table 3. Maximum, minimum, mean, and standard deviation for the gestures

Amplitude Mean StDev Maximum MinimumHigh 55.98 12.52 77.75 41.14Low 31.64 5.25 37.41 22.03

Fig. 5. Mean amplitudes (in inches) of the gestures used in the perception study. Each gestureamplitude was calculated as the maximum of the right-arm and left-arm vectors.

At the conclusion of each session, the subjects completed a twelve-item Likert-scalequestionnaire that assessed the three components of rapport proposed in the Paralin‐guistic Rapport Model [21]: sense of emotional connection, sense of mutual under‐standing, and sense of physical connection. The questions in each component werebalanced with respect to positive and negative responses.

Bigger (Gesture) Isn’t Always Better 615

We expected, comparing results in the subjects’ second interaction sessions, thatsubjects in the experimental condition would interpret the agent’s larger gestures asindicating increased familiarity on the part of the agent. We expected that subjects inthe control condition, where the agent used gestures of similar amplitude in bothsessions, would not sense an increase in familiarity on the part of the agent. Accordingly,we hypothesized that:

1. Subjects in the experimental condition would report higher levels of overall rapportthan subjects in the control condition.

2. Subjects in the experimental condition would report higher levels of rapport for thecomponent of emotional connection than subjects in the control condition.

3. Subjects in the experimental condition would report higher levels of rapport for thecomponent of mutual understanding than subjects in the control condition.

4. Subjects in the experimental condition would report higher levels of rapport for thecomponent of physical connection than subjects in the control condition.

4 Results

Despite our expectations of a positive relationship between gesture amplitude andrapport, the results of the experiment did not confirm the hypotheses. As reported inTable 4, the subjects’ mean ratings for rapport ran consistently in the opposite directionthan expected.

Table 4. Mean rapport scores (1–6) for subjects in high- and low-amplitude conditions

Rapport component AmplitudeHigh Low

Understanding* 3.42 3.73Emotional 3.54 3.78Physical 3.07 3.05Overall 3.34 3.52

While t-tests for the differences in rapport for the emotional and physical rapportcomponents were not significant (p = 0.11 and p = 0.95, respectively, two-tailed test),the t-test for the difference in rapport for the understanding component was significant(p = 0.042, two-tailed test), and the t-test for the difference in overall rapport approachedsignificance (p = 0.052, two-tailed test). Most striking, the difference in across the high-and low-gesture conditions across understanding and emotional rapport components washighly significant (p < 0.01, two-tailed test). In sum, not only was our main claimdisconfirmed, but the evidence indicates, at least for conditions of this study, that therelationship runs in the opposite direction: the larger gestures here led to lower percep‐tions of rapport.

616 D. Novick et al.

5 Conclusion

We hypothesized that rapport would be increased in the larger-gesture condition. However,our results were exactly the opposite: Rapport fell significantly in the larger-gesture condi‐tion. This means that larger gestures are not always better. There are several possible reasonsfor the study’s results being the opposite of what we expected. These reasons include:

• The point of the study was to compare the rapport effects of high- and low-amplitudegestures. But it is possible that the high-amplitude gestures we created were simply toobig. This factor might be clarified by running perception studies of the gestures from thisstudy, plus additional sets of gestures with other amplitudes. In these studies, subjectswould rate the gestures for naturalness, thus providing an empirical basis for character‐izing agents’ gestures as appropriately low- and high-amplitude.

• The agent’s gestures were generated through motion-capture. Some of the gestures,especially the agent’s full-body movement, seemed exaggerated. It may, in fact, be thecase that the particular gestures we used for the agent, especially in the high-amplitudecondition, were simply odd, awkward, or strange. While it is true, cf. [19], that extra‐verted gestures differ from introverted gestures in more ways than just amplitude, for thepurposes of basic research into the impact of gestures on rapport it might be better to usethe same gestures in both high- and low-amplitude conditions, with only amplitudeadjustments.

• The study by Novick and Gris [21] suggested that the difference in gesture amplitudewould produce a difference in the subjects’ feelings of physical rapport, while the currentstudy produced essentially no difference at all in the subjects’ feelings of physicalrapport. That study indicated that the difference in gesture amplitude would not producea difference the subjects’ feelings of understanding and emotional rapport, while thecurrent study produced significant differences—in the unexpected direction. Thissuggests the possibility that the current results can be attributed to statistical anomaly.

The study we report here was subject to several limitations, and the study’s resultsshould be interpreted in light of these limitations, which include:

• While the “Vampire King” application produced high levels of user engagement [21], theuser-agent dialog and the sets of gestures were limited and repetitive. The possibly highersalience for the high-amplitude gestures may have made these limitations more appa‐rent or salient for the high-amplitude condition.

• The agent’s repertoire of six gestures in each condition meant that the results werevulnerable to the presence of even one, or possibly more, unnatural or otherwise unchar‐acteristic gestures. If the repertoire had been much larger, then the effect of the inclusionof an odd or awkward gesture would have been diminished.

• Cronbach’s alpha for the entire survey, combining all three rapport components, was0.76, which indicates reasonable but not strong consistency among the survey elements.The alpha values for each of the three components was under 0.70. While this may bedue simply to having only four questions for each of the components, it may also reflecton the meaningfulness of the survey.

Bigger (Gesture) Isn’t Always Better 617

Acknowledgments. The authors thank Jonathan Daggerhart for permission to adapt his originaltext- based adventure game into “Escape from the Castle of the Vampire King,” and Diego A.Rivera, Mario Gutierrez Baltazar Santaella, Juan Vicario, Joel Quintana and Anuar Jauregui fortheir help in developing the game.

References

1. Anderson, T., Galley, S.: The history of Zork. The New Zork Times 4(1–3) (1985)2. Argyle, M.: Bodily Communication. Taylor & Francis, Milton Park (1988)3. Bickmore, T.W., et al.: Maintaining reality: relational agents for antipsychotic medication

adherence. Interact. Comput. 22(4), 276–288 (2010)4. Bickmore, T., Cassell, J.: Relational agents: a model and implementation of building user

trust. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems,pp. 396–403. ACM (2001)

5. Brixey, J.E.: Virtual rapport with extraverted agents. Master’s thesis, The University of Texasat El Paso (2015)

6. Cafaro, A., Vilhjálmsson, H.H., Bickmore, T., Heylen, D., Jóhannsdóttir, K.R., Valgarðsson,G.S.: First Impressions: Users’ Judgments of Virtual Agents’ Personality and InterpersonalAttitude in First Encounters. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds.) IVA 2012.LNCS (LNAI), vol. 7502, pp. 67–80. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33197-8_7

7. Cassell, J. (ed.): Embodied conversational agents. The MIT Press, Cambridge (2000)8. Cerekovic, A., Aran, O., Gatica-Perez, D.: How do you like your virtual agent? Human-agent

interaction experience through nonverbal features and personality traits. In: Park, H.S., Salah,A.A., Lee, Y.J., Morency, L.-P., Sheikh, Y., Cucchiara, R. (eds.) HBU 2014. LNCS, vol. 8749,pp. 1–15. Springer, Cham (2014). doi:10.1007/978-3-319-11839-0_1

9. Clausen-Bruun, M., Ek, T., Haake, M.: Size certainly matters–at least if You are a gesticulatingdigital character: the impact of gesture amplitude on addressees’ in-formation uptake. In: A,R., Krenn, B., Pelachaud, C., Shimodaira, H. (eds.) IVA 2013. LNCS (LNAI), vol. 8108, p.446. Springer, Heidelberg (2013)

10. Crowther, W., Woods, D., Black, K.: Colossal cave adventure. Computer Game (1976)11. DeVault, D., et al.: Verbal indicators of psychological distress in interactive dialogue with a

virtual human. In: Proceedings of SIGDIAL, August 201312. Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., Werf, R.J., Morency,

L.-P.: Virtual rapport. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA2006. LNCS (LNAI), vol. 4133, pp. 14–27. Springer, Heidelberg (2006). doi:10.1007/11821830_2

13. Gratch, J., Wang, N., Gerten, J., Fast, E., Duffy, R.: Creating rapport with virtual agents. In:Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007.LNCS (LNAI), vol. 4722, pp. 125–138. Springer, Heidelberg (2007). doi:10.1007/978-3-540-74997-4_12

14. Gris, I., Novick, D., Camacho, A., Rivera, D.A., Gutierrez, M., Rayon, A.: Recorded speech,virtual environments, and the effectiveness of embodied conversational agents. In: IntelligentVirtual Agents 2014, Boston, MA, August 2014, pp. 182–185 (2014)

15. Hu, C., Walker, M.A., Neff, M., Tree, J.E.F.: Storytelling agents with personality andadaptivity. In: International Conference on Intelligent Virtual Agents, pp. 181–193 (2015)

16. Ivanov, A.V., Riccardi, G., Sporka, A.J., Franc, J.: Recognition of personality traits fromhuman spoken conversations. In: INTERSPEECH, pp. 1549–1552 (2011)

618 D. Novick et al.

17. McCrae, R.R., Costa Jr., P.T.: Personality trait structure as a human universal. Am. Psychol.52(5), 509 (1997)

18. Mehrabian, A., Ferris, S.R.: Inference of attitude from non-verbal communication in twochannels. J. Couns. Psychol. 31, 248–252 (1967)

19. Neff, M., Wang, Y., Abbott, R., Walker, M.: Evaluating the effect of gesture and language onpersonality perception in conversational agents. In: Allbeck, J., Badler, N., Bickmore, T.,Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS (LNAI), vol. 6356, pp. 222–235.Springer, Heidelberg (2010). doi:10.1007/978-3-642-15892-6_24

20. Neff, M., Kipp, M., Albrecht, I., Seidel, H.P.: Gesture modeling and animation based on aprobabilistic re-creation of speaker style. ACM Trans. Graph. (TOG) 27(1), 5 (2008)

21. Novick, D., Gris, I.: Building rapport between human and ECA: a pilot study. In: Kurosu, M.(ed.) HCI 2014. LNCS, vol. 8511, pp. 472–480. Springer, Cham (2014). doi:10.1007/978-3-319-07230-2_45

22. Swartout, W., et al.: Ada and grace: toward realistic and engaging virtual museum guides. In:Allbeck, J., Badler, N., Bickmore, T., Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS(LNAI), vol. 6356, pp. 286–300. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15892-6_30

Bigger (Gesture) Isn’t Always Better 619