Realizing Real-time Feedbacks on Learners’ Practicefor a Virtual Ballroom Dance Instructor

*Hung-Hsuan Huang, Masaki Uejo, Yuki Seki, Joo-Ho Lee, and Kyoji Kawagoe

College of Information Science & Engineering, Ritsumeikan University, Japan*contact: hhhuang@acm.org

Abstract. The use of virtual conversational agents is awaited in the tutoring ofphysical skills such as sports or dances. This paper describes a project aiming torealize a virtual instructor with real-time feedbacks to the learners for ballroomdance. At first, a human-human experiment between a professional instructor andsix learners is conducted for designing the agent’s instruction model. In orderto give the learner feedbacks in finer granularity, the methods for segmentinglearner’s motion into small meaningful segments, evaluating the learner’s per-formance, and finding the best possible improvement are developed. Finally, theprototype was evaluated in the aspect whether the instructor give on-the-fly feed-backs or only after the whole practice session.

1 Introduction

In learning sophisticated physical skills such as sport, gymnastics, or dance, it requiresthe learner to practice repeatedly for a long period. Virtual agents are thus suitablecandidates for such tasks. The agent (virtual instructor) can be available at any timeand suffers much less constraints in the location where to set up. The learner can thuspractice the tasks for unlimited times at their favorite place at their favorite time. Ifthe knowledge and the interaction scenario of the agent are well designed, the lessonsat acceptable level can be guaranteed constantly. Unlike human instructors, they neverfeel tired caused by long-time service and never loose patience on awkward learners.They also do not need to mind requesting the instructor to repeat the exemplary motionand never feel embarrassed to practice when they are still unskillful.

On the other hand, ballroom dance is becoming a world-wide popular sport. Themotion of ballroom dance involves the whole body of the dancers, head, torso, arm, andlegs have to move simultaneously and synchronized with each other and music. Ball-room dance motion is so sophisticated and require man-to-man instruction and massiverepeated practice for long time. These characteristics of ballroom dance make it anappropriate candidate for research on virtual instructors.

This paper presents an ongoing project aiming to develop such a virtual ballroomdance instructor. For building a believable virtual instructor, the first step is to knowhow a human instructor performs this task. Therefore, this project starts with a human-human tutoring experiment, and the behaviors of the agent are designed based on theanalysis of the collected corpus (Section 3). Second, in order to enable highly interac-tive instruction dialogs, it requires the measurement of the similarity of user’s move-ment and the control of agent’s multi-modal communicative actions in fine granularity

both temporally and spatially. It is then necessary to automatically extract the learner’smotion at the smallest unit for knowledge exchanging. In the case of ballroom dance,a “count” or dance motion segmented by stronger beats of music, can be considered assuch a smallest unit for knowledge exchanging. In section 4, we present a method of au-tomatic segmentation of learner’s dance motion based on the acceleration of learner’sjoints and the comparison of shapes of the trajectory of learner’s motion. Section 5presents a prototype system incorporating both proposed mechanisms. In section 6, theevaluation results of the prototype are presented. Section 7 concludes this paper andprovides a list of future works.

2 Related Works

Using virtual characters for instruction task is not a brand new idea. Previous systemslike [1] proposed by Chua et al. is a virtual reality training system of Tai Chi, the learnerwears a head mounted display (HMD) where one or more virtual instructors are pro-jected to. The authors tried five layouts of the positions of instructor/learner, e.g. oneinstructor standing in front of the learner, four instructors surrounding the learner, etc.In this system, there was no interactive instruction and feedbacks, and the virtual in-structor did not behave like an instructor but only worked like a video clip allowing thelearner to mimic. The learner’s motion is captured and compared to template motionbased on Euclidean distance. Nakamura et al. proposed [2], a dance training systemusing demonstration video instead of an agent projected on a screen . The authors com-pared learners’ performance between a fixed screen and a moving screen synchronizedto the dance motion. Chan et. al. [3] evaluated the effects of three different ways of feed-back to the learner’s performance in a dance training system: an avatar of highlightedlimbs where the learner did badly, a slow motion replay, and numeric scores. However,these present systems merely use CG characters for demonstrating exemplar motionsbut have no thought for actually utilizing the character as a “virtual instructor” teachingthe learner like how a human instructor does. All of these systems do not have interac-tive instruction and the conversation between the virtual instructor and the learner. Thevirtual character, Steve [4] who teaches the procedure of operating a complex instru-ment is a pioneer work of virtual instructor for teaching a physical skill. However, theballroom dance teaching task involves a much more complex and fast body motion thaninstrument operation.

3 Construction of the Virtual Instructor’s Tutoring Model

In order to build a believable virtual instructor, our first step is to investigate how ahuman professional instructor tutors the learner. Also, since the 2D virtual instructor islimited in the screen and can not touch the learner which is considered as quite normalbehaviors in teaching physical skills. A human-human tutoring experiment is there-fore conducted to collect the dance tutoring activities corpus in a simulated situation.Six male students of the dancesport club of our university are recruited as the learnersubjects. Four of them have one-year experience in learning ballroom dance and twoof them have two-year experience. The instructor and the learners are in two different

Greeting

Interruption

Progress

Dance evaluation

Praise

Finalization

Instructions

Demonstration

Instruction of specific part

Between emphasized counts

Overall

Demonstration

Instruction of specific part

Comments about the whole

dance sequence

Transition of dialogue focus

Dance request

Demonstration

Advise

Between countsLoop of improvements of dance skill

Fig. 1.General interaction flow of the tutoring model

rooms and can not see each other directly. These two rooms are then connected by thetelecommunication software, Skype so that the subjects can see and hear each otherremotely. The learner’s dance motion is recorded by NaturalPoint OptiTrack opticalmotion capture (captures at 100 fps) system for further analysis. They are instructedto dance a sequence of basic rumba steps which contains six counts. The interactionbetween the instructor and the learner is controlled to end around 30 minutes.

The corpus collected in the experiment is then labeled by two coders who are knowl-edgeable with ballroom dance but are not directly involved in this project. They areinstructed to label the verbal and non-verbal behaviors both of the instructor and thelearner subjects. From the analysis results, most of the learner’s actions were the re-sponses to the instructor’s instruction, the requests for the repetition or the confirmationof what the instructor just said or demonstrated. This may be caused by the relativelystrict social relationship between the instructor and the learners of the ballroom dancecommunity in Japan.

According to the analysis results, we designed a state transition model of the in-structor’s tutoring behaviors. Fig. 1 shows the general interaction flow of the state tran-sition model. The interaction starts with the greeting from the virtual instructor, it at firstmakes an introduction of what it is going to teach, demonstrate how to perform it, andthen request the learner to mimic its motion. While the learner is dancing, the instruc-tor evaluates the learner’s performance and make real-time feedbacks. If the dancer isdancing well, the instructor praises the learner. If the learner’s performance is too bad,the instructor may interrupt the learner’s dance and repeat the explanation or demon-stration. The process is repeated as a loop until the learner’s performance is improved tosome satisfying degree, then the virtual instructor progresses next steps. The interactionends when all specified steps reach a satisfying level.

4 Measurement of the Learner’s Dance Motion

An essential characteristic of ballroom dance is the dance motion must be synchronizedwith the accompanied music. The timings when the dancer stretches out or draws in his/ her limbs to the extent must match the rhythm of the music where the sound is strong,

t

t

Speed of

instructor’s rootS

peed of learner’s root

0.5 sec. window to search

Breakpoints (count candidates)

Comparison of joints’ moving trajectory

Count found

Counts in the instructor’s motion

Fig. 2.Length and timing differences in comparing exemplar motion and the learner’s one

i.e. the positions called beats. Note that the duration between beats is always the samein one piece of music. A motion segment that is bounded by two beats is called acountin ballroom dance’s terminology.

In order to realize smooth interaction between the virtual instructor and the learner,a count is supposed to be a suitable candidate of the smallest meaningful unit of motionfor the instructor-learner dialog. For the virtual instructor, what to instruct to the learnercan be designed and defined prior to the interaction with the learner, and thus exemplarmotion which completely matches the beats of music can be prepared by recordingthe motion of a ballroom dance expert in advance. The exemplar motion is then usedto drive virtual character animation during the agent-user interaction. For the learner’smotion, automatic on-line count extraction is therefore required in a real-time system.Fine-grained instruction then relies on the comparison between the learner’s motion andthe exemplar motion, count by count. Whether a count is started at the same time withthe corresponding beat of the music is the most important factor to evaluate whether adance motion is “correct” from observation. If it is not performed at right place then thefollowing dance sequence crashes.

Since the learner is not supposed to be able to perform dance motion correctly, thesituations shown in Fig. 2 may occur. The counts may be performed by the learnerat wrong timing with wrong length. Also, the virtual instructor needs to identify thepart when the learner is mimicking the correct motion from the unstop motion capturestream.

The basic idea of proposed automatic segmentation method is by searching mostsimilar segments with the counts of a specific exemplar motion sequence. The similaritybetween a candidate count and a exemplar count is computed based on AMSS (AngularMatrix for Shape Similarity [5]).The most similar segment is searched by the followingsteps: (1) Compensate missing marker data if the markers are hidden by the learner’sbody so that the cameras could not see it. (2) Smooth jagged raw data. (3) Compute thebone lengths of the skeleton formed by the rigid bodies representing the waist and thejoints of the learner’s arms and legs. (4) Align exemplar motion to learner’s motion byshifting joint coordinates according to bone lengths.(5) The break points (the candidatesof counts) where the speed of the learner’s root is close to zero are detected by MDPP

(Minimal Distance/Percentage Principle [6]) method. (6) For the first count (assumeits span to next count isl frames) in the correct motion sequence, its AMSS similarityin the range between 0.5l and 1.2l with the learner’s motion is computed by a 0.5-second sliding window. AMSS similarity is computed on the two moving trajectories ofthe same joint in resized learner’s motion. The candidate with highest average (amongall joints) similarity is extracted as the corresponding count. (7) The second count isidentified from the beginning of the remaining part of the learner’s motion to reducecomputation. Then the third count, fourth count, ..., and so on.

The algorithm is implemented and tested on a PC with a 4-core 2.8 GHz Intel Corei7 CPU. The test dance sequence which contains six counts (count span is about two sec-onds) could be computed after an average delay, 872 msec including the 0.2l overheadof the above mentioned algorithm. The speed is not very fast but should be sufficientfor an on-line system.

In order to realize the target virtual instructor system, the agent has to figure outthe badly performed portion of the learner’s motion and to give the advices of how tofix the problematic motion. In current prototype system, the agent evaluates the perfor-mance of the learner count by count, and may talk (praise, advice, or emergent stop ofthe practice) to the learner according to the the correlation of the learner’s motion andpre-recorded correct motion. These behaviors rely on the identification of the worst per-formed portion of the learner’s motion in each count. This algorithm has been integratedinto the prototype agent system. (1) Substitute the transform parameters of intermediatejoints (shoulders, elbows, hip, knees, neck, etc) with the ones from correct motion oneby one (2) Recompute the “improved new trajectory” of effected body part(s) of thelearner if (s)he can move exactly the same as the correct motion at that joint (3) Findthe body part (according to the joint) which can have highest correlation gain after itssubstitution (4) Find the difference vector in the frame and position where the new andthe original trajectory have largest difference (5) Report the body part from (3) and thedifference vector from (4) to the learner.

5 Virtual Ballroom Dance Instructor Prototype

Fig. 3 shows a learner practicing basic ballroom dance steps in front of the virtual in-structor. The virtual instructor is projected as life-size on a 100 inch screen by an ultrashort-focus projector. A large-enough space is reserved for natural interaction with theagent and dance practice. Eight motion capture cameras are used for acquiring the mo-tion input from the learner. Six cameras are fixed on six poles at high position one byone, while the two poles at the two sides of the learner are attached with two additionalcameras at low position for ensuring the field of view for the motion of the learner’slower body. The floor is covered by a mattress (5 m * 5 m) to prevent the reflection ofinfrared ray beans launched by the motion camera and to indicate the motion capturearea to the learner. The whole system is implemented with a blackboard based architec-ture. Instead of NaturalPoint’s generic motion capture suit where markers are attachedon the limbs, the markers composing 10 rigid bodies are attached on the joints and thewaist of the learner. The major components, motion measuring component (MMC), au-tomatic speech recognizer component (ASRC), dialogue manager component (DMC),

Learner with motion capture suite

Ultra short focus projector

Virtual instructor

Optical motion capture

Fig. 3.Learner practicing ballroom dance in front of the virtual instructor

and CG animator are connected with each other and all data are shared on the black-board. Totally there are 810 different user behavior input/ agent behavior output pairsdefined for various combinations of body parts and various correcting instructions.

6 Evaluation on Feedback Timings

In order to evaluate the benefits of incorporating count identification into the virtualdance instructor system. 12 students from the dance club of our university are recruitedas the learner subjects. They are instructed to receive the lessons from two virtual in-structors for about 10 minutes each. One of them gives feedbacks to the learners attimings of each count (real-time feedbacks) and after each learner practice requestedby the instructor. The other one only gives feedbacks to the learner after the learner’spractice motion ends. The feedbacks made at counts are according to the performance(correlation to the professional dancer) of the learner. The instructor says “nice” if theperformance is good at that count (the motion segment between last count and thiscount), or “hmm...” when the performance is only at a fair level. The agent may evenstops the learner’s practice if the learner’s performance is too bad.

Table 1 shows the questionnaire results of the subject’s impression. From these re-sults, the general user impression to such a virtual instructor system is encouraging.Although the system is not very useful to improve the learner’s skill yet, but it is cer-tainly welcomed. The virtual instructor seems to be more appreciated if it behaved morelearner oriented, that is, gives more and specific advices according to each learner andperforms more friendly behaviors. Current instructor is evaluated behaving like a robot,i.e. its behaviors are quite determined and its reactions to the learners were frank.

At the same time, there were no significant differences between the two virtualinstructors. Only two questions were statistically meaningful. The instructor with real-time feedbacks was evaluated less intelligent (Q16), this may comes from the feedbackswere perceived by the subjects as quite silly and the instructor without real-time feed-backs was perceived as more deliberative. The system with real-time feedbacks was per-ceived as more smooth. This implies that the subjects do expect some feedbacks from

Table 1.Questionnaire evaluation on subject’s perception on the virtual instructor with real-timefeedbacks and the virtual instructor without real-time feedbacks. “Cat.” denotes the categoriesof the questions where the utterance timings (T), the instruction contents (C), impression on thevirtual instructor (I), the quality of the dance lesson (Q), and the overall utility of the system (U)are evaluated. All scores range from 1 (worst) to 7 (best) and are already reversed if necessary

Cat. ID Questionw/ R.T. feedbacksw/o R.T. feedbacks

t testAvg. Dev. Avg. Avg. Dev. Avg.

T

1 The utterance timing was abrupt 5.33 1.03

5.27

5.50 0.96

5.38

n.s.

2 The utterance timing was tardy 5.17 1.07 5.50 0.65 n.s.

3 The utterance timing was appropriate 5.42 0.95 5.25 1.16 n.s.

4 The utterance was annoying 5.17 1.28 5.25 1.23 n.s.

C

5 There were unnecessary instructions 6.42 0.86

5.01

6.75 0.60

5.00

n.s.

6 More advices were desired 3.25 1.36 3.67 1.25 n.s.

7 The advices were appropriate 5.50 0.65 5.42 1.04 n.s.

8 The instructor indicated my weak point 5.83 1.07 5.58 1.04 n.s.

9 The instructor’s behaviors were weird 5.42 1.19 5.25 1.23 n.s.

10 The instructor acted according to the observation on me 5.08 0.86 5.50 0.76 n.s.

11 The dance animation was comprehensive 3.83 1.52 3.33 1.65 n.s.

12 The instructor was comprehensive 4.75 1.36 4.50 1.66 n.s.

I

13 The instructor was friendly 4.25 1.48

4.63

4.08 1.55

4.70

n.s.

14 The instructor was annoying 5.67 1.18 5.58 0.86 n.s.

15 The instructor feels like a human 4.08 1.19 3.75 1.16 n.s.

16 The instructor was intelligent 4.17 1.67 5.08 0.95 <.10

17 The instructor was trustable 5.00 1.08 5.00 1.22 n.s.

Q

18 My skill was improved during the dance lesson 5.33 0.94

5.33

5.00 1.35

5.03

n.s.

19 My skill will be improved if I continue to use this system 5.33 1.03 5.50 1.04 n.s.

20 I felt something different to my previous dance learning experience4.58 1.55 4.25 1.64 n.s.

21 I enjoyed the dance lesson 5.67 1.31 5.58 1.44 n.s.

22 The flow of this dance lesson was smooth 5.75 0.60 4.83 1.07 <.01

U

23 It will be convenient if there is such a system nearby 5.92 0.76

5.33

5.58 0.76

5.22

n.s.

24 This system can substitute regular dance lessons 4.33 1.43 4.58 1.32 n.s.

25 This system is possible to substitute regular dance lesson if it isimproved

5.75 0.92 5.50 0.76 n.s.

the system while they are practicing, but current feedbacks (“nice dance,” “hmm...,” andinterruptions) may not have good enough quality.

Table 2 depicts the learner performance before and after the dance lesson. The re-sults shows that the learner’s dance did not necessarily become closer to the profes-sional dancer after the instruction. Actually in both settings, the correlation degraded.From the observation during the experiment, the learners tended to dance more slowlyand more carefully and thus caused the degradation. The instructor with real-time feed-backs caused significantly less degradation (p<.01), this possible reason is: the learnersbecame more conscious to timings at that setting.

Table 2. Evaluation on subject performance (correlation to the professional dancer) before andafter the dance lesson with the two virtual instructors where “A1” means the instructor with real-time feedbacks, and “A2” means the one without real-time feedbacks

Subject 1 2 3 4 5 6 7 8 9 10 11 12 Avg.

A1

Before 0.133 0.106 0.338 0.025 0.226 0.113 0.329 0.329 0.326 0.050 0.026 -0.059 0.168

After 0.001 0.233 0.217 0.152 0.200 0.123 0.282 0.282 0.341 0.040 -0.071 0.013 0.151

Difference-0.133 0.127 -0.122 0.127 -0.026 0.010 -0.047 -0.047 0.015 -0.009 -0.097 0.072 -0.011

A2

Before 0.267 0.035 0.327 0.238 -0.071 0.019 0.117 0.117 0.005 0.295 0.076 0.304 0.144

After -0.032 0.029 -0.124 0.268 -0.025 -0.035 0.140 0.140 -0.007 0.081 -0.049 0.136 0.044

Difference-0.299 -0.006 -0.451 0.031 0.046 -0.054 0.022 0.022 -0.012 -0.214 -0.125 -0.168 -0.101

7 Conclusion and Future Works

In the case of ballroom dance, the smallest meaningful unit of dance steps is calledcountwhich is synchronized with the beats of accompanied music. In this paper, themethods of the identification of counts, the evaluation of the learner’s performance be-tween counts, and the identification of best possible improvements for the instructor’sadvice are proposed. After that, a prototype virtual instructor was built and evaluated onthe aspect whether the instructor give feedbacks to the learner count by count or onlyafter the whole practice session. The results showed that such a system is welcomed,but the quality of current system is not very satisfying and it may not really improve thelearner’s skill in short period. The first future work is to improve the interactiveness ofthe system, i.e. give more personalized advices to the learner in finer grams. We wouldlike to realize more sophisticated behaviors/ interactions such as synchronizing verbalexplanation and instructing dance movements in variable speed, for example, in the sit-uation to dance with the learner together. Also, we would like to have a more long-termevaluation on the learning results with the system.

References

1. Chua, P.T., Crivella, R., Daly, B., Hu, N., Schaaf, R., Ventura, D., Camill, T., Hodgins, J.,Pausch, R.: Trainning for physical tasks in virtual environments: Tai chi. In: IEEE VirtualReality (VR 2003). (2003)

2. Nakamura, A., Tabata, S., Ueda, T., Kiyofuji, S., Kuno, Y.: Dance training system with ac-tive vibro-devices and a mobile image display. In: IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS 2005). (2005)

3. Chan, J.C., Leung, H., Tang, J.K., Komura, T.: A virtual reality dance training system usingmotion capture technology. IEEE Transactions on Learning TechnologiesPP(99) (2010)

4. Rickel, J., Johnson, W.L.: Steve: An animated pedagogical agent for procedural trainning invirtual environments. SIGART Bulletin8 (1998) 16–21

5. Nakamura, T., Makio, K., Uehara, K.: Discovering and translating skills from motion data.Technical Report CS24-2006-3, Department of Computer and Systems Engineering, KobeUniversity, Japan (2006)

6. Mori, T., Uehara, K.: Extraction of primitive motion and discovery of association rules fromhuman motion. In: 10th IEEE International Workshop on Robot and Human Interactive Com-munication. (2001) 200–206