Dance Generation with Style Embedding: Learning and Transferring LatentRepresentations of Dance Styles

Xinjian Zhang1 , Yi Xu1 , Su Yang1 , Longwen Gao2 , Huyang Sun2

1School of Computer Science, Fudan University, China2Bilibili, China

{zhangxj17, yxu17}@fudan.edu.cn, {gaolongwen, sunhuyang}@bilibili.com

AbstractChoreography refers to creation of dance steps andmotions for dances according to the latent knowl-edge in human mind, where the created dance mo-tions are in general style-specific and consistent.So far, such latent style-specific knowledge aboutdance styles cannot be represented explicitly in hu-man language and has not yet been learned in pre-vious works on music-to-dance generation tasks.In this paper, we propose a novel music-to-dancesynthesis framework with controllable style em-beddings. These embeddings are learned represen-tations of style-consistent kinematic abstraction ofreference dance clips, which act as controllable fac-tors to impose style constraints on dance genera-tion in a latent manner. Thus, the dance styles canbe transferred to dance motions by merely modi-fying the style embeddings. To support this study,we build a large music-to-dance dataset. The qual-itative and quantitative evaluations demonstrate theadvantage of our proposed framework, as well asthe ability of synthesizing diverse styles of dancesfrom identical music via style embeddings.

1 IntroductionDancers move elegantly following the rhythm of music. Be-hind dancing is the hard labor in terms of choreography.Consequently, automatic music-to-dance choreography hasemerged as a new topic in multimedia [Lee et al., 2019;Ye et al., 2020; Ren et al., 2020a]. Besides, music-to-dance synthesis investigations are vital to machine percep-tion of the latent knowledge regarding dance moves as well asrhythm (beat) and emotion (intensity) matching between mu-sic and dance, namely, styles. The generated dance motionscan be transferred to realistic-like videos by video generationmodels [Chan et al., 2019; Ren et al., 2020b] to enable realapplications like virtual character generation, games creationand teaching assistant.

Historically, most dance generation methods pay efforts tomodel the relationship between music and dance in featurespace or latent space. Early studies [Shiratori et al., 2006;Ofli et al., 2008; Fan et al., 2011; Asahina et al., 2016] aremainly retrieval based, which are focused on selecting the

Popping dance

Anime dance

Locking dance

Music

Figure 1: Beat and intensity of popping, anime, and locking dancescorresponding to the same music. All dancers’ moving rhythm andamplitude match the music well with remarkably different motionpatterns in terms of the kinematic representations.

best matching pair from database based on music and dancefeatures. Due to the retrieval nature, these methods cannotgenerate new dance sequences. In recent years, deep gener-ative models [Alemi et al., 2017; Tang et al., 2018; Lee etal., 2019; Ye et al., 2020; Ren et al., 2020a] are introduced toalleviate the problem. [Tang et al., 2018; Ren et al., 2020a]utilize the temporal indexes and contrastive cost function topromote the model’s ability of generation. [Lee et al., 2019;Ye et al., 2020] attempt to decompose dances into a series ofbasic dance units and learn to compose a dance by organizingmultiple basic dancing units. Yet, machine-generated chore-ography is far behind human works in terms of diversity be-cause the professional knowledge behind choreography, suchas dance styles, is not exploited in previous works.

Human choreographers with different dance experiencecan compose various styles of dances. As shown in Fig-ure 1, the dance moves accompanying the same music arecomposed of sharp tuning with diverse-style variations whenmatched to the music in terms of both beat and intensity.

arX

iv:2

104.

1480

2v1

[cs

.MM

] 3

0 A

pr 2

021

Moreover, the advanced choreographer can also learn a par-ticular dance style from a short dance clip and apply it to fu-ture dance creation. So far, the mechanism of human percep-tion is not explicit. In this context, style control and transferin dance generation has been remaining an open problem yet.

To generate dances with controllable styles, we introducethe controllable style embeddings into our dance generationframework. The whole dance generation framework containstwo modules: A style embedding producer and a music-to-dance generator. The style embedding producer learns thestyle information in a latent manner from motion semanticsof reference dance clips so as to output style embedding.The music-to-dance generator contains a music encoder anda dance generator. The former is formulated as a set of trans-former encoders [Vaswani et al., 2017] to obtain semanticand temporal representation from the input music. The latterincorporates both the music representation generated by themusic encoder and the style embedding as input to produceoutputs the generated dance. Considering that the relation-ship between music and dance is frame-aligned, the frame-work generates dance motions in a frame-by-frame way. Ben-efitting from style embeddings, our framework is capable ofcreating different styles of dances accompanying the samemusic by modifying style embedding.

To support this study, we construct a large music-to-dancedataset with style annotation for training and evaluation. Weselect three modern dancing categories: Anime dance, pop-ping dance, and locking dance to build the dataset, whichhave a higher degree of freedom in terms of choreogra-phy. We compare our framework with several baselines fromthe following perspectives: Beat matching, emotion match-ing, style consistency, and diversity. Besides, we visual-ize the choreography by using a realistic video generationmodel [Wang et al., 2018; Chan et al., 2019] for a better qual-ititative evaluation.

Our contributions are summarized as follows: (1) We firstpropose a novel frame-by-frame dance generation frameworkwith controllable style embedding to guarantee style con-sistence and allow flexible style modification on generateddances. (2) We develop a method to represent and learn dancestyles in a latent space spanned by a base of prototypes, whichare learnt from real dance videos in an end-to-end manner,and in this framework, the style embedding is computed au-tomatically as a linear combination of such prototype vectorswith varying weights to indicate different styles. Then, suchlatent knowledge in the form of style-related weighting of theprototype vectors is transferred to dance generation so as toguarantee style consistence between the real and generateddance videos. (3) We establish a large music-to-dance datasetthat contains choreographies of three dance styles. (4) Thequalitative and quantitative results validate that the proposedframework generates more diverse and realistic dances in re-gard to input music.

2 Related Work2.1 Music-to-dance GenerationThe music-to-dance generation models can be roughly di-vided into two categories: Retrieval-based methods and

learning-based methods. Previous works [Shiratori et al.,2006; Kim et al., 2009; Fan et al., 2011; Lee et al., 2013] ondance generation carefully design musical feature extractors,and then the dance motion is selected from the database orlocally modified by feature matching. These retrieval-basedmethods suffer from the lack of machine learning to capturethe generic correlation between music and dance and cannotgenerate novel dances unlimitedly.

Recently, [Alemi et al., 2017; Tang et al., 2018; Lee etal., 2019; Ye et al., 2020] attempt to apply deep genera-tive models to generate more creative dances. [Alemi et al.,2017] establishes the connection between music and dancemotions through the Factored Conditional Restricted Boltz-mann Machines (FCRBM) and Recurrent Neural Networks(RNN) models. [Yalta et al., 2019; Tang et al., 2018] designalternative LSTM Auto-Encoder models. The former methodconcatenates the previous motion with current hidden statesto reduce the feedback error. The latter applies the musicbeat feature as mask to weight the hidden states only to re-duce the feature dimensionality as well as the likelihood ofoverfitting. [Ye et al., 2020; Lee et al., 2019] decompose thedance into dance units conditioned on the music beat and letthe models learn to recompose them according to input music.

The aforementioned works ignore the knowledge in dancemotion referred to as style in this study. Here, we propose adance generation model with controllable style embedding toapply the latent knowledge learnt from specific-style dances.

2.2 Realistic Video Generation from SkeletonsWe introduce a rendering model to generate realistic dancevideos from kinematic representations of human body. Theexisting video generation works utilize conditioned genera-tive adversarial networks (CGAN) [Liu et al., 2019; Ma etal., 2017; Chan et al., 2019; Ma et al., 2018; Ren et al.,2020b] or Variational Auto-Encoder [Dong et al., 2018]. [Maet al., 2017] proposes a U-Net generator with a coarse-to-finestrategy to generate 256× 256 images. [Liu et al., 2019] uti-lizes Liquid Warping Block (LWB) in body mesh recoveryand flow composition, and a GAN module to tackle differ-ent tasks. [Chan et al., 2019] uses Pix2PixHD [Wang et al.,2018] and special Face GAN to achieve body imitation forgenerating a more realistic target video. In this work, we sim-plify [Chan et al., 2019] for realistic dance video generation.

3 Dataset and PreprocessingWe build a large music-to-dance dataset with three popularrepresentative categories: Popping dance, anime dance, andlocking dance. We do not choose traditional dance styles dueto the limited movements. All these videos in the dataset areuploaded onto web by professional dancers, with clear high-quality music, and solo-dance. The Dance videos are resizedto 640 × 480 to extract the musical features and motion fea-tures. Our novel dataset containing 70 popping dance videos,150 anime dance videos, and 65 locking dance videos, with1.48M carefully selected frames.

Musical Features Extraction. We first normalize themusic volume according to root mean square by FFM-PEG. Then, the musical features are extracted via an au-

GRU

Multi-headSelf-

attention

Conv2D List

Reference Dance 𝑫𝒔 Motion Encoder Style Generator

Repeat & Add

InputMusic Bi-LSTM

Generated Dance "𝑫𝒔

Multi-headSelf-

attention

LayerNorm

FeedForward

LayerNorm FC

Music Encoder ×𝑵 Dance Generator

Initial Pose𝒅𝒔

Recon Pose"𝒅𝒔Encoder Decoder

Repeat & Concatenate

Style Embedding Producer

Music-to-Dance Generator

𝑫𝒔𝒉𝒊𝒇𝒕𝒔

𝑴

𝐳𝐦

𝐳𝐢𝐧𝐢𝐭

𝐜𝐢𝐬

𝐳𝐦𝐨𝐭𝐢𝐨𝐧𝐬

𝑾𝒂𝒕𝒕𝒏

Initial Pose VAE

𝐿+,-./

𝐿0/01,-./, 𝐿0/0123

𝐜𝐣𝐬

𝐿-

memory

Reference Signal

Sampl𝑖𝑛𝑔

Figure 2: The architecture of controllable style embedding dance generation framework. The framework consists of two parts: A styleembedding producer and a music-to-dance generator.

dio analysis library Librosa [McFee et al., 2015], includ-ing 20-dimensional Mel Frequency Cepstral Coefficients(MFCC), 12-dimensional constant-Q transform chromagram,1-dimensional pitch, 1-dimensional root-mean-square en-ergy (RMSE), and 1-dimensional onset strength.

Motion Features Acquisition. All raw videos are fed intoOpenpose [Cao et al., 2017] for 2D keypoints detection. Toensure that the skeleton information in the dance motion se-quences is of high quality, we manually filter out the videoclips whose skeletons are not detected correctly. We finallychoose 21 joints most relevant to kinematic expressivenessto represent the dancing motion, including nose, neck, midhip, left and right ears, shoulders, elbows, wrists, hips, knees,hand, and ankles.

4 MethodologiesOur goal is to generate multi-style dances conditioned on theinput musical features, and initial poses. The style of a se-quence of generated dance should be adhered to the stylelearnt from the reference dance clips. We define the inputmusical features as {Mt ∈ RL|t = 1, . . . , T}, and L = 36 isthe dimension of the musical features of a single frame. Let Jbe the number of body joints, each of which is represented byits pixel coordinates (x, y). Hence, a dance motion sequenceis formulated as {Ds

t ∈ R2J |t = 1, . . . , T} with a style indi-cator s ∈ {anime dance, popping dance, locking dance},where J = 21 indicates all joint locations.

As illustrated in Figure 2, we propose a novel end-to-enddance generation framework, which consists of a style em-bedding producer to learn the style information from refer-ence dance clips in a latent manner, and a music-to-dance

generation model to map musical features to the dance mo-tions. In the style transfer phase, the style embedding pro-ducer acts to summarize the attributes of reference dance mo-tions into a style embedding in the form of a vector indicat-ing the category of dance of interest, which can be regardedas a latent knowledge with varying numerical values to al-ter style from slightly to remarkably. Then, the music en-coder in the music-to-dance generator compresses the musi-cal features into semantic representations. The dance gen-erator final generates the dances conditioned on the musicrepresentation, the style embedding, and the initial pose inthe forms of a hidden vector resulting from an initial poseVAE model [Lee et al., 2019]. In the inference phase, for aspecific-style dance generation, we can easily apply a styleembedding to the music-to-dance generation model and al-ter the values of the style embedding to control the style soas to enable diversity of styles. Under the control of styleembedding, the music-to-dance generation model can gen-erate dances with any expected style. In the following, wefirst present the music-to-dance generation model with styleembedding. Then, the mechanism of style transfer via styleembedding is described. Finally, we demonstrate a new lossthat functions to force the learnt representations of style em-beddings fall into discriminative clusters of different styles.

4.1 Music-to-Dance GenerationSince the relationship between music and dance is frame-aligned, we provide a music-to-dance generator module inan Auto-Encoder architecture to generate dance in a frame-by-frame way. As descripted in Figure 2, we apply a set oftransformer music encoders with a Bi-LSTM dance generatorin music-to-dance generator.

Taking advantage of the special temporal structure com-posed of fully connected layers, the music encoder can obtaina long-range temporal representation from the input musicalfeatures M ∈ RT×L. For notational simplicity, we ignorethe temporal indicator t in the following. Aside from that,supported by the multi-head attention mechanism, the musicencoder can capture different musical semantic informationat the distinct heads of attention, which can help dance gen-erator better understand the music. The final output of theencoders is the music representation zm ∈ RT×dmodel , wheredmodel is the dimension of the output of music encoder.

To facilitate the long-term sequential generation, i.e., thelast pose of a current dance sequence can be used as the ini-tial pose of the next one, where we embed the dance motionsinto a latent distribution Zinit ∼ N (0, I) via initial poseVAE (green dashed part in Figure 2). We train this modelon the dance poses ds that are sampled from the referencedance Ds. We enforce a reconstruction L1 loss Lrcon

init and aKL loss LKL

init to enable the reconstruction after encoding anddecoding:

Lrconinit = |ds − ds|,

LKLinit = KL(N (0, I)‖zinit), (1)

where KL(u‖v) = −∫u(z) log u(z)

v(z)dz. Utilizing the initialpose VAE, we can randomly sample initial poses and encodethe last pose of a current dance clip for the next dance clipgeneration to enable long-term dance sequential generationat inference stage.

After all, the dance generator receives a music represen-tation zm from music encoder, a hidden vector zinit for aninitial pose from initial pose VAE, a style embedding cs fromstyle generator which will be mentioned in section 4.2. Werepeat both style embedding cs and the hidden vector zinit ofthe initial pose at the time dimension to align with the shapeof zm. Then we add the expanded style embedding to musicrepresentation, and concat with the expanded hidden vector ofthe initial pose. The composite representation will be passedto the dance generator module for dance generation. Finally,we supervise the generated dance Ds by using reconstructionloss:

LrconG = |Ds −Ds|. (2)

4.2 Style Embedding ProducerIn fact, human’s dance style knowledge is formed in the pro-cess of continuous practice and perception of dance motions.Such memory of dance styles as a complex pattern is in gen-eral the composition of prototype patterns. Motivated by this,we represent the style embedding as a linear combination ofa couple of prototype patterns. The prototype patterns arelearnt from the whole corpus of real dance videos and onesolution of the weights of such prototype patterns in the lin-ear combination of them indicates a specific dance style fora given video example. Ideally, a remarkable variation of theweights will cause style change while a slight one will not. Inthe following, we present how to learn the prototype patternsand represent style embedding through the weighted sum ofsuch prototypes in an end-to-end manner.

As illustrated in Figure 2, the style embedding producerfirst computes the first-order difference between two sequen-tial motion frames as dynamic motion feature: Ds

shift =

|Dst+1 − Ds

t |, t = 1, . . . , T , where t indicates the time step,and s is the label of the corresponding dance style. In the mo-tion encoder, a stack of 2-D convolutional layers and a GRUlayer map the dynamic motion feature Ds

shift into a fixed-length motion representation vector zsmotion ∈ R1×dgru ,which is the last state of the GRU layer, where dgru is thehidden size of the GRU layer and the setting of dgru shouldbe equal to that of dmodel in the music-to-dance generationmodule. Thus, zsmotion encodes the reference dance motion’sspatial and temporal features locally, serving as the referencesignal fed to the style generator.

Then, we design a learnable matrix referred to as stylememory to learn the prototype patterns from the motion rep-resentations globally. As shown in Figure 2, we feed thedance clips Ds to the style embedding producer to extractthe motion representation zsmotion as the reference signal forguiding the dance style memory W ∈ Rdw×dgru writes in thetraining phase and reads W in the inference phase, where dwis a hyper-parameter. Here, each row of W can be regardedas a prototype vector to span a latent space in order to repre-sent any given signal corresponding with dance moves usingdw base vectors, where both the prototype vectors and theweights to form the linear combination of such prototype pat-terns are learnt in an end-to-end manner. By means ofW , anygiven motion signal can be represented as a linear combina-tion of the dw prototype vectors stored in W . In terms of in-ference, we leverage the multi-head self-attention module tocalculate the similarity between the reference signal zsmotionand each prototype row vector in W to represent the givensignal in the form of a linear combination of the prototypevectors according to how much it is close to each prototypevector. The attention module outputs attn ∈ R1×dw auto-matically under predefined loss function, which functions toweight the contribution of each prototype style pattern. Theweighted sum of such prototype vectors, namely style em-bedding, is passed to the music-to-dance generator moduleas conditioning at every time step. So, the style embeddingcs ∈ R1×dgru is calculated as follow:

attn =Softmax(zsmotionW

T )√dgru

,

cs = Sum(attnW ). (3)

Overall, we jointly train the style embedding producer andmusic-to-dance generation model by optimizing the follow-ing objective:

L = LrconG + λrconinit Lrcon

init + λKLinitLKL

init, (4)

where λrconinit ,λKLinit are the weights of related loss terms.

4.3 Learning Representations Converging toStyle-related Clusters via a Contractive Loss

In the learning of the prototype patterns and the weight sum ofthem to form the style embedding, the loss function plays an

important role. A proper loss function makes the leant rep-resentations of dance styles fall into compact and separableclusters while a bad one will cause overlap of the representa-tions over different style categories. In the following, we firstvisualize how a bad one works, and then, how a proper oneworks.

To enable visualization of the high-dimensional style em-bedding {Cs} of all the examples, we reduce the style embed-dings’ dimension with principal components analysis (PCA)to observe the relationship between the style embeddings re-sulting from inappropriate loss function and their labels ofstyle categories. As shown in Figure 3 (a), confusion amongdifferent style categories is obvious. To alleviate that, we de-velop a contrastive loss Lc to introduce weak label informa-tion to the style embedding producer to force the learnt rep-resentations of styles {Cs} fall into separable clusters corre-sponding with different style categories. The contractive lossLc is defined as follows:

Lc(csi , c

sj) =

1

2(1− Y )(Dis)2 +

1

2(Y ){max(0,m−Dis)}2, (5)

where the (csi , csj) is a pair of style embeddings, and the i, j

are just used to identify the different style embeddings. Ifthe pair of style embeddings (csi , c

sj) are generated from the

reference dances of an identical style, Y = 0. OtherwiseY = 1. Dis is the Euclidean distance between csi and csj . mis margin. We set it to 1 to separate different style samples.Thus, the overall objective L changes to:

L = LrconG + λrconinit Lrcon

init + λKLinitLKL

init + λstyleLc, (6)

where λstyle is the weight of related loss terms.As a result, Figure 3 (b) illustrates that the style embed-

dings obtained under such loss function fall into differentclusters with a strong correlation to the style labels, whichenable us to perform style control through style embedding.Finally, we visualize an example of the generated dance clipcorresponding to the style embedding calculated from the ex-ample reference dance in Figure 3 (c). As indicated in Fig-ure 3 (c), though the generated dance and reference dance aretotally different choreographies, the moving direction (moveleft then right) of the generated dance is same as that ofthe reference dance, and both the generated and the refer-ence dance have the same rotation behaviors. (More casesdisplayed in supplementary materials show that the styleembeddings can capture distinct attributes of dance motions,such as moving speed, moving direction, and appearance,which are commonly used to characterize the style of dance.Based on that, we can successfully control the styles of thegenerated dances by the style embeddings lerant from the in-put reference dance clips.)

5 ExperimentsTo evaluate our method, we conduct extensive experimentson our music-to-dance dataset. Our experiments include bothquantitative and qualitative evaluations. We compared our

Reference dance Generated dance

(c)(b)(a)

Contractive Loss

Anime dance Locking dance Popping dance

Figure 3: The visualization of the PCA of style embeddings, an ex-ample of reference dance clip, and the generated dance clip: (a) ThePCA of style embeddings without style label supervision. (b) ThePCA of style embed-dings with style label supervision. (c) At left isthe reference dance clip used to produce style embedding for dancegeneration in the framework with supervision of style labels. Atright is the dance clip generated with the style embedding producedby the reference dance clip at left.

method with the LSTM method proposed in [Shlizerman etal., 2018], MelodyNet [Tang et al., 2018], and the state-of-artmethod Dancing2Music [Lee et al., 2019]. More qualitativeevaluation results and the implementation are detailed in sup-plementary materials.

5.1 Quantitative EvaluationBeat and intensity. Whether the beat of dance matches mu-sic beat greatly affects the quality of the generated dance.Given all the music and generated dances, we gather the mu-sic beatsBm, the number of total dance beatsBd, the numberof dance beats that are aligned with music beats noted as Ba.We use Ba/Bm as the beat hit rate to evaluate how generateddance match the rhythm of music. The music beat informa-tion Bm can be easily extracted from music onset features byLibrosa. For dance, we regard the frames that the movementdrastically slows down as a dance beat event. In practice, wecompute the motion magnitude between neighboring dancemotions, and track the magnitude trajectories to locate whena dramatic decrease in the motion magnitude appears. Asshown in Table 1, our method outperforms Dancing2Musicby 1.3% in terms of beat hit metric, due to the frame-by-frametransformer module to align the dance motions with the mu-sical features.

We also introduce the intensity metric to evaluate the gen-erated dances in terms of how much the emotion of music hasbeen exhibited by dance. For music, the RMSE feature canrepresent the intensity of the music. For dance, we apply thesliding window to average over the magnitudes of dance mo-tions as the intensity of the dance. As shown in Figure 4, we

Method Beat Hit Style Consistency Diversity

Real Dance 63.6% 2.5 -

LSTM 21% 5.3 -MelodyNet 49% 3.6 -Dancing2Music 68.5% 2.9 2.5Ours 69.8% 2.65 2.9

Table 1: The quantitative evaluation results.

Inte

nsity

0

0.2

0.4

0.6

0.8

Frame Index1 21 41 61 81 101 121 141 161 181

Music Intensity Popping Dance Intensity Anime Dance Intensity Locking Dance Intensity

Figure 4: Intensity of the music and generated dance.

draw the intensity curves to describe the emotion of the mu-sic and the generated dance. We find that the generated danceresponses timely to the input music in terms of emotion, asthe dance moves will be more energetic when the music ap-pears to be more intense, and on the contrary, the moves willbe more slow and deliberate.

Style Consistency and Dance Diversity. We use theFrechet Inception Distance (FID) to estimate the style con-sistency for the generated dances with the same styles. TheFID score is also used to measure the diversity of the gener-ated dances conditional on the same music but with differentinitial poses. To evaluate the style consistency, we extract thestyle embeddings of the synthesized dances and those of thetraining dances. As depicted in Table 1, our style consistencyscore is lower than those of baselines which means our modelcan generate the precise style of dances. We also attempt togenerate dances on the 20 test music clips with 50 randomly-picked initial poses and calculate the FID score between thestyle embeddings of generated dances as diversity score. Theresults in Table 1 show that our model is significantly betterthan the Dancing2Music method in terms of diversity score,which means more generative. The reason is that the gener-ated dances from Dancing2Music are based on a global musicstyle representations from the music classifier while the samestyle music have the almost same style features. In contrast,our model considers the frame-level musical features that arestrictly aligned with the dance motion frames according todifferent weights from the transformer encoder.

Generate with anime dance style embedding

Generate with popping dance style embedding

Generate with locking dance style embedding

Figure 5: The visualization of the generated dances with our rendermodels.

50%

92%85%

58%

33%

50%

8%15%

42%

67%

0%

20%

40%

60%

80%

100%

Real / Real Ours  / LSTM Ours / MelodyNet Ours / Dancing2Music Ours / Real

Styl

e C

onsi

sten

cy P

refe

renc

e

Real World Ours LSTM MelodyNet Dancing2Music

Figure 6: The preference results on style consistency.

5.2 Qualitative EvaluationStyle Transfer with Controllable Style Embeddings. Toshow that our model can generate accurate styles of dancescorresponding to the style embeddings. we generate threestyle embeddings with an anime dance clip, a locking danceclip, and a popping dance clip. Then, we put these embed-dings to the dance generation model and generate dances withthe same music and initial pose. The output dances are con-verted into real videos via the rendering models. The ren-dered results are shown in Figure 5, in which we can find thatthe model can produce a corresponding style of dance fol-lowing style embedding. More results are described in thesupplementary material.

User Study on Style Consistency. We conduct a userstudy using a pairwise comparison scheme to verify the per-formance of our music-to-dance framework in style consis-tency. Given a pair of dances generated from the differentmodels which are required to generate specific-style dances,the users are asked to select which one is closer to the re-quested dance style. Figure 6 shows the user study results,where our approach outperforms the baselines in style consis-tency. However, compared with the real dance, there is still agap. The distortions of limbs in some special cases make thegenerated dances unnatural.

6 ConclusionIn this work, we propose a novel style controllable dancegeneration framework that can learn different style patternsfrom the reference dances and transfer them to the generateddances in end-to-end manner. We also introduce a new large-scale music-to-dance dataset. Combing the dance generationframework with a rendering model, we create a successfulcross-modal generation framework that can directly generatedance videos from music. It is the first work to adapt the styleof real dance videos to generated videos. In future works, in-stead of joint coordinates, we will attempt to design a dancegeneration network directly based on the dance video framesand expand the framework to generate multi-people dances.

References[Alemi et al., 2017] Omid Alemi, Jules Francoise, and

Philippe Pasquier. Groovenet: Real-time music-drivendance movement generation using artificial neural net-works. networks, 8(17):26, 2017.

[Asahina et al., 2016] Wakana Asahina, Naoya Iwamoto,Hubert PH Shum, and Shigeo Morishima. Automaticdance generation system considering sign language infor-mation. In ACM SIGGRAPH 2016 Posters, pages 1–2.2016.

[Cao et al., 2017] Zhe Cao, Tomas Simon, Shih-En Wei, andYaser Sheikh. Realtime multi-person 2d pose estimationusing part affinity fields. In Proceedings of the IEEE con-ference on computer vision and pattern recognition, pages7291–7299, 2017.

[Chan et al., 2019] Caroline Chan, Shiry Ginosar, TinghuiZhou, and Alexei A Efros. Everybody dance now. In Pro-ceedings of the IEEE International Conference on Com-puter Vision, pages 5933–5942, 2019.

[Dong et al., 2018] Haoye Dong, Xiaodan Liang, Ke Gong,Hanjiang Lai, Jia Zhu, and Jian Yin. Soft-gated warping-gan for pose-guided person image synthesis. In Advancesin neural information processing systems, pages 474–484,2018.

[Fan et al., 2011] Rukun Fan, Songhua Xu, and WeidongGeng. Example-based automatic music-driven conven-tional dance motion synthesis. IEEE transactions on vi-sualization and computer graphics, 18(3):501–515, 2011.

[Kim et al., 2009] Jae Woo Kim, Hesham Fouad, John LSibert, and James K Hahn. Perceptually motivated auto-matic dance motion generation for music. Computer Ani-mation and Virtual Worlds, 20(2-3):375–384, 2009.

[Lee et al., 2013] Minho Lee, Kyogu Lee, and JaeheungPark. Music similarity-based approach to generating dancemotion sequence. Multimedia tools and applications,62(3):895–912, 2013.

[Lee et al., 2019] Hsin-Ying Lee, Xiaodong Yang, Ming-YuLiu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang,and Jan Kautz. Dancing to music. In Advances in NeuralInformation Processing Systems, pages 3586–3596, 2019.

[Liu et al., 2019] Wen Liu, Zhixin Piao, Jie Min, WenhanLuo, Lin Ma, and Shenghua Gao. Liquid warping gan:A unified framework for human motion imitation, appear-ance transfer and novel view synthesis. In Proceedings ofthe IEEE International Conference on Computer Vision,pages 5904–5913, 2019.

[Ma et al., 2017] Liqian Ma, Xu Jia, Qianru Sun, BerntSchiele, Tinne Tuytelaars, and Luc Van Gool. Pose guidedperson image generation. In Advances in neural informa-tion processing systems, pages 406–416, 2017.

[Ma et al., 2018] Liqian Ma, Qianru Sun, Stamatios Geor-goulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Dis-entangled person image generation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recog-nition, pages 99–108, 2018.

[McFee et al., 2015] Brian McFee, Colin Raffel, DawenLiang, Daniel PW Ellis, Matt McVicar, Eric Battenberg,and Oriol Nieto. librosa: Audio and music signal analysisin python. In Proceedings of the 14th python in scienceconference, volume 8, pages 18–25, 2015.

[Ofli et al., 2008] Ferda Ofli, Yasemin Demir, Yucel Yemez,Engin Erzin, A Murat Tekalp, Koray Balcı, Idil Kızoglu,Lale Akarun, Cristian Canton-Ferrer, Joelle Tilmanne,et al. An audio-driven dancing avatar. Journal on Mul-timodal User Interfaces, 2(2):93–103, 2008.

[Ren et al., 2020a] Xuanchi Ren, Haoran Li, Zijian Huang,and Qifeng Chen. Self-supervised dance video synthe-sis conditioned on music. In Proceedings of the 28thACM International Conference on Multimedia, pages 46–54, 2020.

[Ren et al., 2020b] Yurui Ren, Xiaoming Yu, Junming Chen,Thomas H Li, and Ge Li. Deep image spatial transfor-mation for person image generation. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pat-tern Recognition, pages 7690–7699, 2020.

[Shiratori et al., 2006] Takaaki Shiratori, Atsushi Nakazawa,and Katsushi Ikeuchi. Dancing-to-music character anima-tion. In Computer Graphics Forum, volume 25, pages449–458. Wiley Online Library, 2006.

[Shlizerman et al., 2018] Eli Shlizerman, Lucio Dery, Hay-den Schoen, and Ira Kemelmacher-Shlizerman. Audio tobody dynamics. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, pages 7574–7583, 2018.

[Tang et al., 2018] Taoran Tang, Jia Jia, and Hanyang Mao.Dance with melody: An lstm-autoencoder approach tomusic-oriented dance synthesis. In Proceedings of the26th ACM international conference on Multimedia, pages1598–1606, 2018.

[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, NikiParmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Łukasz Kaiser, and Illia Polosukhin. Attention is all youneed. In Advances in neural information processing sys-tems, pages 5998–6008, 2017.

[Wang et al., 2018] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.High-resolution image synthesis and semantic manipula-tion with conditional gans. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,2018.

[Yalta et al., 2019] Nelson Yalta, Shinji Watanabe, KazuhiroNakadai, and Tetsuya Ogata. Weakly-supervised deep re-current neural networks for basic dance step generation. In2019 International Joint Conference on Neural Networks(IJCNN), pages 1–8. IEEE, 2019.

[Ye et al., 2020] Zijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu,Wei Chen, Fanbo Meng, and Yanfeng Wang. Choreonet:Towards music to dance synthesis with choreographic ac-tion unit. In Proceedings of the 28th ACM InternationalConference on Multimedia, pages 744–752, 2020.