![Page 1: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/1.jpg)
Motion–Sound Mapping by Demonstration
Jules Françoise
{Sound Music Movement} Interaction
UMR STMS
—
IRCAM, CNRS, UPMC
Jury:
Catherine Achard
Olivier Chapuis
Thierry Dutoit
Rebecca Fiebrink
Sergi Jordà
Marcelo Wanderley
UPMC
INRIA, Orsay
U. of Mons
Goldsmiths U. of London
UPF, Barcelona
U. McGill, Montreal
(examinatrice)
(examinateur)
(Rapporteur)
(examinatrice)
(examinateur)
(Rapporteur)
Advisors:
Frédéric Bevilacqua (Ircam)
Thierry Artières (UPMC, AMU)
phd defense — Ircam — 18/03/2015
![Page 2: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/2.jpg)
2
Augmented Violin
Pedagogy
New Interfaces
Gaming
Music Games
!
May 13, 2011
For$a$Violinist,$Success$Means$A$New$Low$Point$By MATTHEW GUREWITSCH$
!
Librado Romero/The New York Times More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of �subharmonics,� the violinist Mari Kimura has been exploring the new horizons opened up by an electronic motion sensor, a half-glove equipped with electrodes that monitor the angle and speed of her bowing arm.
!
SINCE!Pythagoras,!musicians!and!scientists!have!known!(or!thought!they!knew)!that!the!lowest!pitch!a!string!stretched!taut!can!produce!�!the!fundamental!�!is!the!pitch!it!������������� ���� ��������������������� ����������������������� �������d!the!vibrating!section!sounds!at!a!higher!pitch.!Theoretically,!there!is!no!ceiling.!As!with!���� ����������������� � ��������������� ������������������!
For!the!last!five!centuries,!give!or!take,!the!range!of!a!violin!bottomed!out!on!the!G!below!middle!C,!the!pitch!of!the!open!G!string.!But!fiddlers!are!not!like!dancers!after!all.!For!nearly!two!decades,!the!Japanese!violinist!Mari!Kimura,!48,!has!been!exploring!unsuspected!subterranean!sounds!as!much!as!an!octave!deeper.!!
Complementing!the!familiar!concept!of!harmonics!(pitches!drawn!from!the!overtones!of!a!given!fundamental),!Ms.!Kimura!has!named!her!freshly!discovered!sonorities!���������� �������������������astered!the!subharmonic!octave,!third,!second!and!
Movement Learning
Music Rehabilitation
Background | {Sound Music Movement} Interaction team
2
![Page 3: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/3.jpg)
3
Motion & Sound
Relationships
mapping
![Page 4: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/4.jpg)
44
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
Models
Applications
![Page 5: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/5.jpg)
55
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
Models
Applications
![Page 6: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/6.jpg)
Sound Synthesis
Motion Capture
Motion Param.
SoundParam.
6
Related Work | motion-sound mapping: from wires to models
feedback
?
Hunt, A., Wanderley, M. M., & Kirk, R. (2000). Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference.
![Page 7: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/7.jpg)
Explicit Mapping
Sound Synthesis
Motion Capture
Motion Param.
SoundParam.
6
Related Work | motion-sound mapping: from wires to models
feedback
Motion parameters directly wired to sound control parameters
Hunt, A., Wanderley, M. M., & Kirk, R. (2000). Towards a Model for Instrumental Mapping in Expert Musical Interaction. In Proceedings of the 2000 International Computer Music Conference.
![Page 8: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/8.jpg)
Implicit Mapping
Use an intermediate model of interaction
> Physical models
> Geometrical approaches
> Machine Learning
Sound Synthesis
Motion Capture
Motion Param.
SoundParam.
Mapping Model
7
Related Work | motion-sound mapping: from wires to models
feedback
[Momeni et al., 2006]
[Van Nort et al., 2014]
[Caramiaux, 2013]
![Page 9: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/9.jpg)
Machine Learning
8
Discrete gesture recognition
Continuous gesture recognition & temporal mapping
Regression
Related Work | motion-sound mapping: from wires to models
Sound Synthesis
Motion Capture
Motion Param.
SoundParam.
Mapping Model
User Examples
feedback
[Gillian, 2011]
[Bevilacqua al., 2011]
[Caramiaux, 2013]
![Page 10: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/10.jpg)
Related Work | interactive machine learning
9
Machine Learning
Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.
![Page 11: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/11.jpg)
Related Work | interactive machine learning
9
Machine Learning
Training DataMachine Learning
Model
training
Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.
![Page 12: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/12.jpg)
Related Work | interactive machine learning
9
Machine Learning
Training DataMachine Learning
Model
training
new input
predictions
Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.
![Page 13: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/13.jpg)
Related Work | interactive machine learning
9
Machine Learning
Training DataMachine Learning
Model
training
new input
predictions
evaluate results
Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.
![Page 14: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/14.jpg)
Related Work | interactive machine learning
9
Interactive Machine Learning
create training
examples
Training DataMachine Learning
Model
training
new input
predictions
adjust parameters
& train
evaluate results
Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.
![Page 15: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/15.jpg)
Related Work | interactive machine learning
9
Interactive Machine Learning
create training
examples
adjust parameters
& train
evaluate results
Fiebrink, R. A. (2011). Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD Dissertation.
![Page 16: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/16.jpg)
1010
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
Models
Applications
![Page 17: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/17.jpg)
11
Related Work | mapping through listening
Caramiaux, B., Françoise, J., Schnell, N., & Bevilacqua, F. (2014). Mapping Through Listening. Computer Music Journal, 38(3), 34–48.
Sound Synthesis
Motion Capture
Motion Param.
SoundParam.
feedback
?
![Page 18: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/18.jpg)
11
Related Work | mapping through listening
listening as a starting point for mapping design
Caramiaux, B., Françoise, J., Schnell, N., & Bevilacqua, F. (2014). Mapping Through Listening. Computer Music Journal, 38(3), 34–48.
Sound Synthesis
Motion Capture
Motion Param.
SoundParam.
?
listening
![Page 19: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/19.jpg)
Mapping-by-Demonstration
=
mapping through listening
+
interactive machine learning
![Page 20: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/20.jpg)
Robot Programming by Demonstration
Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. Handbook of Robotics.
![Page 21: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/21.jpg)
14
Mapping by Demonstration | Overview
demonstration performance
![Page 22: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/22.jpg)
14
Mapping by Demonstration | Overview
Listening
demonstration performance
![Page 23: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/23.jpg)
14
Mapping by Demonstration | Overview
Listening
Acting
demonstration performance
![Page 24: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/24.jpg)
14
Mapping by Demonstration | Overview
Listening
Acting
demonstration performance
Mappingtraining
![Page 25: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/25.jpg)
14
Mapping by Demonstration | Overview
Listening
Acting
demonstration performance
Mappingtraining
Performance
![Page 26: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/26.jpg)
14
Mapping by Demonstration | Overview
Listening
Acting
demonstration performance
Mappingtraining
Performance
Feedback
![Page 27: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/27.jpg)
14
Mapping by Demonstration | Overview
Feedback
Listening
Acting
Performing
training
![Page 28: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/28.jpg)
example
Françoise, J., Schnell, N., & Bevilacqua, F. (2013). A Multimodal Probabilistic Model for Gesture-based Control of Sound Synthesis. In Proceedings of the 21st ACM international conference on Multimedia (MM’13) (pp. 705–708). Barcelona, Spain.
![Page 29: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/29.jpg)
![Page 30: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/30.jpg)
![Page 31: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/31.jpg)
16
Mapping by Demonstration | requirements
quick training
from 1 to several examples
continuous & causal inference
(real-time)
![Page 32: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/32.jpg)
Recognition Paradigm
Generation Paradigm
vs
17
Mapping by Demonstration | 2 criteria
![Page 33: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/33.jpg)
Recognition Paradigm
Generation Paradigm
vs
17
Mapping by Demonstration | 2 criteria
Probabilistic Models
Synthesisers
modelX
modelY
modelZ
SynthX
SynthY
SynthZ
![Page 34: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/34.jpg)
Recognition Paradigm
Generation Paradigm
vs
17
Mapping by Demonstration | 2 criteria
like
liho
od
s
movement parameters
Probabilistic Models
Synthesisers
modelX
modelY
modelZ
SynthX
SynthY
SynthZ
motion
![Page 35: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/35.jpg)
Recognition Paradigm
Generation Paradigm
vs
17
Mapping by Demonstration | 2 criteria
generatedsound parameters
like
liho
od
s
movement parameters
Probabilistic Models
Synthesisers
modelX
modelY
modelZ
SynthX
SynthY
SynthZ
motion
![Page 36: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/36.jpg)
Recognition Paradigm
Generation Paradigm
Instantaneous
Temporal
vs
vs
17
Mapping by Demonstration | 2 criteria
inputmotion
instantaneous
![Page 37: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/37.jpg)
Recognition Paradigm
Generation Paradigm
Instantaneous
Temporal
vs
vs
17
Mapping by Demonstration | 2 criteria
temporal
inputmotion
![Page 38: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/38.jpg)
Recognition Paradigm Generation Paradigm
Inst
an
tan
eo
us
Tem
po
ral
17
Mapping by Demonstration | 2 criteria
![Page 39: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/39.jpg)
18
Mapping by Demonstration | 4 models
Inst
an
tan
eo
us
Tem
po
ral
Gaussian Mixture Models gmm
Gaussian Mixture Regression gmr
Hierarchical Hidden Markov Models
hhmm
Hierarchical Hidden Markov Regression
hhmr
Recognition Paradigm Generation Paradigm
![Page 40: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/40.jpg)
1919
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR HHMR
Models
Applications
![Page 41: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/41.jpg)
GMR HHMR
2020
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR HHMR
Models
Applications
![Page 42: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/42.jpg)
GMR HHMR
2020
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR
HHMR
Models
Applications
![Page 43: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/43.jpg)
2121
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR HHMR
Models
Applications
GMR
HHMR
![Page 44: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/44.jpg)
22
Probabilistic Movement Models | Gaussian Mixture Models
time
mo
vem
en
t p
ara
me
ter
mo
tio
n p
ara
me
ter
![Page 45: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/45.jpg)
22
Probabilistic Movement Models | Gaussian Mixture Models
time
mo
vem
en
t p
ara
me
ter
mo
tio
n p
ara
me
ter
![Page 46: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/46.jpg)
23
Probabilistic Movement Models | Gaussian Mixture Models
likelihoods
motionparameter
inputmotion
![Page 47: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/47.jpg)
mo
vem
en
t p
ara
me
ter
24
Probabilistic Movement Models | Hidden Markov Models
mo
tio
n p
ara
me
ter
time
![Page 48: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/48.jpg)
mo
vem
en
t p
ara
me
ter
24
Probabilistic Movement Models | Hidden Markov Models
mo
tio
n p
ara
me
ter
time
![Page 49: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/49.jpg)
Probabilistic Movement Models | Hidden Markov Models
25
motionparameter
![Page 50: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/50.jpg)
Probabilistic Movement Models | Hidden Markov Models
25
motionparameter
likelihoods
inputmotion
![Page 51: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/51.jpg)
Probabilistic Movement Models | Hidden Markov Models
25
motionparameter
likelihoods
inputmotion
time
predictedsound
parameters
time
![Page 52: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/52.jpg)
26
Probabilistic Movement Models | GMMs & HMMs for interaction
Interactive Machine Learning
Training
> 1 to several examples
User-defined complexity
> number of states/gaussians
![Page 53: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/53.jpg)
26
Probabilistic Movement Models | GMMs & HMMs for interaction
Interactive Machine Learning
Training
> 1 to several examples
User-defined complexity
> number of states/gaussians
User-defined Regularization
> numerical errors
> generalization on few examples
![Page 54: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/54.jpg)
26
Probabilistic Movement Models | GMMs & HMMs for interaction
Interactive Machine Learning
Training
> 1 to several examples
User-defined complexity
> number of states/gaussians
User-defined Regularization
> numerical errors
> generalization on few examples
![Page 55: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/55.jpg)
26
Probabilistic Movement Models | GMMs & HMMs for interaction
Interactive Machine Learning
Training
> 1 to several examples
User-defined complexity
> number of states/gaussians
User-defined Regularization
> numerical errors
> generalization on few examples
Continuous Recognition
> Forward inference in HMMs
![Page 56: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/56.jpg)
27
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.
![Page 57: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/57.jpg)
27
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.
![Page 58: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/58.jpg)
27
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.
![Page 59: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/59.jpg)
27
1 2 3
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.
![Page 60: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/60.jpg)
27
1 2 3
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.
![Page 61: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/61.jpg)
27
1 2 3
ROOT
transitionprior
exit
hidden state
exit state
SignalLevel
SegmentLevel
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Françoise, J., Caramiaux, B., & Bevilacqua, F. (2012). A Hierarchical Approach for the Design of Gesture-to-Sound Mappings. SMC’2012.
![Page 62: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/62.jpg)
1 2 3
ROOT
transitionprior
exit
hidden state
exit state
SignalLevel
SegmentLevel
28
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Dynamic Bayesian Networks
Linear time inference
Forward algorithm
> causal estimation of signal & segment state probabilities
![Page 63: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/63.jpg)
1 2 3
ROOT
transitionprior
exit
hidden state
exit state
SignalLevel
SegmentLevel
28
Probabilistic Movement Models | Hierarchical Hidden Markov Models
Dynamic Bayesian Networks
Linear time inference
Forward algorithm
> causal estimation of signal & segment state probabilities
![Page 64: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/64.jpg)
29
P A S R
Example gesture
PASR decomposition
Preparation
Attack
Sustain
Release
(P)
(A)
(S)
(R)
> anticipation gesture
> retraction gesture
4 Phases:
Probabilistic Movement Models | HHMM — PASR
![Page 65: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/65.jpg)
Probabilistic Movement Models | HHMM — PASR
30
P A S R
Root
p = 0.5 p = 0.5
p = 1.0 p = 1.0 p = 1.0
p = 0.5
p = 0.5
Exit
p = 1.0
Possibility of authoring the transition parameters
> prior : P et A states
> exit : S et R states
2 playing modes :
> entire gesture with preparation and release
> quick transitions from sustain to attack
![Page 66: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/66.jpg)
31
example: demonstration
![Page 67: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/67.jpg)
31
![Page 68: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/68.jpg)
31
![Page 69: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/69.jpg)
32
example: performance
![Page 70: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/70.jpg)
32
![Page 71: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/71.jpg)
32
![Page 72: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/72.jpg)
3333
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR HHMR
Models
Applications
GMR
HHMR
![Page 73: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/73.jpg)
34
Probabilistic Motion-Sound Models | background: speech & motion
Similar applications from Speech processing
Acoustic-articulatory Inversion
Silent Speech Interfaces
Speech-driven animation
Robotics & Animation
Programming-by-Demonstration (Motor behavior)
Animation
Probabilistic Approaches
GMM-based / HMM-based
[Zhang & Renals, 2008]
[Hueber and Badin, 2011]
[Ding, 2014]
[Calinon, 2010]
[Tilmanne, 2013]
![Page 74: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/74.jpg)
Probabilistic Motion-Sound Models | General Approach
35training performancedemonstration
![Page 75: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/75.jpg)
Probabilistic Motion-Sound Models | General Approach
Motion Features
Sound Features
Joint Features
35training performancedemonstration
![Page 76: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/76.jpg)
Probabilistic Motion-Sound Models | General Approach
Motion Features
Sound Features
Joint Features
Training Joint density distribution
35training performancedemonstration
![Page 77: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/77.jpg)
Probabilistic Motion-Sound Models | General Approach
Motion Features
Sound Features
Joint Features
Training Joint density distribution
35
Conditional density
distribution
training performancedemonstration
![Page 78: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/78.jpg)
Probabilistic Motion-Sound Models | General Approach
Motion Features
Sound Features
Joint Features
Training Joint density distribution
35
Conditional density
distribution
training performancedemonstration
![Page 79: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/79.jpg)
Probabilistic Motion-Sound Models | Gaussian Mixture Regression
36
![Page 80: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/80.jpg)
Probabilistic Motion-Sound Models | Gaussian Mixture Regression
36
observations
x = [x (m) , x (s)]
x = [x (m) , x (s)]
x = [x (m) , x (s)]
![Page 81: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/81.jpg)
Probabilistic Motion-Sound Models | Gaussian Mixture Regression
36
observations
x = [x (m) , x (s)]
joint pdf
µk =�µ(m)
k ;µ(s)k
�(1)
�k =��(mm)
k �(ms)k
�(sm)k �(ss)
k
�
(2)
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)
x = [x (m) , x (s)]
x = [x (m) , x (s)]
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)prior
mean
covariance
![Page 82: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/82.jpg)
Probabilistic Motion-Sound Models | Gaussian Mixture Regression
36
observations
x = [x (m) , x (s)]
Conditional pdf
p�x (s) | x (m),�
�=
K�
k=1�k (x (m)) N
�x (s) ; µ̂(s)
k (x (m)),�̂(ss)k
�
joint pdf
µk =�µ(m)
k ;µ(s)k
�(1)
�k =��(mm)
k �(ms)k
�(sm)k �(ss)
k
�
(2)
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)
x = [x (m) , x (s)]
x = [x (m) , x (s)]
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)prior
mean
covariance
responsibility
mean
covariance
�k (x (m))
µ̂(s)k (x (m)) =µ(s)
k +�(sm)k
��(mm)
k
��1 �x (m) �µ(m)
k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
![Page 83: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/83.jpg)
Probabilistic Motion-Sound Models | Gaussian Mixture Regression
36
observations
x = [x (m) , x (s)]
Conditional pdf
p�x (s) | x (m),�
�=
K�
k=1�k (x (m)) N
�x (s) ; µ̂(s)
k (x (m)),�̂(ss)k
�
joint pdf
µk =�µ(m)
k ;µ(s)k
�(1)
�k =��(mm)
k �(ms)k
�(sm)k �(ss)
k
�
(2)
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)
x = [x (m) , x (s)]
x = [x (m) , x (s)]
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)prior
mean
covariance
responsibility
mean
covariance
�k (x (m))
µ̂(s)k (x (m)) =µ(s)
k +�(sm)k
��(mm)
k
��1 �x (m) �µ(m)
k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
![Page 84: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/84.jpg)
Probabilistic Motion-Sound Models | Gaussian Mixture Regression
36
observations
x = [x (m) , x (s)]
x̂ (s) =K�
k=1�k (x (m))µ̂(s)
k (x (m))
Least Squares EstimateConditional pdf
p�x (s) | x (m),�
�=
K�
k=1�k (x (m)) N
�x (s) ; µ̂(s)
k (x (m)),�̂(ss)k
�
joint pdf
µk =�µ(m)
k ;µ(s)k
�(1)
�k =��(mm)
k �(ms)k
�(sm)k �(ss)
k
�
(2)
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)
x = [x (m) , x (s)]
x = [x (m) , x (s)]
p(x i | θ) =K∑
k=1wkN
(x i ; µk ,Σk
)prior
mean
covariance
responsibility
mean
covariance
�k (x (m))
µ̂(s)k (x (m)) =µ(s)
k +�(sm)k
��(mm)
k
��1 �x (m) �µ(m)
k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
![Page 85: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/85.jpg)
37
Probabilistic Motion-Sound Models | GMR — number of gaussians
![Page 86: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/86.jpg)
38
Probabilistic Motion-Sound Models | GMR — regularization
![Page 87: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/87.jpg)
39
Probabilistic Motion-Sound Models | Hidden Markov Regression
![Page 88: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/88.jpg)
movement parameter
sou
nd
p
ara
me
ter
39
Probabilistic Motion-Sound Models | Hidden Markov Regression
![Page 89: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/89.jpg)
movement parameter
sou
nd
p
ara
me
ter
time
39
Probabilistic Motion-Sound Models | Hidden Markov Regression
![Page 90: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/90.jpg)
likelihoods
movement parameter
movement parameters
sou
nd
p
ara
me
ter
time
39
Probabilistic Motion-Sound Models | Hidden Markov Regression
![Page 91: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/91.jpg)
predictedsound
parameters
likelihoods
movement parameter
movement parameters
sou
nd
p
ara
me
ter
time
39
Probabilistic Motion-Sound Models | Hidden Markov Regression
![Page 92: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/92.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
joint observation pdf:
µk =�µ(m)
k ;µ(s)k
�(1)
�k =��(mm)
k �(ms)k
�(sm)k �(ss)
k
�
(2)
mean
covariance
p (x t | zt = k) =N�x t ;µk ,�k
�
![Page 93: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/93.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
HMM with Conditional Observation probability distribution
mean
covariance
p�
x (s)t | x (m)
t , zt = k�=N
�x t ; µ̂k (x (m)),�̂
(ss)k
�conditional observation pdf:
µ̂(s)k (x (m)
t ) =µ(s)k +�(sm)
k
��(mm)
k
��1 �x (m)
t �µ(m)k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
![Page 94: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/94.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
HMM with Conditional Observation probability distribution
mean
covariance
µ̂(s)k (x (m)
t ) =µ(s)k +�(sm)
k
��(mm)
k
��1 �x (m)
t �µ(m)k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
online estimation (forward)
p(x (s)t |x (m)
1:t ,�) =
![Page 95: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/95.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
HMM with Conditional Observation probability distribution
mean
covariance
µ̂(s)k (x (m)
t ) =µ(s)k +�(sm)
k
��(mm)
k
��1 �x (m)
t �µ(m)k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
online estimation (forward)
p(x (s)t |x (m)
1:t ,�) = N�
x (s) ; µ̂(s)i (x (m)
t ),�̂(ss)i
�
![Page 96: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/96.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
HMM with Conditional Observation probability distribution
mean
covariance
µ̂(s)k (x (m)
t ) =µ(s)k +�(sm)
k
��(mm)
k
��1 �x (m)
t �µ(m)k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
online estimation (forward)
p(x (s)t |x (m)
1:t ,�) = N�
x (s) ; µ̂(s)i (x (m)
t ),�̂(ss)i
��(m)
t (i )
![Page 97: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/97.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
HMM with Conditional Observation probability distribution
mean
covariance
µ̂(s)k (x (m)
t ) =µ(s)k +�(sm)
k
��(mm)
k
��1 �x (m)
t �µ(m)k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
online estimation (forward)
p(x (s)t |x (m)
1:t ,�) = N�
x (s) ; µ̂(s)i (x (m)
t ),�̂(ss)i
��(m)
t (i )N�
i=1
![Page 98: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/98.jpg)
Probabilistic Motion-Sound Models | Hidden Markov Regression
Joint Features
Training
HMM with Joint
Observation probability distribution
40
HMM with Conditional Observation probability distribution
mean
covariance
µ̂(s)k (x (m)
t ) =µ(s)k +�(sm)
k
��(mm)
k
��1 �x (m)
t �µ(m)k
�(1)
�̂(ss)k =�(ss)
k ��(sm)k
��(mm)
k
��1�(ms)
k (2)
x̂ (s)t =
N�
i=1�(m)
t (i )µ̂(s)k (x (m)
t )
Least Squares Estimate
![Page 99: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/99.jpg)
41
Probabilistic Motion-Sound Models | Hidden Markov Regression
Interactive Machine Learning implementation
Number of Hidden States
> complexity / temporal grain
Regularization
> smoothness / generalization
Hierarchical Extension
> high-level structure
Evaluation on Tai Chi Movement Sequences
![Page 100: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/100.jpg)
4242
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR HHMR
Models
Applications
GMR
HHMR
![Page 101: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/101.jpg)
Playing Sound Textures | siggraph’14 Studio installation
Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.
Corpus-based Descriptor-driven Sound Synthesis
CataRT (Schwarz)
Sound Corpora from DIRTI
User Studio, Diemo Schwarz, Roldan Cahen
![Page 102: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/102.jpg)
Playing Sound Textures | siggraph’14 Studio installation
Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.
Corpus-based Descriptor-driven Sound Synthesis
CataRT (Schwarz)
Sound Corpora from DIRTI
User Studio, Diemo Schwarz, Roldan Cahen
![Page 103: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/103.jpg)
Advantages
Complex gestures
> implicit sonification strategies
Adaptive
> personalize gestures
> personalize sonification
44
Playing Sound Textures | texture sonification for gesture learning
![Page 104: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/104.jpg)
Protocol
> reproduce a gesture as precisely as possible
> training from audio-visual demonstration
> reproduction: 3 recording blocks (10 trials)
silent vs sonified
> 11 participants
> 4 gestures (2 sonified / 2 silent)
Analysis
> correlation distance to reference
45
Playing Sound Textures | texture sonification for gesture learning
![Page 105: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/105.jpg)
Playing Sound Textures | texture sonification for gesture learning
*
46
![Page 106: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/106.jpg)
4747
outline
Related Work
Mapping-by-Demonstration
Probabilistic Movement Models
Probabilistic Models for Sound Parameter Generation
Playing Sound Textures
Motion–Sound Interaction through Vocalization
Background & Concepts
GMM HHMM
GMR HHMR
Models
Applications
GMR
HHMR
![Page 107: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/107.jpg)
48
Sonification through Vocalization | imitation game
Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.
![Page 108: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/108.jpg)
48
Sonification through Vocalization | imitation game
Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.
MicrophoneWireless Motion Sensor
![Page 109: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/109.jpg)
48
Sonification through Vocalization | imitation game
Françoise, J., Schnell, N., & Bevilacqua, F. (2014). MaD: Mapping by Demonstration for Continuous Sonification. In ACM SIGGRAPH 2014 Emerging Technologies.
MicrophoneWireless Motion Sensor
![Page 110: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/110.jpg)
![Page 111: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/111.jpg)
![Page 112: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/112.jpg)
50
Sonification through Vocalization | imitation game
![Page 113: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/113.jpg)
Imitation Game: adaptation
51
Sonification through Vocalization | imitation game
![Page 114: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/114.jpg)
Imitation Game: adaptation
51
Sonification through Vocalization | imitation game
![Page 115: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/115.jpg)
Imitation Game: adaptation
51
Sonification through Vocalization | imitation game
![Page 116: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/116.jpg)
Imitation Game: adaptation
51
Sonification through Vocalization | imitation game
![Page 117: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/117.jpg)
Sonification through Vocalization | imitation game
52
Evaluation
Sequence game
> 59 participants recorded at Siggraph’14
Analysis
> investigate log-likelihoods during gesture reproduction
Results
> sensitive to user expertise
> fast adaptation with sound feedback
==> learning
![Page 118: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/118.jpg)
53
Sonification of movement
qualities
HMR System trained with Expert
performances
Exploratory workshop with dancers
> integration in teaching session
Promising for pedagogy
> Further evaluation planned
Vocalization | Application in dance
Françoise, J., Fdili Alaoui, S., Schiphorst, T., & Bevilacqua, F. (2014). Vocalizing Dance Movement for Interactive Sonification of Laban Effort Factors. Designing Interactive Systems (DIS’14), Vancouver, Canada.
![Page 119: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/119.jpg)
54
Vocalization| movement sequence analysis (tai chi)
MO Sensors(Accel. + Gyro.)
(a)
(b) (c)
Tai Chi
> long sequences
> expertise (consistency over trials)
Motion Capture
> 3 inertial sensors (6DOF: acc + gyro)
Task
> 10 performances of 45s sequence
> compare expert & student
Françoise, J., Roby-Brami, A., Riboud, N. & Bevilacqua, F. (2013). [SUBMITTED]
![Page 120: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/120.jpg)
54
Vocalization| movement sequence analysis (tai chi)
MO Sensors(Accel. + Gyro.)
(a)
(b) (c)
Tai Chi
> long sequences
> expertise (consistency over trials)
Motion Capture
> 3 inertial sensors (6DOF: acc + gyro)
Task
> 10 performances of 45s sequence
> compare expert & student
Françoise, J., Roby-Brami, A., Riboud, N. & Bevilacqua, F. (2013). [SUBMITTED]
![Page 121: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/121.jpg)
54
Vocalization| movement sequence analysis (tai chi)
MO Sensors(Accel. + Gyro.)
(a)
(b) (c)
Tai Chi
> long sequences
> expertise (consistency over trials)
Motion Capture
> 3 inertial sensors (6DOF: acc + gyro)
Task
> 10 performances of 45s sequence
> compare expert & student
Françoise, J., Roby-Brami, A., Riboud, N. & Bevilacqua, F. (2013). [SUBMITTED]
![Page 122: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/122.jpg)
Movement Sequence Analysis | vocalization in performance
55
![Page 123: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/123.jpg)
Movement Sequence Analysis | vocalization in performance
55
![Page 124: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/124.jpg)
Movement Sequence Analysis | vocalization in performance
55
![Page 125: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/125.jpg)
conclusions
![Page 126: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/126.jpg)
57
Conclusions
Mapping by Demonstration
> mapping through listening + interactive machine learning
> powerful tool for designing sonic interactions
> leverages expertise
![Page 127: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/127.jpg)
57
Conclusions
Mapping by Demonstration
> mapping through listening + interactive machine learning
> powerful tool for designing sonic interactions
> leverages expertise
GMM HHMM
GMR HHMM
recognition paradigm
generative paradigm
instantaneous temporal
General framework for designing
motion & sound relationships
with probabilistic models
> learning from few examples
> continuous & causal inference
> generative approach
![Page 128: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/128.jpg)
57
Conclusions
Mapping by Demonstration
> mapping through listening + interactive machine learning
> powerful tool for designing sonic interactions
> leverages expertise
GMM HHMM
GMR HHMM
recognition paradigm
generative paradigm
instantaneous temporal
XMM library
open-source C++ library
Max implementation
General framework for designing
motion & sound relationships
with probabilistic models
> learning from few examples
> continuous & causal inference
> generative approach
![Page 129: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/129.jpg)
Modeling and sonifying complex movements
> music
> dance
> rehabilitation
MOCO: International Workshop on Movement & Computing
Vocalization
Domain-specific Expertise
> Sonic Interaction Design
> Dance & Movement Practice
58
Applications & Perspectives
(Greg Beller’s Synekine project)
(Moving Stories Project, Thecla Schiphorst)
(LEGOS project & Labex Smart)
(SkAT-VG EU project)
![Page 130: Motion–Sound {Sound Music Movement} Interaction Mapping · More than one string to her bow: apart from her remarkable acoustic discoveries in the realm of subharmonics, the violinist](https://reader036.vdocuments.net/reader036/viewer/2022081601/6114cb3f0f4f5255db6728d0/html5/thumbnails/130.jpg)
thank you
… and many thanks to collaborators (incomplete pseudo-alphabetical list):
Frédéric Bevilacqua, Omid Alemi, Pablo Arias, Thierry Artières, Caroline Barbot, Greg
Beller, Riccardo Borghesi, Éric Boyer, Amélie Briaucourt, Baptiste Caramiaux, Fabien
Cesari, Olivier Chapuis, Sarah Fdili Alaoui, Masha Fedorova, Emmanuel Fléty, Sylvain
Hanneton, Olivier Houix, Stacy Hsueh, Ianis Lallemand, Jean-Philippe Lambert,
Benjamin Matuszewski, Nicolas Rasamimanana, Natacha Riboud, Sébastien
Robaszkiewicz, Agnès Roby-Brami, Victor Saiz, Kevin Sanlaville, Thecla Schiphorst,
Norbert Schnell, Diemo Schwarz, Alejandro Van Zandt-Escobar, …