ctp431- music and audio computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40...
TRANSCRIPT
![Page 1: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/1.jpg)
CTP431- Music and Audio Computing Sound Synthesis
Graduate School of Culture TechnologyKAIST
Juhan Nam
1
![Page 2: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/2.jpg)
Outlines
§ Additive Synthesis
§ Subtractive Synthesis– Analog synthesizers– Singing voice synthesis
§ Nonlinear Synthesis– Ring modulation / Frequency modulation – Wave-shaping
§ Physical Modeling
§ Sample-based Synthesis
2
![Page 3: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/3.jpg)
Additive Synthesis
§ Synthesize sounds by adding multiple sine oscillators– Also called Fourier synthesis
3
OSC
OSC
OSC
.
.
.
Amp(Env)
Amp(Env)
Amp(Env)
.
.
.
+
![Page 4: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/4.jpg)
Telharmonium
§ The first additive synthesizer using electro-magnetic “tone wheels” (Cahill, 1897)
§ Transmitted through telephone lines – Subscription only– The business failed
![Page 5: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/5.jpg)
Theremin
§ A sinusoidal tone generator
§ Two antennas are remotely controlled to adjust pitch and volume
Theremin(byLéonTheremin,1928)
![Page 6: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/6.jpg)
6
Theremin (Clara Rockmore)https://www.youtube.com/watch?v=pSzTPGlNa5U
![Page 7: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/7.jpg)
Hammond Organ
7
§ Drawbars– Control the levels of individual
tonewheels
![Page 8: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/8.jpg)
Sound Examples
§ Web Audio Demo– http://femurdesign.com/theremin/– http://www.venlabsla.com/x/additive/additive.html
§ Examples (instruments)– Kurzweil K150
• https://soundcloud.com/rosst/sets/kurzweil-k150-fs-additive– Kawai K5, K5000
8
![Page 9: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/9.jpg)
Subtractive Synthesis
§ Synthesize sounds by filtering wide-band oscillators– Source-Filter model– Examples
• Analog Synthesizers: oscillators + resonant lowpass filters• Voice Synthesizers: glottal pulse train + formant filters
9
5 10 15 20−60
−50
−40
−30
−20
−10
0
10
20
Frequency (kHz)
Mag
nitu
de (d
B)
5 10 15 20−60
−50
−40
−30
−20
−10
0
10
20
Frequency (kHz)
Mag
nitu
de (d
B)
0 0.5 1 1.5 2 2.5x 104
−60
−50
−40
−30
−20
−10
0
10
20
Frequency (kHz)
Mag
nitu
de (d
B)
Source Filter FilteredSource
![Page 10: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/10.jpg)
Moog Synthesizers
10
![Page 11: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/11.jpg)
https://www.youtube.com/watch?v=usl_TvIFtG0
![Page 12: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/12.jpg)
Moog Synthesizers
12
Envelope
Envelope LFO
Wheels Slides PedalPhysicalControl
FilterOscillators Amp
Keyboard
AudioPath
SoftControl
Parameter=offset+depth*control (e.g.filtercut-offfrequency)
(staticvalue) (dynamicvalue)
Parameter Parameter Parameter
![Page 13: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/13.jpg)
Oscillators
§ Classic waveforms
§ Modulation– Pulse width modulation– Hard-sync– More rich harmonics
13
0 50 100 150 200−2
0
2
5 10 15 20−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B)
−6dB/oct
0 50 100 150 200−1
0
1
5 10 15 20−60−40−20
020
Frequency (kHz)M
agni
tude
(dB)
−6dB/oct
0 50 100 150 200−2
0
2
5 10 15 20−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B)
−12dB/oct
Sawtooth TriangularSquare
![Page 14: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/14.jpg)
Amp Envelop Generator
§ Amplitude envelope generation– ADSR curve: attack, decay, sustain and release– Each state has a pair of time and target level
14
NoteOn NoteOff
Attack Decay Sustain
Release
Amplitude(dB)
![Page 15: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/15.jpg)
Examples
§ Web Audio Demos– http://www.google.com/doodles/robert-moogs-78th-birthday– http://webaudiodemos.appspot.com/midi-synth/index.html– http://aikelab.net/websynth/– http://nicroto.github.io/viktor/
§ Example Sounds– SuperSaw– Leads– Pad– MoogBass– 8-Bit sounds: https://www.youtube.com/watch?v=tf0-Rrm9dI0– TR-808: https://www.youtube.com/watch?v=YeZZk2czG1c
15
![Page 16: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/16.jpg)
Modulation Synthesis
§ Modulation is originally from communication theory– Carrier: channel signal, e.g., radio or TV channel – Modulator: information signal, e.g., voice, video
§ Decreasing the frequency of carrier to hearing range can be used to synthesize sound
§ Types of modulation synthesis– Amplitude modulation (or ring modulation)– Frequency modulation
§ Modulation is non-linear processing– Generate new sinusoidal components
16
![Page 17: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/17.jpg)
Ring Modulation / Amplitude Modulation
§ Change the amplitude of one source with another source– Slow change: tremolo– Fast change: generate a new tone
17
OSC
OSC
(1+ am (t))Ac cos(2π fct)am (t)Ac cos(2π fct)
AmplitudeModulation
xCarrier
Modulator
OSC
OSC
xCarrier
Modulator
+
RingModulation
![Page 18: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/18.jpg)
Ring Modulation / Amplitude Modulation
§ Frequency domain – Expressed in terms of its sideband frequencies– The sum and difference of the two frequencies are obtained according to
trigonometric identity– If the modulator is a non-sinusoidal tone, a mirrored-spectrum with regard to the
carrier frequency is obtained
18
fc+fmfcfc-fm
am (t) = Am sin(2π fmt))
carriersidebandsideband
![Page 19: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/19.jpg)
Examples
§ Tone generation– SawtoothOsc x SineOsc– https://www.youtube.com/watch?v=yw7_WQmrzuk
§ Ring modulation is often used as an audio effect– http://webaudio.prototyping.bbc.co.uk/ring-modulator/
19
![Page 20: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/20.jpg)
Frequency Modulation
§ Change the frequency of one source with another source– Slow change: vibrato– Fast change: generate a new (and rich) tone– Invented by John Chowning in 1973 à Yamaha DX7
20
Ac cos(2π fct +β sin(2π fmt))
β =Amfm
IndexofmodulationOSC
OSC
Carrier
Modulator
frequency
![Page 21: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/21.jpg)
Frequency Modulation
§ Frequency Domain– Expressed in terms of its sideband frequencies– Their amplitudes are determined by the Bessel function– The sidebands below 0 Hz or above the Nyquist frequency are folded
21
y(t) = Ac Jk (k=−∞
k=−∞
∑ β)cos(2π ( fc + kfm )t)
fc+fmfc fc+2fm fc+3fmfc-fmfc-2fmfc-3fm
carriersideband1sideband1
sideband2sideband2sideband3sideband3
![Page 22: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/22.jpg)
Bessel Function
22
Jk (β) =(−1)n (β
2)k+2n
n!(n+ k)!n=0
∞
∑
0 50 100 150 200 250 300 350−0.5
0
0.5
1
beta
J_(k
)
CarrierSideband 1Sideband 2Sideband 3Sideband 4
![Page 23: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/23.jpg)
Bessel Function
23
![Page 24: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/24.jpg)
The Effect of Modulation Index
24
0 500 1000 1500 2000−1
0
1
Time (Sample)
Ampl
itude
0.2 0.4 0.6 0.8 1−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B) Beta = 0
0 500 1000 1500 2000−1
0
1
Time (Sample)
Ampl
itude
0.2 0.4 0.6 0.8 1−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B) Beta = 1
0 500 1000 1500 2000−1
0
1
Time (Sample)
Ampl
itude
0.2 0.4 0.6 0.8 1−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B) Beta = 10
0 500 1000 1500 2000−1
0
1
Time (Sample)
Ampl
itude
0.2 0.4 0.6 0.8 1−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B) Beta = 20
fc = 500, fm = 50
![Page 25: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/25.jpg)
25
Yamaha DX7 (1983)
![Page 26: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/26.jpg)
“Algorithms” in DX7
26
http://www.audiocentralmagazine.com/yamaha-dx-7-riparliamo-di-fm-e-non-solo-seconda-parte/yamaha-dx7-algorithms/
![Page 27: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/27.jpg)
Examples
§ Web Audio Demo– http://www.taktech.org/takm/WebFMSynth/
§ Sound Examples– Bell– Wood – Brass– Electric Piano– Vibraphone
27
![Page 28: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/28.jpg)
Non-linear Synthesis (wave-shaping)
§ Generate a rich sound spectrum from a sinusoid using non-linear transfer functions (also called “distortion synthesis”)
§ Examples of transfer function: y = f(x)– y = 1.5x’ – 0.5x’3– y = x’/(1+|x’|)– y = sin(x’)– Chebyshev polynomial: Tk+1(x) = 2xTk(x)-Tk-1(x)
28
0 50 100 150 200−1
0
1
Time (Sample)
Am
plitu
de
5 10 15 20−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B)
0 50 100 150 200−1
0
1
Time (Sample)
Am
plitu
de
5 10 15 20−60−40−20
020
Frequency (kHz)
Mag
nitu
de (d
B)
−1 −0.5 0 0.5 1−1
−0.5
0
0.5
1
Time (Sample)
Ampl
itude
T0(x)=1,T1(x)=x,T2(x)=2x2-1,T2(x)=4x3-3x
x’=gx:gcorrespondtothe“gainknob”ofthedistortion
![Page 29: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/29.jpg)
Physical Modeling
§ Modeling Newton’s laws of motion (i.e. 𝐹 = 𝑚𝑎) on musical instruments– Every instrument have a different model
§ The ideal string
– Wave equation: 𝐹 = 𝑚𝑎à 𝐾 '()'*(
= 𝜀 '()
'*((𝐾: tension, 𝜀: linear mass density)
– General solution: 𝑦 𝑡, 𝑥 = 𝑦0(𝑡 −*3) + 𝑦5(𝑡 +
*3)
àLeft-going traveling wave and right-going traveling wave
29
![Page 30: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/30.jpg)
Physical Modeling
§ Waveguide Model– With boundary condition (fixed ends)
§ The Karplus-Strong model
30
-1.0 -0.99
Z-M+x(n)
DelayLine
y(n)
NoiseBurstLowpassFilter
![Page 31: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/31.jpg)
Physical Modeling
§ The Extended Karplus -Strong model
31https://ccrma.stanford.edu/~jos/pasp/Extended_Karplus_Strong_Algorithm.html
![Page 32: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/32.jpg)
Sample-based Synthesis
32
Synthogy IvoryIIPianoFoley(filmmaking) Ringtones
§ The majority of digital sound and music synthesis today is accomplished via the playback of stored waveforms– Media production: sound effects, narration, prompts– Digital devices: ringtone, sound alert– Musical Instruments
• Native Instrument Kontakt5: 43+ GB (1000+ instruments) • Synthogy Ivory II Piano: 77GB+ (Steinway D Grand, ….)
![Page 33: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/33.jpg)
Why Don’t We Just Use Samples?
§ Advantages– Reproduce realistic sounds (needless to say)– Less use of CPU
§ Limitations– Not flexible: repeat the same sound again, not expressive – Can require a great deal of storage– Need high-quality recording– Limited to real-world sounds
§ Better ways– Modify samples based on existing sound processing techniques
• Much richer spectrum of sounds– Trade-off: CPU, memory and programmability
33
![Page 34: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/34.jpg)
Sample-based Synthesis
34
Off-LineProcessingRecording Storing
intableOff-line
OnlineProcessing
SampleFetchingOn-line
Meta-data:pitch,loudness,action,text
Userinput
AudioEffect
- Sampleediting- Pitchchange- Filter,EQ,envelope- Normalization
- PitchChange- Filter,EQ- Envelope
- Delay-basedeffects(e.g.roomEffects)
- Pitch,velocity,Timbre- Speed,strength- text
ReadSamplesfromthetable
![Page 35: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/35.jpg)
Wavetable Synthesis
§ Playback samples stored in tables– Multi-sampling: choose different sample tables depending on input conditions such
pitch and loudness• Velocity switching
§ Reducing sample tables in musical synthesizers– Sample looping: reduce the size of tables– Pitch shifting by re-sampling: avoid sampling every single pitch – Filtering: avoid sampling every single loudness
• e.g. low-pass filtering for soft input
35
![Page 36: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/36.jpg)
Sample Looping
§ Find a periodic segment and repeat it seamlessly during playback– Particularly for instruments with forced oscillation (e.g.
woodwind)– Usually taken from the sustained part of a pitched musical note
§ It is not easy to find an exactly clean loop– The amplitude envelopes often decays or modulated:
• e.g. piano, guitar, violin – Period in sample is not integer à non-integer-size sample table?
36
Attack Loop
Playbackusinglooping
![Page 37: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/37.jpg)
Sample Looping
§ Solutions– Decaying amplitude: normalize the amplitude
• Compute the envelope and multiply it inverse • Then, multiply the envelope back later
– Non-integer period in sample• Use multiple periods for the loop such that the total period is close to integers
* e.g. Period = 100.2 samples à 5*Period = 501 samples – Amplitude modulation
• Crossfade between the end of loop and the beginning of loop meet
§ Automatic loop search– Pitch detection and zero-crossing detection: c.f. samplers
37
![Page 38: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/38.jpg)
Concatenative Synthesis
§ Splicing sample segments based on input information– Typically done in speech synthesis: unit selection
§ Sample size depends on applications– ARS: limited expression and context-dependent
• word or phrase level– TTS: unlimited expression and context-independent
• phone or di-phone (phone-to-phone transition) level
38
![Page 39: CTP431- Music and Audio Computingjuhan/ctp431/2016/slides... · −2 0 2 5 10 15 20 −60 −40 −20 0 20 Frequency (kHz) Magnitude (dB) −6dB/oct 0 50 100 150 200 −1 0 1 5 10](https://reader033.vdocuments.net/reader033/viewer/2022060519/604dc88aa58b7f65d734c504/html5/thumbnails/39.jpg)
Summary
39
Memory(Storage)
Programmability(by # of parameters)
Reproducibility of natural sounds
Interpretabilityof parameters
Computationpower
Additive ** ***** **** **** ****
Subtractive * *** ** *** **
Non-linear * *** ** ** **
Physical model *** ** **** ***** *** ~ *****
Sample-based ***** * ***** N/A * ~ ***