Nov 30, Fall 2006 IAT 410 1

Audio

SoundAudio synthesis

Nov 30, Fall 2006 IAT 410 2

Audio Perception

Sound: Pressure waves in frequencies between 50Herz - 22,000Herz

Lower frequencies more felt by the whole body than heard

Sounds can be perceived as coming from a location– Not terribly accurate– Cone of confusion

Nov 30, Fall 2006 IAT 410 3

Cone of confusion– Cone-shaped zones in front of and

behind head 3D Audio cues:

– Interaural Time Difference– Interaural Intensity Difference– Pinnae filtering– Body filtering

3D Audio Perception

Nov 30, Fall 2006 IAT 410 4

3D Audio Perception

Goal for 3D sound is “Spatialization” The sense that the

– Sound originates outside your head– Sound has a direction

Interaural Time Difference– The more extremely left or right, the greater

the difference– Time difference < 5ms

Nov 30, Fall 2006 IAT 410 5

3D Audio Perception

Interaural Intensity Difference– Head absorbs and reflects sound energy– The first ear to get sound gets loudest

sound– “Head Shadow”

Cone of confusion:– Time difference too small to detect– Intensity is similar in both ears

Nov 30, Fall 2006 IAT 410 6

Pinnae Filtering

Outer ear (Pinna) shape filters sound based on its direction

Childhood learning trains brain to associate filtering effects with direction

Unique per person Record directional white noise

– Microphone in ear canal– Sounds from speakers located about head

Nov 30, Fall 2006 IAT 410 7

Pinnae Filtering

A “Generic” Pinna can be simulated Record directional white noise

received by dummy head Body filtering

– Reflection and absorption– Included in Pinna model

Nov 30, Fall 2006 IAT 410 8

Head-Related Transfer Function

HRTF is the general term Transformation of “real” sound to

spatialized sound Best delivered by earphones

Nov 30, Fall 2006 IAT 410 9

Environmental Effects

Sound exists in an environment– Bounces off objects– Is absorbed by objects

Simple effects– Reverb: Simulate the environmental echo

• Echo is the attenuated signal• Gives a richer room-like feeling• Larger room has longer time delay

Nov 30, Fall 2006 IAT 410 10

Audio signals

Nyquist limit:– Must sample signal at least twice as

frequently as highest reproducible frequency

– Audio: 44.1KHz (CD)– 22KHz– 11KHz (Analog AM Radio)– 8KHz (Telephone)

Nov 30, Fall 2006 IAT 410 11

Audio - Digital Implications

44,100 Hz– 44,100 Samples/sec– 16-bit samples– Stereo– 172KBytes/sec

Specialized hardware - Sound card

Nov 30, Fall 2006 IAT 410 12

Reproduction

Sampling– Record sounds by whatever means

Synthesis– Analog Synthesis– FM Synthesis– Wavetable Synthesis

Nov 30, Fall 2006 IAT 410 13

Control

MIDI - Musical Instrument Digital Interface

Developed to control music synthesizers– Details of synthesis are controlled by

synthesizer MIDI data

– Sets synthesis parameters– Sets music sequence

Nov 30, Fall 2006 IAT 410 14

Synthesis

Analog Synthesis– Simple sum of frequencies– Select from a palette of source

frequencies– Sum of frequencies is filtered

FM– One frequency is controlled by another

Wavetable– Digitize audio signals

Nov 30, Fall 2006 IAT 410 15

Analog Synthesis

Fourier’s observation– Any signal can be created as the sum of

sine waves– Square wave: Infinite sum– f + 2f + 4f…

Synthesizer:– Collection of

oscillators

Nov 30, Fall 2006 IAT 410 16

Frequency Domain

A perfectly periodic signal plotted in the frequency domain

(Time domain plot)

Nov 30, Fall 2006 IAT 410 17

Spectrum

Spectrum represents the set of frequencies present in the signal

Nov 30, Fall 2006 IAT 410 18

Filters

Eliminate part of the signal by removing certain frequencies

Analog filters don’t have these “square” response shapes

Band pass – Bandwidth

Nov 30, Fall 2006 IAT 410 19

FM Synthesis Modulate the frequency of a wave

Carrier frequency is modulated by Modulator signal

Nov 30, Fall 2006 IAT 410 20

FM Synthesis for Synthesizers

The greater the Modulator amplitude, the greater the Carrier frequency variation– Higher Carrier bandwidth

DX: Carrier and Modulator are “musically-tuned frequency”– Depends on the note you are playing – Controls the harmonic content of a note

Nov 30, Fall 2006 IAT 410 21

Wavetable Synthesis

Collect a sample of the real sound Issues:

– Reduce memory load by looping sample– Shift pitch instead of sampling each

individual note– Apply interpolation techniques to make

pitch shifting work right

Nov 30, Fall 2006 IAT 410 22

Audio Path

Raw Sound(Sample, FM, etc)

Tuned

MIDINote Sequence

Resampling

EnvelopeLoudness Control

Mixing/Combination

Reverb,Environment.Spatialization

Nov 30, Fall 2006 IAT 410 23

Wavetable Synthesis Example

Leyanda (Guitar) Leyanda (CDShaw)

Nov 30, Fall 2006 IAT 410 24

Interactive Sound

Goal– Want to enhance the interactive

experience– Give the user a sense of presence– Add to the emotional content of the

game– Make it more fun

Nov 30, Fall 2006 IAT 410 25

Interactive Sound

Music Sound effects

– Noises– Commentary - Sports

Narrative Conversations

Nov 30, Fall 2006 IAT 410 26

Interactive Problems

Regular music composition has – Beginning– Middle– End

Interactive user control makes this difficult– Some genres have this structure

Nov 30, Fall 2006 IAT 410 27

Interactive Music

Game genres with order– Sports– Racing– Fighting

Semi-Ordered– Puzzle– Adventure

Nov 30, Fall 2006 IAT 410 28

Music Genres

The Infinite Loop– Theme and variation plays forever– Pomp & Circumstance– Diablo

Problems:– 30 second piece repeated over 6 hours!– 720 repetitions!– Diablo example: 12 Repetitions/hour

Nov 30, Fall 2006 IAT 410 29

Repetition Solutions

Make the Dominant theme hard to find– No catchy theme!– Create a variety of textures– Make only transitions stand out

Where repetition is small– Don’t repeat musical phrases

Nov 30, Fall 2006 IAT 410 30

Music Strategies

Play Win or Lose music– Music must be long enough to be

meaningful– Music may be so long that the game

situation changes before completion– Very short music makes little sense

Interrupt current music– Sounds jarring

Nov 30, Fall 2006 IAT 410 31

Modules

Modular chunks– Each segment of the game plays

independently of others– Some thematic relation– Disjointed

Nov 30, Fall 2006 IAT 410 32

Music Strategies

Compose many themes in parallel Switch between themes Connect modular components

together

Nov 30, Fall 2006 IAT 410 33

Analogy: Parallel Trains

N trains of music running in parallel Each train serves an emotional

purpose– Train A: Calm– Train B: Rising Excitement– Train C: Climactic moments– Train D: Falling Excitement

Generally, Train A would be most commonly played

Nov 30, Fall 2006 IAT 410 34

Within Each Musical Train

Each “Car” contains a few bars of music Switch between trains when a “Car” is

complete– Don’t switch in the middle of a “Car”

Simple version:– Each musical phrase ends on last bar of

“Car” Complex:

– Notes at end are carried over to next

Nov 30, Fall 2006 IAT 410 35

Parallel Trains

Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32

Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32

Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32

Bars 1-8 Bars 9-16 Bars 17-24 Bars 25-32

Nov 30, Fall 2006 IAT 410 36

Parallel Trains: Shuffle Cars

Shuffle cars – Instead of playing cars in order

Problem: Random cars sound like random radio tuning

Must determine – Appropriate car pairings– Reasonable paths

Nov 30, Fall 2006 IAT 410 37

Repetition In Trains

Use repeated phrases carefully Maybe use a statistical tool to

analyze paths– Bayesian nets

Endings:– Use transitions as an opportunity to

“End” – Use next Car to “Begin” new series

Nov 30, Fall 2006 IAT 410 38

Composing a Train

Create a piece with all layers– Piece can probably survive a layer or two

removed– Variation = piece with layer removed

Be careful with prominent instruments– Fallback: Use instruments with similar

acoustical properties• Piano, Organ, Woodwinds• No Trumpets, Drums or Screaming guitar!