audio for games

Microsoft Confidential

Audio for Games

Scott SelfonXbox Advanced Technology Group


OverviewThe roles of audio in gamesAudio challenges on a game consoleScore, Sound Effects, and Dialog

ChallengesTypical SolutionsContent creation / implementation process

Audio integration and end gameCase studies on XboxLeading edge / future of game audioQ & A


Game Audio Has Come a Long Way…

FidelityTinny mono speaker 5.1 home theater systems1 channel/4 bits 256 channels, 24 bitsFM oscillators 48 kHz wavetable synthesis

Audio processing powerOff-line effects processing real-time DSP effects4-channel mixing Realtime Dolby Digital encodeRealtime 3D positioning

Production pipelineLow level chip programming Separate sound designer/composer and audio programmer


The Roles of Audio in GamesAudio is 50% of the experienceSupport (improve?) the visualsProvide information about what we don’t seeProvide emotional support for gameplay

Rewards/encouragement

The best audio implementations are rarely noticed by the gamer

1995 MIT studyGame reviews and audio


Audio Challenges on a Game Console: Aesthetic

Competing with other ‘living room’ experiencesBut much less linear

Events may be skipped, repeat, play in unpredictable orderPlayers expect same kind of realism as they get from non-interactive media


Audio Challenges on a Game Console: Technical

Finite ResourcesStorage space (not as much of an issue currently)Memory (actual playing sound data, audio library size, etc.)Processing power (realtime effects, mixing, etc.)

InteractivityAvoiding repetition is perhaps most importantSecondary goal: compelling non-linear audioNeed solutions that don’t require a lot of development time


Audio Challenges on a Game Console: Organizational

“Post-Production” mentality towards audio – scheduled late

Often appropriate in film, television

Often viewed in budgeting as expense rather than investment

Tradition & expectations for game audio - hasn’t been always been of highest quality (see previous slide)

No dedicated audio programming resourceOften up to composer/sound designer to understand and determine what’s possible and feasibleCross-platform audio solutionsSimplest-to-implement programmatic solutions used


Audio Role Comparison

Film/TV/RadioSound DesignerComposer/arranger(s)Orchestrator(s)Recording Engineer(s)Mixing/Mastering Engineer(s)Music SupervisorMusic EditorSFX EditorADR/Dialog EditorFoley Artist(s)Casting (for voices)

GamesSound Designer/Composer (maybe 2 separate people)Audio programmer (part time)


Hiring a Content CreatorGenerally ‘buy-out’ (work for hire)

Composer paid straight fee (per platform?)Developer/publisher generally owns all rights to musicMay or may not include cost of recording, musicians, mixing, etc.

May negotiate for publishingPer copy sold royalty (rare)Paid for TV/radio spotsRoyalty for thematic use in sequels, spinoffsOpportunity for game soundtrack CD


Budgeting for AudioMusical underscore

Live, synth, or mix? And how much?How does it start and stop? (linear, looping, branching)

Licensing existing musicSound Effects / Ambience

Live recordings or sample libraries?Creation of new modeled sounds

Sub-contractingOften essential for large projects


Laying out an Attack PlanLook at all ways system can make noiseWork with technical document to determine what’s preferable/do-able

E.g. MIDI or streamed (CD/DVD/HD) musicReal-time effects (reverb, filtering, etc.)What other game resources are being used

Maximize audio capabilities while minimizing impact on rest of game

Often different for different parts of game


Tools, Tools, Tools…

Evaluating for technical and aesthetic criteriaMake authoring and programming pipelines relatively independentMaintain creative flexibility

Linear audioAuthoring: Same as for other mediaProgramming: Cross-platform and platform specific options

Non-linear audio = generally proprietaryPC: DirectMusic or ‘in-house’ enginesConsoles: Platform-specific or ‘in-house’ solutions


Underscore ChallengesFilm

~90 minutes, often <60 minutes of scoreLinear, unchanging storyline

Game10+ hours of gameplayGameplay events may be quite variable (in order, duration, etc.)Don’t need “music” everywhere (but at least ambience)Score: Often linear tracks several minutes long (may loop)


Typical Underscore Solutions for Consoles

Create several long linear tracksMay loopMay be played ‘soundtrack’ style (no interaction with gameplay)

Save purely musical score for key one-time game points

Use ambience instead to set moodAvoids repetition of recognizable thematic material

Advanced: Non-linear audioAlternate versions with looping and branchingVarious intensity levels of music that crossfade based on game state


Licensed Music

Not appropriate for all gamesOften difficult to make interactive/dynamicDisconnects score from moment-to-moment gameplay

Can be effective in some situationsName recognition/cross-marketing potentialCan set titles up for user soundtrack support

Typical usesExisting brands/licenses (esp. movies and TV)Heavily used by racing games


Composing a Game ScoreSpotting session

Where can music support action?

Compose cues for linear sections of game (for instance, cutscenes)

Need final cut for accurate timings (hah hah)

Record and mixPre-rendered: Synth, live players, mixtureReal-time rendered: MIDI, individual pads, etc.

Implement playback in gameTriggers to start, stop, transition, etc.

Edit to match ‘final’ animations


Sound Effects ChallengesNo pre-existing soundtrack to work withSame sound event may be heard hundreds of times

VariationEnvironmental interaction

Tying FX to game physics and AISheer number of sound elements to create

Combinations and permutationsEverything that can make a noise should


Typical Sound Effects SolutionsVariation

Implement basic pitch randomizationSeveral versions of each waveBreak a sound into components and record versions of each of those

Tying effects to game physics / AIImplementation varies per-titleOften vast majority of dev time spent here

Advanced: Dynamic processed soundsEnvironmental effects (reverb)Filtering (force of impacts, radioizing, etc.)


Creating Ambience and Sound Effects

Spotting session (if game playable), or look at artwork (environments, objects)

List everything that can make noise, prioritizeObject interaction matrix

Often start with organic soundsSound effects library ($$$, can be generic)Create own samples (free, unique)

Sound processing (pre-rendered or dynamic)

Reverb, filtering, chorus, effects


Dialog Challenges

Casting – producer may want involvementDirecting / CoachingStitching (first | and | 10 | from the | 25)Dialog repetition is easily recognizedLocalization

Re-recording voice talentEngine integrationGrammatical differencesRecreating pre-rendered processing


Typical Dialog SolutionsTriggers for dialog (just like music)Variation for common lines (often too few)Stitching – coach the talentLocalization

Solutions vary per-titleVery careful asset managementSometimes source assets can just be swapped

Advanced: Sophisticated dialog engineVariation tracking to avoid repetitionPeople don’t always have to say anything


Prepping for DialogContent creator works with producer, game designer, and/or writer to:

List and describe charactersWrite each character’s scripts

Cast person for each characterReal voice talent makes a differenceMany actors can do multiple roles


Dialog ProductionRecord all dialog

Encourage improvisationCoaching for creating convincing stitchable dialog

Cut up dialog into separate filesClean up for stitching – sound designer for sports title can spend majority of time here

Post-recording processingDocument everything for localization


Squeezing it all into 5 bytes

Best to determine resources up frontMemory and CPU availableDisk space and bandwidth available

Inevitably, not enough roomAsset swapping (programming time)CompressionUse of streaming where possibleCut assets


And the Handoff…Source assets given to game

Typically high fidelity + compressed versions deliveredFreelance composer/sound designer: often end of work

Programmer integrates audio assets into titleOften tasked with mixing (either with or without content creator feedback)Implement runtime sound processing (reverb, effects, filtering, etc.) unless engine natively supports

Testers confirm sounds are in placeAsset/trigger lists, scripts, and/or manual triggersVariation testing?


After the Fact: More Challenges

Archive, archive, archiveCreate a recording of typical gameplay for referenceDisaster always strikes at the 11th hourKeep an organized collection of any sounds created (spreadsheet, database)

LocalizationVoice talent recasting, re-recordingRecreate processing (unless real-time used)

Releasing a CD


Audio Case Study:Project Gotham Racing

Stereo ambience, with some 3D positioned foreground ambient elementsPlayer’s car

Stereo engine ambienceEngine sound varies with RPM, velocity, physics, etc.Tire sound varies with surface (sidewalk, stairs, street, etc.)

3D positioning on other carsEngines (varies with RPM, velocity, physics)Car hornsCornering, tire squeals, collisions, etc. (all with some variation)

Dynamic soundsTire squeals (randomized between several versions and pitches)Collisions (depend on speed, material involved, etc.)


Project Gotham Racing: Music

Game disc-supplied licensed songs (WMA)User Soundtrack support

Allows for personalized driving music mixes

Music uniquely integrated into gameUse of ‘car radio’ or ‘CD’ playback‘Radio’: Location-specific DJ interstitials


Audio Case Study: Splinter CellDetailed ambiences, foley, sound effectsAudio as a gameplay element (stealth)

Stealth: Noise tied to game AIFootsteps, interaction with objects, weapons

Audio as an informational toolInteractive musical scoreTells player when enemies alertedUse of musically appropriate boundaries for transitions


Audio Case Study: Halo

4,311 triggerable events (soundtags) 11,629 source waves (soundfiles)2.5 GB uncompressed audio

22 kHz mono and 44 kHz stereo

80 minutes of source music20+ hours of gameplay

12 voice talents, 7,735 lines of dialog…all in 2-MB memory footprint


Halo: Audio Tools UsedPre-rendered linear music/FX = traditional standard audio post production tools

Pro Tools, Studio Vision, Peak, Digital Performer

Dynamic audio = custom tool/engine, randomization/permutation for each event

Pick one or more waves to playRandom variation in pitchWeighting for how often a sound will be used


Halo Audio Engine: DialogSimulation of actual speech patterns

Close connection to game AILarge number of events a character can respond to, multiple phrases per actionCharacters don’t always speak

Tracking speechMultiple voice talents for multiple charactersCharacter only says one thing at a timePositioning (3D versus locked-to-speaker)Repetition avoidance


Halo Audio Engine: Sound EffectsVariations in all sound effects

Weighting (frequency for multiple versions of sound)Randomized pitch control

Variations in ambienceVarious loops of different lengthsLayered 3D-positioned detail sounds (birds, crickets, machinery hum, etc.)

“Cascading” sound effectsIf many instances of an effect are playing, play a single more complex sound

Response to game environmentSpecific conditions/events/game physicsEnvironment interaction (reverb)3D spatial positioning versus static


Halo Audio Engine: MusicTypically linear or looping

If looped, alternate versions with varying lengthsDynamic execution (“spotted” with programmer)

Varying transition types based on situationBranchingCrossfadesGame event tags

Appropriate absence of musicSilence is golden - not scored wall to wallTimeouts


Halo 5.1 UsageSound effects typically 3D-positioned (4 channels)

Some ambience 3D-positioned as well

“Quad” music and ambienceDoubled stereo (fronts 100% / rears 50%)Crossfades based on location

Center channelCortana (“first person” dialog)Radio communication

LFE channelUsed to emphasize more massive sounds

Heartbeat (low health)Explosions


Halo Environmental Reverb Usage

Reverb settings changed based on environment

Indoors versus outsideConfined spaces versus large spacesCaves versus hallways

Sound interactions with reverbEspecially 3D-positioned sound effectsSome ambient elements also


Halo Implementation (1)

2-MB in-memory footprint22-kHz mono ADPCM for 3D sounds44-kHz stereo ADPCM for music + ambienceADPCM = ~3.5:1 compression

Use of Xbox hard diskCopy all sound data to drive at start of level (faster to read than DVD)Pull small chunks into memory rather than entire banks (pseudo-”streaming”)Drive persistence = faster load next time


Halo Implementation (2)

Sound during load screensSmall looped sampleMultiple instances played with pitch shifting and filtering as load progresses“Dynamic” – speed and actual sound vary with read time


Leading Edge Audio ChallengesVoice integration (esp. for online titles)

Chat and gameplay implicationsManipulation (masking, radioizing, etc.)

Voice recognitionCommand and control

Further multichannel audio usageFuller 5.1 implementationsDynamic audio mixing

Physical modelingStill too expensive (CPU, programming effort) for most games


Wrap Up:The State of Game Audio

Lots of independent composers/sound designersMany coming from outside game industryG.A.N.G. (www.audiogang.com)

Still typical for developers to reinvent audio engine with each title (newer tools try to solve)Xbox efforts

Xbox Registered Developer and Registered Content Creator ProgramsTools and dedicated support for audio content creators and programmersGame Designer/Producer education


Q & [email protected]

G.A.N.G. (www.audiogang.com)Game Audio Pro e-mail list (Yahoo)DirectMusic (www.microsoft.com/directx)2003 Game Developer Conference lectures

audio for games

Documents

audio library size

game consolescore

game soundtrack cdbudgeting

gamesaudio challenges

audioaudio challenges

sound data

whats possible

whats preferabledoablee