Linux Sound

Programming

Jan Newmarch

Apress*

Contents

JAbout the Author xxi

About the Technical Reviewer xxiii

Chapter 1: Basic Concepts of Sound 1

Sampled Audio 1

Sample Rate 1

Sample Format 2

Frames 2

Pulse-Code Modulation 2

Overrun and Underrun 3

Latency 3

Jitter 4

Mixing 5

Conclusion 5

Chapter 2: User-Level Tools.... 7

Players 7

MPIayer 7

VLC 7

Totem 8

V

CONTENTS

Sound Tools 8

sox 8

FFmpeg/avconv 9

GStreamer 9

Audacity 9

Conclusion 9

Chapter 3: Sound Codecs and File Formats.......... 11

Overview 11

PCM 11

WAV 12

MP3 12

OggVorbis 12

WMA 13

Matroska 13

Conclusion 14

Chapter 4: Overview of Linux Sound Architecture 15

Resources 15

Components 15

Device Drivers 16

Sound Servers 17

Complexities 18

Conclusion 19

Chapter 5: ALSA 21

Resources 21

User Space Tools 21

alsamixer 21

alsactl 22

speaker-test 22

aplay/arecord 23

Identifying ALSA Cards 23

vi

CONTENTS

Device Names 24

ALSA Configuration Files 27

alsa-info 27

Applications Using ALSA 32

Programming ALSA 33

Hardware Device Information 34

PCM Device Information 37

Configuration Space Information 40

ALSA Initialization 46

Capture Audio to a File 46

Playback Audio from a File 49

Using Interrupts 52

Managing Latency 52

Playback of Captured Sound 52

Mixing Audio 58

Mixing Using dmix 58

Mixing Using PulseAudio 59

Simple Mixer API: Volume Control 59

Writing an ALSA Device Driver 60

Conclusion 60

Chapter 6: PulseAudio 61

Resources 61

Starting, Stopping, and Pausing PulseAudio 61

User Space Tools 62

paman 62

pavumeter 64

pavucontrol 64

Gnome Control Center (Sound) 66

parec/paplay/pacat 67

pactl/pacmd..... 68

Device Names 69

vii

CONTENTS

Loopback Module 69

PulseAudio andALSA 70

Programming with PulseAudio 70

Simple API 70

Play a File 71

Record to a File 73

Play from Source to Sink 76

Asynchronous API 78

List of Devices 79

Monitoring Ongoing Changes: New Sources and Sinks 84

Record a Stream 89

Play a File 93

Play a File Using I/O Callbacks 101

Controlling Latency 112

Play Microphone to Speaker 120

Setting the Volume on Devices 126

Listing Clients 128

Listing Client Sources and Sinks 133

Controlling the Volume of a Sink Client 138

Conclusion 141

Chapter 7: Jack 143

Resources 143

Starting Jack 143

User Tools 146

Applications Using Jack 146

mplayer 146

VLC 146

TiMidity 147

Jack-Supplied Programs 147

Other Jack Programs 147

viii

CONTENTS

Using a Different Sound Card 147

How Can I Use Multiple Sound Cards with Jack? 148

Mixing Audio 148

Writing Audio Applications with Jack 149

Compiling 150

Port Information 151

Copy Input to Output 153

Delaying Audio 157

Audacity with Jack 163

Play a Sine Wave 165

Saving Input to Disk 170

Interacting with ALSA Devices 177

Conclusion 177

Chapter 8: Session Management ...179

Resources 179

Session Management Issues 179

jack_connect 181

LASH 181

Jack Sessions 181

LADISH 184

Jack Session API 189

LADISH API 196

Conclusion 196

Chapter 9: Java Sound 197

Resources 197

Key Java Sound Classes 197

Information About Devices 198

Playing Audio from a File 199

ix

CONTENTS

Recording Audio to a File 202

Play Microphone to Speaker 206

Where Does JavaSound Get Its Devices From? 210

Conclusion 210

Chapter 10: GStreamer 211

Resources 211

Overview 211

Command-Line Processing 213

gst-inspect 213

gst-discoverer 215

gst-device-monitor 216

gst-play 216

gst-launch 217

C Programming 217

Writing Plug-ins 221

Conclusion 221

Chapter 11: libao 223

Resources 223

libao 223

Conclusion 225

Chapter 12: FFmpeg/Libav 227

Resources 227

The FFmpeg/Libav Controversy 227

FFmpeg Command-Line Tools 228

Programming 228

Decoding an MP3 File 229

Conclusion 234

X

CONTENTS

Chapter 13: OpenMAX IL 235

Resources 235

Quotes 236

OpenMAX IL Concepts 236

OpenMAX IL Components 237

Implementations 237

Raspberry Pi 237

Bellagio 238

LIM 238

Hardware-Supported Versions 239

Implementations of Components 239

Getting Information About an IL Component 244

Playing PCM Audio Files 260

State 260

Threads 260

Hungarian Notation in OpenMAX IL 262

Callbacks 262

Component Resources 262

Setting the Output Device 265

Main Loop 265

Complete Program 266

Conclusion 275

Chapter 14: LADSPA 277

Resources 277

User-Level Tools 277

The Type LADSPA_Descriptor 279

Loading a Plug-in 280

A Mono Amplifier Client 280

A Stereo Amplifer with GUI 286

xi

CONTENTS

The amp Program 291

Conclusion 292

Chapter 15: Displaying Video with Overlays Using Gtk and FFmpeg 293

FFmpeg 293

Basic Gtk 297

Versions of Gtk 299

Displaying the Video Using Gtk 300

Pixmaps 300

Pixbufs 300

X,Wayland, and Mir 300

Gtk 3.0 301

Gtk 2.0 301

Threads and Gtk 302

The Code 302

Overlaying an Image on Top of an Image 306

Alpha Channel 308

Using Cairo to Draw on an Image 309

Drawing Text Using Pango 312

Conclusion 315

Chapter 16: MIDI 317

Resources 317

Components of a MIDI System 317

Synthesizer 317

Sequencers 318

Other Components 318

MIDI Events 318

Program Change Events 318

Note Events 318

Meta Events 318

Conclusion 319

xii

CONTENTS

Chapter 17: User-Level Tools for MIDI 321

Resources 321

Sound Fonts 321

TiMidity 322

TiMidity as a Server 322

Setting TiMidity Output Device 323

TiMidity and Jack 323

GStreamer 323

fluidsynth 324

fluidsynth as a Server 324

Rosegarden 324

WildMIDI 324

Comparison 324

VLC 325

Conclusion 325

Chapter 18: MIDI Java Sound 327

Resources 327

Key Java Sound MIDI Classes 327

Device Information 328

Dumping a MIDI File 330

Playing a MIDI File 334

Changing the Soundbank 340

Changing Pitch and Speed 340

Using TiMidity Instead of the Default Gervill Synthesizer 342

Conclusion 342

Chapter 19: MIDI ALSA 343

Resources 343

ALSA Sequencer Clients 343

aconnect 344

xiii

CONTENTS

seqdemo 345

aplaymidi 346

Raw MIDI Ports 347

Raw MIDI Physical Devices 347

Raw MIDI Virtual Devices 347

Mapping MIDI Clients into MIDI Raw Space 348

Turning Off All Notes 349

Conclusion 349

Chapter 20: FluidSynth 351

Resources 351

Players 351

Play MIDI Files 352

Python 353

Conclusion 353

Chapter 21: TiMidity 355

TiMidity Design 355

Turning TiMidity into a Library 356

Managed Environments Hook 356

Building the Library 356

Library Entry Points 357

A Minimal Application 357

Playing a Background Video to a MIDI File 358

Building a New Interface 360

Shared Objects 360

Entry Point 360

ControlMode 361

Include Files 362

My Simple Interface 362

xiv

CONTENTS

Running My Simple Interface 367

Playing a Background Video to a MIDI File 367

Conclusion 368

Chapter 22: Overview of Karaoke Systems 369

Chapter 23: Karaoke User-Level Tools 371

Video CD Systems 371

CD+G Discs 372

MP3+G Files 372

Buying CD+G or MP3+G Files 373

Converting MP3+G to Video Files 373

MPEG-4 Files 373

Karaoke Machines 374

MIDI Players... 375

Finding MIDI Files 375

KAR File Format 376

PyKaraoke 377

kmid3 377

Microphone Inputs and Reverb Effects 379

Conclusion 379

Chapter 24: MP3+G 381

File Organization 384

Song Information 384

Song Table 386

Favorites 390

All Favorites 393

Swing Song Table 395

Playing Songs 401

XV

CONTENTS

VLC402

Playing Songs Across the Network 402

Conclusion 405

Chapter 25: Karaoke Applications Using Java Sound 407

Resources 408

KaraokePlayer 408

MidiPlayer 409

DisplayReceiver 413

MidiGUI 415

AttributedLyricPanel 420

PianoPanel 423

MelodyPanel 425

Sequencelnformation 430

PinYin 434

Karaoke Player with Sampling 435

Comments on Device Choices 441

Performance 441

Conclusion 441

Chapter 26: Subtitles and Closed Captions 443

Resources 443

Subtitle Formats 443

MPIayer 443

VLC 444

Gnome Subtitles 444

Substation Alpha 445

Karaoke Effects in ASS Files 446

Multiline Karaoke 449

libass 451

Converting KAR Files to MKV Files with ASS Subtitles 451

xvi

CONTENTS

HTML5 Subtitles 463

Conclusion 464

Chapter 27: Karaoke FluidSynth 465

Resources 465

Players 465

Play MIDI Files 465

Extending FluidSynth with Callbacks 467

Displaying and Coloring Text with Gtk 470

Playing a Background Video with Gtk 482

Conclusion 497

Chapter 28: TiMidity and Karaoke 499

TiMidity and Jack 500

TiMidity Interface 502

Getting the List of Lyrics 504

TiMidity Options 505

Playing Lyrics Using Pango + Cairo + Xlib 506

Playing a Background Video with Gtk 512

Background Video with TiMidity as Library 519

Background Video with TiMidity as Front End 521

Adding Microphone Input 521

Conclusion 521

Chapter 29: Jack and Karaoke 523

Using Jack Rack for Effects 523

Playing MIDI 526

TiMidity Plus Jack Rack 527

Customizing TiMidity Build 528

Playing MP3+G with Jack Rack Pitch Shifting 528

Conclusion 532

xvii

CONTENTS

Chapter 30: Streaming Audio 533

HTTP 533

HTTP Servers 533

HTTP Clients 533

Streaming vs. Downloading 534

HTML5 534

DLNA 535

Icecast 535

Flumotion 535

Conclusion 535

Chapter 31: Raspberry Pi 537

Resources 537

The Basics 537

Hardware 537

Alternative Single-Board Computers 538

Distros 539

No Sound 539

ALSA 539

Sampled Audio Players 541

MPIayer 541

VLC 541

alsaplayer 541

omxplayer 541

Is It X Using the CPU? 541

Sampled Audio Capture 541

ALSA 541

MIDI Players 542

TiMidity 542

pykaraoke 542

FluidSynth/qsynth 542

xviii

CONTENTS

Java Sound 543

PulseAudio 544

Java MIDI 545

OpenMAX 545

Conclusion 545

aChapter 32: Conclusion 547

Where Did I Start? 547

Where Did I Get To? 547

How Did I Get There? 547

Appendix A: Decoding the DKD Files on the Sonken Karaoke DVD 549

Introduction 549

Format Shifting 550

Files on the DVD 550

BACK01.MPG 551

DTSMUSOO.DKD to DTSMUS07.DKD 551

DTSMUS10.DKD 551

DTSMUS20.DKD 551

Decoding DTSMUS20.DKD 551

Song Information 551

Beginning/End of Data 553

Chinese Songs 553

Other Languages 554

Programs 554

The Data Files 568

General 568

My Route into This 569

The Superblock 570

Song Start Tables 571

Locating Song Entry from Song Number 571

xix

CONTENTS

Song Entries 571

Song Data 572

Decoding MIDI Files 572

Lyric Block 572

Playing MIDI Files 599

Playing WMA Files 599

KAR Format 600

Playing Songs with pykar 600

Tempo 600

Language Encoding 600

Songs with No Notes 601

Conclusion 601

Index 603

XX