EMERGING WEB TECHNOLOGIES AND SONIC ARTS

Lonce WyseArts and Creativity Lab

National University of Singapore

Net-Music 2013: The Internet as Creative Resource in Music

The Web Platform Some new emerging standards from the

World Wide Web Consortium What’s so exciting about standards? Functionality that becomes standard in the

browser Becomes cross platform Runs natively Requires no plug-in (end user effort)

Related technology developments

WebAudio API

Native audio engine in browsers with API called through javascript.

“Node” / connection flow-graph construction model

Create “Nodes” Osc.connect(Filter); Filter.connect(Conext.destination)

WebAudio Features Sample-accurate scheduling for built-in

components (nodes, parameter automation) Spatialization (sound cones, obstruction, oriented

“listeners”) Buffer Nodes, Gain Nodes Oscillators, Delay, Channel management,

Common Filter types, compression, waveshaping Convolution, Time and frequency analysis (Predefined) Parameters with automation

Outstanding Issues “Roll your own” ScriptProcessorNodes for

generation and processing of audio. Not treated the same as system Audio Nodes

Timing latency and jitter Can’t attach WebAudio Paramters to them.

Callbacks from audio processing thread

One API fits all (sound designers – sound users)

Future (ever-increasing number of node types?)

Related developments WebRTC (also W3C emerging standard)

Oriented toward P2P media movement (eg video chat). Access to user media – audio and video

No more plug-ins!

Plays nicely with WebAuido. Also running on some mobile browsers

javascript

First introduced in 1995 to add short interpreted scripts in HTML for dynamic elements – it has come a long way in a short time.

Google V8 engine (chrome, safari, node.js) brought huge increases in speed.

Imag

e: h

ttp://

sss.c

s.pur

due.

edu/

proj

ects

/dyn

js/

Excellent browser debugging tools.

Libraries – thousands upon thousands. Phone sensor data, large project support.

Still some issues that make it a less-than-ideal educational tool.

Server side applications

Javascript on Google V8 virtual machine

Single-threaded, asynchronous event handling

Works with socket.io (a wrapper for web sockets) for real-time apps with two-way open communications.

Now one language for network, clients, app, sound synthesis….

for network apps

Current Projects jsaSound

Sound development tools

Anticipatory Score Shared graphical scrolling score paradigm Network control signals with node.js server

Augmented Storytelling Inter-browser communication for synthesis and control

jsaSound Library for creating WebAudio sound models.

Building tools Simple and consistent API

play(), stop(), release(), setParam(), setParamNorm(), getParam(["name", "type", "val", "normval", "min","max"]

In action: animatedsoundworks.com/jsaSound* Open source

https://github.com/lonce/jsaSound.git

[* Tested with Canary Browser]

Synth1 Patch

Synth2 Patch

Server Patch

Synth2 Patch

Synth1 Patch

Server Patch

Node.js Server

NOW

FUTURE

FUTURE

NOW

LOCATION 1

LOCATION 2

FUTURE NOW HISTORY

FUTURE NOW HISTORY

SHARED SCROLLING SCORE WORKSPACE

Anticipatory Scores

Storytelling and oral traditions Putting sound interaction and voice

transformation in to the hands of the storyteller

Oral traditions: Shareabiltiy Mutability

Story Telling & SharingSound Server* Controller

Synthesis

[* Tested with Canary Browser]

Just another audio environment? And a much less than perfect one, at that. But … Creative possibilities

The integration with web brings Accessibility (no apps or plug-ins to download) Cross platform (or will our hearts be broken again…) Possibilities for massive participation Access to unprecedented data flows to integrate into

artworks Dynamic and programmable at all levels, not just audio

… and it even says right in the WebAudio specification:

Artistic Audio ExplorationThere are many creative possibilities for artistic sonic environments for installation pieces.

Resources W3C Web Audio draft spec

https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html Working grouop mailing list public-audio@w3.org

HTML5 Audio Blog http://www.html5audio.org/

Node.js http://nodejs.org/