immersive audio for vr - world wide web consortium · • vr audio is more than head-tracked...
TRANSCRIPT
![Page 1: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/1.jpg)
© 2015 DOLBY LABORATORIES, INC.
Immersive Audio for VR
Nicolas Tsingos, Mike Assenti
Dolby Laboratories
![Page 2: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/2.jpg)
© 2015 DOLBY LABORATORIES, INC.
VR landscape
![Page 3: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/3.jpg)
© 2015 DOLBY LABORATORIES, INC.
Soundfield and object-based representations
packaged productchannel-based, PCM only
ingredients+instructionsPCM+metadata
![Page 4: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/4.jpg)
© 2015 DOLBY LABORATORIES, INC.
Soundfield and object-based representations
Direct capture Full artistic mix
![Page 5: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/5.jpg)
© 2015 DOLBY LABORATORIES, INC.
5
Be cool, honey bunny
![Page 6: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/6.jpg)
© 2015 DOLBY LABORATORIES, INC.
6
Strength In Numbers
![Page 7: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/7.jpg)
© 2015 DOLBY LABORATORIES, INC.
7
Live and Let Die
![Page 8: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/8.jpg)
© 2015 DOLBY LABORATORIES, INC.
Mixing workflow for post-produced VR
![Page 9: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/9.jpg)
© 2015 DOLBY LABORATORIES, INC.
Live VR workflows
• Mixing for multiple viewpoints
![Page 10: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/10.jpg)
© 2015 DOLBY LABORATORIES, INC.
![Page 11: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/11.jpg)
© 2015 DOLBY LABORATORIES, INC.
Artistic controls for VR audio• Choose how each element behaves– Diegetic vs. non-diegetic; Effects, Dialog, Music, etc.– Headtracked vs. non-headtracked– Rendering modes (stereo, binaural)
• Binaural processing impacts timbre– Stereo can be a good thing
• Interactivity – Gaze-based activation or emphasis
![Page 12: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/12.jpg)
© 2015 DOLBY LABORATORIES, INC.
Panning interfaces for VR
• Matching the video/interaction space– Cartesian coordinate automation is
good for room reference
![Page 13: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/13.jpg)
© 2015 DOLBY LABORATORIES, INC.
Panning interfaces for VR
• Matching the video/interaction space– Cartesian coordinate automation is
good for room reference
![Page 14: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/14.jpg)
© 2015 DOLBY LABORATORIES, INC.
Automated panning
![Page 15: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/15.jpg)
© 2015 DOLBY LABORATORIES, INC.
Mixing VR in VR ?
![Page 16: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/16.jpg)
© 2015 DOLBY LABORATORIES, INC.
![Page 17: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/17.jpg)
© 2015 DOLBY LABORATORIES, INC.
![Page 18: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/18.jpg)
© 2015 DOLBY LABORATORIES, INC.
VR Object Audio on the Web
• Web Audio v1 is incredibly powerful…
• …but it’s missing some integration with delivery mechanisms
• Consider a Linear content stream– 360 Video bitstream
– Object Audio bitstream
• How do we actually render this packaged object audio?
![Page 19: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/19.jpg)
© 2015 DOLBY LABORATORIES, INC.
Mixing workflow for post-produced VR
![Page 20: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/20.jpg)
© 2015 DOLBY LABORATORIES, INC.
Components of VR Object Audio Playback
Decoder RendererBitstream
Audio Objects
Metadata
PlaybackConfiguration
Rendered Audio
Orientation& Position
![Page 21: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/21.jpg)
© 2015 DOLBY LABORATORIES, INC.
Render at decoder
Decoder Renderer
Bitstream
Metadata
PlaybackConfiguration
Rendered Audio
Orientation& Position
JavaScript
Native
Audio Objects
![Page 22: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/22.jpg)
© 2015 DOLBY LABORATORIES, INC.
Render in Web Audio
Decoder Renderer
Bitstream
Metadata
PlaybackConfiguration
Rendered Audio
Orientation& Position
JavaScript
Native
Audio Objects
![Page 23: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/23.jpg)
© 2015 DOLBY LABORATORIES, INC.
Conclusions
• VR audio is more than head-tracked spatialization over headphones
• Physically-based audio is great but audio production is also art
![Page 24: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/24.jpg)
© 2015 DOLBY LABORATORIES, INC.
Conclusions
• VR audio is more than head-tracked spatialization over headphones
• Physically-based audio is great but audio production is also art
• Ego-centric, single-point, audio capture and delivery can be limiting.
• Creating compelling VR experiences requires realism , NOT reality
![Page 25: Immersive Audio for VR - World Wide Web Consortium · • VR audio is more than head-tracked spatialization over headphones • Physically-based audio is great but audio production](https://reader034.vdocuments.net/reader034/viewer/2022042400/5f0f4f687e708231d44386fb/html5/thumbnails/25.jpg)
© 2015 DOLBY LABORATORIES, INC.
Conclusions
• VR audio is more than head-tracked spatialization over headphones
• Physically-based audio is great but audio production is also art
• Ego-centric, single-point, audio capture and delivery can be limiting.
• Creating compelling VR experiences requires realism , NOT reality
• Web Audio v1 gets us very close to streaming delivery of linear VR
• Looking forward to making further steps with v.next!