open standards for 3d and ar on the web - khronos group · open standards for 3d and ar on the web...
Post on 02-Apr-2018
219 Views
Preview:
TRANSCRIPT
© Copyright Khronos Group 2017 - Page 1
Open Standards for
3D and AR on the WebNeil Trevett | Khronos President
NVIDIA VP Developer Ecosystem
ntrevett@nvidia.com | @neilt3d
AWE, May 2017
© Copyright Khronos Group 2017 - Page 2
Khronos Mission
Software
Silicon
Khronos is an International Industry Consortium of over 100 companies creating
royalty-free, open standard APIs to enable software to access hardware acceleration for
3D graphics, Virtual and Augmented Reality, Parallel Computing,
Neural Networks and Vision Processing
© Copyright Khronos Group 2017 - Page 3
Khronos Standards Ecosystem
VR, Vision, Neural Networks- VR/AR system portability
- Tracking and odometry
- Scene analysis/understanding
- Neural Network inferencing
Real-time 2D/3D- Cross-platform gaming and UI
- VR and AR Displays
- CAD and Product Design
- Safety-critical displays
3D for the Web- Real-time apps and games in-browser
- Efficiently delivering runtime 3D assets
Parallel Computation- Machine Learning acceleration
- Embedded vision processing
- High Performance Computing (HPC)
© Copyright Khronos Group 2017 - Page 4
Augmented Reality - Native Acceleration
Vision
sensor(s)
Tracking and
Positioning
Geometric scene
reconstruction
Semantic scene
understanding
(Neural Networks)
Generate Low Latency
3D Augmentations for
display by optical system
Interact with sensor, haptic
and display devices
Download 3D
augmentation object
and scene data
AR Application
How can we bring
ACCELERATED AR to
the Web?
© Copyright Khronos Group 2017 - Page 5
glTF – Cross-Platform 3D Assets
glTF – OpenGL Transmission FormatEfficient transmission of 3D assets
Laugh Engine running on Vulkanhttps://github.com/jian-ru/laugh_engine
Now glTF 2.0 has PBR!Cool, portable materials
Rendering API independence
Coming to Microsoft Office!
glTF enables portable,
efficient, sophisticated 3D
assets for all 3D applications
© Copyright Khronos Group 2017 - Page 6
Khronos 3D APIs already
powering VR rendering
The standard for desktop VR today
On millions of mobile VR devices
Powering WebVR in browsers
The future of explicit, low-latency VR
Khronos APIs for Virtual Reality
But What About
INPUT Hardware for AR/VR..Device discovery
Multiple sensor tracking
Device Events
Haptics
Parameters for optics corrections
etc. etc…
OpenXR!Cross-Platform, portable AR/VR
© Copyright Khronos Group 2017 - Page 7
OpenXR – Solving AR/VR Fragmentation
Proprietary
Engine
VR App
1
VR App
2
VRApp
4
VRApp
3
VR Device
1
VR Device
2
VR Device
3
VR Device
5
VR Device
4
After OpenXR Wide interoperabilityof
VR apps and devices
VR App
1
VR App
2
VRApp
4
VRApp
3
Proprietary
Engine
VR Device
1
VR Device
3
VR Device
5
VR Device
2
VR Device
4
Before OpenXRVR Market
Fragmentation
Device Layer
Application Interface
© Copyright Khronos Group 2017 - Page 8
OpenXR Working Group Members
Design work started in December 2016
Typically 12-18 months to develop a V1.0 specification
© Copyright Khronos Group 2017 - Page 10
Pervasive WebGL 1.0• WebGL on EVERY major desktop and mobile browser
http://caniuse.com/#feat=webgl
© Copyright Khronos Group 2017 - Page 11
WebGL 2.0 Spec• Enhanced visual quality, performance, features
- Instancing Multiple render targets| Uniform buffers | Transform feedback
- Multisampled Renderbuffers| 3D textures | NPOT textures
- More texture formats | Occlusion queries | Vertex array objects
- Sampler objects | Sync objects | Fragment depth | Primitive restart | …
• WebGL 2.0 now available in Chrome/Firefox!
- Chrome: Releasing to desktop and Android
- Firefox: Released on all platforms
- Edge/Safari: plan to ship WebGL 2.0
• Desktop support at 60% - rising fast http://webglstats.com/webgl2
WebVR Running Over WebGL 2.0
© Copyright Khronos Group 2017 - Page 12
WebGL Timeline
2003
1.02004
1.12007
2.02012
3.02014
3.1
DriverUpdate
SiliconUpdate
SiliconUpdate
DriverUpdate
Compute Shaders
32-bit integers and floats
NPOT, 3D/depth textures
Texture arrays
Multiple Render Targets
Programmable
Vertex and
fragment shadersFixed function
Pipeline
2015
3.2
SiliconUpdate
Tessellation and geometry shaders
ASTC Texture Compression
Floating point render targets
Debug and robustness for security
Epic’s Rivalry demo using full Unreal Engine 4on mobile
https://www.youtube.com/watch?v=jRr-G95GdaM
2011
WebGL 1.0
GDC 2017!
WebGL 2.0
Conformance Testing is vital for
Cross-Platform ReliabilityWebGL 2.0 conformance tests are very thorough
10x more tests than WebGL 1.0 tests
NL +
Possible Compute Shader Extension
‘WebGL Next’
© Copyright Khronos Group 2017 - Page 13
Vulkan Explicit GPU Control
GPU
High-level Driver
AbstractionLayered GPU Control
Context managementMemory allocation
Full GLSL compiler
Error detection
ApplicationSingle thread per context
GPU
Thin DriverExplicit GPU Control
ApplicationMemory allocation
Thread managementSynchronization
Multi-threaded generation of command buffers
Multiple Front-end
CompilersGLSL, HLSL etc.
Loadable debug and
validation layers
Vulkan 1.0 provides access to
OpenGL ES 3.1 / OpenGL 4.X-class GPU functionality
but with increased performance and flexibility
SPIR-V pre-compiled shaders
Complex drivers
cause overhead
and inconsistent
behavior across
vendors
Always active
error handling
Full GLSL
preprocessor and
compiler in
driver
OpenGL vs.
OpenGL ES
Resource management
offloaded to app:
low-overhead, low-latency
driver
Consistent behavior:
no ‘fighting with driver
heuristics’
Validation and debug layers
loaded only when needed
SPIR-V intermediate
language: shading language
flexibility
Multi-threaded command
creation. Multiple graphics,
command and DMA queues
Unified API across all platforms with feature set
flexibility
Vulkan = high performance
and low latency 3D – Ideal
for VR/AR applications
© Copyright Khronos Group 2017 - Page 14
Vulkan – Portable New Generation GPU API
Cross Platform
7
All Major GPU Companies shipping Vulkan Drivers – for Desktop and Mobile Platforms
Mobile, Embedded and Console Platforms Supporting Vulkan
Embedded LinuxAndroid TVAndroid 7.0 Nintendo Switch
Vulkan support
in Unity 5.6
Vulkan support in V5.4
Late May 2017
DOTA 2 on Vulkan
port of Source 2
DOOM shipping on
Vulkan port of id Tech 6
Initial Vulkan mobile
renderer in 4.12
Vulkan support
since V1.8
© Copyright Khronos Group 2017 - Page 15
WebGL and WASM (Web Assembly)
C/C++ Code with
Native API calls
Emscripten
Clang/LLVM
JVM
WASM Modules
+ JS Glue Code
JavaScript
JavaScript
calls WASM
Modules
WASM modules
access Web APIs
(like WebGL)
through JS call-outs
For calling Web APIs in WASM Modules there needs to be: 1) a native API for WASM source code
2) a JS API for run-time invocation
3) a simple mapping between them to minimize JS Glue Code
JavaScript WebAPI
WebGL Next will need:1) a portable native APIs that can run on any system
2) a JS binding of that API for use via JavaScript
3) a simple mapping between the two
Mozilla’s Obsidian WebGL Next Proposal: 1) defines as Vulkan Subset as a portable native API
2) binds the Vulkan subset to JavaScript
3) for a simple mapping between the two
© Copyright Khronos Group 2017 - Page 16
Vulkan Portability – Call for Participation
‘WebGL Next’- Lift ‘Portability API’ to
JavaScript and use in
WebAssembly native code
- Nexgen graphics and
GPU compute for the Web
API
Overlap
Analysis
Portable ‘Vulkan
Subset’ API
Specification
A Portability Solution needs to address
APIs and shading languages
Open source compilers/translators for
shading and intermediate languages
MIR
DX IL
GLSL
MSL
HLSL
Vulkan Portability SolutionC/C++ Portability API Library
+ Shading Language tools
All open source
Use Feature
Sets to remove
non-portable
functionality
Use Extensions
to add
functionality
e.g. security and
robustness for
the Web
Vulkan is non-proprietary and is
already designed to be portable
No single API
on all systems
© Copyright Khronos Group 2017 - Page 17
OpenVX - Portable, Efficient Vision Processing• OpenVX developers express a graph of vision and Neural Network operations (‘Nodes’)
- Using a C API
• Graphs isolate the programmer from hardware and enable significant optimizations
- Implementers can optimize under the high-level graph abstraction
• Nodes can be executed on any hardware or processor coded in any language
- Including Vulkan or WebGL Next Compute Shaders
VisionNode
VisionNode
VisionNodeCNN Nodes
OpenVX graph mixing CNN nodes with traditional vision nodes
‘WebGL Next’
‘WebVX’??
Possible Native StackC API to Build Graph
Nodes accelerated by Vulkan
Potential Web Stack??JS API to Build Graph
Nodes accelerated by WebGL Next
© Copyright Khronos Group 2017 - Page 18
Augmented Reality - Web Acceleration
Vision
sensor(s)
Tracking and
Positioning
Geometric scene
reconstruction
Semantic scene
understanding
(Neural Networks)
Generate Low Latency
3D Augmentations for
display by optical system
Interact with sensor, haptic
and display devices
Download 3D
augmentation object
and scene data
AR Application
‘WebGL Next’
‘WebVX’
‘WebGL Next’
© Copyright Khronos Group 2017 - Page 19
Please Get Involved!• Information on all these standards at Khronos
- www.khronos.org
• Khronos is driving to new levels of community engagement
- Specifications and resources increasingly open for industry input and feedback
• Any company or organization is welcome to join Khronos
- For a voice and a vote in any of these standards
• If joining is not possible – ask about a Khronos Advisory Panel
- Free of charge – enables design reviews, requirements and contributions
• Neil Trevett
- ntrevett@nvidia.com
- @neilt3d
top related