xflow - web3d2012.web3d.org declarative data processing for the web felix klein, kristian sons,...

35
Xflow Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek 1

Upload: dangnhi

Post on 23-May-2018

224 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Xflow Declarative Data Processing

for the Web

Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov,

Stefan John, Philipp Slusallek

1

Page 2: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

3

Motivation Design Implementation Results

Motivation

Page 3: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

3D applications on the Web

Web evolved into a full-fledged

application platform

3D graphics HW support is

now everywhere (incl. mobile)

But still hard to create 3D

application for the Web

WebGL is too low-level for web

developers!

4

Motivation Design Implementation Results

We need declarative 3D graphics for the Web

Page 4: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

XML3D

3D Scene Graph inside of

HTML document

Maximize reuse of Web

technology (DOM integration)

Keep generic to match

modern graphics hardware

Limitation: animations

Modify transformations via JS

for rigid body animation

But what about more

complex mesh animations?

5

Motivation Design Implementation Results

HTML Document

xml3d

group

mesh

defs

light

shader

lightshader

group group

data data data

Page 5: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

XML3D + Mesh Animation

Want to support mesh

deformation methods

Skinning

Morphing

High-performance data

processing tasks

Should be applied

somewhere between mesh

and data source.

6

Motivation Design Implementation Results

mesh

light

shader

lightshader

data

data data

Apply mesh

animation here.

Page 6: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Options for data processing

on the Web

JavaScript

To slow for many use-cases

Low-level API (WebCL, GLSL)

Flexible, high entry barrier for Web developers

Web Workers

Thread-Level Parallelism

River Trail

Data-Level Parallelism with JS, internally OpenCL

All these options are imperative

7

Motivation Design Implementation Results

Page 7: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Advantages of declarative

approach

8

Motivation Design Implementation Results

Part of the Web document

Can take advantage of all Web technology

Most intuitive for web developers

Can be accessed via DOM

Analyze data processing for visualization

Modify data processing via JS at run time

HTML Document

xml3d

group

mesh

defs

light

shader

lightshader

group group

data data data

Data

Processing

Page 8: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Xflow Declarative Data Processing

9

Motivation Design Implementation Results

Declare dataflows to describe data modifications

Similar to other approaches

X3D, SVG Filter Effects

Improved dataflow design

Efficiency

Flexibility

Usability

Match modern hardware, parallelized

No domain-specific functionality

Ease of use for Web developer

Page 9: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

10

Motivation Design Implementation Results

Design

Page 10: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Xflow as part of XML3D Declared in HTML

document

Set of nodes to declare dataflow Data input

Operations on data

Connected to sink nodes <mesh>, <shader>,

<lightshader>

11

Design Overview

Motivation Design Implementation Results

HTML Document

xml3d

group

mesh

defs

shader

group group

data data

data data

mesh

data

data Xflow

Page 11: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Data Input Elements

12

Generic data types

Each data element has a name

Declare sequence of input elements

Useful for key frame data

Motivation Design Implementation Results

float

float2

float3

float4

float4x4

float3x3

int

bool

texture

Code

<data id="baseData"> <float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="normal" >0 1 0 ...</float3>

<float2 name="texcoord" >0.0 0.0 ...</float2>

</data>

<float3 name="translation" seqnr=”0.0”>0 0 0 ...</float3>

<float3 name="translation" seqnr=”1.2”>0 3 0 ...</float3>

<float3 name="translation" seqnr=”2.0”>0 6 0 ...</float3>

Page 12: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Data Compositing

14

Combine data from nested elements

Compositing rules

Values with the same name are

overridden

Values can be renamed to avoid

conflict

Motivation Design Implementation Results

Code

<data id="dataA"> <int name="index" >0 1 2 ...</int>

<float2 name="texcoord" >1.0 0.5 ..</float2>

<data id="baseData"> <float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="normal" >0 1 0 ...</float3>

<float2 name="texcoord" >0.0 0.0 ...</float2>

</data> </data>

float3 position

float3 normal

float2 texcoord

#baseData

float3 position

float3 normal

float2 texcoord

#dataA int index

int index

float3 position

float3 normal

float2 texcoord

float2 texcoord

Page 13: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Data References & Reuse

15

Reference whole <data> element via src

attribute

Still possible to modify reused data by

adding / replacing individual fields

Motivation Design Implementation Results

#dataC

int index

float3 position

float3 normal

float2 texcoord

int index

Code

<data id="baseData"> <float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="normal" >0 1 0 ...</float3>

<float2 name="texcoord" >0.0 0.0 ...</float2>

</data>

<data id=”dataB” src="#baseData" />

<data id="dataC" > <data src="#baseData" /> <int name="index" >10 11 12 ...</int>

</data>

float3 position

float3 normal

float2 texcoord

#baseData

float3 position

float3 normal

float2 texcoord

#dataB

float3 position

float3 normal

float2 texcoord

Page 14: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Data Processing

16

Apply operator on <data> via compute attribute

Assign argument via list or name-map

Result of operators is added to result set of <data>, overriding values by name

Motivation Design Implementation Results

float3 position float3 posAdd2

float2 weight2 float3 posAdd1

float2 weight1

float3 posAdd2

float2 weight2

xflow.morph

float3 position

float3 posAdd1

float weight1

float3 position

xflow.morph

float3 posAdd1

float2 weight1

<data compute="position = xflow.morph({

pos: position,

posAdd: posAdd2,

weight: weight2})" >

<float3 name="posAdd2" >1.0 0.0 0.0 ...</float3>

<float name="weight2" >0.6</float>

<data compute="position = xflow.morph(

position, posAdd1, weight1)" >

<float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="posAdd1" >0.0 1.0 2.0 ...</float3>

<float name="weight1" >0.35</float>

</data> </data>

Code

Page 15: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Small & Generic Operators

17

Avoid too specific operators:

Motivation Design Implementation Results

float3 position

xflow.skinning float3 position

float3 normal

float3 normal

Int4 boneIndices

float4 boneWeights

float4x4 pose

float3 position

xflow.skinPosition

Better: small generic operators:

float3 position

Int4 boneIndices

float4 boneWeights

float4x4 pose Int4 boneIndices

float4 boneWeights

float4x4 pose

xflow.skinDirection

float3 normal

float3 position

Int4 boneIndices

float4 boneWeights

float4x4 pose

float3 normal

xflow.skinDirection

float3 tangent

float3 normal

float3 position

Int4 boneIndices

float4 boneWeights

float4x4 pose

float3 tangent

Int4 boneIndices

float4 boneWeights

float4x4 pose

Flexibility

Only works for

position and

normal

Page 16: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

<proto id="doubleMorph"

compute="position = xflow.morph(position, posAdd2,w1)" > <data compute="position = xflow.morph(pos, posAdd1, w2)" > <float3 param="true" name="position" ></float3> <float3 param="true" name="posAdd1" ></float3> <float3 param="true" name="posAdd2" ></float3> <float param="true" name="w1" ></float> <float param="true" name="w2" ></float> </data> </proto>

<data id="instanceA" proto="doubleMorph" > <data src="#meshData1" > <float name="w1" >0</float>

<float name="w2" >0</float>

</data>

<data id="meshData1" > <int name="index" >0 1 2 ...</int>

<float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="posAdd1" >0.0 1.0 2.0 ...</float3>

<float3 name="posAdd1" >0.0 1.0 2.0 ...</float3>

</data>

18

Prototypes

Motivation Design Implementation Results

Code

proto

#doubleMorph

#meshData1

int index

float3 position

float3 posAdd1

float3 posAdd2

#instanceA

float3 w1

float3 w2

#meshData2

int index

float3 position

float3 posAdd1

float3 posAdd2

#instanceB

float3 w1

float3 w2

Instantiate dataflow with different input data sets

Page 17: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Connect data to sink elements: <mesh>, <shader>, <lightshader> etc. just like <data>, but no further output

19

Connection to Scene Graph

Motivation Design Implementation Results

Code

<defs> <data id="#meshData" > <int name="index">0 1 2 ...</int>

...

</data> <shader id="wood" script="urn:xml3d:shader:phong"> <float3 name="diffuseColor">0.4 0.3 0</float3>

<float3 name="specularColor">0 0 0</float3>

...

</shader> </defs>

<group style="transform: url(#t1); shader: url(#wood)" > <mesh type="triangles" src="#meshData" /> </group> <group style="transform: url(#t2); shader: url(#wood)" > <mesh type="triangles" > <data src="#meshData" /> <float3 name="color">0.5 0.24 0.6 ...</float3>

</mesh> </group>

Generic data block

Inline declaration

Refer to a data block via doc id

Refer to a data block and extend

Page 18: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

20

Motivation Design Implementation Results

Implementation

Page 19: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

System Overview

Two implementations:

JavaScript + WebGL

Native Implementation

Separate Xflow library

Communicates with

scene graph library

Create dataflow

Optimizes computation

Request processed

data

21

Motivation Design Implementation Results

XML3D-DOM

(Chromium)

Scene graph library

(RTSG2)

Xflow Library

Dataflow

Structure & Input

Composed &

processed Data

Page 20: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Dataflow Execution Model

Push-based update

phase

Nodes are only marked

as out-of-sync when

modified / created

No processing is

triggered

Two states

Structure out-of-sync

Data out-of-sync

22

Motivation Design Implementation Results

skinPosition

Create

Scene Graph

skinDirection

mul4x4

forwardKinematics

lerp slerp

createTransform

structure

structure

structure

structure

structure structure

structure

Page 21: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

skinPosition

skinDirection

mul4x4

forwardKinematics

lerp slerp

Dataflow Execution Model

Push-based update

phase

Nodes are only marked

as out-of-sync when

modified / created

No processing is

triggered

Two states

Structure out-of-sync

Data out-of-sync

23

Motivation Design Implementation Results

Add fields

Change names

Attach operator

Scene Graph

structure

structure

structure

createTransform

Page 22: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

skinPosition

skinDirection

mul4x4

forwardKinematics

lerp slerp

structure

structure

structure

Dataflow Execution Model

Push-based update

phase

Nodes are only marked

as out-of-sync when

modified / created

No processing is

triggered

Two states

Structure out-of-sync

Data out-of-sync

24

Motivation Design Implementation Results

Change data

only

Scene Graph

createTransform

data

data

data

Page 23: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

skinPosition

skinDirection

mul4x4

forwardKinematics

createTransform

lerp slerp

structure

structure

structure

data

data

data

Dataflow Execution Model

Pull-based execution

phase

Request data from a

node

Trigger computation of

whole sub-graph

Optimize internal

structures beforehand if

structure-out-of-sync

25

Motivation Design Implementation Results

Request Data

Scene Graph

optimize

optimize

optimize

compute

compute

compute

compute

compute

compute

No computation

Page 24: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Dataflow Optimizations

Cache already

computed results

Skip operators that are

not required to be

executed

Request subset of data

by providing name set

26

Motivation Design Implementation Results

float3 position

xflow.skinPosition float3 position

Int4 boneIndices

float4 boneWeights

float4x4 pose Int4 boneIndices

float4 boneWeights

float4x4 pose

Scene Graph

Request “pose”

No need to

compute.

Page 25: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

create shader code

skinPosition

skinDirection

mul4x4

forwardKinematics

lerp slerp

createTransform

Mapping on the GPU

Integrate dataflow into

render pipeline

Create vertex shader

from subgraph

Only includes per-vertex

operations

Scene graph provides

information for vertex

attribute transformation

Required for connection

with fragment shader

27

Motivation Design Implementation Results

Scene Graph

Request Vertex

Shader

compute

compute

compute

compute

compute

Page 26: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

28

Motivation Design Implementation Results

Results

Page 27: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

29

Motivation Design Implementation Results

Page 28: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

30

Motivation Design Implementation Results

Page 29: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Morphing Performance

31

Motivation Design Implementation Results

Scene #triangles FPS (CPU) FPS (GPU)

16x cloned movement 639,552 92 74

16x individual movement 639,552 32 68

32x individual movement 1,279,104 19 41

Page 30: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Future Work

Xflow everywhere

Surface Shading (Shading graph)

Post-Processing (Tone Mapping, SSAO)

Image Processing

Cyclic connections

Important for particles, physics, image processing

Improve River Trail implementation

Image Geometry with Xflow?

32

Motivation Design Implementation Results

Page 31: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Intel

Funding via Intel Visual Computing Institute

Intel Labs, River Trail Team

EU Project: Future Internet PPP – FI-Content

Usage Area: Games & Virtual Environment

Collaboration with Disney Research, BBC ...

EU Project: VERVE

VR-based patient treatment

Realistically animated characters on the Web

33

Acknowledgements

Page 32: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

34

Thank you! Questions?

Page 33: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Performance Native vs. JS

35

Appendix

Instances JavaScript Nativ w/o VS Nativ with VS*

16 45 19 1,4

8 23 9,5 0,7

1 3,3 1,2 0,1

Processing Time on CPU in milliseconds

*only includes operators executed on CPU (Transform computations)

Page 34: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Xflow Filter Syntax

36

Motivation Design Implementation Results

Code

<data filter="rename({pos: position, norm: normal})"> <float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="normal" >0 1 0 ...</float3>

<float2 name="texcoord" >0.0 0.0 ...</float2>

</data>

<data filter="keep(position, normal)"> <float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="normal" >0 1 0 ...</float3>

<float2 name="texcoord" >0.0 0.0 ...</float2>

</data>

<data filter="remove(texcoord)"> <float3 name="position" >1.0 0.04 -0.5 ...</float3>

<float3 name="normal" >0 1 0 ...</float3>

<float2 name="texcoord" >0.0 0.0 ...</float2>

</data>

Page 35: Xflow - web3d2012.web3d.org Declarative Data Processing for the Web Felix Klein, Kristian Sons, Dmitri Rubinstein, Sergiy Byelozyorov, Stefan John, Philipp Slusallek

Scene Graph

2. Render

Pass

Imperative options hard to

integrate into Scene Graph

37

Appendix

Fragment

Shader

Vertex

Shader

Input

Final Image

Extend for

mesh

animation

Extend for

post

processing

DOM modifications only

straight-forward option

Slow for large data-sets

We also want to extend

shaders for data processing

Shaders already managed by

scene graph

Extend scene graph to

support proper integration DOM