abstract shade trees

Abstract Shade TreesAbstract Shade Trees

Morgan McGuire1 George Stathis2

Hanspeter Pfister2,3 Shriram Krisnamurthi1

1Brown University 2Harvard Extension School 3MERL

Shaders

• Programs that color (“shade”) a pixel

• Often execute on a graphics card

• Pervasive in film and game rendering

[Proudfoot01]

Shaders

• Programs that color (“shade”) a pixel

• Often execute on a graphics card

• Pervasive in film and game rendering

[Proudfoot01]

Let’s author a pixel shader

Shading Language

(Renderman, RTSL, HLSL, GLSL, Cg)// glass_ball.glsluniform float etaRatio;uniform float fresnelPower;uniform samplerCube environmentCubeMap;varying vec3 objectSpaceIncomingVector;varying vec3 n, b,t;uniform sampler2D normalMap;varying vec3 tangentSpaceViewVector;varying vec2 textureCoordinates;

void main(void) { float bump = ((texture2D(heightFieldTexture, textureCoordinates).a) + bumpBias) * bumpScale; vec2 bumpCoords = bump * vec2(tangentSpaceViewVector.x, tangentSpaceViewVector.y) + textureCoordinates;

vec3 tangentSpaceNormalVector = texture2D(normalMap, vec2(bumpCoords.x,bumpCoords.y)).xyz * 2.0 - 1.0;

mat3 tangentObjectSpaceMat = mat3(t,b,n); vec3 objectSpaceNormalVector = tangentObjectSpaceMat*tangentSpaceNormalVector; vec3 objectSpaceReflectedVector = normalize(reflect(objectSpaceIncomingVector,objectSpaceNormalVector)); vec4 color1 = textureCube(environmentCubeMap, objectSpaceReflectedVector);

float fresnelRatio = max(0.0,min(1.0, fresnelBias + fresnelScale* pow(1.0 + dot(normalize(objectSpaceIncomingVector), normalize(objectSpaceNormalVector)),fresnelPower))); vec3 objectSpaceRefractedVector = normalize(refract(objectSpaceIncomingVector,objectSpaceNormalVector,etaRatio)); vec4 color2 = textureCube(environmentCubeMap, objectSpaceRefractedVector); vec4 mixedColorOutput = mix(color1,color2,fresnelRatio);

gl_FragColor = mixedColorOutput;}

Shade Tree for Visual Programming[Cook84; Abram+Whitted90; Borau04; Unreal06; Hargreaves04]

Parameter

Function call

Authoring in a production context…

Shader Authoring Problems1. Dependent workflow

– Artists define and use

– Programmers implement

– Frequently block on each other

2. Shaders are rarely modular– Overlapping features [Dijkstra76]

– Exacerbated by restrictive execution environments

– Programmers frequently rewrite from scratch

Artist Programmer

Shader Authoring Problems1. Dependent workflow

– Artists define and use

– Programmers implement

– Frequently block on each other

2. Shaders are rarely modular– Overlapping features [Dijkstra76]

– Exacerbated by restrictive execution environments

– To combine features, programmers frequently rewrite both from scratch

Artist Programmer

void main() {

New authoring idea…

Shade Tree

Abstract…

Abstract Shade TreeSuperposition

Reflection Refraction

Parallax Bump Mapping

Abstract Shade Tree

Superposition

Reflection Refraction

Parallax Bump Mapping

• Node = Code atom– Like functions with funny scopes

(See the paper for details)

• Arrow = Dependency– Not parameters!

• Outline = Feature– Note the overlap at Fresnel

• Requires a weaver to convert into executable code

New Workflow

1. Programmer writes atoms2. Both can visually combine atoms into features3. Artist visually combines features into trees4. Weaver synthesizes:

– Shader entry and exit points– Program structure– Parameter linkage– Type coercions– “Glue” code

5. Rendered in IDE for feedback

Problems Addressed

1. Workflow dependency reduced– Programmer writes atoms once– Artist can independently generate many shaders– Abstract Tree is independent of atom signatures

2. Modularity improved– Visualized feature interaction– Weaver resolves overlapping features– Abstract Tree is independent of atom signatures

Related Feature Work• Features as abstractions

[Parnas72; Dijkstra76; Turner99]

• Feature/Aspect-oriented programming

[Batory92; Kiczales97; Batory04]

• Software product lines

[Clements02]

• Sh Language, Combining shaders

[McCool et al. 02, McCool et al. 04]

• Über-Shader alternative

[Barzel97/Gritz98; Pellacini+Vidimce04; McGuire05]

The Weaver

Weaving Algorithm

• Input: Abstract shade tree• Output: GLSL shader

1. -Rename parameters2. Topologically sort nodes3. Match inputs to outputs4. Generate code5. Generate boilerplate

Match Inputs to Outputs

Lambertian Shading

Normal Map

Some node in the middle of an Abstract Shade Tree

Match Inputs to Outputs

OWT

Lambertian Shading

Transform

Normalize

ON

Normal Map

WN̂

WL̂

WN

12

1ON

Unpack

The abstracted parameters the Weaver must reconstruct

Lambertian Shading

Transform

Normalize

ON

Normal Map

WN̂

WL̂

WN

12

1ON

Unpack

Match Inputs to Outputs

Globals

Basis Change

Upstream Output

Storage Class Change

OWT

The abstracted parameters the Weaver must reconstruct

Matching Algorithm

• Bind each input parameter to the “closest” matching output parameter of some ancestor node

• “Closest” = shortest/fastest type coercion

• Typical GLSL types are just storage classes– e.g., float3 vector color point …– float LambertianShading(float3 normal, float3 light)

– Not much information in “float3”

• Need a richer type system…

Semantic Types

• Encode information like “world-space vector” in types

• Two variables with precisely the same semantic type are likely semantically interchangeable

Semantic Types

1

TLK JavaC/GLSL

Too general Useful Too specificNumber of types

Some Semantic Types for Shading

Float1 LambertianShading(UnitWorldNormalFloat3 normal, UnitWorldVectorFloat3 light)

Dimension: {2, 3, 4, any }

Length: {Unit, any }

Basis: {Tangent, Object, World, Screen, any }

Interpretation: {Color, Texcoord, Normal, Point, Vector, any}

Precision: {float32, int32, Boolean, any }

• Type coercion now includes normalization, basis change, unpacking, storage change, etc.

• “any” allows polymorphism• Could add “meters,” “non-negative,” etc.

Parameter Matching is Semantic Coercion Tree Search

VEC3___O_NOR

NormalWorldToEyeSpace

VEC3___W_NOR

NormalTangentToObjectSpace

VEC3___T_NOR

VEC3___O_NOR

VEC3___E_NOR

VEC3___T_NOR

VEC3___O_NOR

NormalObjectToTangentSpace NormalObjectToWorldSpace

NormalWorldToObjectSpace NormalTangentToObjectSpace

Weaver Properties

• Always terminates• Bounded (and small) memory requirements• Always generates a legal program

• …but sometimes it is the wrong program• Most common error: Parameter aliasing

– Artist missed a dependency– Programmer assigned an overly generic type– Anecdotally infrequent

Results

Bumpy Glass

Bumpy GlassManual

(GLSL)

Automatic

(AST)

Final Source Length 58 lines 188 lines

Assembled 41 instr. 46 instr.

Radeon 9700 @ 512 x 512

245 fps 240 fps

1024 x 768 50 fps 50 fps

User experience?

IDE

Phong Illum. + Parallax Map + Texture Map

Anisotropic + Diffuse Lighting

Phong Illum. + Projective Light + Shadow Map

Summary of Contributions

• Abstract shade trees

• Semantic type system

• Weaver for shader synthesis

• Visual editor

Future Directions

• Consider performance model during parameter matching

• Beyond pixel shaders• Other interesting semantic types

• How do you debug abstract trees?

• Applications to general scripting

Thanks

• NVIDIA

• MERL

• NSF

• Iron Lore Entertainment