Understanding M3G 2.0 and its Effect on Producing Exceptional 3D Java-Based Graphics

Sean EllisConsultant Graphics Engineer

ARM, Maidenhead

222Confidential

IntroductionM3G 1.x RecapARM M3G IntegrationM3G 2.0 UpdateM3G 2.0 Hints and Tips

333Confidential

AudienceWho were we addressing with M3G 1.x?

Java design skill

3D design skill

“TechnicalArtists”

ExperiencedSoftware Engineers

The same people need to be able to use M3G 2.0.

444Confidential

Basic MIDlet ArchitectureLoad assets, find commonly used objects, set up display, initialize game state

paint()Canvas calls Graphics3D to perform rendering

initialize

user input

scene update

request redraw

wait

shut down

Get any user input, network play, etc.

Game logic, animate, align if necessary

Wait to ensure consistent frame rate

Release assets, tidy up

Exit request

Main loop.

Don’t create/destroy objects if possible

Throttle to consistent frame rate

Keep paint() as simple as possible

Be careful with threads

555Confidential

Typical RelativePerformance

Java Virtual MachineOther

Library(OEM specific)

OtherClasses

(OEM specific)

Java with Native Library

Hardware

OS

NetworkLibrary

(J2ME)

InputLibrary

(J2ME)

SoundLibrary

(OEM specific)

NetworkClasses

(J2ME)

InputClasses

(J2ME)

SoundClasses

(OEM specific)

3D Application using SwV classes2D UI

Classes(J2ME)

2D UILibrary

(J2ME)

M3GClasses

M3GLibrary

Scene managementAnimations

3D transformation Lighting

Triangle drawingTexturing

Transfer to 2D

Interaction from user3D object control

FASTMost work donehere - at high performance

666Confidential

Class HierarchyObject3D Node

Mesh

Camera

Light

Group

VertexBuffer VertexArray IndexBuffer TriangleStripArray

Appearance

Material

Fog

PolygonMode

CompositingMode

Texture2D

Background

Image2D

World

SkinnedMesh

MorphingMeshSprite3D

Graphics3D

Transformable

Transform

AnimationTrack AnimationController

KeyframeSequence

Loader RayIntersection

777Confidential

M3G Integration – Top End

C++ Implementation

C Interface Layer

Some C++ libraries pre-linked

JSR184/297 API

C API

Java Interface Layer

Supplied as a linkable library

Interface Description

Example API implementations

shipped as source

Java Application

API interface codeis auto-generated

for KNI, JNI, C

888Confidential

M3G Integration – Bottom End

C++ Implementation

C Interface Layer

Some C++ libraries pre-linked

Supplied as a linkable library

OSAbstraction

GLAbstraction

SurfaceAbstraction

GLES EGL

OS

Example implementations

shipped as source

Abstraction APIsSurface abstraction

needs access to Java VM internals (pixel buffers for

images and Graphics objects)

999Confidential

M3G 2.0 Integration TipGood news, everyone – almost exactly the same as M3G 1.1Some software-renderer-only integration APIs removedNew abstraction interface for DynamicImage2D

101010Confidential

M3G 2.0 New FeaturesCore block

Fixed function (Open GL ES 1.1)M3G 1.x compatibleMali55™ or Mali200™

Advanced blockProgrammable hardwareMali200Can mix modes in API, with some restrictionsCan implement both blocks on OpenGL ES 2.0Must be completely present or absent – less fragmentationJavax.microedition.m3g.shader package is major new addition

DisclaimerSpec still in draft – details subject to change

M3G2.0

Advanced

M3G 2.0 Core

M3G 1.1

M3G 2.0 Core

111111Confidential

NativeVideo Source

New Image Types

Fast pathRenderer

Slow path

Java Video Control

ImageBaseImageCube

DynamicImage

Image2D

121212Confidential

Mesh and its componentsMesh

VertexBuffer

IndexBufferIndexBufferIndexBuffer

AppearanceAppearanceAppearance

VertexArray Vertex positions

VertexArray Vertex normals

VertexArray Vertex colors

VertexArray Texture coords

VertexArray Point sizes

VertexArray User attributes

VertexArray User attributes

Vertex arrays now can contain values of type byte, short, 16.16 fixed point,

float, or half-float

131313Confidential

Mesh and its componentsMesh

VertexBuffer

IndexBufferIndexBufferIndexBuffer

AppearanceAppearanceAppearance

TriangleStripArrayExplicit or sequential vertex indices

Index into…

TriangleArrayExplicit or sequential vertex indices

LineArrayExplicit or sequential vertex indices

PointArrayExplicit or sequential vertex indices

141414Confidential

ShaderAppearance

Appearance and its components

AppearanceBase

Material Diffuse, emissive, specular, ambient colors, shininess

PrimitiveMode

Perspective correction, winding, culling

CompositingMode

Screen buffer write mode, thresholds, depth offset

Fog Fog density, color

Texture2DTexture2DTexture application mode, coordinate transform, wrapping

Image2D

Appearance PointModeLineMode

PolygonMode

151515Confidential

ShaderAppearance

Appearance and its componentsShader

Program

ShaderUniforms

VertexShader

Texture2D TextureCube

Image2D

FragmentShader

ImageCube

161616Confidential

Vertex ShaderFirst, vertex shader transforms the position of each vertex in an object.Often also calculates other parameters

InputObject

TransformedObject

Transformed PositionLit Color

Transformed Texture Coords

PositionNormalColor

Texture Coords

171717Confidential

Vertex ShaderSame vertex shader runs on all vertices in the object.Can assign different shaders to other objects.Results are interpolated across primitives.

InputObject

TransformedObject

TransformedObject

181818Confidential

Fragment ShaderFragment shader takes interpolated (“varying”) variables, and calculates the actual color at each pixel.

TransformedObject

RenderedObject

Texture

191919Confidential

RenderedObject

Fragment ShaderOr it abandons the pixel, rendering nothing, and leaving a hole.

TransformedObject

Texture

202020Confidential

Fragment ShaderSame fragment shader runs on all pixels in the object.Can assign different shaders to other objects.

TransformedObject

RenderedObject

You getthe idea

212121Confidential

M3G 2 Shader Model

Appearance

uniform mat4 modelViewMatrix;attribute vec4 vertexPos;void main(){

gl_Position =modelViewMatrix * vertexPos;

}

uniform mat4 modelViewMatrix;attribute vec4 vertexPos;void main(){

gl_Position =modelViewMatrix * vertexPos;

}

CompleteAppearance

VertexBuffer

UniformVariables

CompiledVertex Shader

CompiledFrag Shader

MeshDisclaimer: Specification

subject to change

CompleteShader Program

222222Confidential

Why keep Appearance?ShaderAppearance can do anything than Appearance canSo why keep it?

Easy transitionMore artists know fixed function pipeline than shaders.Perfectly adequate for a lot of content.Familiarity means faster development.Can use all the legacy content out there.Can mix and match both in one scene.Therefore, shader designers can concentrate effort where it’s going to be noticed.

232323Confidential

Shader Example

242424Confidential

Shader WalkthroughSo, what does a shader look like?

// Astoundingly dull fragment shaderuniform vec4 flatColor;void main(){gl_FragColor = flatColor;

}

Isn’t that just C?

252525Confidential

Same Geometry, Dull Shader

262626Confidential

C-like languageSyntax familiar to most programmersSimplified: no pointers, no charactersNew types: vectors, matrices, samplersNew qualifiers: attribute, uniform, varyingLimited size95% common between Vertex and Fragment shaders

272727Confidential

RenderPassMesh

VertexBuffer

IndexBufferIndexBufferIndexBuffer

AppearanceAppearanceAppearance

Texture2D

Image2D

RenderPass

Renders to…

World

Meshes

Any Node can have a…

282828Confidential

M3G2 Shader TipsTANSTAAFLShaders are not free – they take time, even on hardwareComplex shaders take more timeSpecialize shaders for common parameter values

292929Confidential

The #1 Important Shader Optimization TipDON’T DO IT…yet

Premature optimisation is the root of all evil – C.A.R.Hoare

Profile!Is it really the shaders at all?How fast does my shader run?What’s really holding it up?Then you can really improve things

303030Confidential

M3G2 Shader TipsCompilation time is fast, but not insignificantCompile once, use many times(This is why compilation happens at construction)Don’t create shaders on the fly

It’s a balancing actMore specialized shaders = faster executionMore specialized shaders = slower startup

313131Confidential

M3G2 Shader TipsTools support often at the low levelRequires cooperation from the hardwareMali hardware has debugging/profiling countersLow-level performance analysis tools

FramerateThroughputCache usage

Higher level analysis tools under development:Shader debuggerShader profiler

323232Confidential

Animation ChangesContent developers requested changes to animation system.Designed for ease-of-use.Details currently under discussion in expert group.

333333Confidential

You can contributeEarly Draft Review Spec available at:http://jcp.org/aboutJava/communityprocess/edr/jsr297Or just go to jcp.org and enter 297 in the “Go to JSR” box. The download link is on the resulting page.

We want to hear feedback, especially from content developers

We want you to understand itWe want you to use itWe want you to make good-looking applications

exceptional

343434Confidential

Q&A

http://jcp.org/aboutJava/communityprocess/edr/jsr297