N-Dimensional Presentation States

Progress and Paths from WG 11

Michael Aratow, M.D., FACEPNicholas F. Polys Ph.D.

Web3D Consortium

Convergence in Medical ITNew opportunities in improving quality and

reducing cost of care:• Standards enable new analytics– Reproducible Interactive presentations and

reports– Leveraging the web multimedia ecology (W3C)– Semantics (FMA, SNOMED CT)

• Standards enable new levels of care– Consistent communication between providers and

organizations

Critical Path:Medical Imaging & Informatics

• Interoperability challenges for the presentation of medical images are well-recognized – Accessibility outside the radiology suite– Procedural and Cognitive Education– Informed Consent– Custom prostheses– Surgical Planning– Radiation Therapy and Planning

• Why risk a new wheel? – Proven and unencumbered scenegraph abstractions yield a

quantum jump: the ‘Hopscotch Strategy’

• A decade of experience with 3D and the web in an ISO environment– Aligning with proven web delivery and

presentation technologies

• Leads to the obvious reason these standards bodies are working together:– Existing DICOM requirements overlap existing X3D

functionality – ISO standardization – Open technology

The Hopscotch Strategy

The Web3D Consortium

A not-for-profit organization that develops and maintains ISO standards for the description and transmission of 4D data over telecommunication networks. The X3D, H-Anim and VRML standards specify both a file format and a runtime specification that are open and royalty-free

www.web3d.org

Web3D & X3DProven Innovation (see web3d.org):

• Real time interactive graphics – Lights, cameras, text, meshes, textures and materials– Animations, sensors, volumes & style

• Interoperability with web – distributed resources referred via URL/URI– E.G. images, sounds, videos, models, worlds

• Durability – ISO Specifications:

• Abstract Spec (scenegraph)• Encodings• Bindings• H-Anim

http://www.web3d.org/x3d/specifications/

Highlights

• Abstract scenegraph with multiple encodings and API bindings• Well-defined runtime semantics• ISO (SC-24), Royalty-Free• Multiple open-source and commercial implementations (approx 24)• demonstrated capability for medical applications• covers significant majority of DICOM requirements• remainder being specified through WG11 work item

US ARMY TATRC Contract

• Successfully completed in 2007• MedX3D Draft Specification available to

members• Implementations:– H3D.org – Open Source Haptics incl.

X3D, MedX3D and VRML– Fraunhofer IGD’s Instant Reality

• Semantic Scenegraph inline or referenced• Volume Rendering Demos

Web3D + DICOM Recent History

• Started in WG-17: Web3DC contributed to 3D Surface data structures (compatible with ISO 19775 (X3D))

• Now primarily in WG11: Scoped Work Item on n-Dimensional Presentation States

• Web3D Consortium Medical WG presenting specs and royalty-free technology meeting (and exceeding) requirements

Timeline Recap

DICOM: Critical Mass• Continue to gather momentum, requirements,

alignments with DICOM stakeholders (you!)• Web3D Consortium brings a time-proven

solution: a standardized scenegraph to meet the requirements of the healthcare enterprise

• Recognizing the value of consistent renderings across caregivers and stakeholders, the US Army TATRC provided critical support (spec draft and open source implementations)

We are pleased to report: Significant progress on this work item!

X3D Interactive Presentations use DNA

Basic Interactive Scenes:• 2D & 3D Geometry, Appearance, Text, Layers• Lighting• Sensors• Event Utilities• Interpolators, Sequencers

• No PROTOs, No Scripts

Sample Presentation

• Simple X3D Interactive demos • Data from http://pubimage.hcuge.ch/• Volume to Surfaces by OSIRIX

6-DOF Camera

• Animation includes coordination of transformation graph and behavior graph

• Endoscopy path example:– Timesensor– PositionInterpolator, OrientationInterpolator– Routes [event model]

Specification Plan:Med X3D Profile

1. Specify additional nodes and components for X3D abstract spec

2. Create MedX3D Profile for X3D abstract spec3. Encoding» Create DICOM encoding of MedX3D Profile » Create Binary encoding of MedX3D Profile

X3D Encodings and Bindings

http://www.x3dgraphics.com

Equivalent scenegraphs

can be: encoded inand manipulated by multiple languages

X3D Components and Profiles

• Components– Families of related nodes

• Levels– Of support (fields and

functionality)

• Profiles– Well-defined nodesets

for vertical application functionality

See: http://www.web3d.org/x3d/specifications/ISO-IEC-19775-1.2-X3D-AbstractSpecification/index.html

X3D Volume Rendering Component

Specified and developed w/ US Army TATRC; two implementations (one open-source)

• General transfer function• Shading styles

– Blinn-Phong(including advanced features such as shadows and light scattering)– Cartoon shading– Gooch shading(tone mapped)

• Enhancement styles (enables enhancement of important features in an easy way)– Boundary enhancement– Edge enhancement– Silhouette enhancement

• Maximum , Minimum and Average intensity projection• Iso-surface rendering• Segmented data rendering (render different parts of volume data using different

styles)

WG 11 Direction for n-D PS

• Functional Profile (nodeset): Med X3D– X3D Interactive+ – Volumes & Rendering styles– Cutting/ clipping (i.e. for

Multiplanar Reconstruction (MPR))

• Scenegraph could be encoded in X3D Binary Encoding scenegraph profile in DICOM

Specification: Include Base X3D Concepts

• Include some set of components for meeting requirements:– Grouping: Transformation, Switch, LOD– Viewpoints (Perspective and Ortho)– Lights (Directional, Point, Spot)– 3D, 2D surfaces, lines and point geometries

• including Shape Appearance and Materials• Per-vertex data such as RGBA and ImageTexture maps

– Annotation (Text, Layers)– Interactivity

• Sensors• Event model, animation• Scripting?

Specification Progress: Volumetric Presentation

• Volume Rendering– MedX3D styles have necessary and sufficient

coverage• Masks, subvolumes & segments– MedX3D voxel data structure and access per WG2

and WG12 requirements• Addressing voxels within volume• Volume time series

• UI (Toshiba input) – e.g. Clipping & MPR

• Lines generate plane• Planes• 3D volume

DICOM WG 11 + Web3D Medical WGSpecification Plan:

1. Med X3D Functional Specification (abstract)» Specify additional nodes and components for X3D abstract

spec» Create MedX3D Profile for X3D abstract spec

1. X3D (XML, Binary) encodings exist; DICOM encoding of MedX3D Profile?

Going Forward

• ISO scenegraph technology yields a quantum leap in health care interoperability

• Immediate value in cross-platform transmission and rendering of medical image presentations

• Continue development of DICOM+MedX3D Profile: specification (X3D 3.3) including annotations, implementations, conformance examples

Ways to get Engaged

• WG 11• IEEE VR 2010 in Waltham, MA this month• Web3D Symposium in Los Angeles in July 2010• SIGGRAPH in Los Angeles in July 2010