3-D Visualization of 3-D Visualization of Functional Brain Map DataFunctional Brain Map Data

A.V. Poliakov; E.B. Moore; J.F. Brinkley, A.V. Poliakov; E.B. Moore; J.F. Brinkley, Structural Informatics GroupStructural Informatics Group

Department of Biological StructureDepartment of Biological Structure

University of WashingtonUniversity of Washington

UW Human Brain ProjectUW Human Brain ProjectStructural Informatics Group, Dr. BrinkleyStructural Informatics Group, Dr. Brinkley

CollaboratorsCollaboratorsNeurosurgery, Dr. OjemannNeurosurgery, Dr. OjemannPsychiatry, Dr. CorinaPsychiatry, Dr. CorinaRadiology, Dr. MaravillaRadiology, Dr. Maravilla

Medical ProcedureMedical ProcedureNeurosurgery for intractable epilepsy Neurosurgery for intractable epilepsy

Objective Objective Remove epileptic foci while sparing the patient’s language functionRemove epileptic foci while sparing the patient’s language function

MethodMethodLanguage mapping using electrical stimulationLanguage mapping using electrical stimulation

Information SystemInformation System Help researches organize and analyze the dataHelp researches organize and analyze the dataUnique dataUnique data

Study languageStudy languageEvaluate non-invasive imaging techniques for surgical planningEvaluate non-invasive imaging techniques for surgical planning

Data acquired during surgery Data acquired during surgery Electrical stimulation map Electrical stimulation map PhonogramsPhonograms

Data acquired prior to surgeryData acquired prior to surgery MR scans ofMR scans of cortical anatomy, veins, arteries cortical anatomy, veins, arteries Functional MRIFunctional MRI

Information System for Language Information System for Language Mapping Mapping Primary Sources of DataPrimary Sources of Data

Data acquired during surgery Data acquired during surgery Electrical stimulation map Electrical stimulation map PhonogramsPhonograms Single unit recordingSingle unit recording

Data acquired prior to surgeryData acquired prior to surgery MR scans ofMR scans of cortical anatomy, veins, arteries cortical anatomy, veins, arteries Functional MRIFunctional MRI EEG in grid patientsEEG in grid patients

Information System for Language Information System for Language Mapping Mapping Primary Sources of DataPrimary Sources of Data

Brain MapperBrain Mapper

• Alignment • Masking • Reconstructing 3D models• Mapping CSM Sites

Information System for Language Information System for Language Mapping Mapping

Information System for Language Information System for Language Mapping Mapping

In developmentIn development

fMRI DB and Automated Data fMRI DB and Automated Data Analysis Analysis

Single Unit DBSingle Unit DB

Data ManagementData ManagementTalairach Daemon QueryTalairach Daemon Query

Query the Brain Mapping Repository using XQueryQuery the Brain Mapping Repository using XQuery

Information System for Language Information System for Language Mapping Mapping

SPM

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

Web InterfaceWeb Interface

3D Visualization3D VisualizationSystem ArchitectureSystem Architecture

– Client-serverClient-server– Server-based rendering and data storage Server-based rendering and data storage – Web-based client (e.g. Java or forms-based)Web-based client (e.g. Java or forms-based)

Web Interface

Server

Client #1CGI script

Client #2Java Applet

Client #3. . .

Internet

Graphics Server

snapshot snapshot

Stimulation Sites

3-DModels

3-D Image Volumes

Client

Data

Experiment Management System

Relational Database

2-D Images

Brain Mapper

3D Visualization3D VisualizationRequestsRequestsNew UsersNew Users

Different data modalitiesDifferent data modalities

Various software packages Various software packages

SpectroscopySpectroscopyERP Source SignalERP Source SignalfMRIfMRI

Transcranial Magnetic StimulationTranscranial Magnetic Stimulation

Collaborators Collaborators New features New features

Remote collaborationRemote collaboration

3D Visualization3D Visualization

Limitations of current systemLimitations of current systemRelatively slow software renderingRelatively slow software rendering

Non-portable architecture Non-portable architecture SGI Irix and Linux onlySGI Irix and Linux only

Large code base Large code base Needs to be maintained and developedNeeds to be maintained and developed

Reliance on outdated software toolsReliance on outdated software tools

3D Visualization3D VisualizationOther possible approachesOther possible approaches

OpenGL, DirectX (Windows)OpenGL, DirectX (Windows)VTK, Java3DVTK, Java3DHigher level toolkitsHigher level toolkits

Java3DJava3DCross PlatformCross PlatformSupports hardware acceleration Supports hardware acceleration Off-screen rendering Off-screen rendering JavaJava

Pilot studiesPilot studies Performance Performance Off-screen renderingOff-screen rendering Client-serverClient-server

Proof of conceptProof of concept

BrainJ3D – Next GenerationBrainJ3D – Next Generation

Live DemoLive Demo

BrainJ3DBrainJ3D

BrainJ3D BrainJ3D Features Seen in DemoFeatures Seen in Demo

• CutawayCutaway

• MappingMapping

• fMRI OverlaysfMRI Overlays

• WorkspaceWorkspace

BrainJ3DBrainJ3DOther FeaturesOther Features

• Supports common formats and data types Supports common formats and data types (Analyze, Minc, data created with SPM, FSL, etc.)(Analyze, Minc, data created with SPM, FSL, etc.)– Leverages functionality of other programsLeverages functionality of other programs– Increases user baseIncreases user base– Use Java utilitiesUse Java utilities

• 3D model loaders3D model loaders• NetCDF libraryNetCDF library• 2D image API2D image API

• Configuration and internal files in XMLConfiguration and internal files in XML– More portableMore portable– Easy to customizeEasy to customize

BrainJ3DBrainJ3DOther FeaturesOther Features

• Client/ServerClient/Server– Client/Server mode looks and behaves Client/Server mode looks and behaves

exactly like standalone modeexactly like standalone mode– Reduced performanceReduced performance

•Sending the image is a bottleneckSending the image is a bottleneck

– Takes advantage of built in Java securityTakes advantage of built in Java security– Thin client only requires installation of JavaThin client only requires installation of Java

• Java3D and accelerated graphics card not Java3D and accelerated graphics card not requiredrequired

BrainJ3DBrainJ3DArchitectureArchitecture

BrainJ3DBrainJ3D

Java and Java3D provide viable development Java and Java3D provide viable development environment for scientific visualization environment for scientific visualization applicationsapplications

+ Cross Platform ( Linux, Windows and Solaris)+ Cross Platform ( Linux, Windows and Solaris)+ Supports hardware acceleration and scene optimization+ Supports hardware acceleration and scene optimization+ Supports hardware accelerated off-screen rendering+ Supports hardware accelerated off-screen rendering+ Mature software package that provides needed + Mature software package that provides needed

functionalityfunctionality+ Java+ Java

Modern object-oriented languageModern object-oriented languageNumerous libraries and utilitiesNumerous libraries and utilitiesWeb-based and remote technologies are well supportedWeb-based and remote technologies are well supported

BrainJ3DBrainJ3D

Future DevelopmentFuture DevelopmentDifferent way to access programDifferent way to access program

Java Web StartJava Web Start

HTML GUIHTML GUI

Advanced Visualization TechniquesAdvanced Visualization TechniquesTime series (movie) mode for VolumesTime series (movie) mode for Volumes

Show functional activation on Surface modelShow functional activation on Surface model

Other FeaturesOther FeaturesAnalysis and visualization features requested by usersAnalysis and visualization features requested by users

Current and Future WorkCurrent and Future WorkIntegration with Information System for Language Integration with Information System for Language

MappingMapping

Improving InteroperabilityImproving InteroperabilitySupport for other data modalities Support for other data modalities

Spectroscopy, ERP source signal, Transcranial Magnetic StimulationSpectroscopy, ERP source signal, Transcranial Magnetic Stimulation

Importing data from other Brain Imaging software packages Importing data from other Brain Imaging software packages FreeSurfer and SureFitFreeSurfer and SureFit

* Interoperability is a fast moving target, often beyond our control* Interoperability is a fast moving target, often beyond our control

Refining Client/Server modeRefining Client/Server modeSecuritySecurity

PerformancePerformance

UW Human Brain ProjectUW Human Brain Project

Structural Informatics GroupStructural Informatics Group

http://sig.biostr.washington.edu/http://sig.biostr.washington.edu/

AcknowledgementsAcknowledgements

Human Brain Project grant DC02310Human Brain Project grant DC02310National Institute of Deafness and Other Communication National Institute of Deafness and Other Communication

Disorders and National Institute for Mental HealthDisorders and National Institute for Mental Health