Integrating Integrating OntologiesOntologies with with ThreeThree--Dimensional Models of Dimensional Models of
AnatomyAnatomy
Daniel L. RubinYasser Bashir
David Grossman Parvati Dev
Mark A. Musen
Stanford Medical InformaticsStanford University
Copyright © Daniel L. Rubin 2004 2
Projectile InjuryProjectile Injury
● Penetrating trauma responsible for many civilian deaths; major cause of battlefield fatalities
● Survivability after projectile injury depends on rapid acquisition & interpretation of knowledge
Need to know anatomic structures injured and extent of organ damageNeed to know physiological consequencesNeed to make triage decision (immediate surgery, medi-vac, observe, etc.)
Copyright © Daniel L. Rubin 2004 3
ObjectivesObjectives
● Build computable model of human anatomy
Predict direct anatomic injuriesPredict propagation of injuries
● Develop intelligent applicationsOn-scene diagnosisAssist triage decisions
● Provide graphical display of anatomy, bullet trajectory, and tissue damage
Copyright © Daniel L. Rubin 2004 4
33--D Geometric ModelsD Geometric Models
● Many prior applicationsSurgical simulation, planningMedical visualizationTeaching
● Encode spatial geometric information
● Contain no knowledge about contents of these models
Identity of anatomic structuresTissue propertiesPhysiological status of organs
Copyright © Daniel L. Rubin 2004 5
RequirementsRequirements
● Intelligent applications for injury:3-D geometric dataKnowledge pertinentto injuriesReasoning services
Use geometric data and knowledge to predictconsequences of injuries
Copyright © Daniel L. Rubin 2004 6
Need Variety of KnowledgeNeed Variety of Knowledge
● Anatomic knowledgeWhere do organs lie in the body?What organs are fed by different arteries?What are the subparts of an organ?
● Biomechanical knowledgeWhat are the tissue material properties of an organ?
● Tissue injury knowledgeHow do different organs respond to injury?
Copyright © Daniel L. Rubin 2004 7
Need Variety of Reasoning ServicesNeed Variety of Reasoning Services
● Which organs were directly injured?
● What additional tissue damage will occur as the primary injury propagates?
● How will physiological parameters be altered by the injury?
● How should the subject be treated?
The Virtual Soldier Project
Copyright © Daniel L. Rubin 2004 9
ApproachesApproaches
● Make knowledge explicit and computable1. Use ontologies for knowledge representation
Useful in rich and complex domains
Can reuse existing knowledge sources
● Integrate patient data and canonical knowledge
2. Create patient-specific models
3. Develop reasoning services
1. 1. OOntologiesntologies for knowledge for knowledge representationrepresentation
Copyright © Daniel L. Rubin 2004 11
Knowledge SourcesKnowledge Sources
● Geometric KnowledgeGeometry ontology constructed to describe computer graphics principlesSpecifies data structures used to represent geometric models
● Anatomic KnowledgeFoundational Model of Anatomy (FMA) ontologySpecifies anatomical entities and relationships (e.g., partonomies, continuities, adjacencies)Logical as opposed to spatial model
Copyright © Daniel L. Rubin 2004 12
Ontology of Geometric ModelingOntology of Geometric Modeling
Foundational Model of AnatomyFoundational Model of AnatomyOntology Slots
Wallof Heart
RightAtrium
HollowViscus
Organ
AnatomicalStructure
Heart Parts of the heart
Cavityof Heart
Cavity ofRight Atrium
Wall ofRight Atrium
FossaOvalis Myocardium
SinusVenarum
SANode
Myocardiumof Right Atrium
Part-of
CardiacChamber
OrganComponent
OrganSubdivision
OrganPart
InternalFeature
OrganCavity
Organ CavitySubdivision
AnatomicalSpatial Entity
AnatomicalFeature
BodySpace Viscus
Is-a
Is-a
Copyright © Daniel L. Rubin 2004 15
Ontology ViewsOntology Views
FMAHead
Musculo-skeletal
Neck
AbdomenFemalePelvis
BackNon-Eng
EquivSynonyms
Definition
MemberofConstitu-
tionalpartAttributed
partThorax
HeartLungAdjacency Part-Of
Thorax
HeartLungAdjacency Part-Of
Copyright © Daniel L. Rubin 2004 16
Ontology Views for ReasoningOntology Views for Reasoning
● Catalog of organs; controlled vocabulary of names
● Organ parts and compositionality
● Adjacencies for organs and organ parts
● Connectivity
● Containment
● Arterial supply & regional organ perfusion
Copyright © Daniel L. Rubin 2004 17
Organ SubOrgan Sub--Parts: Artery SegmentsParts: Artery Segments
Copyright © Daniel L. Rubin 2004 18
Subdivision of Myocardial Regions Subdivision of Myocardial Regions Based on FMA KnowledgeBased on FMA Knowledge
Regions of Left Ventricle
Copyright © Daniel L. Rubin 2004 19
Other Knowledge SourcesOther Knowledge Sources
● Biomechanics ontologyTissue material propertiesUseful for predicting trajectory of projectiles
● Tissue injury ontologyTaxonomy of types of injuries
● Physiology ontologyKnowledge of how injuries affects circulatory dynamics
“VSKB” = all pertinent ontologies in VS Project
2. Creating patient2. Creating patient--specific modelsspecific models
Copyright © Daniel L. Rubin 2004 21
A “PatientA “Patient--Specific Model (PSM)”Specific Model (PSM)”
● Contains data specific to the patient at a point in time
Patient-specific geometryDescription of projectile path of damageVital signs
● Links to canonical ontologies
● Provides API for queries, reasoning services, and visualization routines
Copyright © Daniel L. Rubin 2004 23
Building Geometric/Knowledge ModelsBuilding Geometric/Knowledge Models
SegmentedCT Volume Data
of the Chest(or Visible Human
Image set)
3-D GeometricModel
BiomechanicalKnowledge
FMA
Reasoning
Display andUser Queries
Visible Human Raw DataVisible Human Raw Data
LVLV
RVRV
RARA
LALA
RAARAA
Visible Human Segmented DataVisible Human Segmented Data
LVLV
RVRV
RARA
LALA
RAARAA
3. Developing reasoning services3. Developing reasoning services
Copyright © Daniel L. Rubin 2004 27
Reasoning TasksReasoning Tasks
● Predict organ injury
● Predict physiological consequences of organ injury
● Classify injuries (trauma score; ICD-10)
● Predict survival
● Decision supportDiagnosis and triageRecommend additional tests
Copyright © Daniel L. Rubin 2004 28
Predicting Organ InjuryPredicting Organ Injury
● Direct organ injury1. Organs visible on CT: injury predicted by
intersection with “cone of damage”2. Organs not visible on CT: inferred from
adjacency knowledge in FMA
● Propagation of organ injury3. Use knowledge of arterial anatomy to
infer downstream consequences of arterial damage
Copyright © Daniel L. Rubin 2004 29
Injury Caused by “Cone of Damage”Injury Caused by “Cone of Damage”
We infer injured tissues using FMA:• names of injured
tissues• knowledge of
organadjacencies
Copyright © Daniel L. Rubin 2004 30
Bullet entryBullet entrysitesite
Area of injuredArea of injuredtissue surfacetissue surface
Conical volumeConical volumeof injuryof injury
Reasoning about injuriesReasoning about injuriesand their consequencesand their consequences
Copyright © Daniel L. Rubin 2004 31
Reasoning using ClassificationReasoning using Classification
● Some reasoning tasks are classification tasksDiagnosisExtent of injuryConsequences of injury (e.g., vascular damage)Triage and associated actions
● Representation of VSKB in Description Logic enables efficient classification
Represent knowledge conducive to classificationInfer non-obvious relationships among concepts (e.g., ischemia)
● This approach builds on current standards for knowledge interoperability (OWL)
Copyright © Daniel L. Rubin 2004 32
OWL model relates anatomic OWL model relates anatomic structures to vascular supplystructures to vascular supply
This organ is defined here
Anatomy OntologyAnatomy Ontology Concept DefinitionsConcept Definitions
Copyright © Daniel L. Rubin 2004 33
OWL automatically infers where OWL automatically infers where distal blood flow is lostdistal blood flow is lost
Asserted arterial occlusion
Inferred arterial insufficiencyInferred arterial insufficiency
If we assert that the RCA is occluded If we assert that the RCA is occluded between conus a. andbetween conus a. andand diagonal a.and diagonal a., we can infer the ischemic consequences, we can infer the ischemic consequences
Copyright © Daniel L. Rubin 2004 34
OWL automatically infers what OWL automatically infers what structures are damagedstructures are damaged
Predicted Predicted regions regions
of heart that of heart that will be ischemicwill be ischemic
Types ofTypes ofischemicischemicorgansorgans
Copyright © Daniel L. Rubin 2004 35
ImplementationImplementation
● VSKB ontolgies in Protégé
● PSMGeometric data objects in ITK (C++)API to read patient data; visualize outputC++/Java interface to link PSM to ontologies
● OWL-based reasoning deployed as a Web service
● Outputs of reasoning updates PSM
Copyright © Daniel L. Rubin 2004 36
C++/Protégé InterfaceC++/Protégé Interface
● Geometric modeling code in C++; Protégé API in Java
● C++/Protégé interface developed using JACE
● Proxy C++ classes created for core Java classes (KnowledgeBase, etc.)
● JVM invoked in C++; direct Protégé API calls via JACE interface
DEMODEMO
Copyright © Daniel L. Rubin 2004 38
ConclusionsConclusions
● Benefits of integrating geometric models with ontologies
Makes anatomic knowledge and relationships explicit and computer-accessibleUseful for reasoning (e.g., propagation of vascular injury)
● Benefits of integrating additional information in Patient Specific Model
Biomechanical and other data for simulationExtensibility to accommodate future data
Copyright © Daniel L. Rubin 2004 39
AcknowledgementsAcknowledgements
● Defense Advanced Research Projects Agency (DARPA)
● Protégé Resource (LM007885)
Thank you.Thank you.
Contact info:Contact info:[email protected]@smi.stanford.edu
SegmentedData
FMAInjury
AdditionalInvisibleObject
Landmarks Biomechanics
GeometricModels FMA
InvisibleObjects
Biomech
Injury
Pre-processing of linkBetween {geometry (image+mesh), biomechanics, etc} and FMA. How? Spatial objects (abstract geometry objects)
Copyright © Daniel L. Rubin 2004 43
Working with the PSM:Working with the PSM:Reasoning and VisualizationReasoning and Visualization
Physiological effects
Patient-Specific Model (PSM)
Visualization
Reasoning
Indirect effects
Tissue damage
Copyright © Daniel L. Rubin 2004 44
Conical regionof tissue injury
Copyright © Daniel L. Rubin 2004 45
Architecture for Integrating Architecture for Integrating Image Data and KnowledgeImage Data and Knowledge
● Provide reasoning services for current and anticipated requirements
● Use blackboard architecture where all reasoners relate to data available in a Patient-Specific Model (PSM)
● Enable all modules in VSP to read and write to PSM at runtime
Copyright © Daniel L. Rubin 2004 46
OntologicOntologic approach to geometric approach to geometric models of anatomymodels of anatomy
● Input: segmented CT data (pre-injury)● Build an integrated 3-d model of anatomy:
Represent 3-d geometry of anatomic structuresIntegrate tissue physical properties, biomechanics, physiology, and clinical parameters (vital signs)Simulate geometric effects of penetrating injury
● Geometric model links to knowledge sources (e.g., anatomy in FMA ontology)
● (Use physiologic models to predict consequence of the injury)
● Display predicted organ injury
Copyright © Daniel L. Rubin 2004 47
Geometric Model BuildingGeometric Model Building
● Organ parts derived from segmented CT data
● Construct mesh models from segmented organ parts using VTK and ITK (added to PSM)
● Biomechanical and other information added to PSM
Tissue physical properties; density
● Knowledge in VSKB is cached in PSM for efficient computation
E.g., Heart pericardium; LA; LV; RA; RV
● This model is extensible; can include other info
Copyright © Daniel L. Rubin 2004 48
GeometryTissue DensityElasticitySurface area…
VSKB Knowledge VSKB Knowledge Cached in PSMCached in PSM
VSKB
Cachedknowledge Rendering of geometric
data associated with cached geometric
knowledge
Copyright © Daniel L. Rubin 2004 49
What Is An Ontology?What Is An Ontology?
● Enumerates concepts, attributes of concepts, and relationships among concepts
Defines a structure (“model”) for the application areaEncodes knowledgeA “knowledge source”
● Can be comprehended by people and processed by machines
● Provides a “domain of discourse” for characterizing some application area; a common vocabulary (shared understanding)
Copyright © Daniel L. Rubin 2004 50
ProtégéProtégé--20002000
● Ontology editor Model concepts, attributes, and relationships
● ToolsVisualize ontologies and knowledge bases
● StorageArchive ontologies and knowledge bases in a variety of formats
● Java API Link knowledge bases to other applications
● A world-wide community of active users
Copyright © Daniel L. Rubin 2004 51
ChallengesChallenges
● Making knowledge explicit and computableGeometric knowledge: implicitly represented in 3-d modelsAnatomic/physiologic knowledge: usually in head of observerThis separation makes automated reasoning difficult
● Integrating and computing with patient data and canonical knowledge
Data: geometry, biomechanics, vital signsKnowledge: anatomy, tissue strain ↔ injuryCombining and using these in reasoning tasks
Copyright © Daniel L. Rubin 2004 52
The Foundational Model of Anatomy The Foundational Model of Anatomy (FMA) Ontology(FMA) Ontology
Material Physical Anatomical Entity
AnatomicalSpatial Entity
Anatomical Structure
Body Substance
BodyPart
OrganSystem OrganismOrgan
PartOrganCell
OrganSubdivision
OrganComponentTissue
Copyright © Daniel L. Rubin 2004 53
Integrating Knowledge and DataIntegrating Knowledge and Data
Patient-Specific Model (PSM)
Knowledge
Biomechanics
Geometry
Injury dataVital Signs
Copyright © Daniel L. Rubin 2004 54
PSM Links VSKB to PSM Links VSKB to Runtime DataRuntime Data
Cached knowledge in PSM derived from anatomy ontologyCached knowledge in PSM derived from anatomy ontology
Each node above is an object containing Each node above is an object containing patientpatient--specific specific knowledgeknowledge: anatomy, geometry, biomechanics, tissue : anatomy, geometry, biomechanics, tissue
damage, physiologydamage, physiology
Copyright © Daniel L. Rubin 2004 55
Knowledge about Vascular SupplyKnowledge about Vascular Supply
Copyright © Daniel L. Rubin 2004 56
Knowledge for Injury ReasoningKnowledge for Injury Reasoning
Part-ofknowledge
Other usefulknowledge
Copyright © Daniel L. Rubin 2004 57
Key knowledgeused in
reasoning