1

KMeD: A Knowledge-Based KMeD: A Knowledge-Based Multimedia Medical Database SystemMultimedia Medical Database System

Wesley W. ChuWesley W. ChuComputer Science DepartmentComputer Science Department

University of California, Los AngelesUniversity of California, Los Angeles

http://www.cobase.cs.ucla.eduhttp://www.cobase.cs.ucla.edu

2

KMeD

A Knowledge-Based Multimedia Medical Distributed Database System

A Cooperative, Spatial, Evolutionary Medical Database System

Knowledge-Based Image Retrieval with Spatial and Temporal Constructs

Wesley W. ChuWesley W. Chu Computer Science DepartmentComputer Science DepartmentAlfonso F. CardenasAlfonso F. Cardenas Computer Science DepartmentComputer Science DepartmentRicky K. TairaRicky K. Taira Department of Radiological SciencesDepartment of Radiological Sciences

October 1, 1991 to October 1, 1991 to September 30, September 30,

19931993

July 1, 1993 to July 1, 1993 to June 30, 1997June 30, 1997

May 1, 1997 toMay 1, 1997 toApril 30, 2001April 30, 2001

3

Research Team

StudentsJohn David N.

DionisioChih-Cheng HsuDavid JohnsonChristine Chih

CollaboratorsComputer Science

DepartmentAlfonso F. Cardenas

UCLA Medical SchoolDenise Aberle, MDRobert Lufkin, MDRicky K. Taira, MD

4

Significance

Query multimedia data based on image content and spatial predicatesUse domain knowledge to relax and interpret medical queriesPresent integrated view of multiple temporal and evolutionary data in a timeline metaphor

5

Overview

Image retrieval by feature and contentQuery relaxationSpatial query answeringSimilarity query answeringVisual query interfaceTimeline interfaceSample cases

6

Image Retrieval by Content

Featuressize, shape, texture, density, histology

Spatial Relationsangle of coverage, shortest distance, overlapping ratio, contact ratio, relative direction

Evolution of Object Growthfusion, fission

7

8

9

10

11

Characteristics of Medical Queries

MultimediaTemporalEvolutionarySpatialImprecise

12

O O’

01

Om

O O

01

On

Evolution: Object O evolves into a new object O’

Fusion: Object 01, …, Om fuse into a new object

Fission: Object O splits into object 01, …, On

13

Case a:

Case c:

The object exists with its supertype or aggregated type.

The life span of the object starts with and ends before its supertype or aggregated type.

Case b:

Case d:

The life span of the object starts after and ends with its supertype or aggregated type.

The life span of the object starts after and ends before its supertype or aggregated type.

14

Lesion

Micro-Lesion

Micro-Lesion

15

16

17

18

19

Query Modification Techniques

RelaxationGeneralizationSpecialization

Association

20

Generalization and Specialization

More Conceptual Query

Specific Query

Conceptual Query Conceptual Query

Specific Query

Generalization

SpecializationGeneralization

Specialization

21

Type Abstraction Hierarchy

Presents abstract view ofTypesAttribute valuesImage featuresTemporal and evolutionary behaviorSpatial relationships among objects

Provides multi-level knowledge representation

22

TAH Generation for Numerical Attribute Values

Relaxation ErrorDifference between the exact value and the returned approximate valueThe expected error is weighted by the probability of occurrence of each value

DISC (Distribution Sensitive Clustering) is based on the attribute values and frequency distribution of the data

23

TAH Generation for NumericalAttribute Values (cont.)

Computation Complexity: O(n2), where n is the number of distinct value in a clusterDISC performs better than Biggest Cap (value only) or Max Entropy (frequency only) methodsMDISC is developed for multiple attribute TAHs. Computation Complexity: O(mn2), where m is the number of attributes

24

Query Relaxation

RelaxAttribute

Query

Yes

Display

QueryModification

AnswersDatabase

TAHs

No

25

Cooperative Querying for Medical Applications

QueryFind the treatment used for the tumor similar-to (loc, size) X1 on 12 year-old Korean males.

Relaxed QueryFind the treatment used for the tumor Class X on preteen Asians.

AssociationThe success rate, side effects, and cost of the treatment.

26

Type Abstraction Hierarchies forMedical Domain

Age

Preteens

910 1112

Teen Adult

Ethnic Group

Asian

Korean Chinese Japanese Filipino

African European

Tumor (location, size)

Class X

[loc1 loc3]

[s1 s3]

Class Y

[locY sY]

X1

[loc1 s1]

X2

[loc2 s2]

X3

[loc3 s3]

27

Knowledge-Based Image Model

Representation Level(features and contents)

Brain TumorLateral

Ventricle

TAHSR(t,b)

TAHTumor Size

TAHSR(t,l)

TAHLateral

Ventricle

SR: Spatial Relationb: Braint: Tumorl: Lateral Ventricle

Knowledge Level

Schema LevelSR(t,b) SR(t,l)

28

KnowledgeBasedQueryProcessing

Queries

Query Analysis andFeature Selection

Knowledge-BasedContent Matching

Via TAHs

Query Relaxation

Query Answers

29

User Model

To customize query conditions and knowledge-based query processing

User typeDefault Parameter ValuesFeature and Content Matching Policies

Complete MatchPartial Match

30

User Model (cont.)

Relaxation Control Policies

Relaxation Order

Unrelaxable Object

Preference List

Measure for Ranking

31

32

33

Query Preprocessing

Segment and label contours for objects of interestDetermine relevant features and spatial relationships (e.g., location, containment, intersection) of the selected objectsOrganize the features and spatial relationships of objects into a feature databaseClassify the feature database into a Type Abstraction Hierarchy (TAH)

34

Similarity Query Answering

Determine relevant features based on query inputSelect TAH based on these featuresTraverse through the TAH nodes to match all the images with similar features in the databasePresent the images and rank their similarity (e.g., by mean square error)

35

Spatial Query Answering

PreprocessingDraw and label contours for objects of interestDetermine relevant features and spatial relationships (e.g., location, containment, intersection) of the selected objectsOrganize the features and spatial relationships of objects into a feature databaseClassify the feature database into a type abstraction hierarchy (TAH)

36

Spatial Query Answering (cont.)

ProcessingSelect TAH based on t he query conditions and contextSearch nodes to match the query conditionsReturn images linked to the TAH node

37

Similarity Query Answering

PreprocessingSelect objects and specify features of interest in the imageCreate a feature database of the selected objects for all imagesClassify the feature databases as type abstraction hierarchies

38

Similarity Query Answering (cont.)

ProcessingDetermine relevant features based on query inputSelect TAH based on these features (interact with user to resolve ambiguity)Traverse through the TAH nodes to match all the images with similar features in the databasesPresent the images and rank their similarity (e.g., by mean square error)

39

40

Visual Query Language and Interface

Point-click-drag interfaceObjects may be represented iconicallySpatial relationships among objects are represented graphically

41

Visual Query Example

Retrieve brain tumor cases where a tumor is located in the region as indicated in the picture

42

43

44

45

46

47

48

49

50

51

52

Implementation

Sun Sparc 20 workstations (128 MB RAM, 24-bit frame buffer)Oracle Database Management SystemX/Motif Development Environment, C++Mass Storage of Images (9 GB)

53

54

55

56

57

Conclusions

Image retrieval by feature and contentMatching and relaxation images based on featuresProcessing of queries based on spatial relationships among objectsAnswering of imprecise queriesExpression of queries via visual query languageIntegrated view of temporal multimedia data in a timeline metaphor

58

59

60

Semi-Automatic Segmentation of Lung Tumors

classificationclassification seedestimation

seedestimation

adaptive fusionadaptive fusion

regiongrowing

regiongrowing

tumorsegment

tumorsegment

interesting area

61

62