1

KMeD: A Knowledge-Based KMeD: A Knowledge-Based Multimedia Medical Database Multimedia Medical Database

SystemSystem

Wesley W. ChuComputer Science Department

University of California, Los Angeles

http://www.cobase.cs.ucla.edu

2

KMeD

A Knowledge-Based Multimedia Medical Distributed Database System

A Cooperative, Spatial, Evolutionary Medical Database System

Knowledge-Based Image Retrieval with Spatial and Temporal Constructs

Wesley W. Chu Computer Science DepartmentAlfonso F. Cardenas Computer Science DepartmentRicky K. Taira Department of Radiological Sciences

October 1, 1991 to September 30, 1993

July 1, 1993 to June 30, 1997

May 1, 1997 toApril 30, 2001

3

Research Team

StudentsJohn David N.

DionisioChih-Cheng HsuDavid JohnsonChristine Chih

CollaboratorsComputer Science

DepartmentAlfonso F. Cardenas

UCLA Medical SchoolDenise Aberle, MDRobert Lufkin, MDRicky K. Taira, MD

4

A NIH Grant at UCLA (2001-2005)

A Medical Digital library---A Digital File Room for Patient Care, Education, and Research

Wesley W. Chu, PhDWesley W. Chu, PhD

Hooshang Kangarloo, Hooshang Kangarloo, MDMD

Usha Sinha, PhDUsha Sinha, PhD

David B. Johnson, David B. Johnson, PhDPhD

Bernard Churchill, Bernard Churchill, MDMD

5

Significance

Query multimedia data based on image content and spatial predicatesUse domain knowledge to relax and interpret medical queriesPresent integrated view of multiple temporal and evolutionary data in a timeline metaphorRetrieve Scenario Specific Free-text documents in a Medical Digital Library

6

Overview

Image retrieval by feature and contentQuery relaxationSpatial query answeringSimilarity query answeringVisual query interfaceTimeline interfaceRetrieval of scenario specific free text medical documents

7

Image Retrieval by Content

Features size, shape, texture, density, histology

Spatial Relations angle of coverage, shortest distance,

overlapping ratio, contact ratio, relative direction

Evolution of Object Growth fusion, fission

8

9

10

11

12

Characteristics of Medical Queries

MultimediaTemporalEvolutionarySpatialImprecise

13

O O’

01

Om

O O

01

On

Evolution: Object O evolves into a new object O’

Fusion: Object 01, …, Om fuse into a new object

Fission: Object O splits into object 01, …, On

Representing of Temporal and Evolution Objects

14

Case a:

Case c:

The object exists with its supertype or aggregated type.

The life span of the object starts with and ends before its supertype or aggregated type.

Case b:

Case d:

The life span of the object starts after and ends with its supertype or aggregated type.

The life span of the object starts after and ends before its supertype or aggregated type.

Representing of Temporal and Evolution Objects (cont)

15

Lesion

Micro-Lesion

Micro-Lesion

An Example of Temporal and Evolution Object

16

17

18

Spatial Distance and Angle of Coverage of Two Objects

19

20

Query Modification Techniques

Relaxation Generalization Specialization

Association

21

Generalization and Specialization

More Conceptual Query

Specific Query

Conceptual Query Conceptual Query

Specific Query

Generalization

SpecializationGeneralization

Specialization

22

Type Abstraction Hierarchy

Presents abstract view of Types Attribute values Image features Temporal and evolutionary behavior Spatial relationships among objects

Provides multi-level knowledge representation

23

TAH Generation for Numerical Attribute Values

Relaxation Error Difference between the exact value and the

returned approximate value The expected error is weighted by the

probability of occurrence of each value

DISC (Distribution Sensitive Clustering) is based on the attribute values and frequency distribution of the data

24

TAH Generation for Numerical Attribute Values (cont.)

Computation Complexity: O(n2), where n is the number of distinct value in a cluster

DISC performs better than Biggest Cap (value only) or Max Entropy (frequency only) methods

MDISC is developed for multiple attribute TAHs. Computation Complexity: O(mn2), where m is the number of attributes

25

Query Relaxation

RelaxAttribute

Query

Yes

Display

QueryModification

AnswersDatabase

TAHs

No

26

An Cooperative Query Answering Example

Query Find the treatment used for the tumor similar-

to (loc, size) X1 on 12 year-old Korean males.

Relaxed Query Find the treatment used for the tumor Class X

on preteen Asians.

Association The success rate, side effects, and cost of

the treatment.

27

Type Abstraction Hierarchies for Medical Domain

Age

Preteens

910 1112

Teen Adult

Ethnic Group

Asian

Korean Chinese Japanese Filipino

African European

Tumor (location, size)

Class X

[loc1 loc3]

[s1 s3]

Class Y

[locY sY]

X1

[loc1 s1]

X2

[loc2 s2]

X3

[loc3 s3]

28

Knowledge-Based Image Model

Representation Level(features and contents)

Brain TumorLateral

Ventricle

TAHSR(t,b)

TAHTumor Size

TAHSR(t,l)

TAHLateral

Ventricle

SR: Spatial Relationb: Braint: Tumorl: Lateral Ventricle

Knowledge Level

Schema LevelSR(t,b) SR(t,l)

29

Queries

Query Analysis andFeature Selection

Knowledge-BasedContent Matching

Via TAHs

Query Relaxation

Query Answers

Knowledge-based Query Processing

30

User Model

To customize query conditions and knowledge-based query processing

User typeDefault Parameter ValuesFeature and Content Matching Policies Complete Match Partial Match

31

User Model (cont.)

Relaxation Control Policies Relaxation Order Unrelaxable Object Preference List

Measure for Ranking

33

34

Query Preprocessing

Segment and label contours for objects of interestDetermine relevant features and spatial relationships (e.g., location, containment, intersection) of the selected objectsOrganize the features and spatial relationships of objects into a feature databaseClassify the feature database into a Type Abstraction Hierarchy (TAH)

35

Similarity Query Answering

Determine relevant features based on query inputSelect TAH based on these featuresTraverse through the TAH nodes to match all the images with similar features in the databasePresent the images and rank their similarity (e.g., by mean square error)

40

41

Visual Query Language and Interface

Point-click-drag interfaceObjects may be represented iconicallySpatial relationships among objects are represented graphically

42

Visual Query Example

Retrieve brain tumor cases where a tumor is located in the

region as indicated in the picture

43

44

45

46

47

48

49

A Visual Query Example

50

A Visual Temporal Query Example

53

Implementation

Sun Sparc 20 workstations (128 MB RAM, 24-bit frame buffer)Oracle Database Management SystemX/Motif Development Environment, C++Mass Storage of Images (9 GB)

54

55

56

58

Summary I

Image retrieval by feature and contentMatching and relaxation images based on featuresProcessing of queries based on spatial relationships among objectsAnswering of imprecise queriesExpression of queries via visual query languageIntegrated view of temporal multimedia data in a timeline metaphor

59

A Knowledge-based Approach to Retrieve Scenario Specific Free-text in a Medical Digital Library

60

NIH Program Project Grant (2000-2005)

A 5 year $ 10M joint interdisciplinary project between Medical School & CS faculty

Project 1-- teleradaiology infrastructure

Project 2-- neuroradiology workstation

Project 3-- multimedia information architecture

Project 4-- natural language processing for medical reports

Project 5-- medical digital library

61

Project 5 Personnel

Graduate students:Victor Z. LiuWenlei MaoQinghua Zou

Consultants:Hooshang Kangaloo, M.D.Denies Aberle, M.D.

Project leader: Wesley W. Chu

62

Data in a Medical Digital Library

Structured data (patient lab data, demographic data,…)--CoBaseImages (X rays, MRI, CT scans)--KMeDFree-text Patient reports Teaching files Literature News articles

63

System Overview

Patient reports

Medical literature

Medical Digital Library(MDL)

Teaching materials

Query results

Ad-hoc query

Patient report for content correlation

News Articles

64

Treatment-related articles

??? How to treat the disease

Diagnosis-related articles

??? How to diagnose the disease

Scenario Specific Retrieval

…Tissue Source:LUNG (FINE NEEDLE

ASPIRATION) (LEFT LOWER LOBE)

…FINAL DIAGNOSIS:

- LUNG NODULE, LEFT LOWER LOBE (FINE NEEDLE ASPIRATION):- LUNG CANCER, SMALL CELL, STAGE II.

…

65

Challenge I: Indexing

Extracting domain-specific key concepts in the free text for indexing Free-text: Lung cancer, small cell, stage II

Concept terms in knowledge source: stage II small cell lung cancer

Conventional methods use NLP Not scalable Cannot adapt to various forms of word permutation

66

Challenge II: Terms used in the query are too general

Expanding the general terms in the query to specific terms that are used in the document

Query: lung cancer, diagnosis options

Document: … the effectiveness of chest x-ray and bronchography on patients with lung cancer …

?√

Query: lung cancer, chest x-ray, bronchography, …

67

Challenge III: Mismatching between terms used in query and documents

Example

Query: … lung cancer, …

Document 3: anti-cancerdrug combinations…

?? ?Document 1: … lung carcinoma …

Document 2: … lung neoplasm …

81

Application: Query Answering via Templates

Sample templates:“<disease>, treatment,”“<disease>, diagnosis ”

QueryExpansion

…Template:“<disease>, treatment”

lung cancer

lung cancerradiotherapychemotherapycisplatin

relevant documents

IndexFinder

lung cancer,treatment

Phrase-basedVSM

82

Application: Scenario Specific Content Correlation

Query Templates Scenario

Selection

e.g. treatment, diagnosis, etc.

PatientReport

QueryExpansion

…

relevant documents

Phrase-basedVSM

IndexFinder

83

Summary of MDL

Knowledge based (UMLS) approach provides scenario-specific medical free-text retrieval IndexFinder – use word permutation as well as syntactic and

semantic filtering to extract domain-specific key concepts in the free text for indexing

Knowledge-based query expansion – transform general terms in the query into the scenario specific terms used in the documents, giving the query a higher probability of matching with the relevant documents

Phrase based indexing – transform document indexing into phrase paradigm (concept and its word stems) to improve retrieve effectiveness

84

Acknowledgement

This research is supported in part by NIC/NIH Grant#4442511-33780

88

Demo http://fargo.cs.ucla.edu/umls/search.aspx

Test Texts

• Technically successful left lower lobe nodule biopsy.

• Preliminary localization CT images again demonstrate a left lower lobe nodule adjacent to the posterior segmental bronchus.

• CT scans obtained during biopsy demonstrate the coaxial cannula adjacent to the proximal aspect of the nodule.

• Surrounding pulmonary parenchymal hemorrhage as a result of the biopsy is also noted.

• There may be a tiny left apical air collection in the pleural space lateral to the apical bulla.

• Formal cytologic evaluation of the withdrawn specimen is pending at this time, although abnormal appearing "spindle" cells were identified during on-site cytopathologic evaluation of specimen adequacy.

102

107

108

109

110

111

112