complex question answering based on a semantic domain model of clinical medicine dina demner-fushman...
Post on 21-Dec-2015
230 views
TRANSCRIPT
COMPLEX QUESTION ANSWERING BASED ON
A SEMANTIC DOMAIN MODEL OF CLINICAL MEDICINE
Dina Demner-Fushman
24 August 2006
2
Informed clinical decision making
Evidence Based Medicine• Combine:
– the best published medical research findings
– clinical judgment– expertise and experience
• Use systematic approach:– Formulate specific and
relevant questions– Know where to look for an
answer– Answer questions
iteratively
clinical stateand circumstances
patient’spreferences
information resources
3
Outline
• Motivation • Evidence Based Medicine• Hypotheses• Clinical Question Answering system• Evaluation
– System components• extractors• Document re-ranking
– Answers• Multi-tier answers• Best answers
• Contributions, Limitations, Future work
4
Real-life questions
• How do we diagnose prostatitis?– Interactive multi-level answers:
Diagnosis of chronic abacterial prostatitis: evaluate infection inflammation biochemistry ultrasonography …
• How much better is Amiodarone in controlling fast atrial fibrillation with rapid ventricular response compared to cardizem?
• Zosyn dosage regimens: 3.375g or 4.5g?– Best answer:
The usual total daily dose of Zosyn for adults is 3.375 g every six hours. Patients with nosocomial pneumonia should start with Zosyn at a dosage of 4.5 g …
5
Evidence search and appraisal
• Convert information needs into focused questions
• Track down the best evidence with which to answer them
• Identify “bottom-line” recommendations, supporting evidence, and its strength
6
Information sources
• Paper (books, desk references, journals): 50%• Colleagues: 40%• Clinical librarians or services: 32%• Online Resources: 25%
– Primary sources:Bibliographic databases (MEDLINE)
– Secondary sources:Systematic reviews (Cochrane collaboration, American College
of Physicians Journal Club)
Databases of expert answers to clinical questions (FPIN, BMJ Clinical Evidence)
7
Question frameHow much better is Amiodarone in controlling fast atrial fibrillation with rapid ventricular response compared to cardizem?
Etiology Diagnosis
Task: Therapy Prognosis
Population: [unspecified]Problem: Atrial fibrillation with
rapid ventricular response Intervention 1: Amiodarone Intervention 2: Cardizem Outcome: achieve rate control
Users' Guides to Evidence-based Medicine (JAMA series)
P
ICO
8
Strength of Evidence
• A– Meta-Analysis– Randomized Controlled Trials
• B – Cross-Sectional Studies– Retrospective Studies
• C– Case Report– Animal Studies
9
Hypotheses
• Document frames based on three main EBM components (clinical task, PICO, SoE) are sufficient to answer questions
– Document frames could be generated using a hybrid statistical/knowledge-based approach to leverage existing resources
– Complex clinical questions could be answered through semantic matching of the question-document frames
10
MEDLINE
Clinical QA system architecture
Entities & relations
annotation
PubMed Document Retrieval
Query termsE-Utilities
citationsMetaMap SemRep
Semantic matchingAnswer
Generation
Document frame
Question frame
Answer
Annotated citations
PICO Query Formulation
UMLSEBM
Domain Model
Semantic processing
Knowledge Extraction
Clinical Task Classification
Strength of Evidence
Classification
11
Component architecture
Search Engine Wrapper
Question Processing
MetaMap Wrapper
Semantic Matcher
Answer Generator
Citations
Query
MEDLINEAnnotated Document
Semantic processor
Task Classifier
Strength of Evidence Classifier
Problem Extractor
Population Extractor
Intervention Extractor
Outcome Extractor
ESearchEFetch
Question frame
Document frame
Answer
12
Semantic processor
Task Classifier
Strength of Evidence Classifier
Problem Extractor
Population Extractor
Intervention Extractor
Outcome Extractor
Semantic processing example
… Patients with atrial fibrillation (n = 57), … were randomly assigned to one of three intravenous treatment regimens.
Amiodarone versus diltiazem for rate control in critically ill patients with atrial tachyarrhythmias.
Group 1 received diltiazem … group 2 received amiodarone ….
Sufficient rate control can be achieved in critically ill patients with atrial tachyarrhythmias using either diltiazem or amiodarone …
Task: Therapy
Strength of Evidence: A (RCT)
13
Outcome Extractor
Classifiers
Cue-terms
Naïve Bayes
N-gram
Position
Heuristic
Length
Multiple Linear
Regression
Score: 0.99Sufficient rate control can be achieved in critically ill patients with atrial tachyarrhythmias using either diltiazem or amiodarone.
Score: 0.75Although diltiazem allowed for significantly better 24-hr heart rate control, this effect was offset by a significantly higher incidence of hypotension requiring discontinuation of the drug.
Problem Extractor
Population Extractor
Intervention Extractor
Training: 275 manually annotated abstracts
(x)PαMLR(x) k
K
kk
14
Extractor accuracyExtractor Test
set N=Correct Unknown Wrong
Problem 50 90% 5% 5%
Population 100 79% 11% 10%
Intervention 100 77% 23%
Outcome 358 90% 10%
15
Outline
• Motivation • Evidence Based Medicine• Hypotheses• Clinical Question Answering system• Evaluation
– System components• extractors• Document re-ranking
– Answers• Multi-level answers• Best answers
• Contributions, Limitations, Future work
16
Semantic Matcher Doc3 frameProblem: atrial fibrillationIntervention: coronary surgery Outcome: ……Outcome score: Pico score: Task: THERAPY score: SoE score:
Document re-ranking
Question frameTask: THERAPYProblem: atrial fibrillation Intervention: Amiodarone
Cardizem
Doc Score = λPSPICO + λSSSoE + λTSTask
SPICO = λpSproblem + λptSpopulation
+ λiSintervention + λoSoutcome
SSoE = λjSjournal + λsSstudy + λdSdate
STask=ΣλiTask_Indicator(i)
Doc2 frameProblem: arterial hypertensionIntervention: Warfarin Outcome: ……Outcome score: Pico score: Task: THERAPY score: SoE score:
Doc1 frameProblem: atrial fibrillation Intervention: diltiazem, amiodaroneOutcome: ……Outcome score: 0.79Pico score: 0.89Task: THERAPY score: 0.64SoE score: 0.32
17
Document re-ranking evaluation
baseline filtering components
Relevance judgments for 24 FPIN questions by Dr. CS
18
Answer Generation
Intervention Extractor
Outcome Extractor
Semantic Clustering
Imaging by method
[ultrasound][Doppler studies]
Transrectal ultrasound (TRUS) offers a valuable complement to digital rectal examination (DRE) in diagnosing prostate diseases. A sensitivity of 90.6% and a specificity of 64.2% was reached.
Automated analysis and interpretation of transrectal ultrasonography images in patients with prostatitis.
Eur Urol. 1995;27(1):47-53. Metadata SoE Task
Cluster label
Intervention
Outcome
Title
19
UMLS Semantic Clustering
Magnetic resonanceDoppler studies
MRI of abdomen Specific ultrasound
studies
Imaging by method
Imaging by body site Ultrasound scan
Evaluation procedure
Procedure by method
Diagnostic imaging …
Procedures
Investigations
SNOMED Clinical Terms
Operations, procedures and interventions
Read Codes
Pruned top
Interior nodes
Extracted interventions
Ultrasonography
20
Q1
Q2
30
Q1
Q2
30
Q1
Q2
25
Cluster selection for evaluation
UMLS (latest,3 largest clusters)
User (latest,3 best clusters)
Pubmed (3 latest)
Imaging by method
biochemistry
good OK bad
infection
inflammation
21
Answer evaluation
Clinical Evidence categories
Beneficial Harmful
B LB T U+N H
0.13 0.25 0.13 0.46 0.01
0.23 0.27 0.12 0.37 0.01
0.35 0.28 0.11 0.26 -
Distribution for 25 Clinical Evidence questions (cluster selection and judgment by Dr CA)
Cluster selection strategy
Evidence support
good OK bad
PubMed 0.57 0.14 0.27
UMLS 0.72 0.09 0.19
User 0.85 0.08 0.07
22
Outline
• Motivation • Evidence Based Medicine• Hypotheses• Clinical Question Answering system• Evaluation
– System components• extractors• Document re-ranking
– Answers• Multi-level answers• Best answers
• Contributions, Limitations, Future work
23
Answer precision at 5
221 answers to 24 questions judged by Drs. CS and KWH
24
Contributions• Leveraging semantic domain model as a foundation for
an end-to-end clinical question answering system.
• Identification of the domain-model components necessary and sufficient for system development.
• Demonstration of applicability of the system architecture for complex question answering in the clinical domain.
• Methods for combining information extraction based on statistical and knowledge-based methods.
• Adaptation of question answering evaluation methods for the clinical domain.
• Development of test collections for information extraction and question answering evaluation.
25
Limitations
• No user interface
• Manual question processing
• PubMed for document retrieval
• Processing speed of automatic semantic annotation
26
Future work
• Combining knowledge-based and corpus-based methods beyond outcome extractor
• Developing a corpus-based stopping condition for hierarchical ontological clustering
• In-depth study of PICO frame alternatives
• Combining ranking results of different search engines
27
Thanks to my advisory cloud!
Douglas Oard
Jimmy Lin
Philip Resnik
Dagobert Soergel
Ben Shneiderman
Susan Hauser
Thomas Rindflesch
George Thoma
Alan Aronson
Susanne Humphrey