student :erez shalom
DESCRIPTION
An Evaluation of a Methodology for Specification of Clinical Guidelines at Multiple Representation Levels. Student :Erez Shalom. Supervisors: Prof. Yuval Shahar Dr. Meirav Taieb-Maymon. Talk Roadmap :. Background Methods Results Conclusions Future Directions. Clinical Guidelines. - PowerPoint PPT PresentationTRANSCRIPT
An Evaluation of a Methodology for Specification of Clinical Guidelines at Multiple Representation Levels
Student :Erez Shalom
Supervisors:
Prof. Yuval Shahar
Dr. Meirav Taieb-Maymon
ISE Dep. Seminar 26/4/06
Background Methods Results Conclusions Future Directions
Talk Roadmap :
ISE Dep. Seminar 26/4/06
Clinical Guidelines Textual documents describing “state of
the art” patient management A powerful method to standardize and
improve the quality of medical care Usually specify diagnostic and/or
therapeutic procedures
ISE Dep. Seminar 26/4/06
Subsections Describing
patient diagnosis and
treatment
ISE Dep. Seminar 26/4/06
The Need for Automation of Clinical Guidelines
Automatic support provides: Visual specification Search and retrieval Application of a GL Retrospective quality assurance
However: Most GLs are text based and electronic inaccessible at the point of care
ISE Dep. Seminar 26/4/06
The Required Infrastructure A machine-comprehensible GL representation
ontology (e.g., Asbru ontology) Runtime GL application and QA tools
A preliminary engine, namely , Spock was already developed in our lab by [Young,2005]
Support for a gradual structuring of the GL (from text to an executable code)
ISE Dep. Seminar 26/4/06
The Structuring ProcessThe Guideline as a tree of plansThe Guideline as a text document
Involves 2 main types of knowledge:
Procedural knowledge – e.g. Regimen A for administer the two medications in parallel
In parallel
D D
Declarative knowledge - e.g. 2 g IV
Regimen A
Cefotetan Doxycline
P
PP
ISE Dep. Seminar 26/4/06
Sample GL Modeling Methods
Method Knowledge Acquisition tool EON and SAGE, Prodigy, GLIF
A Protégé-based interface
PROforma Arrezo
GEM GEM-Cutter
GLARE "CG_AM" graphical interface
GUIDE NEWGUIDE
AsbruAsbru-View, GMT, Stepper
ISE Dep. Seminar 26/4/06
//Check HGBIf(HGB > 12)
{...}
ExpertPhysician
Collaboration KnowledgeEngineer
Semanticmarkup
Adding controlstructure
Formalizing toexecutable code
DeGeL KB
//Check HGBIf(HGB > 12)
{...}
Free textGuidelines
Semi- StructuredLevel
Semi- FormalLevel
FormalLevel
Expert Physician Knowledge Engineer
The Hybrid Representation ModelGradually structuring the GL using increasingly formal representation levels
Implemented as part of the Digital Electronic Guideline Library) DeGeL)
Used within the URUZ GL markup tool
ISE Dep. Seminar 26/4/06
Asbru- the Underlying Guideline-Representation Ontology
Conditions KR-Class (e.g., the filter condition, and the abort condition)
Plan-body KR-Class for the GL’s Control structures (e.g., sequential, concurrent, and repeating combinations of actions or sub-guidelines),
GL’s Goals KR-Class (e.g. process and outcome intentions),
Context KR-Class of the activities in the GL (e.g. actors, clinical-context).
Includes semantic Knowledge Roles (KRs) organized in KR-Classes such as:
ISE Dep. Seminar 26/4/06
URUZ (I): Specification of declarative knowledge
Expert physician Selects “filter condition” knowledge role
ISE Dep. Seminar 26/4/06
URUZ (cont’d) :Specification of Procedural Knowledge
Expert physician decomposing the GL into tree of plans
ISE Dep. Seminar 26/4/06
GL Specification: Core Issues
√ Expert Physicians (EPs) - Knowledge Engineers (KEs) collaboration
√ Incremental Specification√ Treatment of Multiple Ontologies √ Distributed Collaboration and Sharing√ Text Based Source
√ Knowledge Conversion
ISE Dep. Seminar 26/4/06
Several unresolved issues: Definition of the necessary steps for the GL
specification process Use optimally of EPs and KEs in the process Evaluation is crucial for quantify the markups quality
To Achieve high quality of markups there is a Need for:• An overall process of guideline specification • A complete evaluation methodology
ISE Dep. Seminar 26/4/06
√ Background Methods Results Conclusions Future Directions
Talk Roadmap
ISE Dep. Seminar 26/4/06
The Overall Process of Guideline Specification
The activities in the markup process include three main phases :
1) Preparations before the markup activities
2) Actions during the Markup activities
3) After Markup activity
ISE Dep. Seminar 26/4/06
A Methodology Specification of Clinical Guidelines
Evaluatingtogether
the Markups
EP
KE
Af ter Markup
9.
KE
KE
Before Markup
1.
KE ChooseSpecification
Language
KEsInstructing
the EPs
2.
EP
KE
Choosingtogether GL
forSpecification
EP
KE
3.Creating
together anOntology-Specific
Consensus
4.
EP
KE
EPs Trainingin the
Markup Tool
5.
EP
KE
Creatingtogether
GoldStandardMarkup
EP
6.
KE
KE
EPs Classifythe GL
EPs PerformMarkup
During Markup
EPEP
7. 8.
Creating a consensus is a crucial, mandatory step before markup
ISE Dep. Seminar 26/4/06
The Importance of Using an Ontology-Specific Consensus (OSC)
An OSC is a structural document that describes schematically the clinical directives of the GL
Described by the semantic of the specification ontology
Prevent disagreement and a great deal of variability among the EPs
ISE Dep. Seminar 26/4/06
Methodology for Creation of OSC
The OSC is created in a iterative fashion by performing the following steps 1. First, we create a preliminary structure of the clinical
pathway
2. The KE adds procedural, control structure
3. The KE adds declarative concepts for each defined step
4. After some iteration of steps 2 and 3, an OSC is formed
ISE Dep. Seminar 26/4/06
The second stage in forming a consensus
YESNOis PID
severe? (*)
Patient Treatment and evaluation
Hospitalization anddischarged(sequential)
OutpatientTreatment
Outpatient Treatment andevaluation (parallel)
Outpatient Evaluationand follow up
See 1.2See 1.7
See 1.8
Hospitalization
Post DischargedSee 1.5
ISE Dep. Seminar 26/4/06
Evaluation Design
Amount of Expertise The acquired knowledge domain The Ontology Specific Consensuses The Gold Standard markup for each GL The Markups for each GL The Evaluation of markups
Considered some specific Criteria :
ISE Dep. Seminar 26/4/06
Evaluation Design (cont’d)
Three GLs in different domains were used :Pelvic Inflammatory Disease (PID)Chronic Obstructive Pulmonary
Disease (COPD) Hypothyroidism(HypoThyrd)
ISE Dep. Seminar 26/4/06
Evaluation Design (cont’d)
HypoThyrd
COPD
PID
GLSources
EP
KE
HypoThyrd
COPD
PID
One OntologySpecific Consensus
For each GL
EP
MarkupMarkup
EP
MarkupMarkup
MarkupMarkup
HypoThyrd
HypoThyrd
COPD COPD
PIDPID
Two markupsfor each GL
EP
PI DGoldStandardGold
Standard
GoldStandard
HypoThyrd
COPD
One Gold StandardMarkup f or each GL
PI D
EP
KE
Each markup isevaluated compare to
the Gold Standard
EP
ISE Dep. Seminar 26/4/06
Research Questions Is markup feasible by EPs? Is there a difference between the EPs
editing the same GLs , and same EPs editing difference GLs?
Is there a difference between the KRs across all EPs?
Is there a difference in the amount of errors when using different OSC?
ISE Dep. Seminar 26/4/06
Evaluation of markups
Subjective Measures - Questionnaires were administered for finding the EPs attitude regarding the specification process
Objective Measures – in two scales (compared to the GS):
* Completeness of the markup * Correctness of the markup
ISE Dep. Seminar 26/4/06
The Objective Measures - Completeness
A CB
GS Markup
ISE Dep. Seminar 26/4/06
* Clinical Measure (CM) – measure the clinical correctness of the content* Asbru Semantic Measure (ASM) - measure the semantics correctness of the content ( Asbru semantic in our case)
The Objective Measures - Correctness
ISE Dep. Seminar 26/4/06
Resolution of Measure Mean (weighted) Quality Score (MQS) for: GLs - to find common trends in a GL, and in all GLs
EPs - to find trends in between the markups of the
EPs across the same GL and between GLs KRs - to find trends in a specific KR type and
common trends across KRs and KR classes across one markup, GL and in all GLs
ISE Dep. Seminar 26/4/06
General errors classified into two types, and thus into two corresponding scales:
Clinical errors: Clinical content not accurate Clinical semantics not well specified Clinical content not complete.
Asbru semantics errors: Asbru semantics content not accurate Asbru semantics content not well specified The content does not includes mappings to standard terms The necessary knowledge is not defined in the guideline
knowledge when it should be.
The Objective Measures – Types of Errors
ISE Dep. Seminar 26/4/06
Specific errors for each KR Type a specific error, for example :
Conditions /Intentions KRs: There are no And/Or operators between the different criteria.
Simple Action Plan-Body Type: Has no text content describing the plan Has no single atomic action semantics with clear specification
and description for the action to be performed.
Plan Activation Plan-Body Type: Plan name is not defined Defined plan does not exist in DeGeL.
The Objective Measures – Types of Errors
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool (Cont’d)
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool (Cont’d)
ISE Dep. Seminar 26/4/06
√ Background√ Methods Results Conclusions Future Directions
Talk Roadmap
ISE Dep. Seminar 26/4/06
Results – Subjective Measures PurposeStageResult
1Finding the Aspects that most helped the EPs' when creating an OSC
after creating the OSC
Using their medial knowledge and their understanding of the specification ontology (Asbru, in our case) Vs. specification Tools
2Finding the Aspects that most helped the EPs' making a markup
after markupSpecification Tools is considered as more helpful
3Finding how well The EPs Understand Asbru KRs
before markupDeclarative KRs are more easy to understand (such as filter condition)
4Finding what were the difficulties of the EPs' in structuring the Asbru KRs
after markupProcedural KRs are more easy to structure (such periodic plan)
5System Usability Scale (SUS) for URUZ
after markupSUS=47 ; Not Usable!
• Non significant correlation between results 1 and 2
• Significant correlation between results 3 and 4
ISE Dep. Seminar 26/4/06
Results – Objective measures
• Mean Completeness for all markups of EPs of 91%
•All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75)
Markup is feasible by EPs
Number of specified plans:
Measures Summary:
ISE Dep. Seminar 26/4/06
Results –Difference between EPs
•Any EP can perform markup with high completeness
•There is wide variability between the EPs in the correctness measure with a range of [0.13,0.58] on a scale of [-1,1]
√Difference in correctness measure between different GLs editing the same EPs4
Difference in Correlation measure between EPs editing the same GLs in most GLs (except the PID)3
√Difference in correctness measure between EPs editing the same GLs in most GLs (except the Hypo)
2
nonsignificant (P>0.05)
Difference between the proportions of completeness measure between EPs editing the same GLs
1
significant (P<0.05)
Issue
√
√
ISE Dep. Seminar 26/4/06
Results – Difference between KRsThere was significant difference (P<0.05) between homogenous groups of KRs
EPs has difficulty to structure procedural KRs than declarative ones
ISE Dep. Seminar 26/4/06
Results – Types of errorsMarkup Error Rate(MER) per KR, per EP
0.55
0.41
0.12
0.31 0.31
0.09
0.24
0.090.03
0
0.1
0.2
0.3
0.4
0.5
0.6
PID COPD HypoThyrdGLs
ME
R
Total Ontological Clinical
360
199
360
112
360
87
194
79
194
61
194
1898
12
98
9
98
3
2*.
.
i
i
GLinKRsofNo
GLinerrorsofNo= MERi.
The differences in total between the three GLs were highly significant in a proportion test (P<0.001)
The more detailed and structured the OSC was, the lower the total number of errors committed by the EPs for each KR
ISE Dep. Seminar 26/4/06
√ Background√ Methods√ Results Conclusions Future Directions
Talk Roadmap
ISE Dep. Seminar 26/4/06
Four main aspects:
Creation of an Ontology-Specific Consensus (OSC)
The essential aspects needed to learn to support the specification process by EPs
The medical and computational qualifications needed for specification
The characteristics of the KA tool needed for this kind of specification
ISE Dep. Seminar 26/4/06
Creation of an Ontology-Specific Consensus (OSC)
Should be made as detailed as possible, including all relevant procedural and declarative concepts
The OSC is independent of the specification tool
Saving the OSCs in an appropriate digital library for re-using and sharing
ISE Dep. Seminar 26/4/06
The Essential Aspects Needed to Learn to Support the Specification Process by EPs creating an OSC and performing the markups are two
different tasks which require teaching two different aspects
Teaching the “difficult” KRs in particular, the procedural KRs
Short test should be administered before the EPs perform markups
A help manual and a small simulation of marking up a GL should be included in the teaching session
ISE Dep. Seminar 26/4/06The Medical and Computational Qualifications Needed for Specification
Senior EPs and KEs together should work on the tasks of selecting a GL for specification and making the OSC
Any EP (senior, non-senior or a general physician) can structure the GL's knowledge in a semiformal representation completely
To specify it correctly, a more available EP should be selected, perhaps from among residents, interns or even students
ISE Dep. Seminar 26/4/06
The Characteristics of the KA Tool Needed for This Kind of Specification A robust, graphical, highly usable framework is
needed More intuitive, graphic, user friendly interfaces
should be used for acquiring the “difficult” KRs , especially the procedural ones
Need to bridge the gap between the initial structuring of the EP and the full semantics of the specification language
GESHER - A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels
ISE Dep. Seminar 26/4/06
Limitations and Advantages of the research
Small of the number of EPs and GLs, But, in fact, 196 sub-plans and 326 KRs in total were structured by all of the EPs together in all markups
Lack of careful measurement of the required time , but, obtain more realistic results, since the interaction with most of the EPs took place in their own "playground"
ISE Dep. Seminar 26/4/06
√ Background√ Methods√ Results √ Conclusions Future Directions
Talk Roadmap
ISE Dep. Seminar 26/4/06
The GESHER’s Main Features
User friendly graphical client application Support specification at multiple
representation levels Support to multiple specification languages
(GL ontologies) Access centralized resources such as
DeGeL and a knowledge base
ISE Dep. Seminar 26/4/06
GESHER: Semi-Structured LevelThe Hybrid
Ontology Treeshowing KRs
at allrepresentation
levels
ISE Dep. Seminar 26/4/06
GESHER(II) :Semi-Formal Widgets
ISE Dep. Seminar 26/4/06
√ Background√ Methods√ Results√ Conclusions√ Future Directions
Talk Roadmap
ISE Dep. Seminar 26/4/06
Summary The need for gradual GL specification
Making an ontology-specific consensus as first step Use a well defined methodology for the overall process
Markup is feasible by EPs Any EP can perform markup with high completeness We should use methodology for increase quality of markups Use GESHER as the new framework for specification Ongoing new research is being conducted (Pre-Eclampsia GL)
based on this research results
ISE Dep. Seminar 26/4/06
Research Publications 2005סביבת עבודה גראפית במס' רמות ייצוג, הכנס הישראלי למ"מ רפואיות Shalom, E. and Shahar Y. (2005). A Graphical Framework for Specification
of Clinical Guidelines at Multiple Representation Levels. AMIA Annual Fall Symposium, Washington DC, USA
2006האם רופאים מסוגלים להבנות ידע רפואי? הכנס הישראלי למ"מ רפואיות Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A,
B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E. (2006) A Methodology for Evaluation of A Markup-Based Specification of Clinical Guidelines Submitted to AMIA , Washington DC, USA
Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A, B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E.(2006) The Importance of Creating an Ontology-Specific Consensus Before a Markup-Based Specification of Clinical Guidelines , Submitted to ECAI06 ,Tronto, Italy
JAMIA journal paper is in preparation
ISE Dep. Seminar 26/4/06
Acknowledgments Prof. Yuval Shahar and Dr. Meirav Taib-Maymon All Medical informatics research center members Our colleagues at Soroka’s university medical center :
Prof. Eitan Lunenfeld, Dr. Avi Yarkoni, Dr. Guy Bar, Prof. Yair Liel and Dr. Tal Marom
Our colleagues at Stanford and VA hospital: Drs. Mary Goldstein, Susana Martins, Lawrence Basso, Herbert Kaizer, Laszlo Tudor Vaszar
NLM award No LM-06806 Contact info : [email protected] Visit our web site : http://medinfo.ise.bgu.ac.il/medlab/
ISE Dep. Seminar 26/4/06
Questions?
ISE Dep. Seminar 26/4/06
EPs create a Clinical consensus :
EPs and KE adds procedural knowledge :
EP+KE add declarative knowledge :
Ontology- specific consensus
Doxycycline Metronidazole
Order :parallel
Filter condition for drug :Is patient not sensitive to Doxycycline and the drug available?
Give Doxycycline 100 mg orally or IV every 12 hours Plus Metronidazole 500 mg IV every 8 hours
Methodology for creating an Ontology-Specific Consensus
ISE Dep. Seminar 26/4/06
Textual Source Parenteral Regimen A
Cefotetan 2 g IV every 12 hours
OR Cefoxitin 2 g IV every 6 hours PLUS Doxycycline 100 mg orally or IV every 12 hours.
Parenteral Regimen B
Clindamycin 900 mg IV every 8 hours
PLUS Gentamicin loading dose IV or IM (2 mg/kg of body weight) followed by a maintenance dose (1.5 mg/kg) every 8 hours. Single daily dosing may be substituted.
Alternative Parenteral Regimens
Ofloxacin 400 mg IV every 12 hours
OR Levofloxacin 500 mg IV once daily WITH or WITHOUT Metronidazole 500 mg IV every 8 hours OR Ampicillin/Sulbactam 3 g IV every 6 hours PLUS Doxycycline 100 mg orally or IV every 12 hours.
ISE Dep. Seminar 26/4/06
The first stage in forming a consensus
Parenteral Regimen A
Cefotetan 2 g IV every 12hours
ORCefoxitin 2 g IV every 6hours
PLUSDoxycycline 100 mg orallyor IV every 12 hours.
Parenteral Regimen B
Clindamycin 900 mg IV every8 hours
PLUSGentamicin loading dose IV orIM (2 mg/kg of body weight)followed by a maintenancedose (1.5 mg/kg) every 8 hours.Single daily dosing may besubstituted.
Alternative Parenteral Regimens
Ofloxacin 400 mg IV every 12hours
ORLevofloxacin 500 mg IV oncedaily
WITH or WITHOUTMetronidazole 500 mg IV every8 hours
ORAmpicillin/Sulbactam 3 g IVevery 6 hours
PLUSDoxycycline 100 mg orally or IVevery 12 hours.
Inpatient treatment -Give one of the following regimens
ISE Dep. Seminar 26/4/06
The second stage in forming a consensus
ISE Dep. Seminar 26/4/06
The third stage in forming a consensus
ISE Dep. Seminar 26/4/06
Results – Subjective measures
ISE Dep. Seminar 26/4/06
ISE Dep. Seminar 26/4/06
ISE Dep. Seminar 26/4/06
Results – Objective measures
• Mean Completeness for all markups of EPs of 91%
•All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75
Markup is feasible by EPs
ISE Dep. Seminar 26/4/06