NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Comprehensive Evaluation of Radiation Oncology Information
Systems (ROIS)Luis Fong, Ph.D.
Michael Herman, Ph.D.
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Several Layers of Complexity
• Different types of cancer• Different treatment techniques
• Several technologies
Consultation Patient Information Prescription
Simulation
Treatment PlanQATreatment
TreatmentR & V
MainHospital
OtherSites
Technological Innovations:• EPID• OBI• CBCT• Other IGRT
Different users:• Physicians• Physicists• Therapists• Dosimetrists• IS Staff• Administrative
Staff
• Research• Clinical activities
Analysis:On-lineOff-line
Multi- vs. single-vendorenvironments Lots, Lots of Information …
Paper vs. Paperless
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Several Layers of Complexity
• Different types of cancer• Different treatment techniques
• Several technologies
Consultation Patient Information Prescription
Simulation
Treatment PlanQATreatment
TreatmentR & V
MainHospital
OtherSites
Technological Innovations:• EPID• OBI• CBCT• Other IGRT
Different users:• Physicians• Physicists• Therapists• Dosimetrists• IS Staff• Administrative
Staff
• Research• Clinical activities
Analysis:On-lineOff-line
Multi- vs. single-vendorenvironments Lots, Lots of Information …
Functionality, Integration, Connectivity,
Robustness, High Performance and Stability
Paper vs. Paperless
Guarantee High Standards of Patient Care
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Some History
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
“X-Otron”PMH/OCI 1958-1983
H.E. Johns et al.(1959)
kV x-ray tube housed
in the treatment
Head
www.imaginis.com/radiotherapy/radio_history.asp
http://www.slac.stanford.edu/cgi-wrap/getdoc/neal97-008.pdf
Clinac 4 control console
http://scaa.usask.ca/gallery/uofs_events/articles/1951.php
Paper Chart
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Varian OBI™Elekta Synergy™
www.hermesmedical.com
TomoTherapy Hi-Art™
www.brachytherapy.com/tomotherapy.html
www.varian.com/prnt/orad/prd131.html
MRI-CT
PET-CT
ROIS
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
InformationManagement
TechnologicalInnovations
PatientTreatment
High Quality Level of
Patient Care
ROIS
The successful implementation of the ROIS depends greatly on a clear understanding of the site-specific processes and IS/IT infrastructure as well as the capacity of the ROIS to match the constraints and clinical
practice of a particular institution1,2.
1Ammenwerth E, Graber S, Herrmann G, et al. ”Evaluation of health information systems-problems and challenges”. International Journal of Medical Informatics 2003;71:125-135.2Lorenzi N.M, and Riley R. T Organizational Aspects of Health Informatics: Managing Technological Change, New York, Spring-Verlag, 1995, pp 11-13.
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Goal• To develop a comprehensive methodology to
systematically assess the processes and information flow as well as the performance of radiation oncology information systems (ROIS).
Clinic Specific Road Map
Clinic SpecificPractice
andInfrastructure
ROIS
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Peopleware
Radiation Oncology Classification Model
Radiation Oncology Practice
Clinical Process
Software - Hardware
ROIS
CP
IM TII
Team of Experts
Evaluation Categories: •Clinical Processes (CP), •Information Management (IM)•Technological Innovations Integration (TII)
Team of Expertsrepresenting the
different areas of the radiation oncology:
Physicians
Therapists
Physicists
IS team
Dosimetrists
AdministrativeStaff
Users
Physicians
Therapists
Physicists
IS team
Dosimetrists
AdministrativeStaff
Users
NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Analytical Methods
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Analytical Methods• Analytical Method I
– IS/IT Infrastructure Maps. The purpose of these maps is to provide a clear understanding of all IS/IT processes and connectivity between the different components and internal systems in the department.
• Analytical Method II– Process Flow Maps. These maps include all the
patient care tasks and activities as well as the management of the patient information. These maps will be the basis for the development of a survey.
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Evaluate and identify potential interoperability problems and lack of functionality and connectivity
Equipmentor
System A
Equipmentor
System B
ConnectivityProcesses
Analytical Method I - IS/IT Infrastructure Map Template
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Analytical Method II - Process Flow MapTemplate
User(s)
Process X
SurveyGeneric:• Tasks• Activities• Sub-Processes
System A
INPUT
System B
OUTPUT
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Three Major Areas• Clinical Processes (CP), • Information Management (IM)• Technological Innovations
Integration (TII)
ROIS XProcess XSurvey
Act 1Act 2
Act N
Tool 1Tool 2
Tool N
Analytical Method II – Survey
I. Quantitative Measurement of System Performance*II.Qualitative Measurement by Recording users Experiential Feedback
*where Performance measures the level of integration between the ROIS and the site-specific clinical practice, needs and infrastructure
NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Analytical Method I IS/IT Infrastructure Maps
Equipmentor
System A
Equipmentor
System B
ConnectivityProcesses
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
• Two Types of Infrastructures:– Clinical Multi-vendor:
• EXCI-IMPAC• 4DITC-IMPAC
– Test Single-vendor• 4DITC-ARIA
IS/IT infrastructures
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
EXCI-IMPAC
IMPAC dB (Pervasive):\\rdosrv01\accessrt\impac\db
RT Plan (*.rtp format):\\rdosrv01\accessrt\apps\Impac\rtp\
RT Image:\\rdosrv01\accessrt\apps\impac\stagedcm\
RT Image (per machine):\\rdosrv01\accessrt\apps\impac\pv*
QREADSMIDIALocally
CT-SIM SIM WS GE AdvantageTM
Eclipse/VARIS Vision
Multi-ACCESS /ViewStationVARiS Images
\\mfad.mfroot.org\rchdept\filedata
Brain Lab TPS
Mayo PACS system
Locally
VARIS dB (Sybase):\\varsrv04
PortalVisionWS
LINACConsole
MLC WS
LINACIAS
MLCController
DataTab
EPID
IMPAC image:\\mfad.mfroot.org\rchdept\impac
Record & Verification
IMPAC WS(Sequencer)
NPV6
IMPACDCM module
Eclipse Process
NAMER
RTP to
IMPAC
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Locally
CT-SIM SIM WS GE AdvantageTM
Eclipse/VARIS Vision
VARiS Images\\mfad.mfroot.org\rchdept\filedata
Brain Lab TPS
Mayo PACS system
RTP to
IMPAC
Locally
VARIS dB (Sybase):\\varsrv04
QREADSMIDIA
IMPAC dB (Pervasive):\\rdosrv01\accessrt\impac\db
RT Plan (*.rtp format):\\rdosrv01\accessrt\apps\Impac\rtp\
RT Image:\\rdosrv01\accessrt\apps\impac\stagedcm\NAMER
IMPAC images:\\mfad.mfroot.org\rchdept\impac
IMPACDCM module
4DITC-IMPAC
Eclipse ProcessMulti-ACCESS /ViewStation
IMPAC WS(Sequencer)
LINACConsole
LINAC
MLCController
VARIAN network switch
CBCTWS
EPID OBI
VerificationRecord
SupervisorLVI
DICOM4DITC
WSIAS
OBIWS
IAS
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Locally
CT-SIM SIM WS GE AdvantageTM
ARIA Brain Lab TPS
Mayo PACS system
Locally
ARIA dB(Sybase)
:\\varsrv05
QREADSMIDIA
4DITC-ARIA
Record
Daemons
ARIA Images
LINACConsole
LINAC
MLCController
VARIAN network switch
CBCTWS
EPID OBI
Verification
SupervisorLVI
4DITCWS
IAS
OBIWS
IAS
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Local
GE Advantage
Eclipse/VARIS Vision
Mayo PACS system
Local
Eclipse Process
IMPAC WS(Sequencer)
LINACConsole
MLCController
VARIAN network switch
CBCTWS
VerificationRecord
SupervisorLVI
DICOM4DITC
WSIAS
4DITCWS
IAS
OBIWS
IAS
OBIWS
IAS
IMPAC dB (Pervasive)
RT Plan-Temporaryfrom TPSs
RT Images-Temporaryfrom TPSs
Multi-ACCESS
IMPAC images (DFS)
NAMER
RTP to
IMPAC
VARIAN Images (DFS)VARIAN dB
(Sybase)
IMPACDCM module
SIM WS
MV
LINACkV
CTSIM
BrainSCAN
IMPAC dB (Pervasive)
RT Plan-Temporaryfrom TPSs
RT Images-Temporaryfrom TPSs
RT Images-Temporaryper machine
Local
GE Advantage
Eclipse/VARIS Vision
Multi-ACCESSVARIAN
Images (DFS)
Mayo PACS system
Local
VARIAN dB (Sybase)
PortalVisionWS
LINACConsole
MLC WS
IAS
MLCController
DataTab
MVIMPAC images (DFS)
Record & Verification
IMPAC WS(Sequencer)
NPV6
IMPACDCM module
Eclipse Process
NAMER
RTP to
IMPAC
SIM WSCTSIM
LINAC
BrainSCAN
Local
GE Advantage
ARIA BrainSCAN
Mayo PACS system
Local
VARIAN dB(Sybase)
DICOM Daemons
VARIAN Images(DFS)
LINACConsole
MLCController
VARIAN network switch
CBCTWS
Record&
Verification
SupervisorLVI
4DITCWS
IAS
4DITCWS
IAS
OBIWS
IAS
OBIWS
IAS
SIM WS
MV
LINACkV
CTSIM
Information System
Workstation(WS)
DFS(Distributed File
System)
Server(Database)
Connectivity Service
Imager
LocalStorage
Symbols Description:
IS/IT Infrastructure Maps EXCI-IMPAC
4DITC-IMPAC4DITC-ARIA
NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Analytical Method II Process Flow Maps
User(s)
Process XSystem A
INPUT
System B
OUTPUT
NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Clinical Processes (CP)
Information Management (IM)
Technological Innovations Integration (TII)
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Site-Specific Clinical Processes
Mayo EMR Mayo GPAS
ROIS
Treatment Planning
Plan Second Check
Treatment Delivery
Weekly Chart CheckWeekly Management VisitCompletion of Treatment
QA (IMRT)
Reports
Rad Onc
IMRT-SReview of Images
Weekly Programming
Physician Consult Desk Notification Treatment Scheduling Simulation
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Mayo GPASFront Desk
Desk Notification
Treatment Scheduling
Tx Scheduler ROIS
Therapist
Simulation
PhysicianROIS
Physician Consult
Mayo EMR
Physician
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Physicist
Plan Second Check
ROIS
Report
Dosimetrist
Treatment Planning
Physician ROIS
Report
IMRT-S
Physicist
IMRT QA
ROIS
IMRTReport
IMRT-S
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Therapist
Treatment Delivery
IMRTReport
Report
ROIS
Physicist ROIS
Weekly Chart Check
Therapist ROIS
Weekly Programming
Weekly Management Visit
Physician ROISMayo GPAS
Review Images
Physician ROIS
Daily
Weekly
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Completion of Treatment
Front Desk
Tx Scheduler
TherapistIMRT
ReportReport
IMRTReport
Report
Storage
ROIS
Mayo EMRDose
Summary
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Clinical Process Flow Map
Patient information and process management systems:(A) Mayo EMR, (B) Physician Internal Electronic Ordering form,(C) Mayo Schedule Manager(D) IMRT QA Manager(ROIS)
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Therapist
Simulation
PhysicianROIS
System: ROIS X
Process: Simulation
Please evaluate the performance of the system for: ScoreAdding patient ID photoAdding field specific set-up photosAdding patient set-up informationCreating Reference FieldsSending reference images from GE Workstation into the systemSending images/field information from Acuity into the systemViewing/Accessing information from previous treatment coursesViewing/Accessing prescription and related notesAdding specific task lists to patient informationBilling appropriate items performed
Information System Evaluation Survey
Example of Process Flow
Map and corresponding
survey
NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Clinical Processes (CP)
Information Management (IM)
Technological Innovations Integration (TII)
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Information Management:I. Database management II. Long-term archiving
solutionIII. Application availabilityIV. Customer supportV. System integrationVI. Data Conversion
Information Management
Survey
NCCAAPM MeetingNCCAAPM Meeting Rochester, MN Spring 2008 Rochester, MN Spring 2008
Clinical Processes (CP)
Information Management (IM)
Technological Innovations Integration (TII)
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Technological Innovations Integration
I. New imaging modalities: I. Cone Beam CT (3D matching),II. EPID-MV (2D matching), III. OBI-kV, kV – MV (2D matching)
II. Portal Dosimetry.
• Implementation
• Connectivity with ROIS
• Operability with ROIS
• Ability to keep updating with ROIS
• Ability to perform off-line review for research purposes
Technological Innovations Integration
Survey
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Information Management
Survey
Clinical ProcessSurvey
Technological Innovations Integration
Survey
ROIS X
Physicians
Therapists
Physicists
IS team
Dosimetrists
AdministrativeStaff
Users
Physicians
Therapists
Physicists
IS team
Dosimetrists
AdministrativeStaff
Users
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Performance-Importance of Patient Care Space
High Medium Low
5 4 3 2 1
• High. The performance of this activity is essential to guarantee an accurate and safe patient treatment. It has a high effect on dose delivery as well as in the information and process flow.
• Medium. The performance of this activity helps in the process and information flow, and has medium dosimetric effect on the patient treatment.
• Low. The performance of this activity has very small effect on process and information flow and does not have any dosimetriceffect at all on the patient treatment
Importance for Patient Care Weighting Factor
ExcellentAbove
Adequate AdequateBelow
Adequate Poor
0 if Not Available 5 4 3 2 1
• Excellent. The system performs the activity and has the capability of improving the efficiency and efficacy of our department processes flow.
• Above Adequate. The system performs the activity with some extra features that make the activity flow smoothly.
• Adequate. The system performs the activity adequately• Below Adequate. The system performs the activity with some
minor difficulties that could be overcome.• Poor. The system performs the activity in a way that could not
be implemented in our department. • Not Available. The system is unable to perform this activity.
Performance Scoring Scale:
Poor Adequate Excellent
Performance
Impo
rtanc
e fo
r Pat
ient
Car
e High
Medium
Low
RejectionRegion
(I)
RejectionRegion
(II)
AcceptanceRegion
(I)
AcceptanceRegion
(II)
Performance Patient Care Factor
Magnitude Direction
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Performance Sorted by Patient Care Priority
PoorBelow
Adequate AdequateAbove
Adequate Excellent
1 2 3 4 5Performance
HighMediumLow
1 2 3 4 5
HighMediumLow
1 2 3 4 5
Importance for Patient Care
Region Evaluate the performance of the system for: PI IPCRR-I Review detailed treatment record 2.50 5.00RR-I Reviewing changes in plan per field 2.50 5.00AR-I Reviewing dose site summary 4.00 5.00AR-I Reviewing overrides 4.00 5.00AR-I Checking the chart since the last time checked 5.00 4.00RR-I If changes were made, tracking the history of the change 2.50 3.50RR-I If overrides were made, tracking the history of the override 2.50 3.50RR-I Reviewing action points 2.75 3.50RR-I Selecting BID patients 3.00 3.50AR-I Selecting patients per machine per site 4.50 3.50AR-I Signing off electronically that chart check was done 5.00 3.50
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Clinic Specific Performance vs. Patient Care
PERFORMANCE
PA
TIE
NT
CA
RE
Poor Adequate Excellent
High
Medium
Low
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Information Management
Survey
Clinical ProcessSurvey
Technological Innovations Integration
Survey
Patient Care Site-Specific Master Priority List in terms
of Importance for Patient Care
Focus on High Importance for Patient
Care and Low Performance
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Information Management
Survey
Clinical ProcessSurvey
Technological Innovations Integration
Survey
Patient Care
Lens 2
Lens 3
Lens n
Alternative Priority Coefficient
nPriorityPerformace ∏×
Site-Specific Master Priority List in terms
of n-Priorities
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Poor Adequate Excellent
High
Medium
Low
Performance
Impo
rtan
ce o
f Pat
ient
Car
e
RR-I
RR-II AR-II
AR-I
Poor Adequate Excellent
High
Medium
Low
Performance
Impo
rtan
ce o
f Pat
ient
Car
e
RR-I
RR-II AR-II
AR-I
Quantitative Results
ROIS A ROIS B
In our practice, both evaluated systems showed, in average, above adequate performance
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Qualitative Results
Clinical Practice• Proficient knowledge and understanding of database management, DICOM RT and the
components, connectivity and infrastructure of any clinically relevant information systems linked to the ROIS is needed by physicist and IS team
Vendor• Development of efficient and effective customer services program as well as training resources
Vendor• It is highly desirable to have the vendors provide a virtual environment (beyond a canned demo)
ready for the users which will allow them to test the tools provided by the ROIS, simulating their current practice.
Clinical Practice &Vendor
• Effective channels of communication among users and between user and the vendor need to be developed in relation with ROIS issues, configuration and upgrades.
Clinical Practice
• Appropriate amount of resources need to be allocated toward the development of a strong IS team. The main role of the IS team will be to provide support for the everyday maintenance and basic trouble shooting of the non-clinical relevant hardware and software (e.g. virus configuration, operating system patches, workstations, etc.)
Clinical Practice &Vendor
• Successful, effective and safe approach to ROIS issues shall be conducted by a team formed by: physics, IS staff and vendor service support.
Clinical Practice &Vendor
• Clear definition of roles and responsibilities of vendor and user related to the management, service and support of ROIS need to be developed prior its deployment.
ResponsibleRecommendation
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Who could benefit from this Evaluation Method?
Specific ClinicalPractice ROIS
Communication Channels&
Taxonomy• Vendors• Developers• Administrators
• Physicians• Physicist• Administrators• All Users
• Prioritize• Develop better
Processes
• Prioritize• Develop better
Solutions and Tools
Ultimately Improve
Patient Care
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Conclusions• This work provides for the first time an objective tool to
understand and analyze clinical information flow and its compatibility with a given ROIS in a clinic specific patient care setting.
• Mapping provides insight into process communication, interoperability and efficiency.
• The hierarchy and importance of various characteristics are customizable to a given clinical practice and thus allow the tool’s broad applicability.
• Proper assessment of information flow and matching to an ROIS will provide a more efficient and more effective care delivery setting
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Acknowledgements
• Lori Buchholtz• James Sorenson • Janelle Miller• Alan Kraling• Jill Rendler• Tim Peters• Jessica Schlichting• Michael Herman, Ph.D.• Jon Kruse, Ph.D.• Terry Harms• Bob Dahl• Janelle Molloy, Ph.D.• Jack T. Cusma, Ph.D.
• Chris Hanna, Ph.D.• Rob Mitchell• Matt Adler• Maureen Thompson• Jeremy Cohen• Richard Nusspickel• David Ackerson• Scott Johnson• Sean Lapinski• Chuck Pearce• Kenneth Dick• Cathy Tenda• Diana Jones
Project supported in part by Varian Medical Systems
Rochester, MN Spring 2008 Rochester, MN Spring 2008 NCCAAPM MeetingNCCAAPM Meeting
Thanks!
Ready for some
questions
?