deploying ai into the clinical radiology workflow
TRANSCRIPT
Dr. Judy W Gichoya
Emory University Department of Radiology & Imaging Sciences
@judywawira
Deploying AI into the clinical radiology workflow – Important considerations for
radiologists and informaticists
• No personal disclosures
• During infrastructure review may mention some of the equipment we have -> Not an endorsement
• To describe pilot deployment of the ACR AI-Lab tool at Emory University
• To describe our architecture from development to production for inhouse and commercially available AI models
• Review ML data pipelines for working with AI in production
Emory Radiology
Clinical Research Education
Emory Department of Radiology and Imaging Sciences
Informatics Team
Revised 8/27/19
Imaging Workflow Special ists
Brenda Hall
Radiology
EUH
Jacqueline McCarty
Radiology
ESJH
Candace Moczarski
Radiology
EJCH
Wilbert Pope
Radiology
ESJH
Steve St. Louis
Radiology
EUH
Imaging Workflow Administrators
Karen Boles
Dir. of Informatics
and Tech Services
April Carter
Enter. Sol. Arch III
Project Manager
Kelli Miller
Project Manager
Project Management Team
Nabile Safdar, MD
Director
Imaging Informatics
Willie Arnold
Administrator,
Clinic Operations
Denise Fennell
Administrative
Assistant
Peter Harri, MD
Imaging
Informatics
Asst. Professor
Adam Prater, MD
Imaging
Informatics
Asst. Professor
Mercy Mutahi
Business Analyst
Judy Gichoya, MD
Imaging
Informatics
Asst. Professor
Marjin Brummer
Staff Scientist
Havi Trivedi, MD
Imaging
Informatics
Asst. Professor
Imaging Informatics
Starla Longfellow
Asst. Director
Imaging Srvcs.
Stacey Adams
Department System
Specialist II
Clinical Bus. Support
Chris Braithwaite
Desktop Support
EU IT Support
Wendy Lybrand
Enterprise Trainer
Trainer
Roslyn Baitey
EUHM
Mammo
Stacey Walker
EUHM
Evenings
Judy Graham
EUHM
Paulette Jackson
EUMHLennex Annor
EUHM
Evenings
Wangail Assamenew
Executive Park
MSK
Latonia Smith
Telerad
Nina Stephenss
EUMHErika Petty
Breast Imaging
(Winship)
Sharon Cain
TeleRadLisa Kappel
Supervisor
IWS
Lynn Coram-Allen
EUH
Interventional
Inez Dupree
EUH
Nuclear Medicine
Henok Abate
EUH
Neuroradiology
Dexter Bostic
EUH
Cardio
Gerald “Rick”
Foster
EUH
Brenda Stokes
TeleRad
Veena Amin
App Solutions
Analyst III
Jeff Trott
Technical Apps
Specialist III
Daria Miller
Enterprise
Solutions Arch II
RadNet Team
Jason Jacob
Dept. Systems
Specialist
Lee Taylor
Technical Apps
Specialist III
Lance Manley
Technical Apps
Specialist IV
Chris Vant
Technical Apps
Specialist III
Dario Rodriguez
Technical Apps
Specialist II
Rick Cobb
Technical Apps
Specialist III
Brian Goertemiller
Imaging System
Software Spec. Lead
Dongqing Shi
Imaging Application
Specialist
Luke Wademan
Enterprise
Solutions Arch III
Imaging Applications Support
Geo Eapen
Enterprise
Solutions Arch III
Collette Erickson
EJCH
Valerie
Fowler-Robinson
ESJH - Mammo
Ursula Handy-Evans
ESJH
Mary Greer Thomas
EJCH
Mammo
Lola Oceanty
ESJH
Outpatient
Sabrina Baida
Intern
IVC filter case study
IVC filter case study
• Complications if delayed removal• Venous thromboembolism
• Stent fracture
• IVC damage
• Death
• Frequent delayed removal:• Lack of patient awareness
• Improper identification on imaging
• Absence of follow up Cleveland Clinic Journal of Medicine. 2018 November;85(11):835-83
Methods - Dataset
828 Radiographs : abdominal, thoracoabdominal, lumbar
Positive: With IVCF348
Negative: Without IVCF480
Strongly labeled with bounding boxesIR Fellowship in-training radiologist
Randomly divided without patient overlap
Training501
Validation127
Test200
Methods - Technique
Retinanet architecture Encoder : Resnet-50
Pretrained : COCO dataset
Batch size : 1 Learning rate : 0.00001
15 epochs
Resolution : smallest side > 800 pixels, largest side < 1333 pixels No data augmentation : preserve high resolution IVCF spatial representation
Focal loss: γ=2 to compensate for pixel class imbalance
Metrics: Classification : AUC, Sensitivity, Specificity
Object Localization : Mean Average precision at IOU > 0.5 (mAP-0.5)
Lin T-Y, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection.
2017 IEEE International Conference on Computer Vision (ICCV). IEEE; 2017. p. 2999–3007.
Methods – Baseline classifier
EfficientNet architecture B0 baseline subtype
Pretrained : Imagenet
Batch size : 8
Learning rate : 0.001 to 0.00001
30 epochs
Image resolution : 512 x 512 pixels
Standard affine data augmentation
BCE loss
Metrics: Classification : AUC, Sensitivity, Specificity
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 2019.
Findings - Performance
Findings – Confusion Matrix
What next?
Metadata
• Started project over 1 year ago
• IRB for data – Retrospective versus umbrella versus prospective ?• Clinical trial ?
• Quality improvement project
• Exempt status
• Obtaining data • Existing IVC filter DB for different project – Accession No -> Manual extraction via
Osirix
• Deidentification pipeline with manual scrubbing -> and then manually verified
• Training – Single GPU 1080 Ti• Collaboration across institutions ?
Metadata
• Operationalizing aka “Lets save lives !”• Research PACS – nightly build
• DICOM Headers – CXR, KUB, Lumbar and Thoracic radiographs
• Containers • .py file VS Jupyter notebook
• Docker for containers
• Anonymization – Needs to be identifiable
• Output / Monitoring • JPEG with bounding box -> Classification tasks ? What output - ? Probability, heat
maps ? segmentation output
• Clinical data integration -> Anticoagulants, Upcoming appts, Labs
Background : Hardware
• 1 Titan X
• 3 Titan Vs
• Lambda
Background: Network
DMZ
Healthcare
Academic
Outside Traffic
Background : Software
• Annotation – Vision tasks, NLP
• Data science – Jupyter Hub + Jupyter notebooks . Pandas • Support ease of sharing + versioning
• DL tools – Tensorflow, Pytorch, Keras, FastAI (python shop)
• Front end – Web based (Flask apps, Bootstrap)
• Code – Versioning (private repos) , Tickets/ Sprints for work management, Parameters tracking for multiple engineers
• Deployment – Containers
• Not cloud interactive
• Open source
Success looks like
IVC filter detector PACS stream HL7 clinical feed Communication
- CXR /KUB/ Spine Xrays
- Follow up imaging
- IVC filter present / removed
Labs Medication list
Problem list Clinic referral
ACR AI-Lab
ACR AI-Lab at Emory
• IRB submission – Umbrella
• Aim for access to duplicate/research PACS
• Authentication
• Anonymization challenge
• Hardware – Inference GPU
Summary
• Avoid #FOMO ….. Manage the hype
• Build team
• T vs N
• Metadata aggregation • New data streams
• Access integration engines
• Resources – Leadership, Align with CMIO , Business case • Build or buy ?
• Don’t forget to “pay the rent”
• Southern AI club?
