medical image analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/c4m_seminar4... · 2017-09-08 ·...
TRANSCRIPT
![Page 1: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/1.jpg)
Medical Image Analysis C4M
Chris McIntosh
![Page 2: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/2.jpg)
Medical Imaging • Enable clinicians to examine anatomy in-vivo (without extracting it) • Some major areas for computer science
– Data visualization – Image Analysis – Computer assisted diagnosis (CAD) – Disease understanding
2
![Page 3: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/3.jpg)
Acquisition
• Many different types of signals – Colour (dermatology, pathology) – Radiodensity (structural, geometrical) – Radioactive isotope uptake over time (functional) – Water and fat (soft-tissue structural) – Water diffusion over time (soft-tissue functional) – High frequency sound wave refraction
3
![Page 4: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/4.jpg)
Colour Images • Acquired by a camera, optionally with aid
– Microscope, Dermatoscope, Endoscope, etc. • Composed of 3 data channels yields an MxNx3 array • Each discrete element is a pixel on the X, and Y axis
Original cell picture: Patho under CC BY-SA 3.0
4
![Page 5: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/5.jpg)
Radiodensity
• CT Imaging • MxNxS where S is the number of slices • Each element is now a voxel in (X,Y,Z)
N
M
S
5
![Page 6: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/6.jpg)
Positron Emission Tomography
• MxNxSxT where T is the amount of time!
6 Hamarneh, McIntosh and Drew, IEEE TMI 2011
55
75
95
115
135
0 2 4 6 8 10 12 14 16 18 20 22 24 26
Phot
on A
ctiv
ity
dynamic Positron Emission Tomography (dPET)
Voxel 1 Voxel 2 Voxel 3
Time
![Page 10: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/10.jpg)
Disease Understanding
10
Tench et al. Horsefield et al. McIntosh et al. Volume Accuracy 97% 86% 93%
80% 85% 90% 95%
100% Volume Accuracy
Tench et al. Horsefield et al. McIntosh et al. Interaction Time 30 5 1
0 10 20 30 40
Min
utes
Interaction Time
McIntosh et al., MICCAI 2006, HISB 2011
![Page 11: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/11.jpg)
Summary • Understanding the end-user
– Not trying to replace medical experts – Augmenting ability
• It’s about standardization and time – Make the region of interest more obvious
(visualization) – Point me to the interesting data (Detection and CAD) – Extract the interesting data faster (segmentation, and
shape analysis for disease understanding) • A patient should receive the same diagnosis and
treatment on a Monday morning in Toronto, or Friday night in Whitehorse
11
![Page 12: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/12.jpg)
What is Vision?
• If the goal is observe or highlight something a person can see in an image, we must first understand how a person can see
12
![Page 13: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/13.jpg)
A Nobel Prize
• Hubel & Wiesel won the Nobel Prize in Physiology or Medicine in 1981
• Inserted microelectrodes into cats, and monkeys, and studied the response of different areas of the brain under different stimuli
Gilbert and Li, Nature Reviews, 2013
13
![Page 14: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/14.jpg)
The First Layer
• Primary visual cortex (V1) • Different neurons respond to differently oriented bright vs
dark bars • We call this edge response
14
![Page 16: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/16.jpg)
Filtering • We need examine every pixel of an MxN image and
compare it to the pattern’s discovered by Hubel and Weise
• We call the pattern a filter, and it will be a [2*k+1,2*k+1] array.
16
![Page 17: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/17.jpg)
Example
17
Source: S. Seitz
Input Image Output Result
1/9 [%1&1&1@1&1&1@1&1&1 (]
Filter *(+,,) -(+,,) .=1
0
![Page 18: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/18.jpg)
0 10
Example
18
Source: S. Seitz
Input Image Output Result
1/9 [%1&1&1@1&1&1@1&1&1 (]
Filter *(+,,) -(+,,) .=1
![Page 19: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/19.jpg)
0 10 20
Example
19
Source: S. Seitz
Input Image Output Result
1/9 [%1&1&1@1&1&1@1&1&1 (]
Filter *(+,,) -(+,,) .=1
![Page 21: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/21.jpg)
Example
21
Source: S. Seitz, S. Fidler
Input Image Output Result
1/9 [%1&1&1@1&1&1@1&1&1 (]
Filter *(+,,) -(+,,) .=1
![Page 22: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/22.jpg)
Algorithms
• Three main variants – 2D Filtering – Convolution
• Same as filtering with a filter flipped in Y and then X to gain a few important mathematical properties
– Normalized cross-correlation or template-matching
• Same as filtering, but the response at each pixel normalized by the magnitude of the filter times the pixel-window
22
![Page 23: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/23.jpg)
0 0 -90 -90 -90 -90 -90 -90 0 0
Edge Response
23
Source: S. Seitz
Input Image Output Result
[%?&?&?@?&?&?@?&?&? (]
Filter *(+,,) -(+,,) .=1
![Page 25: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/25.jpg)
Higher levels of vision
• As we progress through the visual cortex we begin grouping responses from lower levels to create more complex representations
• This grouping gives rise to contours and shapes (Gilbert and Li, Nature Reviews, 2013)
• V4 has shown strong response to texture (Kastner et al., J Neurophysiology, 2000)
25
Gilbert and Li, Nature Reviews, 2013
![Page 26: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/26.jpg)
Vision is challenging • Let’s watch a short video and perform a basic vision task
26
![Page 27: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/27.jpg)
Vision is challenging
• Do both blocks have a gradient?
27
Photo by Dodek, CC BY-SA 3.0
![Page 28: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/28.jpg)
Vision is challenging
• Do both blocks have a gradient?
28
Photo by Dodek, CC BY-SA 3.0
![Page 30: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/30.jpg)
Psychological • Vision is a psychological and perceptual phenomena, not
a physical measurement • Computer vision can emulate human vision • It can also build measured responses to assist human
vision • Measurements lead to quantitative markers that can help
decision making • We call these features (some are perceptual, some are
physical, some are both)
30
![Page 31: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/31.jpg)
Features • In computer vision we call the lower level responses/
structures used to define an object features • Features can be anything • Represent a voxel or group of voxels by a number
– Try and describe the local structure – Bright vs dark – Wavy vs smooth – Round vs square
• The features will enable us to build our applications (e.g. image segmentation)
31
![Page 32: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/32.jpg)
Edge Detection • Our edge filter is local (per pixel) • Edges are connections of strong
responses that group into a logical contour
32
![Page 33: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/33.jpg)
Edges Are Both • Humans implicitly ignore edges that are not relevant to
their perception of an image
33
Martin et al., A Database of Human Segmented Natural Images…, Computer Vision and Pattern Recognition, 2001
![Page 34: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/34.jpg)
Texture
• Repeating intensity patterns in the data
• Texture analysis, called radiomics, in CT has been shown to correlate with genomic features and cancer outcomes (Aerts et al., Nature Methods, 2014)
34
![Page 37: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/37.jpg)
Building a Model
• Once features are extract the next step is to build a model to make predictions based on the features – *(4567895:)=;87<=>5
• Model can be coded based on prior knowledge or learned via machine learning
37
![Page 40: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/40.jpg)
Patients and Treatment
• Relating patient’s anatomical geometry, texture, shape and appearance to radiotherapy treatment
![Page 41: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/41.jpg)
Distribution of Features • Different radiotherapy plans and qualities have different image and plan features • Machine Learning learns to distinguish between the different groups • Learns to:
– Automatically catch low quality plans – Rank plans in order of least-to-most complexity for review
Plan Features
Imag
e Fe
atur
es
Gold standard Breast RT plan Complex patient anatomy Average quality Breast Plan Poor quality plan
Distribution of Features
41 McIntosh, Svistoun, and Purdie, PP: PCT/CA2014/050551
![Page 42: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/42.jpg)
Planning Error Detection • Preliminary breast study, detects 80% of clinically rejected plans • Detected plan error with poor high dose conformity (700 cGy
isodose)
Rejected Accepted
42
![Page 43: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/43.jpg)
Project Example
• Four main tasks: – Compute features – Train a model – Predict centres – Score
43
![Page 44: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/44.jpg)
Template Matching
• Might be difficult to construct a good feature for nuclei • We can use training data to build templates, and find those
templates in the image 1. Extract the template as a square around a nuclei centre 2. Use normalized cross-correlation (match_template in python) to find similar
patterns 3. Find the (x,y) locations of all of the top responses
44
![Page 45: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/45.jpg)
Scoring • True positive: Any nuclei centre with a prediction sufficiently close by
(e.g. 12 pixels) • False negative: Any nuclei centre without a sufficiently close
prediction • False positive: Any prediction not sufficiently close to a nuclei centre
45
![Page 46: Medical Image Analysisc4m.cdf.toronto.edu/cohort1/phase3/seminar4/C4M_seminar4... · 2017-09-08 · Original cell picture: Patho under CC BY-SA 3.0 4 . chris.mcintosh@rmp.uhn.on.ca](https://reader034.vdocuments.net/reader034/viewer/2022050312/5f742fdd11a9e144fa6ecf1c/html5/thumbnails/46.jpg)
Summary • Medical image analysis can aid in standardization and efficiency of
measurements for outcomes, treatments, and disease understanding
• Many image features are built on filtering or convolution, emulating a similar process to the human visual system
• Human perception of colors, gradients, and edges is both psychological and physical
• The best systems will pair the strengths of medical experts (domain knowledge, compassion, understanding, interaction, high-level analysis) with the best of computers (repeatable, high throughput, quantitative)
46