machine learning in health care...machine learning for image content recognition machine learning...

49
A. Criminisi Machine Learning in Health Care

Upload: others

Post on 25-Jun-2020

21 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

A. Criminisi

Machine Learning in Health Care

Page 2: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Features in 2D space Features in 3D space

Data point =

Feature vector

Machine learning for image content recognition

Page 3: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Machine learning for image content recognition

Features in 2D space

induction

Page 4: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Application: Kinect for Xbox gaming

Input test depth image Body part segmentation

image measurements

made relative to pixel

classifier per-pixel prediction

of class label

e.g. depth, color, neighbors

Task: assigning body part labels to each pixel in Kinect-acquired depth videos

J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore,

P. Kohli, A. Criminisi, A. Kipman, and A. Blake, Efficient Human Pose Estimation

from Single Depth Images, in Trans. PAMI, IEEE, 2012

Page 5: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Decision trees

terminal (leaf) node

internal

(split) node

root node0

1 2

3 4 5 6

7 8 9 10 11 12 13 14

A general (binary)

tree structure

Is top

part blue?

Is bottom

part green?Is bottom

part blue?

A decision treeA general learned

predictor

Page 6: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Toy tree learning example

• Try several hyperplanes, chosen at random

• Keep hyperplane that best separates data

• information gain

• Recurse

x1

x2

0

1 2

3 4

Learning a conditional structure of discriminative features.

Page 7: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

x1

x2

Toy tree learning example

0

1 2

3 4

Learning a conditional structure of discriminative features.

• Try several hyperplanes, chosen at random

• Keep hyperplane that best separates data

• information gain

• Recurse

Page 8: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

x1

x2

Toy tree learning example

0

1 2

3 4

Learning a conditional structure of discriminative features.

• Try several hyperplanes, chosen at random

• Keep hyperplane that best separates data

• information gain

• Recurse

Page 9: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

x1

x2

Toy tree learning example

0

1 2

3 4

Learning a conditional structure of discriminative features.

• Try several hyperplanes, chosen at random

• Keep hyperplane that best separates data

• information gain

• Recurse

Page 10: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

• Used to decide which candidate split function is best

• Typically an “information gain” – a very general and flexible formulation

entropy of

examples

at parent

node

entropy of

examples

at child nodes

Training objective function

weighting

left/right

children

0

1 2

3 4

Page 11: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Examples of split functions

“Axis aligned” “Oriented hyper-plane” “Conic section”

Efficient (one feature at a time)

0

1 2

3 4

Page 12: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Decision trees: test time prediction

test input data

probabilistic

prediction

Page 13: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Decision forests

Better generalization than individual trees

Page 14: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Aggregating tree predictions

Page 15: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Dep

th D

=5

Dep

th D

=1

3

Weak learner: axis aligned Weak learner: oriented line Weak learner: conic section

Effect of tree depth and randomness

A. Criminisi and J. Shotton, Decision Forests for Computer

Vision and Medical Image Analysis, Springer, February 2013Free code available!

Page 16: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Are forests sufficient?

• Memory issues: • Number of nodes in trees

grows exponentially with depth

• Amount of training data• Training data is quickly diluted with depth

• Yet, training deeper trees (on enough data) yields highest test accuracy

(several real applications, e.g. Kinect, have “infinite” data available)

Page 17: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

From trees to DAGs: node merging

• Each internal node has 2 children (like in binary trees)

• Each non-root node can have more than 1 parent

Page 18: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Decision jungles

• A “jungle” is an ensemble of rooted decision DAGs

• We train each DAG layer by layer, jointly optimizing both• the structure of the DAG

• the split node features

Page 19: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Properties of jungles

• Limited memory consumption• e.g. by specifying a width at each layer in the DAG

• Potentially improved generalization• fewer parameters

• less “dilution” of training data

Page 20: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

grass

grass

cow

sheep

How do DAGs help in practice?A toy example on classifying images of cows, sheep and grass

Training data

Page 21: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

grass

grass

cow

sheep

brightness

chro

maticity

A toy example on classifying images of cows, sheep and grass

Page 22: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

grass

grass

cow

sheep

brightness

chro

maticity

A toy example on classifying images of cows, sheep and grass

Page 23: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

A toy example on classifying images of cows, sheep and grass

Page 24: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Tree

brightness

chro

maticity

Axis-aligned splits only

A toy example on classifying images of cows, sheep and grass

Page 25: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

Tree

A toy example on classifying images of cows, sheep and grass

Page 26: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

Tree

A toy example on classifying images of cows, sheep and grass

Page 27: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Estimated: cow Estimated: sheep

brightness

chro

maticity

Too many model parameters: overfitting

Tree

A toy example on classifying images of cows, sheep and grass

Page 28: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

DAG

A toy example on classifying images of cows, sheep and grass

Page 29: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

DAG

A toy example on classifying images of cows, sheep and grass

Page 30: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

DAG

A toy example on classifying images of cows, sheep and grass

Page 31: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Node merging

brightness

chro

maticity

DAG

A toy example on classifying images of cows, sheep and grass

Page 32: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

brightness

chro

maticity

DAG

A toy example on classifying images of cows, sheep and grass

Page 33: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Correct generalization: grass

capturing illumination invariance

brightness

chro

maticity

DAG

A toy example on classifying images of cows, sheep and grass

Page 34: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

2

grass

grass

cow

sheep

c s g

c s g

c s g

c s g c s g c s g

Training

patches

c s g

0

1

3 4 5

Trees vs DAGs

A toy example on classifying images of cows, sheep and grass

Merged nodes help capture appearance invariance

Page 35: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Anatomy Localization in 3D Computed Tomography Scans

Input CT scan Output anatomy localization

Page 36: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

sp

lee

nli

ve

r

ga

ll b

lad

de

rle

ft k

idn

ey

High variability in appearance, shape, location, resolution, noise, pathologies …

Anatomy localization: why is it hard?

Page 37: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Different image cropping, noise, contrast/no-contrast, resolution, scanners, body shapes/sizes, patient position…

Anatomy localization: the ground-truth database

Page 38: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Anatomy localization: regression forest

Page 39: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Anatomy localization: automatic landmark discovery

Input CT scan and detected landmark regions

Here the system is trained to detect

left and right kidneys.

The system learns to use bottom of

lung and top of pelvis to localize

kidneys with highest confidence.

Page 40: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Segmentation of

tumorous tissues:

---- Active cells

---- Necrotic core

---- Edema

---- Background

3D MRI input data

T1-gad T1

T2

DTI-p

FLAIR

DTI-q

Automatic segmentation of brain tumour

Page 41: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Tumour

Tissue

Classification

Training a voxel-wise forest classifier

Page 42: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

New Patient,

previously unseen

Tumour

Tissue

Classification

Testing the voxel-wise forest classifier

Page 43: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Glioblastoma segmentation resultspatient 1 patient 2 patient 3 patient 4 patient 5

Page 44: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Glioblastoma segmentation results

Page 45: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Speeding up acquisition of medical scans

Learned voxel predictor

Low-res diffusion MRI(faster acquisition, cheaper)

High-res diffusion MRI

Page 46: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Speeding up acquisition of medical scans

Problem statement

learning to predict the value of the high-res voxels

from the low-res voxels.

- Training data can be easily obtained

- Well defined accuracy measure

Page 47: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Speeding up acquisition of medical scans

Direction-encoded colour FA maps for various reconstructed DTIs Comparison of ground truth NODDI parameter maps with various

fitting techniques

Page 48: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Speeding up acquisition of medical scans

D. Alexander, D. Zikic, J. Zhang, H. Zhang, and A. Criminisi,

Image Quality Transfer via Random Forest Regression:

Applications in Diffusion MRI, in MICCAI, Springer, 2014

Reconstruction errors for DT maps Reconstruction errors for NODDI parameter maps

Page 49: Machine Learning in Health Care...Machine learning for image content recognition Machine learning for image content recognition Features in 2D space induction Application: Kinect for

Modern, efficient machine learning has the potential to revolutionize medicine!