fast magnetic resonance imaging: theory, technique and ......fast magnetic resonance imaging:...

60
Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179 Medical AI Research Center/ Paul C. Lauterbur Research Center for Biomedical Imaging Shenzhen Institutes of Advanced Technology (SIAT)๏ผŒCAS ICCM 2020, 2020/06/18

Upload: others

Post on 24-Feb-2021

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Fast Magnetic Resonance Imaging: Theory, Technique and Application

Dong Liang, PhD

Program number: 5179

Medical AI Research Center/

Paul C. Lauterbur Research Center for Biomedical Imaging

Shenzhen Institutes of Advanced Technology (SIAT)๏ผŒCAS

ICCM 2020, 2020/06/18

Page 2: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Outline

Background

Parallel Imaging

Compressed sensing based methods

Deep Learning based methods

Page 3: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Magnetic Resonance Imaging (MRI)

Page 4: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Slow imaging speed is the bottle neck of MRI

< 1 s> 15 min

MRI CT

Coronary ArteryImaging

Page 5: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Claustrophobia

Motion artifacts High cost

Drawbacks of Long Acquisition Time

Page 6: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Fourier

Ideally๏ผŒthe relationship between MR image and k-space data can be formulated as Fourier transform

๐‘‘ ๐‘˜ = ๐‘€( ๐‘Ÿ) ๐‘’โˆ’๐‘—2๐œ‹๐‘˜ ๐‘Ÿd ๐‘Ÿ

Spatial domain and MR data space

Page 7: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

K-space Sampling

Page 8: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

NS-sampling

Insufficientsampling

larger ฮ”KSmaller FOV

smaller ฮ”Klarger FOV

FOV=1/ฮ”K

Nyquist-Shannon Sampling Theorem in k-space

ฮ”KFOV

Page 9: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Kx

Ky

imageK-space

Inverse Fourier transformN

T = N ร— TRscan time collected lines line time

Example: spin echo

T1-weighted (T1w)๏ผšTR=800ms๏ผŒ N=256 , T=3.4min

T2-weighted (T2w)๏ผšTR=2000ms, N=256, T=8.5min

The factors that determine the time of acquisition

ฮ”K

undersampling

Page 10: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Accelerate MR imaging from under-sampled k-space

full-sampled k-space

Fourier transform๏ผˆFT๏ผ‰

Ky

Kx

Nyquist-Shannon sampling theorem

Kx

Ky

Image reconstruction

โˆš

under-sampled k-space

โ•ณ

Inverse Problem

Page 11: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Accelerate MR imaging from under-sampled k-space

prior information

Some knowledge that were known before scan

full-sampled k-space

Fourier transform๏ผˆFT๏ผ‰

Ky

Kx

Nyquist-Shannon sampling theorem

Kx

Ky

โˆš

under-sampled k-space

Page 12: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Fast magnetic resonance imaging based on prior knowledge

Partial Fourier๏ผˆConjugate symmetry๏ผ‰

1980s

Parallel imaging๏ผˆCoil sensitivity๏ผ‰

2000s

Compressed sensing(Sparse prior)

2007

Artificial intelligence(big-data prior)

2016

Fourier transform๏ผˆ1807๏ผ‰

Multi-channelSampling theorem

๏ผˆ1976๏ผ‰

Compressed sensing๏ผˆ2006๏ผ‰

Deep learning๏ผˆ2006๏ผ‰

Page 13: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

๐‘ ๐‘– ๐‘˜๐‘ฅ, ๐‘˜๐‘ฆ = ๐น๐ถ๐‘–๐‘š ๐‘ฅ, ๐‘ฆ

reconstruction

Inverse FFT

Sensitivity Encoding

๐ถ11

๐ถ12

๐ถ21

๐ถ22

๐‘š๐‘Ž1

๐‘š๐‘Ž1๐‘š๐‘Ž1

๐‘š๐‘Ž4

=

๐ถ11 ๐ถ12๐ถ21 ๐ถ22๐ถ31๐ถ41

๐ถ32๐ถ42

๐‘š1

๐‘š2

reconstruction๏ผšsolve an inverse problemCondition number๏ผšSNR

The formulation of parallel MR imaging

Page 14: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

ๅนถ่กŒ็ฃๅ…ฑๆŒฏๆˆๅƒ้—ฎ้ข˜โ€”โ€”ไฟกๅ™ชๆฏ”ไธ‹้™

Highly ill-posed

4-fold

๐œŒ2

๐œŒ1

๐œŒ๐‘Ž

The problem of parallel MR imaging โ€”low SNR

Spatially variant noise amplification

Page 15: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

RO

PE ๐‘†๐‘Ž๐‘๐‘ 

Physical Coil

Step 1. Conjugate Transpose

๐‘†๐‘Ž๐‘๐‘ ๐‘‡

Conjugate Virtual CoilStep 2.

Kernel

Mapping

Step 2.

Kernel

Mapping๐‘†๐‘Ž๐‘๐‘ 2 ๐‘†๐‘Ž๐‘๐‘ 

๐‘‡ 2

Kernel Mapped Coil Kernel Mapped Coil

Coil 1 Coil 2 Coil 3

๐‘†๐‘Ž๐‘๐‘ 

Coil 1 Coil 2 Coil 3

Readout

Phase

Encoding

Coil 1 Coil 2 Coil 3 Coil 1 Coil 2 Coil 3

๐‘†๐‘Ž๐‘๐‘ ๐‘‡๐‘†๐‘Ž๐‘๐‘ 

2 ๐‘†๐‘Ž๐‘๐‘ ๐‘‡ 2

Increasing #equations by using Virtual Coil

H Wang, S Jia & D Liang et al. PMB

Page 16: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

6X acceleration192x192, 1.8 x 1.8 mm

8X acceleration192x192, 1.8 x 1.8 mm

TGrap

pa

CarR

ace

RO

PE

Time Time

CarRace accelerated Real-Time Cine

Submitted to MRM

Page 17: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

4X๏ผŒ ~85 ms/frame 6X, ~56 ms/frame 8X, ~40 ms/frame

CarRace: accelerated Real-Time Cine

Page 18: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Sparse Sampling based on compressed sensing

Compressed Sensing (CS):The sparse signals can be reconstructed accurately from very few incoherent measurements by a non-linear procedure.

OutcomeRequirement

42000 citations

BenefitImaging innovation: how compressed sensing

would change the world of magnetic resonance

Page 19: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

CS-MRI

Page 20: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Sparsity in MR Images

Most MRI images are sparse or can be sparsely represented in some bases

y

t

y

f

MR Angiography Finite Difference/TV

Cardiac cineTemporal-frequency

Brain Wavelet

Page 21: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

artifacts caused by incoherent undersampling are similar to noise

Image by full sampling

Incoherence

artifacts by coherent sampling

artifacts caused by Incoherent sampling

Image Sparse transform K-space

Page 22: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Long reconstruction time

Complicated parameter-selection

Losing fine details (โ€œfeatureโ€)

Gold standard (full sampled data) CS(R=2.5, 1D variable density)

[1] S.Ravishankar et.al., IEEE TMI,2011 [2] Q. Liu, et.al., IEEE TIP,2013. [3] D. Liang et.al., MRM 2011. [4] V.Patel et.al., IEEE TIP 2012. [5] B. Ning et.al., MRI 2013

Some Issues in CS-based MR Imaging

Page 23: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Reason: nonlinear model, iterative reconstruction

Solution: 1) improve computing power2) use pre-conditioner

Long Reconstruction Time

286

Pentium

GPUCloud computing

[1] F. Ong, et.al., ISMRM2018. [2] J. Gemert, et.al., ISMRM2018

Diagonal pre-conditioner

Circulant pre-conditioner

Page 24: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

[1] K. Khare et al, MRM, 2013. [2] J. Tamir et al, ISMRM2018. [3] K. Bang et al, ISMRM2018. [4] J. Cheng et al, ISMRM2018

Reason: multi constraints๏ผŒnon-convex function

Solution:

1) Test on a big data set and choose the sub-optimal but robust

parameters

2) Parameter-free methods or automatic parameter selection

2min ( ) s.t. p

IJ I F I f

2

1 2,( ) min

2l l l

Dl

J I D R I

Complicated Parameter-selection

โ‘  Noise pre-scanโ‘ก First-order primal dual algorithm

Page 25: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

residual image

Reason

Loosing Details

*

2 0 1 1 || || || ||SI I C I I S C

Reconstruction solution SI : Vector I with only S largest components*I :

Details cannot be sparsely represented enough and are lost after nonlinear filtering

Candes, E. J et al, COMMUNICATIONS ON PURE AND APPLIED MATHEMATICS . 2006; 59 (8): 1207-1223

Page 26: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

The methods of preserving details

Powerful sparse representation patch-based methods (dictionary

learning, Nonlocal constraint)

Tensor-based Sparse transformation

Nonlinear compressed sensing method

Detail restoration

Iterative regularization method (IRM)

Bregman iterative method

[1] S.Ravishankar et.al., IEEE TMI,2011 [2] Q. Liu, et.al., IEEE TIP,2013. [3] D. Liang et.al., MRM 2011. [4] S.Fang et.al., MRM 2010. [5] Y. Yu et.al., PLoS ONE,2014. [6] J He et.al., IEEE TMI,2016. [7] Y. Zhou , et.al., ISBI 2007.[8]http://blog.sina.com.cn/s/blog_89ba75c80101gxgm.html [9]S. Osher et.al, Multiscale Model. Simul., 2005. [10] B. Liu et.al., MRM, 2009. [11] W. Yin et.al., SIAM J. Imaging Sci., 2008. [12] X Ye et.al., IEEE TMI, 2011. [13] S. Wang et. al., PMB 2016. [14] J. Cheng et. al., MRM 2018

Page 27: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research
Page 28: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research
Page 29: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Editorial: the articlesfeatured span some of themost cutting-edge areasand a reflection of the mostinfluential researchpublished in PMB 2016

Physics in Medicine and Biology Highlights of 2016

Page 30: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Application: Whole Brain Vessel Wall Imaging

Current: 24min without acceleration๏ผŒ9min with parallel imaging

MR Angiography MR VW imagingStroke

Challenges๏ผšhigh

resolution, large

FOV and fast scan Lumen wall

Page 31: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

45y male stroke patient๏ผŒeccentric thickening 52y male stroke patient ๏ผŒeccentric thickening

9min 5min 9min 5min

9min 5min 5min enhanced

47y female patient๏ผŒArterial dissection is visible

5 mins๏ผš3D Whole Brain Vessel Wall Imaging

Page 32: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Joint Intracranial and Carotid Arterial Wall Imaging

1

1

2

2

3

3

1

1

2

2

3

3

Parallel Imaging 2.7x in 9:18 min Compressed Sensing 5x in 5 min 0.55mm

A female stroke

patient of 47

years old.

Wall thickening was

detected in MCA

and ECA.

Page 33: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

FOV๏ผš230 x 210 x 192 mmScan resolution๏ผš0.6 x 0.6 x 0.6 ๆฏซ็ฑณCoil๏ผš32-channel Head&NeckTA๏ผš3.5minsA 62-year old male

Joint Intracranial and Carotid Arterial Wall Imaging

S. Jia et al. ISMRM2020

Page 34: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Ultrafast cardiac MR imaging and Vessel Wall Imaging were featured in the 2018 Radiological Society of North America (RSNA) as the highlight of the cooperation between SIAT and United Imaging (UI).

Collaboration with industry

Page 35: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Deep Learning MRI: motivation

offline training

online reconstructionzero-filling reconstruction

Training data

scan reconlinear MRI

scan reconCS-MRI

scanDL-MRI offline learning recon

Page 36: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Deep Learning MRI : Preliminary work

1. S Wang, L. Ying, D Liang, โ€œAccelerating Magnetic ResonanceImaging via Deep Learning,โ€ IEEE-ISBI 2016: 514-517.

3. Yang Y๏ผŒSun J, Li H, Xu Z. Deep ADMM-net for compressive sensing MRI. Advances in Neural Information Processing Systems 2016: 10-18.

2. Hammernik K, et al. Learning a Variational Network for Reconstruction of Accelerated MRI Data. ISMRM 2016.

Page 37: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Deep Learning MRI: Our Roadmap

ๆขๆ ‹๏ผˆDong Liang๏ผ‰๏ผŒๆœฑ็‡•ๆฐ๏ผŒๅดๅž ๏ผŒๅˆ˜ๆ–ฐ๏ผŒ้ƒ‘ๆตท่ฃ๏ผŒๅŸบไบŽๆœบๅ™จๅญฆไน ็š„ๅนถ่กŒ็ฃๅ…ฑ ๆŒฏ ๆˆ ๅƒ GRAPPA

ๆ–น ๆณ• ๏ผŒZL201210288373.4

ๅทฒ่ฝฌ่ฎฉไธŠๆตท่”ๅฝฑ

Dong Liang et al, Machine learning for GRAPPA, ZL201210288373.4

ๆขๆ ‹๏ผˆDong Liang๏ผ‰๏ผŒๆœฑ็‡•ๆฐ๏ผŒๆœฑ้กบ๏ผŒๅˆ˜ๆ–ฐ๏ผŒ้ƒ‘ๆตท่ฃ๏ผŒๅŸบไบŽๆทฑๅบฆๅญฆไน ็š„็ฃๅ…ฑๆŒฏๅฟซ้€Ÿๅ‚ๆ•ฐๆˆๅƒๆ–นๆณ•ๅ’Œ็ณป็ปŸ๏ผŒZL201310617004.X

Dong Liang et al, Deep learning for fast parameter mapping, ZL201310617004.X

ๆขๆ ‹๏ผˆDong Liang๏ผ‰๏ผŒๆœฑ็‡•ๆฐ๏ผŒๆœฑ้กบ๏ผŒๅˆ˜ๆ–ฐ๏ผŒ้ƒ‘ๆตท่ฃ๏ผŒๅŸบไบŽๆทฑๅบฆๅญฆไน ็š„็ฃๅ…ฑๆŒฏๆˆๅƒๆ–นๆณ•ๅ’Œ็ณป็ปŸ๏ผŒZL201310633874.6

Dong Liang et al, Deep learning for fast MR imaging ZL201310617004.X

Page 38: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Deep Learning for Fast MR Imaging

Scan ReconLinear MRI

Scan ReconCS-MRI

ScanDL-MRI Learning Recon

input output

Offline Training

Training data

Data-driven

Offline Training

Training data

Model-driven

Iterative procedure of an algorithm

D Liang et al., IEEE Sig Proc Mag,

37(1):141-151, 2020

Page 39: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

2

2 1( )Ax f F x

๐‘Ÿ ๐‘›+1 = ๐‘ฅ ๐‘› โˆ’ ๐œŒA๐‘‡ A๐‘ฅ ๐‘› โˆ’ ๐‘“

๐‘ฅ ๐‘›+1 = ๐น ๐‘ ๐‘œ๐‘“๐‘ก ๐น ๐‘Ÿ ๐‘›+1 , ๐œƒ

Model-driven Deep Learning

Page 40: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Model-driven Deep Learning

Model Inference

Network architecture

Gradient decent ADMM FISTA PDHG โ€ฆ

CNN RNN GAN โ€ฆ

2

2 1

1

2Ax f x

( , ) ( )d Ax f R x

2

2

1( )

2Ax f R x

Unrolling the iterations of an optimization algorithm to deep network

Learning capability of deep networks

Convergence guarantee of optimization algorithms

A could be Fourier encoding or sensitivity encoding (learnable๏ผ‰

Page 41: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research
Page 42: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

VN (Variational Network)

min ๐ด๐‘ฅ โˆ’ ๐‘ฆ 22 +

๐‘–=1

๐‘๐‘˜

ฮฆ๐‘–(K๐‘–๐‘ฅ), 1

Gradient Decent

๐‘ข๐‘›+1 = ๐‘ข๐‘› โˆ’

๐‘–=1

๐‘๐‘˜

๐พ๐‘–๐‘› ๐‘‡ฮฆ๐‘–

๐‘›โ€ฒ ๐พ๐‘–๐‘›๐‘ข๐‘› โˆ’ ๐œ†๐‘›๐ดโˆ— ๐ด๐‘ข๐‘› โˆ’ ๐‘ฆ

Parameters TransformRegularizer

K. Hammernik et al. MRM2018

Page 43: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

ADMM-CSnet

min ๐ด๐‘ฅ โˆ’ ๐‘ฆ 22 +

๐‘™=1

๐ฟ

๐œ†๐‘™ ๐‘” ๐ท๐‘™๐‘ฅ

l lz D xADMMSolver 1

ADMMSolver 2

z xTransform domain Image domain

Basic-ADMM-CSNet Generic-ADMM-CSNet

ParametersTransformRegularizer

argmin๐‘ฅ

1

2๐ด๐‘ฅ โˆ’ ๐‘ฆ 2

2 +

๐‘™=1

๐ฟ๐œŒ๐‘™2

๐ท๐‘™๐‘ฅ + ๐›ฝ๐‘™ โˆ’ ๐‘ง๐‘™ 22

argmin๐‘ง

๐‘™=1

๐ฟ

๐œ†๐‘™๐‘” ๐‘ง๐‘™ +๐œŒ๐‘™2

๐ท๐‘™๐‘ฅ + ๐›ฝ๐‘™ โˆ’ ๐‘ง๐‘™ 22

argmin๐›ฝ

๐‘™=1

๐ฟ

๐›ฝ๐‘™ , ๐ท๐‘™๐‘ฅ โˆ’ ๐‘ง๐‘™

argmin๐‘ฅ

1

2๐ด๐‘ฅ โˆ’ ๐‘ฆ 2

2 +๐œŒ

2๐‘ฅ + ๐›ฝ โˆ’ ๐‘ง 2

2

argmin๐‘ง

๐‘™=1

๐ฟ

๐œ†๐‘™๐‘” ๐ท๐‘™๐‘ง +๐œŒ

2๐‘ฅ + ๐›ฝ โˆ’ ๐‘ง 2

2

argmin๐›ฝ

๐‘™=1

๐ฟ

๐›ฝ, ๐‘ฅ โˆ’ ๐‘ง

๐‘‹ ๐‘› : ๐‘ฅ ๐‘› =A๐ป๐‘ฆ + ๐‘™=1

๐ฟ ๐œŒ๐‘™๐ท๐‘™๐‘‡ ๐‘ง๐‘™

๐‘›โˆ’1โˆ’ ๐›ฝ๐‘™

๐‘›โˆ’1

A๐ป๐ด + ๐‘™=1๐ฟ ๐œŒ๐‘™๐ท๐‘™

๐‘‡๐ท๐‘™

๐‘ ๐‘› : ๐‘ง๐‘™๐‘›= ๐‘†(๐ท๐‘™๐‘ฅ

๐‘› + ๐›ฝ๐‘™๐‘›โˆ’1

; ๐œ†๐‘™ ๐œŒ๐‘™

M ๐‘› : ๐›ฝ๐‘™๐‘›= ๐›ฝ๐‘™

๐‘›โˆ’1+ ๐œ‚๐‘™(๐ท๐‘™๐‘ฅ

๐‘› โˆ’ ๐‘ง๐‘™๐‘›

๐‘‹ ๐‘› : ๐‘ฅ ๐‘› =A๐ป๐‘ฆ + ๐œŒ ๐‘ง ๐‘›โˆ’1 โˆ’ ๐›ฝ ๐‘›โˆ’1

A๐ป๐ด + ๐œŒ๐ผ

๐‘ ๐‘› : ๐‘ง ๐‘› = ๐œ‡1๐‘ง๐‘›,๐‘˜โˆ’1 + ๐œ‡2 ๐‘ฅ ๐‘› + ๐›ฝ ๐‘›โˆ’1

โˆ’

๐‘™=1

๐ฟ

๐œ†๐‘™ ๐ท๐‘™๐‘‡๐ป ๐ท๐‘™๐‘ง

๐‘›,๐‘˜โˆ’1

M ๐‘› : ๐›ฝ ๐‘› = ๐›ฝ ๐‘›โˆ’1 + ๐œ‚ ๐‘ฅ ๐‘› โˆ’ ๐‘ง ๐‘›

Y Yang, J Sun, Z Xu et al. NIPS 2006,IEEE TPRMI 2018

Page 44: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

ADMM-net-โ…  ADMM-net-โ…ก

ADMM-net-โ…ข

2

2

1

( )L

l l

l

Ax f g D x

( , ) ( )d Ax f R x

argmin๐‘ฅ

1

2๐ด๐‘ฅ โˆ’ ๐‘“ 2

2 +๐œŒ

2๐‘ฅ + ๐›ฝ โˆ’ ๐‘ง 2

2

argmin๐‘ง

๐‘™=1

๐ฟ

๐œ†๐‘™๐‘” ๐ท๐‘™๐‘ง +๐œŒ

2๐‘ฅ + ๐›ฝ โˆ’ ๐‘ง 2

2

argmin๐›ฝ

๐‘™=1

๐ฟ

๐›ฝ, ๐‘ฅ โˆ’ ๐‘ง

argmin๐‘ฅ

๐‘‘(๐ด๐‘ฅ, ๐‘“) +๐œŒ

2๐‘ฅ + ๐›ฝ โˆ’ ๐‘ง 2

2

argmin๐‘ง

๐‘™=1

๐ฟ

๐œ†๐‘™๐‘” ๐ท๐‘™๐‘ง +๐œŒ

2๐‘ฅ + ๐›ฝ โˆ’ ๐‘ง 2

2

argmin๐›ฝ

๐‘™=1

๐ฟ

๐›ฝ, ๐‘ฅ โˆ’ ๐‘ง

๐‘‹ ๐‘› : ๐‘ฅ ๐‘› =A๐ป๐‘“ + ๐œŒ ๐‘ง ๐‘›โˆ’1 โˆ’ ๐›ฝ ๐‘›โˆ’1

A๐ป๐ด + ๐œŒ๐ผ

๐‘ ๐‘› : ๐‘ง ๐‘› = ๐œ‡1๐‘ง๐‘›,๐‘˜โˆ’1 + ๐œ‡2 ๐‘ฅ ๐‘› + ๐›ฝ ๐‘›โˆ’1

โˆ’

๐‘™=1

๐ฟ

๐œ†๐‘™ ๐ท๐‘™๐‘‡๐ป ๐ท๐‘™๐‘ง

๐‘›,๐‘˜โˆ’1

M ๐‘› : ๐›ฝ ๐‘› = ๐›ฝ ๐‘›โˆ’1 + ๐œ‚ ๐‘ฅ ๐‘› โˆ’ ๐‘ง ๐‘›

Learning variable structures

[1] Y Yang et al. IEEE TPRMI 2018[2] J Cheng and D Liang. unpublished data

1

( )( , )L

l l

l

g D xd Ax f

Learning data consistency

๐‘‹ ๐‘› : ๐‘ฅ ๐‘› = ๐‘š1๐‘ฅ๐‘›,๐‘˜โˆ’1 +๐‘š2 (๐‘ง

๐‘›โˆ’1 โˆ’ ๐›ฝ ๐‘›โˆ’1 )

โˆ’๐›พA๐ปฮ“(๐ด๐‘ฅ ๐‘›,๐‘˜โˆ’1 , ๐‘“)

๐‘ ๐‘› : ๐‘ง ๐‘› = ๐œ‡1๐‘ง๐‘›,๐‘˜โˆ’1 + ๐œ‡2 ๐‘ฅ ๐‘› + ๐›ฝ ๐‘›โˆ’1

โˆ’

๐‘™=1

๐ฟ

๐œ†๐‘™ ๐ท๐‘™๐‘‡๐ป ๐ท๐‘™๐‘ง

๐‘›,๐‘˜โˆ’1

M ๐‘› : ๐›ฝ ๐‘› = ๐›ฝ ๐‘›โˆ’1 + ๐œ‚ ๐‘ฅ ๐‘› โˆ’ ๐‘ง ๐‘›

๐ท ๐‘› : ๐‘‘ ๐‘› = ฮ“(๐ด๐‘ฅ(๐‘›โˆ’1), ๐‘“)

๐‘‹ ๐‘› : ๐‘ฅ ๐‘› = ฮ (๐‘ฅ ๐‘›โˆ’1 , ๐‘ง ๐‘›โˆ’1 โˆ’ ๐›ฝ ๐‘›โˆ’1 , A๐ป๐‘‘ ๐‘› )

๐‘ ๐‘› : ๐‘ง ๐‘› = ฮ› ๐‘ฅ ๐‘› + ๐›ฝ ๐‘›โˆ’1

M ๐‘› : ๐›ฝ ๐‘› = ๐›ฝ ๐‘›โˆ’1 + ๐œ‚ ๐‘ฅ ๐‘› โˆ’ ๐‘ง ๐‘›

ADMM-CSnet

Page 45: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

ref ADMM-net-โ…ข ADMM-net-โ…ก ADMM-net-โ…  Zero-fillingR=6

J Cheng and D Liang. http://arxiv.org/abs/1906.08143

ADMM-CSnet

Page 46: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

PDHG-net

[1] J Alder et al. IEEE TMI2018 [2] J Cheng and D Liang. ISMRM2019 [3] J Cheng and D Liang. EMBC2019

Primal Dual Hybrid Gradient algorithm

(Chambolle-Pock algorithm)

min๐‘ฅ{๐น ๐ด๐‘ฅ + ๐‘…(๐‘ฅ)}

MR reconstruction

Data fidelity Regularization

๐’…๐’+๐Ÿ =๐’…๐’ + ๐ˆ ๐‘จ ๐’™๐’ โˆ’ ๐’‡

๐Ÿ + ๐ˆ๐’™๐’+๐Ÿ = ๐’‘๐’“๐’๐’™๐‰ ๐‘น ๐’™๐’ โˆ’ ๐‰๐‘จโˆ—๐’…๐’+๐Ÿ ๐’™๐’+๐Ÿ = ๐’™๐’+๐Ÿ + ๐œฝ ๐’™๐’+๐Ÿ โˆ’ ๐’™๐’

๐’Ž๐’Š๐’๐’Ž

๐Ÿ

๐Ÿ๐‘จ๐’™ โˆ’ ๐’‡ ๐Ÿ

๐Ÿ + ๐€๐‘น ๐’™

MR reconstruction

๐’…๐’+๐Ÿ = ๐’‘๐’“๐’๐’™๐ˆ ๐‘ญโˆ— ๐’…๐’ + ๐ˆ๐‘จ ๐’™๐’

๐’™๐’+๐Ÿ = ๐’‘๐’“๐’๐’™๐‰ ๐‘น ๐’™๐’ โˆ’ ๐‰๐‘จโˆ—๐’…๐’+๐Ÿ ๐’™๐’+๐Ÿ = ๐’™๐’+๐Ÿ + ๐œฝ ๐’™๐’+๐Ÿ โˆ’ ๐’™๐’

Relaxing data consistency

Page 47: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

๐‘‘๐‘›+1 = ฮ“(๐‘‘๐‘› + ๐œŽ๐ด ๐‘ฅ๐‘›, ๐‘“)

๐‘ฅ๐‘›+1 = ฮ›(๐‘ฅ๐‘› โˆ’ ๐œ๐ดโˆ—๐‘‘๐‘›+1) ๐‘ฅ๐‘›+1 = ๐‘ฅ๐‘›+1 + ๐œƒ(๐‘ฅ๐‘›+1 โˆ’ ๐‘ฅ๐‘›)

)๐‘‘๐‘›+1 = ฮ“(๐‘‘๐‘›, ๐ด๐‘ฅ๐‘›, ๐‘“)๐‘ฅ๐‘›+1 = ฮ›(๐‘ฅ๐‘›, ๐ด

โˆ—๐‘‘๐‘›+1

๐‘‘๐‘›+1 =)๐‘‘๐‘› + ๐œŽ(๐ด ๐‘ฅ๐‘› โˆ’ ๐‘“

1 + ๐œŽ)๐‘ฅ๐‘›+1 = ฮ›(๐‘ฅ๐‘› โˆ’ ๐œ๐ดโˆ—๐‘‘๐‘›+1) ๐‘ฅ๐‘›+1 = ๐‘ฅ๐‘›+1 + ๐œƒ(๐‘ฅ๐‘›+1 โˆ’ ๐‘ฅ๐‘›

PDHG-net-โ… 

PDHG-net-โ…ก

PDHG-net-โ…ข

2

2( )Ax f R x

( , ) ( )d Ax f R x

( , ) ( )d Ax f R x

J Cheng and D Liang. MICCAI 2019

PDHG-net

Page 48: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

PSNR 41.3688 40.1125 38.9424 38.5734 34.1730 25.3383

SSIM 0.9926 0.9900 0.9874 0.9875 0.9396 0.6696

R=4

Rec_PF Zero-fillingPDHG-net-โ…ขreference D5-C5PDHG-net-โ…กBasic-ADMM-

CSNet

J Cheng and D Liang. MICCAI 2019 J Cheng and D Liang. http://arxiv.org/abs/1906.08143

PDHG-net

Page 49: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

๐‘Ÿ ๐‘›+1 = ๐‘ฅ ๐‘› โˆ’ ๐œŒA๐‘‡ A๐‘ฅ ๐‘› โˆ’ ๐‘“

๐‘ฅ ๐‘›+1 = ๐น ๐‘ ๐‘œ๐‘“๐‘ก ๐น ๐‘Ÿ ๐‘›+1 , ๐œƒ

๐‘‘ ๐‘›+1 = ๐›ค A๐‘ฅ ๐‘› , ๐‘“

๐‘Ÿ ๐‘›+1 = ๐‘ฅ ๐‘› โˆ’ ๐œŒA๐‘‡๐‘‘ ๐‘›+1

๐‘ฅ ๐‘›+1 = ๐น ๐‘ ๐‘œ๐‘“๐‘ก ๐น ๐‘Ÿ ๐‘›+1 , ๐œƒ

๐‘‘ ๐‘›+1 = ๐›ค A๐‘ฅ ๐‘› , ๐‘“

๐‘Ÿ ๐‘›+1 = ๐›ฌ(๐‘ฅ ๐‘› , ๐ด๐‘‡๐‘‘ ๐‘›+1

๐‘ฅ ๐‘›+1 = ๐น ๐‘ ๐‘œ๐‘“๐‘ก ๐น ๐‘Ÿ ๐‘›+1 , ๐œƒ

ISTA-net-โ… 

ISTA-net-โ…ก

ISTA-net-โ…ข

2

2 1( )Ax f F x

1( , ) ( )d Ax f F x

1( , ) ( )d Ax f F x

J Cheng and D Liang. http://arxiv.org/abs/1906.08143

ISTA-net

J Zhang et al. CVPR2018

Page 50: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Discussion: stability

35

37

39

41

43

45

47

Average PSNR

R=4 R=6 R=8 R=10

Test data: 398 human brain 2D slices from 3D data

3D T1-weighted SPACE sequence, TE/TR=8.4/1000ms, FOV=25.6ร—25.6ร—25.6cm3

Poisson disk sampling Networks trained with R=6

More samples imply better reconstruction quality.

J Cheng and D Liang. http://arxiv.org/abs/1906.08143

Page 51: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

ref

R=6 error spectrum plots (ESP) with Fourier radial

the high frequency part of the image has higher error than low frequency part

state โ…ข has the lowest

J Cheng and D Liang. http://arxiv.org/abs/1906.08143

Discussion: stability

Page 52: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

J Cheng and D Liang. unpublished data

40

40.5

41

41.5

42

42.5

PDHG-net-โ…  PDHG-net-โ… * PDHG-net-โ…ก

Average PSNR R=6 Test data: 398 human brain 2D

slices from 3D data PDHG-net-โ…ก has more

parameters than PDHG-net-โ…  PDHG-net-โ… * has the same

structure as PDHG-net-โ…  but the same number of parameters as PDHG-net-โ…ก

the improvement from state โ…  to state โ…ก is induced by the learned data fidelity rather than the deeper network.

Discussion: stability

Page 53: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

full 3X 4X

ADMM-Net-III Test in Industry

Page 54: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Spine data๏ผˆUIH๏ผ‰ADMM-Net-III Test in Industry

full 3X 4X

Page 55: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

55

LeftMCA 2

RightMCA 2

62 years oldMaleTA = 4:48

The ADMM-Net-III gave better SNR and wall depiction quality.

S Jia and J Cheng

et al. unpublished data

ADMM-Net-III MR VWI in Industry

Page 56: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

56

uCS 5x0.6 mm iso

ADMM-Net 5x0.6 mm iso

basilar artery

The ADMM-Net-III gave slightly better SNR and similar wall depiction quality.

S Jia and J Cheng et al. unpublished data

ADMM-Net-III MR VWI in Industry

Page 57: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Deep Learning MRI: a big picture

Training size

Complex convolution

Data sharing

Model-driven

Methods

Data-driven

Methods

Learn parameters

[1] Y. Yang et al, IEEE PAMI 2018.[2]K. Hammernik, et al, MRM 2018. [3]J. Zhang et al, CVPR 2018. [4]H. K. Aggarwal et al, IEEE TMI 2019. [5]J. Schlemper, et al, IEEE TMI 2018. [6] J. Adler et al, IEEE TMI 2018. [7]J. Cheng et al, MICCAI 2019. [8]C. Qin et al, IEEE TMI 2019. [9]S. Wang et al, ISBI 2016. [10]K. Kwon et al, Med Phys 2017. [11]B. Zhu et al, Nature 2018. [12]T. M. Quan et al, IEEE TMI 2018. [13]M. Mardani et al, IEEE TMI 2019. [14]Y. Han et al, MRM 2018. [15]M. Akcakaya et al, MRM 2019. [16]D. W. Lee et al, IEEE TBME 2018. [17]A. Hauptmann et al, MRM 2019. [18]S. Wang et al, NMR in Biomedicine 2019. [19]T. Eo et al, MRM 2018. [20]C. Cai et al, MRM 2018. [21]F. Liu et al, MRM 2019. [22]J. Yoon et al, Neuroimage 2018. [23]O. CohenMRM 2018.

Page 58: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Roadmap of medical imaging equipment

new applications and New discoveries

New requirementsfor equipment

Next generation equipment

New physical phenomenon/ new principle

Medical imagingequipment

Several supporting techniquesnew information technologies,new material, etc

Mathematics

Page 59: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

โ€ข Collaborators:โ€“Leslie Ying(UB)

โ€“Yuchou Chang(Houston)

โ€“Qiegen Liu(NCU)

โ€“Yio-Chu Chung (Siemens)

โ€ข Funding supportโ€“ National Science Foundation of China

โ€“ Chinese Academy of Sciences

โ€“ Shenzhen Basic Research Program

โ€ข My colleges and

students at SIAT:โ€“ YanJie Zhu

โ€“ Haifeng Wang

โ€“ Shanshan Wang

โ€“ Lei Zhang

โ€“ Sen Jia

โ€“ Jing Cheng

โ€“ Ziwen Ke

Acknowledgement

Page 60: Fast Magnetic Resonance Imaging: Theory, Technique and ......Fast Magnetic Resonance Imaging: Theory, Technique and Application Dong Liang, PhD Program number: 5179Medical AI Research

Thanks!