ieee journal of biomedical and health informatics 1 …
TRANSCRIPT
IEEE P
roof
IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 1
Lung Respiratory Motion Estimation Based onFast Kalman Filtering and 4D CT Image
Registration
1
2
3
Peng Xue , Yu Fu, Huizhong Ji, Wentao Cui, and Enqing Dong , Member, IEEE4
Abstract—Respiratory motion estimation is an important5part in image-guided radiation therapy and clinical diag-6nosis. However, most of the respiratory motion estimation7methods rely on indirect measurements of external breath-8ing indicators, which will not only introduce great estima-9tion errors, but also bring invasive injury for patients. In10this paper, we propose a method of lung respiratory mo-11tion estimation based on fast Kalman filtering and 4D CT12image registration (LRME-4DCT). In order to perform dy-13namic motion estimation for continuous phases, a motion14estimation model is constructed by combining two kinds15of GPU-accelerated 4D CT image registration methods with16fast Kalman filtering method. To address the high compu-17tational requirements of 4D CT image sequences, a multi-
Q118
level processing strategy is adopted in the 4D CT image19registration methods, and respiratory motion states are20predicted from three independent directions. In the DIR-lab21dataset and POPI dataset with 4D CT images, the average22target registration error (TRE) of the LRME-4DCT method23can reach 0.91 mm and 0.85 mm respectively. Compared24with traditional estimation methods based on pair-wise im-25age registration, the proposed LRME-4DCT method can26estimate the physiological respiratory motion more accu-27rately and quickly. Our proposed LRME-4DCT method fully28meets the practical clinical requirements for rapid dynamic29estimation of lung respiratory motion.30
Index Terms—Respiratory motion estimation, 4D CT,31Image registration, Kalman filtering.32
I. INTRODUCTION33
ACCURATE respiratory motion modeling of lung has a34
great potential in many clinical applications, such as35
clinical diagnosis, treatment planning and image-guided inter-36
ventions [1], [2]. In the process of image-guided radiotherapy37
Manuscript received June 23, 2020; revised September 8, 2020;accepted October 7, 2020. This work was supported in part by theFundamental Research Funds for the Central Universities (China), inpart by the National Natural Science Foundation of China under Grants81671848 and 81371635, and in part by Key Research and Devel-opment Project of Shandong Province under Grant 2019GGX101022.(Corresponding authors: Enqing Dong; Wentao Cui.)
The authors are with the Department of Mechanical, Elec-trical and Information Engineering, Shandong University, Weihai264209, China (e-mail: [email protected]; [email protected];
Q2
[email protected]; [email protected]; [email protected]).
Digital Object Identifier 10.1109/JBHI.2020.3030071
(IGRT), an accuracy motion model can effectively reduce treat- 38
ment margins and potentially enable more targeted radiation 39
delivery. In addition, image artifacts caused by irregular mo- 40
tion during four-dimensional computed tomography (4D CT) 41
acquisition can be effectively reduced by constructing a patient- 42
specific respiratory motion model. Previously, most respiratory 43
motion estimation methods relied on indirect measurements of 44
external breathing indicators [3]. As it is difficult to image the 45
motion of interest lung during the procedure, markers are often 46
implanted into the region of interest. In this case the implantation 47
can be invasive, and motion information is not applicable to the 48
whole region of interest. 49
However, the advent of 4D CT techniques and image regis- 50
tration methods enable accurate and reliable respiratory motion 51
estimation of the whole lung. The deformation fields obtained 52
by registration of 4D CT images can effectively describe the 53
relative motion of the corresponding tissue structure between 54
the image sequences, and thus to realize the accurate motion 55
estimation of lung [4]–[6]. At the same time, due to the influence 56
of heart beating and respiratory movements, the local intensity 57
inhomogeneity of the lung 4D CT images and large motions of 58
the fine textures are caused, which brings great challenge for 59
accurate motion estimation of lung. 60
In order to solve the above problems, Rühaak et al. [7] 61
proposed a method for estimating large motion in lung CT by 62
integrating regularized key-point correspondences into dense 63
deformable registration. In addition, Vishnevskiy et al. [8] pro- 64
posed an isoPTV method that used Local Correlation Coefficient 65
(LCC) similarly metric and isotropic total variation regulariza- 66
tion to deal with large deformation and intensity inhomogeneity 67
in 4D CT image registration respectively. In 2019, Xue et al. 68
[9] designed an energy function with high-order cliques to 69
constrain the Jacobian matrix values of the deformation field 70
to be nonnegative. This method can maintain the topological 71
of deformation field and prevent unrealistic transformations 72
such as folding and tearing, so as to effectively improve the 73
registration accuracy. In the same year, Castillo [10] proposed a 74
Quadratic Penalty DIR (QPDIR) method, which minimizes an 75
image dissimilarly term and a regularization term derived from 76
the classical leave-one-out cross-validation statistical method. 77
This method works best in the DIR-lab dataset. 78
Although the above methods can achieve intra-observer ac- 79
curacy even for large motion amplitudes, they only register 80
the specific reference image with moving images, and cannot 81
2168-2194 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
IEEE P
roof
2 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
dynamically process the whole image sequence. In order to82
fully utilize the whole information of 4D CT image sequences to83
continuously correct the respiratory motion state of lung. Metz84
et al. [11] proposed a registration method for motion estimation85
in dynamic medical image data based on Nd+t free-form B-86
spline deformation model. The method performed groupwise87
registration directly on the dynamic images, thus avoiding a88
bias towards a specific reference phase. Based on [11], Guyader89
et al. [12] proposed a groupwise registration method based90
on a total correlation dissimilarly metric to further improve91
the estimation accuracy. In addition to the above estimation92
methods based on groupwise registration, Vlad et al. [13] used93
pairwise registration method to calculate displacement vector94
fields relative to a specified reference image, then constructed95
the respiratory motion model based on Principal Component96
Analysis (PCA). In order to reduce the motion artifacts in 4D CT97
images, Zhang et al. [14] calculated displacement vector fields98
relative to a reference phase by using an in-house deformable99
image registration method, and then used PCA to decompose100
each of the displacement vector fields into linear combinations101
of principal motion bases. Finally, these principal motion bases102
were parameterized using a spline model to allow the recon-103
struction of the displacement vector fields at any given phase in104
a respiratory cycle.105
However, the above methods can only construct patient-106
specific respiratory motion estimation model according to the107
existing data, which cannot meet the requirement of real-time108
applications. In order to realize real-time motion estimation109
in image guided radiotherapy, most methods [15]–[19] used110
block matching technology or feature points to track the motion111
state for the region of interest. Although these methods extract112
feature points and regularize the trajectory of feature points for113
different phases to correct the motion state, they cannot estimate114
the motion of the whole lung. In recent related researches, Ha115
et al. [20] proposed a real-time respiratory motion estimation116
method based on sparse-to-dense image registration, this method117
combined GPU-accelerated image-based real-time tracking of118
sparsely distributed feature points and a dense patient-specific119
motion-model to realize the estimation of respiratory motion.120
In 2018, Foote et al. [21] proposed a patient specific motion121
subspace and deep convolutional neural network to recover122
anatomical positions from a single fluoroscopic projection in123
real-time. In the method, the patient specific motion subspace124
is generated by 4D CT image registration, and then a large125
number of 2D respiratory phase fluorescence images with known126
subspace coordinates are generated from the motion subspace as127
a training set for a deep convolution neural network. Although128
this method can estimate the anatomical position in real-time, the129
subspace coordinates of fluorescence images for the training set130
are indirectly estimated by 4D image registration, which relies131
too much on registration accuracy and limits the accuracy of132
estimation.133
In view of the strong tracking ability of the Kalman filter134
algorithm in image sequence processing, and the HOMRF reg-135
istration method proposed in our previous work [9] has the136
characteristics of effectively using prior knowledge and high137
registration accuracy, therefore, a dynamic respiratory motion138
estimation method based on fast Kalman filtering and 4D CT 139
image registration (LRME-4DCT) is proposed in this paper. 140
In the process of constructing Kalman filtering motion esti- 141
mation model, two kinds of registration methods (isoPTV [8] 142
and HOMRF [9]) are used to register for each phase, and the 143
registration results are used as observation and prediction vectors 144
of the constructed motion estimation model respectively. We see 145
that in this way, the combination of 4D CT image registration 146
and Kalman filtering algorithm can be very cleverly generalized 147
to the dynamic lung respiratory motion estimation of 4D CT 148
sequence images. 149
The remainder of this paper is structured as follows. Section II 150
introduces the general workflow of our motion estimation frame- 151
work, which is based on fast Kalman filtering algorithm and 4D 152
CT image registration methods to estimate the respiratory mo- 153
tion of the whole lung. According to the process of constructing 154
a Kalman filter model, we divide the estimation framework into 155
three stages to describe the estimation process of respiratory 156
motion in detail. Section III introduces implementation details of 157
our estimation framework and discuss how to make parameters 158
selection. Section IV carries out the experiments on the DIR-lab 159
dataset and POPI dataset with the target registration error (TRE) 160
as the evaluation index. Comparing the proposed LRME-4DCT 161
method with several current best methods, the effectiveness of 162
the proposed LRME-4DCT method is verified. Finally, some 163
discussions are presented in section V, together with the existing 164
problems and further research ideas. 165
II. METHODS 166
Four-dimensional computed tomography (4D CT), devel- 167
oped for treatment planning and image-guided radiotherapy, 168
is an imaging technique that allows for the acquisition of 169
time-varying 3D image sequences of the thorax through the 170
respiratory cycle. For a sequence of 3D CT images of patient 171
under free-breathing for T+1 phases {It}t∈{0,1,...,T }, where 172
each image It : Ω→ RN consists of N pixels, our goal is to 173
estimate respiratory motion state Xt : t ∈ {0, 1, . . . T} of each 174
phase successively. For computational efficiency, we select the 175
image corresponding to maximum inhalation phase as reference 176
image IR ∈ {It}t∈{0,1,...,T }.The maximum inhalation phase has 177
been reported to be more reproducible than other phase [22], 178
[23], making it suitable for analyzing respiratory motion. On 179
this basis, we initialize the initial state of respiratory motion 180
X0 = {xp} according to spatial positionxp of each pointp ∈ Ω. 181
Therefore, the state of respiratory motion at t phase can be 182
expressed as: Xt = X0 +Dt,0, and our LRME-4DCT method 183
aims to find an optimal displacement field Dt,0 : Ω→ R3×N 184
that describes respiratory motion between the reference image 185
IR and moving image IM,t ∈ {It}t∈{0,1,...,T } at t phase. 186
In this work, we combine fast Kalman filtering algorithm 187
with 4D image registration methods to realize continuous and 188
accurate estimation of the deformation field Dt,0. The general 189
workflow of our LRME-4DCT framework is illustrated in Fig. 1 190
and consists of three stages (prediction stage, observation stage 191
and estimation stage). Among them, the prediction stage and 192
observation stage can be processed in parallel. The estimation 193
IEEE P
roof
XUE et al.: LUNG RESPIRATORY MOTION ESTIMATION BASED ON FAST KALMAN FILTERING AND 4D CT IMAGE REGISTRATION 3
Fig. 1. Workflow of LRME-4DCT framework. Our LRME-4DCT method consists of three stages (prediction stage, observation stage and estimationstage). In the prediction stage and observation stage, the HOMRF continuous registration method and isoPTV registration method are used to obtainthe predicted deformation field and the observed deformation field respectively. The estimation stage is based on the above two stages and usesfast Kalman filtering algorithm to estimate the state of respiratory motion.
stage is based on the above two stages and uses fast Kalman194
filtering algorithm to estimate the state of respiratory motion.195
For the prediction stage at t phase, the estimated deformation196
field D̂t−1,0 (red deformation field grid in Fig. 1) between197
IR and IM,t−1 of t− 1 phase is taken as a prior knowledge,198
then a predicted deformation field Dpt,0 (blue deformation field199
grid in Fig.1) between IR and IM,t is obtained by HOMRF200
continuous registration method (blue box in Fig. 1). Therefore,201
the prediction vector of respiratory motion at t phase can be202
expressed as: Xpt = X0 +Dp
t,0. In the observation stage, a203
traditional registration method based on continuous optimiza-204
tion (isoPTV method, green box in Fig. 1) is used to obtain205
observed deformation fieldDot,0 (green deformation field grid in206
Fig. 1) between IR and IM,t directly. Similarly, the observation207
vector of respiratory motion at t phase can be expressed as:208
Xot = X0 +Do
t,0. Subsequently, the prediction equation and209
observation equation are constructed by using Xpt and Xo
t210
respectively. On this basis, the estimated deformation field D̂t,0211
(red deformation field grid in Fig. 1) at t phase is obtained212
through a constructed Kalman filtering model (red box in Fig. 1).213
A. High-Order MRF Continuous Registration Method214
During the prediction stage, the predicted displacement field215
Dpt,0 between IR and IM,t can be obtained by a suitable IVD CT216
image registration method. In this work, we choose the HOMRF217
continuous registration method [9], which can effectively use218
prior knowledge to obtain predicted deformation field Dpt,0.219
In addition, the HOMRF continuous registration method can220
also maintain topology of the displacement field and avoid local221
optimal solutions effectively. In high-order MRF, estimating a222
3-dimensional displacement field is commonly formulated as223
the following optimization problem: 224
Dpt,0 = argmin
Dpt,0
ED(Dpt,0; IR, IM,t) + λER(D
pt,0) (1)
where ED(Dpt,0; IR, IM,t) represents image similarly 225
metrics,ER(Dpt,0) represents the regularization term (include 226
smoothing constraint term and topological preserving constraint 227
term), λ controls the amount of regularization. In the process 228
of 4D CT image acquisition, due to the influence of heart beats 229
and respiratory movements, motions within the lung can often 230
be larger than the scale of the features (vessels and airways), 231
which brings a great challenge to HOMRF registration method. 232
For large motions of fine texture within the lung, [9] employed 233
a multi-level processing strategy and used a large range of 234
label sets to describe the displacement field Dpt,0between 235
IR and IM,t. However, too many levels of the multi-level 236
processing strategy also bring a heavy computation burden. In 237
order to reduce the number of levels in multi-level processing 238
strategy and avoid directly processing the large motions of the 239
small features between IR and IM,t, the previous estimated 240
deformation field D̂t−1,0 at t− 1 phase can be regarded as a 241
prior knowledge. According to this, we can define a continuous 242
registration energy function as follows: 243
E(Dpt,0) = ED(D̂t−1,0 +Dp
t,t−1; IR, IM,t)
+ λER(D̂t−1,0 +Dpt,t−1) (2)
where Dpt,0 = D̂t−1,0 +Dp
t,t−1, Dpt,t−1 represents a deforma- 244
tion field that describes respiratory motion between t phase and 245
t− 1 phase. In this way, the displacement field Dpt,0 is divided 246
into a prior part D̂t−1,0 and a part to be predicted Dpt,t−1. 247
Therefore, we only need to select a small range of label set 248
IEEE P
roof
4 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
TABLE IAVERAGE TRES FOR VARIOUS METHODS AND COMPUTATION TIME OF LRME-4DCT METHOD IN POPI DATASET
TABLE IIAVERAGE TRES OF 75 LANDMARKS AND COMPUTATION TIME OF LRME-4DCT METHOD IN DIR-LAB DATASET
instead of a large label set to describe the relative respiratory249
motion between t phase and t− 1 phase.250
In addition, irregular respiratory motion of lung will lead251
to local intensity inhomogeneity of the images. In this case,252
some traditional similarly metrics, such as Sum of Absolute253
Differences (SAD) and Sum of Squared Differences (SSD), are 254
not suitable for registration. In order to solve the above problems, 255
Modality Independent Neighborhood Descriptors (MIND) [24] 256
were used to measure the similarity between IR and IM,t. On 257
this basis, we use the same way as [9] to impose smoothing 258
IEEE P
roof
XUE et al.: LUNG RESPIRATORY MOTION ESTIMATION BASED ON FAST KALMAN FILTERING AND 4D CT IMAGE REGISTRATION 5
TABLE IIIAVERAGE TRES OF 300 LANDMARKS FOR VARIOUS METHODS IN DIR-LAB DATASET
TABLE IVAVERAGE COMPUTATION TIME FOR DIFFERENT METHODS
constraints and topology preserving constraints on the displace-259
ment field Dpt,0.260
For the complexity of the continuous energy function with261
high-order cliques form, only a few discrete optimization al-262
gorithms can be used for optimization. In [9], Markov Chain263
Mote Carlo (MCMC) based optimization algorithm was used to264
solve the optimization problem of the designed energy functions.265
At the same time, since the MCMC algorithm is optimized266
based on stochastic sampling, which consumes a lot of time267
to optimize the energy function with high-order cliques. In268
this paper, Iterated Conditional Modes (ICM) [25] optimization269
algorithm is used instead of the traditional MCMC optimization270
algorithm. In the process of optimizing the energy function, the271
ICM optimization algorithm can consider all candidate labels of272
the control points. Therefore, it can greatly reduce the number273
of iterations and ensure the robustness of the algorithm.274
B. Deformable Registration Method Based on275
Continuous Optimization276
In order to ensure the robustness of fast Kalman filtering algo-277
rithm, the selection of the registration methods for obtaining ob-278
servation vectors and prediction vectors should be independent279
of each other. According to this, observation values are directly280
obtained by using the publicly available isoPTV method [8],281
which shows higher accuracy and speed than most methods in282
the DIR-lab and COPD datasets. In observation stage, since only283
a fixed reference image IR is used for registration to all phases 284
of the 4D CT image sequence and the isoPTV method uses 285
continuous optimization algorithm to solve the designed energy 286
function. Therefore, the displacement fieldDot,0 between IR and 287
IM,t obtained using the isoPTV method has some independence 288
compared with HOMRF continuous method. Subsequently, ac- 289
cording to the initial respiratory motion state X0, the observa- 290
tion valueXot at t phase can be expressed as:Xo
t = X0 +Dot,0. 291
Similarly, in order to improve the calculation speed of isoPTV 292
method, the number of levels used in the multi-layer processing 293
strategy is less than [8]. On this basis, the selection of other 294
parameters of the isoPTV method is the same as [8]. 295
C. Fast Kalman Filtering Motion Estimation Model 296
The Kalman filtering algorithm has been widely used in 297
many tracking applications ranging from Global Positioning 298
Systems to biomechanics experiments to track body motion 299
and position [26], [27]. In the process of estimating by using 300
classical Kalman filtering algorithm, the prediction equation and 301
observation equation of the state vector Xtcan be expressed as: 302
Xpt = F tX̂t−1 +Q (3)
Xot = HtXt +R (4)
whereXpt andXo
t represent the predication vector and observa- 303
tion vector respectively,F t andHt represent the state transition 304
matrix and observation matrix at t phase respectively, Q and R 305
represent the variance matrix of prediction noise and observation 306
noise at t phase. On this basis, the Kalman filtering model can 307
sequentially estimate the state vector Xt through following two 308
steps. In the first step, the filtering model estimates the state 309
vector Xt in the following form: 310
Xpt = F tX̂t−1 (5)
Σ−t = F tΣt−1F Tt +Q (6)
where X̂t−1 andΣt−1 represent the optimal estimate state vector 311
and covariance matrix of Xt−1 at t− 1 phase respectively, Σ−t 312
represent the covariance matrix of Xt at t phase. In the second 313
step, the filtering model uses the observation vectorXot to update 314
IEEE P
roof
6 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
its estimated state vector in the previous step as follows:315
X̂t = Xpt +Kt(X
ot −HtX
pt ) (7)
Σt = (I −KtHt)
−∑t
(8)
where Kt is called Kalman gain matrix which is calculated as316
follows:317
Kt = Σ−t HTt (HtΣ
−t H
Tt +R)−1 (9)
The pseudo code of our proposed LRME-4DCT method is318
present in Algorithm 1. As shown in Algorithm 1, each step t,319
LRME-4D CT method returns the estimation vector X̂t (by line320
13) and covariance matrix Σt (by line 7). The whole estimation321
process of LRME-4DCT method can be divided into three322
stages. In prediction stage (lines 4-5), the HOMRF continuous323
registration method is used to obtain the predicted deformation324
field Dpt,0. On this basis, the prediction vector Xp
t and state325
transition matrix F t are calculated. Similarly, in observation326
stage (lines 7-8), the observed deformation fieldDot,0 is obtained327
by using isoPTV registration method. Then, the observation328
vector Xot and observation matrix Ht are calculated. During329
estimation stage (lines 10-14), Kalman filtering equation (equa-330
tion (5)-(8)) is used to update the estimation vector X̂t and331
covariance matrix Σt.332
For a specific patient, the state of respiratory motion at t phase333
can be represented as triplet of matrices, denoted by Xt =334
{Xt;x,Xt;y,Xt;z} ∈ R3×N , where matrices Xt;x, Xt;y and335
Xt;z represent the 3D spatial position matrices, along the left-336
right (LR), anterior-posterior (AP), and superior-inferior (SI)337
directions respectively. In Cartesian coordinate system, the three338
directions of LR, AP, SI are orthogonal, therefore, the state of339
respiratory motion can be estimated independently along the340
above three directions. Without any loss of generality, we use341
our fast Kalman filtering model to estimate the motion state of342
lung along the LR direction, which also are applied to the other343
two directions.344
Given a moving image IM,t at t phase, the prediction vector345
Xpt;x and observation vector Xo
t;x of respiratory motion along346
LR direction can be expressed as:347
Xpt;x = X0;x +Dp
t,0;x (10)
Xot;x = X0;x +Do
t,0;x (11)
where X0;x represents initial state of respiratory motion along348
LR direction, Dpt,0;x and Do
t,0;x represent the deformation field349
along LR direction obtained by HOMRF continuous method350
and isoPTV method respectively. Affected by respiratory move-351
ments and heart beats, respiratory motion between different352
areas of the lung are often inconsistent and non-linear. Therefore,353
it is difficult to find an appropriate state transition matrix F t;x354
and observation matrix Ht;x along LR direction to describe the355
complex motion between neighbor phases. In order to solve the356
above problems and construct a fast Kalman filtering model,357
we use 4D CT image registration process instead of the state358
transition matrix F t;x and observation matrix Ht;x to get the359
prediction vector Xpt;x and observation vector Xo
t;x directly.360
Algorithm 1: LRME-4D CT.
1: Initialize X̂0, Q, R, Σ0
2: for t = 1→T do3: Prediction Stage4: Dp
t,0← HOMRF(argminEHOMRF ;X̂t−1)
5: Xpt = X0 +Dp
t,0;F t = Xpt /X̂t−1
6: Observation Stage7: Do
t,0← isoPTV(argminEisoPTV )8: Xo
t = X0 +Dot,0;Ht = Xo
t/Xt
9: Estimation Stage10: Xp
t = F tX̂t−111: Σ−t = F tΣt−1F T
t +Q12: Kt = Σ−t H
Tt (HtΣ
−t H
Tt +R)−1
13: X̂t = Xpt +Kt(X
ot −HtX
pt )
14: Σt = (I −KtHt)∑−
t
15: end for16: return X̂t
Meanwhile, the motion state of each pixel is estimated inde- 361
pendently to reduce the calculational complexity of covariance 362
matrix. 363
In addition, since the high resolution of 4D CT images, it will 364
consume a lot of time to process millions of dimensions of the 365
state vector by using the classical Kalman filtering algorithm, 366
especially for the calculation of covariance matrix (equation (8) 367
and (9)). For the covariance matrix Σ−t with the size of N × 368
N , it requires huge memory in the estimation process, which 369
will bring a great challenge in practical application. In general, 370
the motion state of each point along the LR direction is only 371
related to its neighbor points. Therefore, the covariance matrix 372
Σ−t can be expressed in a sparse form and there are two non-zero 373
elements in each row of Σ−t except the diagonal. Although the 374
memory consumption can be greatly reduced by sparse matrix, it 375
will take a long time to calculate a sparse matrix with millions of 376
dimensions, as a result, the calculation of the covariance matrix 377
with millions of dimensions makes the Kalman filter algorithm 378
slower in the estimation process. 379
In fact, the constraint terms of each points (such as smooth 380
constraint, topology preservation constraint) along all directions 381
have been considered in the process of registration. In order to 382
avoid excessive constraints and reduce the computation time, 383
the respiratory motion state of each point is considered to be 384
independent and the covariance matrix Σ−t is set as the diagonal 385
sparse matrix. Similarly, in order to avoid excessive constraints, 386
the state transition matrix F t;x and observation matrix Ht;x is 387
set as the diagonal sparse matrix, which can be easily calculated 388
by matrix operation. Therefore, the term (HtΣ−t H
Tt +R) of 389
equation (9) is also in the form of diagonal matrix, and its inverse 390
matrix can be easily obtained by simple reciprocal operation. In 391
this way, a fast Kalman filtering model for continuous estimation 392
of respiratory motion at each phase can be constructed rapidly. 393
D. Parameter Selection and Implementation Details 394
In our estimation framework, parameters are mainly com- 395
posed of two parts, one is the parameters of HOMRF continuous 396
IEEE P
roof
XUE et al.: LUNG RESPIRATORY MOTION ESTIMATION BASED ON FAST KALMAN FILTERING AND 4D CT IMAGE REGISTRATION 7
registration method and isoPTV registration method, the other397
is the parameters of fast Kalman filtering model. In High-Order398
MRF continuous registration model, the selection of parameters399
for smooth term and topology preservation term are the same400
as that in [9], and the maximum iterations of ICM algorithm401
is set to 50. In addition, for all our experiments in this 4D CT402
respiratory motion estimation, displacement vectors are selected403
densely from three orthogonal directions to form a label set of404
each control point.405
For the fast Kalman filtering algorithm, the prediction noise406
and observation noise are assumed to be Gaussian distribution407
with mean value of 0 and standard deviation of σ. In addition,408
since the motion state of each pixel is estimated independently,409
the variance matrix R and Q are diagonal sparse matrix and410
their diagonal values can be expressed as σ2o and σ2
p respectively,411
where σo and σp represent the standard deviation of observation412
noise and prediction noise respectively. In general, the selection413
of R and Q is related to the observed and predicted values. For414
high-precision observed and predicted values, we should choose415
smaller values of R and Q to reduce the impact of noise on the416
estimation accuracy. In our proposed LRME-4DCT method, we417
use two kinds of different GPU-accelerated image registration418
methods (HOMRF registration method and isoPTV registration419
method) to obtain the predicted values and observed values of420
respiratory state respectively. These two registration methods421
show higher speed and accuracy (≈1mm) than most methods in422
the DIR-lab and COPD datasets. According to this, the diagonal423
values of R and Q are empirically set as 0.01 to estimate the424
respiratory motion state of lung.425
In the process of estimation by using fast Kalman filtering426
algorithm, the size of the predicted deformation field Dpt,0 and427
the observed deformation fieldDot,0 obtained by different 4D CT428
image registration methods are usually inconsistent. To resolve429
this inconsistency, linear interpolation algorithm is needed to430
interpolate Dpt,0and Do
t,0 to the same size. On this basis, the431
fast Kalman filtering algorithm is used to estimate the optimal432
respiratory state X̂t at t phase. In addition, since the multi-level433
processing strategy of the HOMRF registration is to mesh the434
moving image and reference image from coarse to fine before435
registration, in the prediction stage of t+ 1 phase, it is necessary436
to down-sample the optimal estimated deformation field D̂t,0437
of t phase to the initial grid size of the multi-level processing438
strategy, and then take the subsampled deformation field as439
a prior knowledge of HOMRF continuous image registration440
method.441
E. GPU Implementation442
In order to meet the requirements of real-time and dynamic443
estimation for respiratory motion in clinical applications, we444
use GPU to accelerate our proposed LRME-4DCT method. In445
general, the estimation of respiratory motion for the whole lung446
takes about 2 minutes on a CPU (Intel Xeon CPU at 3.5GHz (12447
cores) with 128GB RAM) by using our LRME-4DCT method.448
After analysis, it is found that the pre-calculation of the similarly449
metric is the main time-consuming unit. For HOMRF continuous450
registration method, in order to reduce the computation time of451
sum of square differences of MIND descriptors, convolution 452
filters [20] are used to realize the parallel calculation of the 453
similarly metric. According to [20], the calculation of the sum 454
of the squared differences between the two MIND control points 455
can be decomposed into the addition of three convolutional terms 456
which can be calculated in parallel. In this way, the correlation 457
between all control points in moving image and reference im- 458
age can be calculated via convolution computation on GPU at 459
once. In addition, GPU is also used to accelerate the calcula- 460
tion of LCC similarly metric for isoPTV registration method 461
directly. 462
III. EXPERIMENTS 463
To evaluate our proposed LRME-4DCT method and com- 464
pare it with state-of-the-art methods, we use the following two 465
publicly available datasets with manually annotated landmarks: 466
POPI Dataset and 4D CT DIR Dataset. The target registration 467
error (TRE) and the mean TRE of each phase in the experiments 468
are the mainly objective evaluation methods of lung respiratory 469
motion estimation. Among them, TRE is defined as the Eu- 470
clidean distance between the coordinate of the expert landmarks 471
and the coordinate of the landmarks estimated by deformation 472
field transformation, the mean TRE of each phase is used to 473
summarize the estimation accuracy. In addition, to access the 474
statistical significance of differences between overall average 475
TRE of different methods, we performed paired t-tests with a 476
significance level of 5% (p < 0.05) by paring the patient-specific 477
mean TRE. 478
A. Evaluation on POPI Dataset 479
The POPI dataset consists of 10 3D CT reconstructions of dif- 480
ferent phases of a single breathing cycle (case1). Each breathing 481
phase image has a resolution of 0.98mm× 0.98mm× 2mm 482
and is complemented with 40 corresponding anatomical land- 483
marks. In addition, the POPI dataset also provides 3 validation 484
cases of 4D CT images with 100 landmarks(case2-case4). The 485
average voxel resolution of the validation dataset is 0.94mm× 486
0.94mm× 2mm and the average image size is 512× 512× 487
161 voxels. In order to achieve high precision registration, we 488
resize all images to an isotropic 1mm× 1mm× 1mm resolu- 489
tion. Then, we select the image corresponding to the maximum 490
inhalation phase (T10 for case1, T00 for case 2-4) as the initial 491
reference image to track the lung motion of different phase for 492
one breathing cycle. In order to reduce the computation time as 493
much as possible while maintaining the registration accuracy, 494
the number of levels for multi-level processing strategy is set 495
to 3. Meanwhile, the dense sampling range is defined to be 496
{0,±1,±2,±3,±4} voxels. On this basis, the constructed fast 497
Kalman filtering model is used to track the respiratory motion 498
state of each pixel under the above resolution. 499
In order to quantify the degree of coincidence between differ- 500
ent trajectories, we define a Euclidean Distance (ED) to represent 501
the similarly between the trajectories of the landmarks and the 502
IEEE P
roof
8 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
Fig. 2. The motion trajectory of T00-T90 phase with 40 landmarks for POPI dataset. The red lines in the figure represent the motion trajectoriesof anatomical landmarks for T00-T90, and the green lines represent the estimated trajectories using LRME-4DCT method. The magenta linesrepresent that the motion trajectories of anatomical landmarks and the motion trajectories of the estimated trajectory using LRME-4DCT arebasically completely coincident (ED value less than 1 mm).
Fig. 3. Coronal overlay images for case1(a-d) of POPI dataset and case5(e-h) of DIR-lab dataset. (a, e) Overlay of Non-registered image andreference image. (b, f) Overlay of registered image by HOMRF method and reference image. (c, g) Overlay of registered image by isoPTV methodand reference image. (d, h) Overlay of registered image by LRME-4DCT method and reference image.
estimated trajectories of LRME-4DCT:503
ED =1
2
⎛⎝√√√√ 3∑
i=1
(ait − bit)2+
√√√√ 3∑i=1
(ait−1 − bit−1)2
⎞⎠ (12)
where at = (a1t , a2t , a
3t ) and at−1 = (a1t−1, a
2t−1, a
3t−1) repre-504
sent the spatial positions of each landmarks at t phase and505
t− 1 phase under the same anatomical structure respectively,506
bt = (b1t , b2t , b
3t ) and bt−1 = (b1t−1, b
2t−1, b
3t−1) represent the spa-507
tial positions of the estimated points at t phase and t− 1 phase508
respectively. A small value of ED indicates a high degree of509
coincidence between different trajectories.510
Fig. 2 shows the motion trajectory of 40 landmarks for511
T00-T90 phase of POPI dataset. It can be seen from Fig. 2(a)512
that most of the estimated trajectories basically coincide with513
the trajectories of anatomical landmarks, which indicates that514
LRME-4DCT method can accurately estimate the moving posi-515
tion of the landmarks for each phase. In addition, Fig. 3 shows the516
lung CT coronal overlay images of the proposed LRME-4DCT517
method, HOMRF method, and isoPTV method. The first row518
is for the case1 of POPI dataset, the second row is for the519
case5 of DIR-lab dataset. The overlay images are obtained by520
superimposing the Non-registered image (T50 phase image) or521
the registered image and the reference image (T10 phase image522
in POPI dataset or T00 phase image in DIR-lab dataset). In 523
the overlay image, the gray portion indicates complete reg- 524
istration, however the magenta portion and the green portion 525
respectively indicate image under registration and image over 526
registration. 527
It can be seen from Fig. 3(a) that the T10 phase image (the 528
reference image) and the T50 phase image (the Non-registered 529
image) have a large difference, in particular, there is a large 530
displacement in the lower part of the lung, and there are many 531
magenta regions in the Fig.3, indicating severe under registra- 532
tion. Compared with Fig. 3(a), the overlay images of Fig. 3(b) 533
and Fig. 3(c) obtained by HOMRF method and isopTV method 534
respectively are obviously improved, but there are still over 535
registration and under registration. Comparing LRME-4DCT 536
method with HOMRF method and isoPTV method, the un- 537
der registered magenta area and over registered green area 538
in Fig. 3(d) are significantly less than those in Fig. 3(b) and 539
Fig. 3(c). 540
We also use the TRE indicator to further evaluate the ef- 541
fect of our proposed estimation framework. We compare the 542
displacement fields of case1 generated by the FFD [28] and 543
Demons [27] registration methods published by the dataset orga- 544
nizer in the evaluation. In addition, in order to comprehensively 545
analyze the performance of the proposed LRME-4DCT method, 546
IEEE P
roof
XUE et al.: LUNG RESPIRATORY MOTION ESTIMATION BASED ON FAST KALMAN FILTERING AND 4D CT IMAGE REGISTRATION 9
the predicted values and observation values of the Kalman547
filtering model are also used to comparative analysis for all548
cases. Table I lists average TREs and computation time for all549
4 cases. Among them, each breathing phases image of case1550
is complemented with 40 corresponding anatomical landmarks,551
and the other cases have 100 corresponding landmarks of each552
breathing phases image. As can be seen from Table I, for case1,553
the average TRE of the proposed method is 0.85 mm, which is554
optimal among all methods. And through the phase by phase555
analysis of all methods in Table I, it can be found that 7556
phases of the proposed LRME-4DCT method are optimal. At557
the same time, the t-test results of case1 show that the TREs of558
our proposed LRME-4DCT method is significantly better than559
HOMRF method (p-value=0.040) and isoPTV method (p-value560
= 0.004). Similarity, for case2-case4 with 100 corresponding561
landmarks, our proposed LRME-4DCT is also significantly562
better than HOMRF method and isoPTV method which shows563
that our method can effectively improve the estimation accuracy.564
Moreover, for the 4D CT image with size of 512× 512× 161,565
the calculation time of GPU-accelerated LRME-4DCT is about566
11s, showing great potential for clinical application.567
B. Evaluation on 4D CT DIR-Lab Dataset568
In addition to the POPI dataset, we also evaluate our method569
on DIR-lab dataset. The DIR-lab dataset consists of 10 different570
cases labeled from 4D CT1-4D CT10, each of which is further di-571
vided into ten respiratory phase sequences from T00-T90, where572
the maximum inspiratory phase (T00) and maximum expiratory573
phase (T50) provide full 300 expert landmarks, in addition, a574
subset of 75 landmarks has been propagated onto each of the575
expiratory phase (T00-T50). The average voxel resolution of the576
dataset is 1mm× 1mm× 2.5mm and the average image size is577
256× 256× 100 voxels. Similar to the POPI dataset, we resize578
all images to an isotropic 1mm× 1mm× 1mm resolution and579
choose the image corresponding to the maximum inhalation580
phase (T00) as the initial reference image to track the lung581
motion of different phase for one breathing cycle. Compared582
with the POPI data, the DIR-lab dataset has a smaller amplitude583
of motion. In order to ensure the estimation accuracy, the dense584
sampling range is defined to be {0,±1,±2,±3} voxels, and585
the number of levels for multi-level processing strategy is set586
to 2.587
In order to show the effectiveness of estimation more intu-588
itively, we take case 4 in the DIR-lab dataset as an example, and589
give the motion trajectories of 75 landmarks for T00-T50 phases590
(Fig. 4). The representation in the Fig. 4 is similar to Fig. 2. The591
magenta trace lines in Fig. 4 are much more than the green592
and red trace lines, which means our proposed LRME-4DCT593
method can accurately estimate the moving position of the594
landmarks. Similarly, we respectively give the overlay images595
of the proposed LRME-4DCT method, HOMRF method, and596
isoPTV method for the case5 of the DIR-lab dataset. Fig. 2(e) is597
the overlay image of the T00 phase image (the reference image)598
and the T50 phase image (the Non-registered image). Compared599
with Fig. 2(e), the overlay images of Fig. 2(f) and Fig. 2(g)600
obtained by HOMRF method and isopTV method respectively601
are obviously improved. Comparing LRME-4DCT method with602
Fig. 4. The motion trajectory for T00-T50 phase with 75 landmarksof case 4 in DIR-lab dataset. The motion trajectory represented by thevarious color curves in the figure is consistent with that in Fig. 2.
HOMRF method and isoPTV method, the under registered 603
magenta area and over registered green area in Fig. 2(h) are 604
significantly less than those in Fig. 2(f) and Fig. 2(g). In the red 605
boxes in Fig.2(b)-(d) and Fig. 2 (f)-(h), the differences in several 606
registration methods can be seen. 607
In order to comprehensively analyze the performance of 608
the proposed LRME-4DCT method, Table II lists the TREs 609
of 75 landmarks from T00-T50 phases and the computation 610
time of each case, the main purpose is to show the effect of 611
motion estimation at various stages (No regis., isoPTV, HOMRF, 612
LRME-4DCT). In average TREs of T00-T50 phases for all 10 613
cases, the proposed LRME-4DCT method (0.92mm) is signif- 614
icantly better than HOMRF method (predicted value 1.19mm) 615
and isoPTV method (observed value 0.95 mm) in estimating 616
the real state of lung motion. In addition, through analysis of 617
the average TREs for the expiratory phases (T00-T50 phases), 618
compared with the HOMRF method and isoPTV method, the 619
proposed LRME-4DCT method is optimal except for the T30 620
phase. Moreover, for the 4D CT image of DIR-lab dataset with 621
average size of 256× 256× 100, the calculation time of GPU- 622
accelerated LRME-4DCT is about 9.2s, which is approximately 623
11 times faster than that without using GPU. 624
In addition, Table III lists the average TREs of various meth- 625
ods for full 300 landmarks. Among all methods in Table III, the 626
proposed method (0.91mm) is inferior to the optimal QPDIR 627
method [10] (0.90mm). Compared with the optimal QPDIR 628
method, among the results of the proposed method, 2 cases are 629
equivalent and 3 cases are superior. Similarly, by comparing 630
with our previous HOMRF method, the LRME-4DCT method 631
is superior to the HOMRF method [9] in 5 cases. In average 632
TREs of all methods for 10 cases, the average TRE of proposed 633
LRME-4DCT method is 0.91mm, even without lung masked 634
pretreatment, which is better than the optimal masked method 635
[34]. 636
C. Evaluation of Computation Time on Different 637
Datasets for Various Methods 638
In this section, we evaluate the computation time of different 639
methods for POPI dataset and DIR-lab dataset on a 12 Core 640
Intel Xeon CPU at 3.5GHz (with NVIDIA TITAN GPU). In 641
addition, in order to simulate practical clinical application, we 642
extract an area with the size of 130× 250× 100 (left lung) in 643
IEEE P
roof
10 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
4D CT images of the POPI dataset to evaluate the computation644
time of different methods. Table IV lists the average computa-645
tion time on different datasets for various methods. For POPI646
dataset, it takes about 11s to calculate each phase of whole647
lung volume by using GPU-accelerated LRME-4DCT method,648
which is approximately 14 times faster than isoPTV method649
and 60 times faster than HOMRF method. For the DIR-lab650
dataset, it takes about 9.2s to calculate each phase of whole651
lunge volume by using GPU-accelerated LRME-4D CT method,652
which is approximately 10 times faster than isoPTV method and653
98 times faster than HOMRF method. In addition, for the 4D654
CT image with size of 130× 250× 100, the calculation time655
of LRME-4D CT method is 0.29s / phase. In general, it is well656
known that a respiratory cycle is 3-4 seconds. In order to avoid657
excessive irradiation in clinical application, the number of 3D658
images collected in one respiratory cycle is usually less than659
10 phases, so the average interval between neighbor phases is660
0.3s-0.4s. In this case, the method we proposed (0.29s/phase)661
can fully meet the practical clinical requirements for real-time662
estimation of lung respiratory motion.663
IV. DISCUSSIONS664
In the process of estimating respiratory motion state by us-665
ing classical Kalman filtering algorithm, it is difficult to find666
an appropriate state transition matrix and observation matrix667
to describe the complex motion between neighbor phases. At668
the same time, for high-resolution 4D CT images, it will take669
a lot of time to process millions of state vector dimensions670
using the classic Kalman filter algorithm. In order to solve the671
above problems, the prediction vector and observation vector of672
the Kalman filtering algorithm are directly obtained by using673
4D CT image registration methods. In this way, not only can674
complex motions between neighbor phases be described, but675
the constraints contained in the registration can reduce the676
computational complexity of the covariance matrix.677
In order to ensure the effectiveness of Kalman filtering algo-678
rithm, it is necessary to select appropriate values of R and Q,679
and ensure that the observation vectors and prediction vector are680
independent of each other. In general, the higher the accuracy681
of prediction and observation, the higher estimation accuracy of682
Kalman filtering algorithm. At the same time, the selection of R683
and Q also affects the estimation accuracy. For high-precision684
observed and predicted values, choosing smaller values of R685
and Q can reduce the influence of noise on estimation accuracy.686
Therefore, for different registration methods, the appropriate687
values of R and Q should be selected to ensure the registration688
accuracy.689
In lung 4D CT images, due to the influence of heart beats and690
respiratory motions, it will cause local intensity inhomogeneity691
of images and sliding motions between organs. In order to692
solve the above problems, we use MIND and LCC metrics to693
describe the similarity between different images. At the same694
time, since the previous information of respiratory motion is695
fully considered in the HOMRF continuous registration method,696
only the respiratory motion between neighbor phases needs to be697
considered. In general, time interval between neighbor phases698
of 4D CT image sequence is very small, so the motion amplitude699
of respiratory changes little in such a small interval. According 700
to this, in multi-level processing strategy, we employ fewer 701
levels than [9] for registration. On this basis, we use the same 702
linear interpolation algorithm to interpolate the deformation 703
field of multi-level control points as in [8]. Unlike more common 704
cubic B-splines [37], the linear interpolation algorithm can avoid 705
over-smoothing effects and describe the sliding motions between 706
different organs accurately. 707
In order to reduce the computation time, we use GPU to 708
accelerate the processing of our proposed framework, At the 709
same time, ICM optimization algorithm is used to replace the tra- 710
ditional MCMC optimization algorithm to optimize the designed 711
energy function with high-order cliques. In clinical applications, 712
an accuracy respiratory model can reduce treatment margins 713
and enable more targeted radiation delivery. In addition, with 714
the introduction of real-time 4D ultrasound (US) and magnetic 715
resonance imaging (MRI) technology in radiotherapy and high 716
intensity focused ultrasound system (HIFU), compared with off- 717
line respiratory estimation methods based on deformable image 718
registration, the computation speed of respiratory motion model 719
needs to be greatly improved under the premise of ensuring the 720
accuracy. In this case, our proposed GPU-base LRME-4D CT 721
method based on GPU can track the motion state of specific 722
interest regions in real-time, which shows great potential in 723
clinical application. Over past few years, deep learning-based 724
methods have been gradually applied to image registration, 725
image segmentation and feature extraction, among which con- 726
volution neural network is the most representative [38]. The 727
constant time inference capability of convolutional neural net- 728
work allows real-time estimation of the anatomical position of 729
tumor target using interventional images, thus greatly reducing 730
the computation time. In our future research, also within the 731
framework of the proposed LRME-4DCT method, at first con- 732
sider using of lung mask preprocessing [39], and then use the 733
4D CT images of the pre-treatment stage to generate a training 734
set and use the convolution neural network to constructed a 735
patient specific-respiratory model [40]. Subsequently, the pre 736
construct patient specific-respiratory model is used instead of 737
isoPTV method to quickly obtain the observation vector. On 738
this basis, the fast Kalman filtering algorithm is used to estimate 739
the real-time motion states of lung tumor. 740
V. CONCLUSION 741
In this work, we propose a novel respiratory motion estima- 742
tion method based on fast Kalman filtering and 4D CT image 743
registration (LRME-4DCT). This method combines the 4D CT 744
image registration method with fast Kalman filter algorithm in 745
a skillful way. By using two different kinds of GPU-accelerated 746
image registration methods, the predicted values and observed 747
values of the current respiratory state are obtained respectively, 748
then the respiratory state of whole lung can be estimated in few 749
seconds by using constructed fast Kalman filtering model. Com- 750
pared with previous pair-wise registration methods, the proposed 751
LRME-4DCT method can fully utilize the neighboring phase 752
information in the 4D CT image sequences, thus achieve con- 753
tinuous correction of the estimated respiratory state. In the pre- 754
calculation of the similarity metric, the GPU is used to accelerate 755
IEEE P
roof
XUE et al.: LUNG RESPIRATORY MOTION ESTIMATION BASED ON FAST KALMAN FILTERING AND 4D CT IMAGE REGISTRATION 11
the strategy of parallel calculation of three convolutional terms,756
which further reduces the calculation time of the model. For757
clinical applications such as image-guided radiotherapy, only758
the local tumor area is tracked and estimated. In our experiment,759
the tracking time of the area with a size of 130× 250× 100760
can reach 0.29 s/phase. Therefore, the proposed method can761
fully meet the actual clinical requirements for rapid dynamic762
estimation of lung respiratory motion. The experimental results763
in DIR-lab dataset and POPI dataset indicate that our proposed764
LRME-4DCT method can achieve a high accuracy and rapid765
motion estimation.766
REFERENCES767
[1] J. R. McClelland, D. J. Hawkes, T. Schaeffter, and A. P. King, “Respiratory768motion models: A review,” Med. Imag. Anal., vol. 17, no. 1, pp. 19–42,769Jan. 2013.770
[2] S. Khachira, F. Kallel, and A. B. Hamida, “A comparative study of motion771estimation algorithms in cardiac ultrasound sequences,” in Proc. 2th Int.772Conf. Adv. Tech. Sign. Imag. Process., 2016, pp. 119–124.773
[3] A. Siqueira, A. F. Spirandeli, R. Moraes, and V. Zarzos, “Respiratory774waveform estimation from multiple accelerometers: An optimal sensor775number and placement analysis,” IEEE J. Biomed. Health Inform., vol. 23,776no. 4, pp. 1507–1515, Jul. 2019.777
[4] J. Vandemeulebroucke, S. Rit, J. Kybic, P. Clarysse, and D. Sarrut, “Spa-778tiotemporal motion estimation for respiratory-correlated imaging of the779lungs,” Med. Phys., vol. 38, no. 1, pp. 166–178, Jan. 2011.780
[5] G. R. Wu, Q. Wang, J. Lian, and D. G. Shen, “Estimating the 4D respiratory781lung motion by spatiotemporal registration and super-resolution image782reconstruction,” Med. Phys., vol. 40, no. 3, pp. 1–17, Mar. 2013.783
[6] D. D. Yu et al., “Fast rotation-free feature-based image registration using784improved N-SIFT and GMM-based parallel optimization,” IEEE Trans.785Biomed. Eng., vol. 63, no. 8, pp. 1653–1664, Aug. 2016.786
[7] J. Ruhaak et al., “Estimation of large motion in lung CT by integrating787regularized keypoint correspondences into dense deformable registration,”788IEEE Trans. Med. Imag., vol. 36, no. 8, pp. 1746–1757, Aug. 2017.789
[8] V. Vishnevskiy, T. Gass, G. Szekely, C. Tanner, and O. Goksel, “Isotropic790total variation regularization of displacements in parametric image regis-791tration,” IEEE Trans. Med. Imag., vol. 36, no. 2, pp. 385–395, Feb. 2017.792
[9] P. Xue, E. Dong, and H. Ji, “Lung 4D CT image registration based on793high-order markov random field,” IEEE Trans. Med. Imag., vol. 39, no. 4,794pp. 910–921, Apr. 2020.795
[10] E. Castillo, “Quadratic penalty method for intensity-based deformable796image registration and 4DCT lung motion recovery,” Med. Phys., vol. 46,797no. 5, pp. 2194–2203, May 2019.798
[11] C. T. Metz, S. Klein, M. Schaap, T. van Walsum, and W. J. Niessen,799“Nonrigid registration of dynamic medical imaging data using nD plus800t B-splines and a groupwise optimization approach,” Med. Imag. Anal.,801vol. 15, no. 2, pp. 238–249, Apr. 2011.802
[12] J. M. Guyader et al., “Groupwise image registration based on a total cor-803relation dissimilarity measure for quantitative MRI and dynamic imaging804data,” Sci. Rep., vol. 8, pp. 1–14, Aug. 2018.805
[13] V. Boldea, G. C. Sharp, and S. B. Jiang, “4D-CT lung motion estimation806with deformable registration: Quantification of motion nonlinearity and807hysteresis,” Med. Phys., vol. 35, no. 3, pp. 1008–1018, Mar. 2008.808
[14] Y. B. Zhang, J. Z. Yang, L. F. Zhang, L. E. Court, P. A. Balter, and L.809Dong, “Modeling respiratory motion for reducing motion artifacts in 4D810CT images,” Med. Phys., vol. 42, no. 11, pp. 6768–6768, Apr. 2013.811
[15] B. D. de Senneville, A. E. Hamidi, and C. Moonen, “A direct PCA-based812approach for real-time description of physiological organ deformations,”813IEEE Trans. Med. Imag., vol. 34, no. 4, pp. 974–982, Apr. 2015.814
[16] M. Seregni, C. Paganelli, P. Summers, M. Bellomi, G. Baroni, and M.815Riboldi, “A hybrid image registration and matching framework for real-816time motion tracking in MRI-guided radiotherapy,” IEEE Trans. Biomed.817Eng., vol. 65, no. 1, pp. 131–139, Jan. 2018.818
[17] J. H. Choi and S. Lee, “Real-time tumor motion tracking in 3D using plan-819ning 4D CT images during image-guided radiation therapy,” Algorithms,820vol. 11, no. 10, pp. 1–14, Oct. 2018.821
[18] J. B. Yi, H. Yang, X. Yang, and G. L. Chen, “Lung motion estimation by 822robust point matching and spatiotemporal tracking for 4D CT,” Comput. 823Biol. Med., vol. 78, pp. 107–119, Nov. 2016. 824
[19] G. L. Xiong, C. Z. Chen, J. Z. Chen, Y. Q. Xie, and L. Xing, “Tracking the 825motion trajectories of junction structures in 4D CT images of the lung,” 826Phys. Med. Bio., vol. 57, no. 15, pp. 4905–4930, Aug. 2012. 827
[20] I. Y. Ha, M. Wilms, H. Handels, and M. P. Heinrich, “Model-based sparse- 828to-dense image registration for realtime respiratory motion estimation in 829image-guided interventions,” IEEE Trans. Biomed. Eng., vol. 66, no. 2, 830pp. 302–310, Feb. 2019. 831
[21] M. D. Foote, B. Zimmerman, A. Sawant, and S. Joshi, “Real-time 2D-3D 832deformable registration with deep learning and application to lung radio- 833therapy targeting,” in Proc. Inf. Process. Med. Imag., 2019, pp. 265–275. 834
[22] J. J. Sonke, J. Lebesque, and M. Van Herk, “Variability of four-dimensional 835computed tomography patient models,” Int. J. Radiat. Oncol. Biol. Phys., 836vol. 70, no no. 2, pp. 590–598, Feb. 2008. 837
[23] Y. Seppenwoolde et al., “Precise and real-time measurement of 3D tumor 838motion in lung due to breathing and heartbeat, measured during radio- 839therapy,” Int. J. Radiat. Oncol. Biol. Phys., vol. 53, no. 4, pp. 822–834, 840Jul. 2002. 841
[24] M. P. Heinrich et al., “MIND: Modality independent neighbourhood 842descriptor for multi-modal deformable registration,” Med. Imag. Anal., 843vol. 16, no. 7, pp. 1423–1435, Oct. 2012. 844
[25] J. Besag, “On the statistical-aanlysis of dirty pictures,” J. R. Stat. Soc. B. 845Met., vol. 48, no. 3, pp. 259–302, 1986. 846
[26] A. Alshareef et al., “Application of trilateration and kalman filtering 847algorithms to track dynamic brain deformation using sonomicrometry,” 848Biomed. Signal Process. Ctrl., vol. 56, Feb. 2020. 849
[27] C. F. Baumgartner, C. Kolbitsch, J. R. McClelland, D. Rueckert, and A. P. 850King, “Autoadaptive motion modelling for MR-based respiratory motion 851estimation,” Med. Imag. Anal., vol. 35, pp. 83–100, Jan. 2017. 852
[28] B. Delhay, P. Clarysse, J. Lotjonen, T. Katila, and I. E. Magnin, “Evaluation 853of two free form deformation based motion estimators in cardiac and chest 854imaging,” in Proc. Int. Workshop Func. Imag. Mod. Heart, Barcelona, 855Spain, 2005, pp. 467–476. 856
[29] D. Sarrut, V. Boldea, S. Miguet, and C. Ginestet, “Simulation of four- 857dimensional CT images from deformable registration between inhale and 858exhale breath-hold CT scans,” Med. Phys., vol. 33, no. 3, pp. 605–617, 859Mar. 2006. 860
[30] S. Hermann and R. Werner, “High accuracy optical flow for 3D medical 861image registration using the census cost function,” in Proc. Pacific-Rim 862Symp. Imag. Video Technol., 2014, pp. 23–35. 863
[31] J. Rühaak, S. Heldmann, T. Kipshagen, and B. Fischer, “Highly accurate 864fast lung CT registration,” Proc. SPIE, vol. 8669, pp. 1–9, Mar. 2013. 865
[32] T. Polzin et al., “Combining automatic landmark detection and variational 866methods for lung CT registration.” in Proc. 5th Int. Workshop Pulmonary 867Imag. Anal., 2013, pp. 85–96. 868
[33] S. Hermann, “Evaluation of scan-line optimization for 3D medical image 869registration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 8702014, pp. 3073–3080. 871
[34] L. Konig and J. Rühaak, “A fast and accurate prallel algorithm for non- 872linear image registration using normalized gradient fields,” in Proc. 11th 873IEEE Symp. Biomed. Imag., Apr./May 2014, pp. 580–583. 874
[35] V. Vishnevskiy, T. Gass, G. Szekely, and O. Goksel, “Total variation 875regularization of displacements in parametric image registration,” in 876Proc. Int. Conf. Med. Imag. Comput. Comput-Assist. Intervent., 2015, 877pp. 338–345. 878
[36] M. P. Heinrich, M. Jenkinson, M. Brady, and J. A. Schnabel, “MRF-based 879deformable registration and ventilation estimation of lung CT,” IEEE 880Trans. Med. Imag., vol. 32, no. 7, pp. 1239–1248, Jul. 2013. 881
[37] W. M. Yu, M. Tannast, and G. Y. Zheng, “Non-rigid free-form 2D-3D 882registration using a B-spline-based statistical deformation model,” Pattern. 883Recognit., vol. 63, pp. 689–699, Mar. 2017. 884
[38] D. G. Shen, G. R. Wu, and H. I. Suk, “Deep learning in medical image 885analysis,” Annu. Rev. Biomed. Eng., vol. 19, pp. 221–248, 2017. 886
[39] J. D. Song et al., “Lung lesion extraction using a toboggan based growing 887automatic segmentation approach,” IEEE Trans. Med. Imag., vol. 35, no 888no. 1, pp. 337–353, Jan. 2016. 889
[40] G. S. Karanasiou et al., “Stents: Biomechanics, biomaterials, and in- 890sights from computational modeling,” Ann. Biomed. Eng., vol. 45, 891no. 4, pp. 853–872, Apr. 2017. 892