enhanced image modeling for em/mpmenhanced image...
TRANSCRIPT
Enhanced Image Modeling for EM/MPMEnhanced Image Modeling for EM/MPM Segmentation of 3D Materials Image Data
Dae Woo Kim, Mary L. ComerSchool of Electrical and Computer EngineeringSchool of Electrical and Computer Engineering
Purdue University
Presentation Outline
- Motivation
- SEM Imaging Modes
2D d 3D Bl i M d l- 2D and 3D Blurring Models
- Expectation-Maximization/Maximization of the Posterior Marginals
(EM/MPM) Segmentation
- 2D and 3D Joint Deconvolution/Segmentation (JDS)
- New prior model: Minimum Area Increment (MAI)
3D EM/MPM- 3D EM/MPM
- Results & Conclusions
Motivation
• Scanning electron microscope (SEM) images have blurring due in part to complexScanning electron microscope (SEM) images have blurring due in part to complexelectron interactions during acquisition
• One particular problem that arises during segmentation is necking: the merging ofti l th t d t t t h i th i i l i d tparticles that do not appear to touch in the original image data
• We incorporate model for blurring degradation into the original EM/MPM method inorder to reduce neckingg
• We also introduce a new prior model called minimum area increment to reduce necking
Current model in EM/MPM has smoothing parameter βCurrent model in EM/MPM has smoothing parameter β
(a) Original image (b) Ground truth (c) Original (d) Original
3
(a) Original image (b) Ground truth (c) Original EM/MPM (β=3.0)
(d) Original EM/MPM (β=1.2)
SEM Imaging Modes
• Secondary Electrons (SE):
XSEBSE BSE
PE
AE
Secondary Electrons (SE):Due to SE’s low energy, they can escape only from a thin surface layer of a few nanometers. In this mode, blurring degradation can be modeled with a
R
SEBSE AE, g g2D blurring filter.
• Backscattered Electrons (BSE) :
ctro
n ra
nge ( )
Information depth in BSE mode is deeper than in SE mode. If we capture electrons with small energies below 1keV, we can make the exit depth of
Ele
g pBSE have the same order as of SE. Therefore, we can conclude that the interactions for low-energy electrons can be modeled with a 2D filter while the interactions for high-energy electrons can be modeled with 3D blurring filter.
Diffusion cloud of electron range R for normal incidence of the primary electron (PE)1.
41L. Reimer. Scanning Electron Microscopy: Physics of Image Formation and Microanalysis , 2nd Edition. Springer-Verlag, Berlin, 1998
2D and 3D Blurring Filter Coefficients
• 2D filter coefficients :2D filter coefficients : The lateral number of generated SE can be modeled as an exponential.
ated
SE 10000
8000
r of g
ener 6000
4000
• 3D filter coefficients :We propose 3D filter which has coefficients as below: N
umbe
r
Lateral distance from impact
2000
0-400 -300 -200 -100 0 100 200 300 400
The lateral distribution of generated
Lateral distance from impact point [nm]
gSE (Monte Carlo simulation, silicon, 5kV)2
52 Günter Wilkening, Ludger Koenders. Nanoscale Calibration Standards And Methods: dimensional and related measurements in the micro and nanometer range, 1st Edition, WILEY-VCH, Weinheim, 2005
Original Image Models for EM/MPM
• Use the Markov Random Field as the prior model
• Use the Gaussian distribution
• Use Bayes’ rule to combine the these two models into the posterior distribution function
data term regularization term
6
EM/MPM Segmentation
• Use the Maximizer of the Posterior Marginal (MPM) criterion as the optimization objective. – Minimizes the expected number of misclassified pixels.
• Use the Expectation/Maximization (EM) algorithm to estimate model parameters.- The unknown parameter vector contains means and variances for the Gaussian image model.
7
2D JDS(Joint Deconvolution/Segmentation) Method : Definition
• We define label field x, observed image y and blurring vector hWe define label field x, observed image y and blurring vector h.
- Let the set of all lattice point S be [1, · · · ,M]2 and the order of the pixel of the label field x and the observed image y be raster scan order as below:label field x and the observed image y be raster scan order as below:
- We can make the blurring matrix H having window size (2W + 1) × (2W + 1) be avector h using raster scan order, so that
8
2D JDS Method : Image Model & Posterior Model 3
• Image Model :g
• Posterior Model:
93 D.W. Kim and M.L. Comer, “Joint deconvolution/segmentation of microscope image of materials,” in IEEE Statistical Signal Processing Workshop, Ann Arbor, MI, USA, August 2012.
2D JDS Method : EM algorithm
• In EM iteration we can get closed form solution of variance But for theIn EM iteration we can get closed form solution of variance. But for the mean, we get L linear equations from which we can obtain estimates of the means
10
3D JDS Method : Definition
• We define label field xn and observed image yn of the n-th slice in a stack of image. n g yn gAnd we define 3D blurring vector h3D.
xn : label field of the n-th imageyn : observed n-th image.h3D : blurring vector having coefficient h3D(s1,s2,m)
yT
yn
y1
11
3D JDS Method : Image Model & Posterior Model
• New Image Model :g
• New Posterior Model:• New Posterior Model:
12
Results : Test Sequence
Slice166(Bottom) Slice167 Slice168Slice166(Bottom) Slice167 Slice168
Slice169 Slice170(Top)
• Series of five René 88 DT images The light colored phase is γ' the gray13
• Series of five René 88 DT images. The light-colored phase is γ the graymatrix is γ. We applied our method from the bottom image to the top.
Results : 3D Blurring Image Model
(c) Original EM/MPM (β=3.0)PMP 5 35%
(b) Ground Truth(a) Original imageSli 170 PMP = 5.35%Slice 170
(d) blurring image model (β=3.0, ω=0.10, δ =0.5)
(e) Preprocessed ImageMATLAB ‘deconvlucy’
(ω=2 0)
(f) Preprocessed EM/MPM (β=3.0)
PMP = 4 76%
(d) blurring image model (β=3.0, ω=0.15, δ =0.5)
(d) blurring image model (β=3.0, ω=0.20, δ =0.5)
(d) blurring image model (β=3.0, ω=0.25, δ =0.5)
(d) blurring image model (β=3.0, ω=0.30, δ =0.5)
(d) blurring image model (β=3.0, ω=0.35, δ =0.5)
(d) blurring image model (β=3.0, ω=0.40, δ =0.5)
(d) blurring image model (β=3.0, ω=0.45, δ =0.5)
(d) blurring image model (β=3.0, ω=0.50, δ =0.5)
(d) blurring image model (β=3.0, ω=0.55, δ =0.5)
(d) blurring image model (β=3.0, ω=0.60, δ =0.5)
(d) blurring image model (β=3.0, ω=0.65, δ =0.5)
(d) blurring image model (β=3.0, ω=0.70, δ =0.5)
(d) blurring image model (β=3.0, ω=0.75, δ =0.5)
(d) blurring image model (β=3.0, ω=0.80, δ =0.5)
(d) blurring image model (β=3.0, ω=0.85, δ =0.5)
(d) blurring image model (β=3.0, ω=0.90, δ =0.5)
(d) blurring image model (β=3.0, ω=0.95, δ =0.5)
(d) blurring image model (β=3.0, ω=1.00, δ =0.5)
(d) blurring image model (β=3.0, ω=1.05, δ =0.5)
(d) blurring image model (β=3.0, ω=1.10, δ =0.5)(d) 3D JDS EM/MPM(β=3.0, ω=0.52, δ =0.5)
PMP = 4 08%
14PMP (percentage of misclassified pixels)
(ω 2.0) PMP = 4.76%PMP = 4.08%
MAI(Minimum Area Increment)
• To further reduce object necking, we propose a minimum area incrementconstraint This assigns a penalty for the merging of two or more large objectsconstraint. This assigns a penalty for the merging of two or more large objects
• Connecting point: A point where two or more disconnected areas of thesame class exist in a pre-defined neighborhood around the pointsame class exist in a pre defined neighborhood around the point
4-neighbor 12-neighbor
• C id 4 i hb fi ti Th t i l i th l ft fi i t
neighborhood neighborhood
• Consider a 4-neighbor configuration: The center pixel in the left figure is nota connecting point; the center pixel in the right figure is a connecting point
1 1 001 1 1 0012 class example 1 1 0
1 0
0 0
0
0 0 11
0
1
0
00
0
00xr
0 0 1
0
1 1 0
1 0
0 0
0
0 0 11
1
1
1
00
0
00xr
0 1 1
0
p0 : class 01 : class 1
Minimum area increment
15
Minimum area incrementwindow size ws = 5 Not connecting point
when either xr = 0 or 1Connecting pointwhen either xr = 0 or 1
Area Increment Measuring Function
• Area increment measuring function gws,r(xr) : The increase in area of theg f gws,r( r)largest-area region in a window of size ws×ws around pixel location r if theclass label assigned to pixel r is xr ,
- If one class is a background class (assume this is class 0), then we letIf one class is a background class (assume this is class 0), then we letgws,r(0)=0 for all r- If pixel r is not a connecting point then gws,r(xr)=0
• Consider the following 3-class example, with class 0 a background class
3 l l 1 1
1
11
1
xr
1
1 1 2
1
2
11
1
1
1
2
2
2xr
1 1
1 1 2
1
1
2
1 1 11
1
1
1
21
2
2xr
1 1
1
1 1
1
1 1
1
1 11
1
1
1
11 11xr
1 1
1
3 class exampleblank : class 0
1 : class 12 : class 2
11
g2,r(1) = 0g2,r (2) = 0
11
g2,r(1) = 4g2,r(2) = 1
1 1 111
g2,r(1) = 0g2,r(2) = 1
1 111
g2,r(1) = 9g2,r(2) = 0
Minimum area incrementwindow size ws = 5
16
New prior model: MRF and MAI
• We propose new prior model by incorporating MAI constraint into existingMRF prior model as below:
• To make proposed prior model more effective, we applied SA(simulated annealing) scheme. We gradually increase the β value of the classes which have no necking problemhave no necking problem.
17
Results : MAI
(c) Original EM/MPM (β 3 0) PMP 6 37%
(b) Ground Truth(a) Original imageSli 017
no MAI MAI
(β=3.0) PMP = 6.37%Slice 017
(d) 2D JDS
τ = 1.5ws = 712-neighbor
method with…
g
23th EM iteration 23th EM iteration24th EM iteration 24th EM iteration25th EM iteration 25th EM iteration26th EM iteration 26th EM iteration27th EM iteration 27th EM iteration28th EM iteration 28th EM iteration29th EM iteration 29th EM iteration30th EM iteration 30th EM iteration
18
23th EM iterationβ(0) = 2.3, β(1) = 3.0
23th EM iterationβ(0) = 2.3, β(1) = 3.0
24th EM iterationβ(0) = 2.4, β(1) = 3.0
24th EM iterationβ(0) = 2.4, β(1) = 3.0
25th EM iterationβ(0) = 2.5, β(1) = 3.0
25th EM iterationβ(0) = 2.5, β(1) = 3.0
26th EM iterationβ(0) = 2.6, β(1) = 3.0
26th EM iterationβ(0) = 2.6, β(1) = 3.0
27th EM iterationβ(0) = 2.7, β(1) = 3.0
27th EM iterationβ(0) = 2.7, β(1) = 3.0
28th EM iterationβ(0) = 2.8, β(1) = 3.0
28th EM iterationβ(0) = 2.8, β(1) = 3.0
29th EM iterationβ(0) = 2.9, β(1) = 3.0
29th EM iterationβ(0) = 2.9, β(1) = 3.0
30th EM iterationβ(0) = 3.0, β(1) = 3.0
PMP =2.97%
30th EM iterationβ(0) = 3.0, β(1) = 3.0
PMP=2.75%
Results : MAI
(b) Ground Truth(a) Original imageSli 170
(c) Original EM/MPM (β 3 0) PMP 5 35%Slice 170 (β=3.0) PMP = 5.35%
no MAI MAI(d) 2D JDS
τ = 1.5ws = 712-neighbor
method with…
g
23th EM iteration 23th EM iteration24th EM iteration 24th EM iteration25th EM iteration 25th EM iteration26th EM iteration 26th EM iteration27th EM iteration 27th EM iteration28th EM iteration 28th EM iteration29th EM iteration 29th EM iteration30th EM iteration 30th EM iteration
19
23th EM iterationβ(0) = 2.3, β(1) = 3.0
23th EM iterationβ(0) = 2.3, β(1) = 3.0
24th EM iterationβ(0) = 2.4, β(1) = 3.0
24th EM iterationβ(0) = 2.4, β(1) = 3.0
25th EM iterationβ(0) = 2.5, β(1) = 3.0
25th EM iterationβ(0) = 2.5, β(1) = 3.0
26th EM iterationβ(0) = 2.6, β(1) = 3.0
26th EM iterationβ(0) = 2.6, β(1) = 3.0
27th EM iterationβ(0) = 2.7, β(1) = 3.0
27th EM iterationβ(0) = 2.7, β(1) = 3.0
28th EM iterationβ(0) = 2.8, β(1) = 3.0
28th EM iterationβ(0) = 2.8, β(1) = 3.0
29th EM iterationβ(0) = 2.9, β(1) = 3.0
29th EM iterationβ(0) = 2.9, β(1) = 3.0
30th EM iterationβ(0) = 3.0, β(1) = 3.0
PMP=4.35%
30th EM iterationβ(0) = 3.0, β(1) = 3.0
PMP=3.70%
3D EM/MPM with JDS
• Image Model :
• Posterior Model:Posterior Model:
20
2D EM/MPM VS 3D EM/MPM
2D EM/MPM 3D EM/MPM
yT
2D EM/MPM 3D EM/MPM
IyT
yn
ImageModel
y1
Prior
pixel ofnext frame
PriorModel pixel of
previous frame
• 3D EM/MPM needs a large amount of memory. So we apply 3D EM/MPMmethod for 3 frames and save the result of the middle frame and then move tothe next 3 frames which are one frame shifted from the previous 3 frames
21
the next 3 frames which are one frame shifted from the previous 3 frames.
Results : 3D JDS with 3D EM/MPM
3D JDS with 3D EM/MPM( = 1 5 = 0 75 ω = 0 3
3D JDS with 2D EM/MPM ( = 1 5 ω = 0 3 δ = 0 1
NiAlCr ground truth( = 1.5, = 0.75, ω = 0.3, δ = 0.1, PMP = 6.03%)
( = 1.5, ω = 0.3, δ = 0.1, PMP = 6.31%)
slice 027
3D JDS with 3D EM/MPM ( = 1.5, = 0.75, ω = 0.5, δ = 0.5, PMP = 4.24%)
3D JDS with 2D EM/MPM ( = 1.5, ω = 0.5, δ = 0.5,
PMP = 4.38%)
Rene88 slice170
ground truth
22
δ 0.5, PMP 4.24%))
Results : 3D EM/MPM with JDS and MAI
NiAlCr slice number 1 to 59 (59slices, Image Size 194 X 149 pixels)
3D JDS & MAI i h 3D EM/MPMEM/MPM 3D JDS & MAI 2D EM/MPM
23
3D JDS & MAI with 3D EM/MPM(β= 1.5, ω = 0.3, δ = 0.1, τ = 1.5)
Running time : 6638sec (82x)
EM/MPM (β= 1.5)
Running time4 : 81sec 4 Intel i7 CPU 2.4GHz, Memory 8GB
3D JDS & MAI 2D EM/MPM(β= 1.5, ω = 0.3, δ = 0.1, τ = 1.5)
Running time : 2260sec (28x)
Results : 3D EM/MPM with JDS and MAI
Rene88 slice number 143 to 188 (46slices, Image Size 194 X 149 pixels)
3D JDS & MAI i h 3D EM/MPMEM/MPM 3D JDS & MAI 2D EM/MPM
24
3D JDS & MAI with 3D EM/MPM(β= 1.5, ω = 0.5, δ = 0.5, τ = 1.0)
Running time : 5175sec
EM/MPM (β= 1.5)
Running time : 63sec
3D JDS & MAI 2D EM/MPM(β= 1.5, ω = 0.5, δ = 0.5, τ = 1.0)
Running time : 1762sec
Conclusions
• In this research, we propose a blurring model to improve pixel mis-In this research, we propose a blurring model to improve pixel misclassification originating from blurring in SEM images. The proposed methodincorporates physical modeling of electron interactions into a blurring imagemodelmodel
• We also propose a new prior model including minimum area incrementt i t d l it ith SA hconstraint and apply it with SA scheme.
• In addition, we apply JDS and MAI in the 3D EM/MPM.
• Experimental results demonstrate that the proposed methods can be used toreduce necking in the segmentation of microscope images of materialsg g p g
25
Th kThank you
Appendix B: EM estimation for the new image model (1/4)model (1/4)
• The EM algorithm is an iterative procedure. At each iteration expectation step and maximization step are performed. In the expectation step the following function is computed.
• In the maximization step, we can estimate θ(p) which maximize Q(θ(p), θ( 1))θ(p-1))
Appendix B: EM estimation for the new image model (2/4)model (2/4)
• Similarly, by differentiating with parameter σkwe can get
Appendix B: EM estimation for the new image model (3/4)model (3/4)
Therefore,
Appendix B: EM estimation for the new image model (4/4)model (4/4)
• Let
then we can getg