texture based image segmentation - mit csailpeople.csail.mit.edu/jchang7/pubs/presentations/... ·...
TRANSCRIPT
![Page 1: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/1.jpg)
Texture Based
Image Segmentation
Jason Chang
Massachusetts Institute of Technology
Master’s Degree Candidate
October 15, 2008
![Page 2: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/2.jpg)
Outline
• The Goal
• Level Set Methods
• Previous Work
• Extensions / Improvements• Extensions / Improvements
– Bias Field Estimation
– Texture Based Segmentation
![Page 3: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/3.jpg)
Image Segmentation
• Separate the image into separate regions
• Focus on Binary Segmentation (two regions, one curve)
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
![Page 4: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/4.jpg)
Level Set Methods
• Curve evolution is defined by an energy functional to minimize over
• Allows for easy manipulation
• Implicitly define a curve on the image with a surface in 3D
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
![Page 5: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/5.jpg)
Level Set Methods
• Define a height at every pixel in the image
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
The SurfaceThe Level Sets / Contours
of the Surface
![Page 6: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/6.jpg)
Level Set Methods
• The zero level set represents the 2D curve
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
![Page 7: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/7.jpg)
Some Notation
Notation Meaning Comments
The pixel location
The intensity value at pixel j
j
( ) jx j x≡
{ }1,2,...,j N∈
{ }0,1,...,255j
x ∈
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
The level set function at pixel j
The label assigned to pixel j
The segmented regions
The curve that segments the
image (zero level set)
( ) ( )signj jL j L φ≡ = { }1, 1j
L ∈ + −
{ }| 1j
j LR± = = ±
R+
R−
( ) jjφ φ≡
{ }| 0j
C j φ= =
C
![Page 8: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/8.jpg)
Segmentation Criterion
• Maximize mutual information between pixel intensity and labeling
• Approximate MI by using a Kernel Density Estimate to find the PDFs
( );J J
I x L
( ) ( )|ˆ
1ˆ
J
j s
jx J R x
s
j
R
x xp x K
hp x
hR±
±
±
∈∈±
− ≡ =
∑
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
{ }~ 1, ,J U N…
• Use the Gaussian Kernel±
( )2
2
2
1 x
K x eπ
−=
J. Kim, J.W. Fisher III, A. Yezzi, M. Cetin, and A.S. Willsky. A nonparametric statistical method for image segmentation using information theory and
curve evolution. Image Processing, IEEE Transactions on Image Processing, 14(10):1486-1502, Oct. 2005.
![Page 9: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/9.jpg)
• Minimize the energy functional
• Gradient flow that minimizes the energy
Curve Length Penalty
Junmo’s Algorithm
j C∀ ∈
( )( )
( )( )
( )( )
1 1lo
ˆ
ˆ ˆ ˆg
j i j i jj
R R
xx K x K xp x x
N Nt p p p
di diR Rx x x
ακφ
+ −
+
−−
+ −+
∂= + − −
− −
∂
∫ ∫
� �
( ) ( );ˆJ J
CE C N LI x dsα= − ⋅ + ∫�
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
• Approximate Gradient Descent
( ) ( ) ( )ˆ ˆ ˆRjx
Rj jx x
t p p pR Rx x x+ −−+
∂
∫ ∫
Computationally Intensive
( )( )
lˆ
ˆog
jx
x
j
j
pN N
x
t xp
φακ
+
−
∂−
∂≈
� �
j C∀ ∈
J. Kim, J.W. Fisher III, A. Yezzi, M. Cetin, and A.S. Willsky. A nonparametric statistical method for image segmentation using information theory and
curve evolution. Image Processing, IEEE Transactions on Image Processing, 14(10):1486-1502, Oct. 2005.
![Page 10: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/10.jpg)
Junmo’s AlgorithmThe Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
J. Kim, J.W. Fisher III, A. Yezzi, M. Cetin, and A.S. Willsky. A nonparametric statistical method for image segmentation using information theory and
curve evolution. Image Processing, IEEE Transactions on Image Processing, 14(10):1486-1502, Oct. 2005.
![Page 11: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/11.jpg)
Comments
• Why is the Scalar Segmentation Algorithm Good?
– Segmentation based on non-parametric densities
– No training required
• What needs to be improved?
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
– Does not perform well on images with lighting effects
– Only supports grayscale image segmentation
– Textured images can not be segmented
![Page 12: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/12.jpg)
Bias Field Estimation
• Assume the observed image is the product of an intrinsic image and
a multiplicative gain field
x =
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
Basic Intrinsic Image Gain Field Observed Image
x =
b x g = x
![Page 13: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/13.jpg)
Bias Field Estimation
• Bias field is the log of the Gain field
• Assume that the intrinsic image pixels, bj, are i.i.d. conditioned on
knowing the regions R±
• Find the MAP estimate of β for a given segmentation
j j jx b g= ×
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
( ) ( ) ( ) ( )log log log logj j j j j jy x b g b β= = + = +
• Find the MAP estimate of β for a given segmentation
• Use a fixed-point iteration to find β
( ) ( )( )
( )MAP
ˆPr | ,ˆ ,
ˆPr | ,
jj y j j ji
jj y j j ji
L i p y L
L i p y L
β
β
β
β
∂∂ = = Λ = =
∑∑
β f β f β
( ) ( )( )1ˆ ˆk k
β
+= Λβ f β
W.M. Wells, III, W.E.L. Grimson, R. Kikinis and F.A. Jolesz. Adaptive segmentation of MRI data. IEEE Transactions on Medical Imaging, Vol. 15, pp.
429-442, 1995.
![Page 14: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/14.jpg)
Bias Field Estimation
Segmentation Algorithm with Bias Field Estimation
1. Assume that the bias field, β, is zero and the intrinsic image is just
the observed image
2. Segment the estimated intrinsic image
3. Estimate the bias field, β, given the current segmentation
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
3. Estimate the bias field, β, given the current segmentation
4. Find the estimated intrinsic image, b, from the estimated bias field
5. Repeat from Step 2 until convergence
![Page 15: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/15.jpg)
• Alternate between segmentation and bias field estimation
Bias Field EstimationThe Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
![Page 16: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/16.jpg)
Vector Segmentation
• Extending the formulation to vector values (notice the bold vector ri,
instead of the scalar xi)
• Vector-valued images can be segmented
( )( )
ˆlog
ˆ
jx
x
j
j
pN N
t p
φακ
+
−
∂≈ − ∂
v
v
� �
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
• Vector-valued images can be segmented
– Color images are segmented using
– Texture images can be segmented by representing each pixel
with a texture vector
[ ], ,j
R G B=v
![Page 17: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/17.jpg)
Steerable Pyramid
Recursively replace the gray box at the red dot
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
Φ0
Φ2
Φ1
x
…
ΦD-1
xΘ0
Θ2
Θ1
ΘD-1
z0
zD-1
z1
z2
…
+
+
+
v0
vD-1
v1
v2
Eero P. Simoncelli and William T. Freeman. The steerable Pyramid: A flexible architecture for multi-scale derivative computation. In International
Conference on Image Processing, volume 3, pages 444-447, 23-26 Oct. 1995, Washington DC, USA, 1995
![Page 18: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/18.jpg)
Steerable Pyramid
Varying OrientationOriginal
High-Pass (v0)
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
VaryingScale
Low-Pass (vD)
(v1, v2, v3, … , vD-1)
![Page 19: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/19.jpg)
Dimensionality Reduction
• Estimating PDFs using the Improved Fast Gauss Transform (Yang
et al.) has complexity:
D – dimensionality M – # target points N – # source points
( )( )cO D M N+
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
• Problems with high dimensionality
– Takes ~1 hour to do a KDE on 14 dimensions
– Sparse data in 14 dimensions provide for poor estimate
C. Yang, R. Duraiswami, N. Gumerov, and L. David. Improved fast gauss transform and efficient kernel density estimation. IEEE International
Conference on Computer Vision, pages 464-471, 2003.
![Page 20: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/20.jpg)
Dimensionality Reduction
• We can reconstruct the original image from the outputs
• Approximately reconstruct x by using a subset of the outputs
• Define the error of the reconstruction as the MSE
0 0
D D
i i i
i i= =
= Θ =∑ ∑x z v
{ }1
0
ˆ 0,1D
D
i i
i
u−
=
= ∈∑x v u
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
• Define the error of the reconstruction as the MSE
• Perform the following optimization
( ) ( ) ( )2
2ˆ ˆ ˆ ˆ,
Te = − = − −x x x x x x x x
{ }( )
0,1
1
0
ˆarg min ,
. .
D
D
i
i
e
s t u d
∗
∈
−
=
=
= =∑
u
u x x
u
![Page 21: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/21.jpg)
Pyramid Subset Results
Ground Truth
Oriented Stripes Scaled Stripes
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
Scaled Checkerboard Different Textures
(Using d=3)
![Page 22: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/22.jpg)
Smoothly Varying Textures
• Try to capture a texture that varies smoothly in orientation and scale
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
![Page 23: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/23.jpg)
Smoothly Varying TexturesThe Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
Smoothly changing
orientation
![Page 24: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/24.jpg)
Smoothly Varying TexturesThe Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
Outputs at 4 orientations and 1 scale
![Page 25: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/25.jpg)
Smoothly Varying Textures
Junmo’s Scalar Segmentation
The Goal
Level Set Methods
Previous Work
Bias Field Estimation
Texture Segmentation
Vector Segmentation (Pyramid Subset)
Vector Segmentation (Smoothly Varying Textures)
![Page 26: Texture Based Image Segmentation - MIT CSAILpeople.csail.mit.edu/jchang7/pubs/presentations/... · Texture Segmentation • Approximate Gradient Descent x j x xj j Computationally](https://reader033.vdocuments.net/reader033/viewer/2022042807/5f805e626050b07370169abc/html5/thumbnails/26.jpg)
Thanks!
Questions / Comments?