5-binary image analysis
TRANSCRIPT
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ngC
ompu
ter V
isio
n W
S 08
/09
Computer Vision
Binary Image Analysis
Bastian LeibeRWTH Aachenhttp://www.umic.rwth-aachen.de/multimedia
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Binary Images
Just two pixel values Foreground and background Regions of interest
Slide credit: Kristen Grauman2
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Industrial Inspection
R. Nagarajan et al. A real time marking inspection schemefor semiconductor industries, 2006
3
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Document Analysis, Text Recognition
Source: Till Quack, Martin Renold
Natural text (after detection)
Handwritten digits
Scanned documents
4
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Medical/Bio Data
Source: D. Kim et al., Cytometry 35(1), 1999
5
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Blob Tracking & Motion Analysis
Frame Differencing
Source: Kristen Grauman
Background Subtraction
Source: Tobias Jggli6
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Shape Analysis, Free-Viewpoint Video
Medial axis
Silhouette
Visual Hull Reconstruction
Blue-c project, ETH Zurich
7
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Intensity Based Detection
Looking for dark pixels
8
fg_pix = find(im < 65);
Slide Credit: Kristen Grauman
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Uses: Color Based Detection
Looking for pixels within a certain color range
9
fg_pix = find(hue > t1 & hue < t2);
Slide Credit: Kristen Grauman
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Issues
How to demarcate multiple regions of interest? Count objects Compute further features per object
What to do with noisy binary outputs? Holes Extra small fragments
10Slide Credit: Kristen Grauman
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Outline of Todays Lecture
Convert the image into binary form Thresholding
Clean up the thresholded image Morphological operators
Extract individual objects Connected Components Labeling
Image Source: D. Kim et al., Cytometry 35(1), 199911
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Thresholding
Grayscale image Binary mask Different variants
One-sided
Two-sided
Set membership
otherwise ,0
, if ,1,
TjiFjiFT
otherwise ,0
, if ,1, 21
TjiFTjiFT
otherwise ,0
, if ,1,
ZjiFjiFT
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/12
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
13
Thresholding
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
load trees BW = im2bw(X,map,0.4); imshow(X,map), figure, imshow(BW)
14
Thresholding
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Selecting Thresholds
Typical scenario Separate an object from a distinct background
Try to separate the different grayvalue distributions Partition a bimodal histogram Fit a parametric distribution (e.g. Mixture of Gaussians) Dynamic or local thresholds
In the following, I will present some simple methods. We will see some more general methods in Lecture 6
15
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
A Nice Case: Bimodal Intensity Histograms
Source: Robyn Owens
Ideal histogram,light object on dark background
Actual observedhistogram with noise
16
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Not so Nice Cases
How to separate those?
Threshold selection is difficult in the general case Domain knowledge often helps E.g. Fraction of text on a document page ( histogram quantile) E.g. Size of objects/structure elements
Source: Shapiro & Stockman
Multiple modesTwo distinct modes Overlapping modes
17
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Global Binarization [Otsu79]
Search for the threshold T that minimizes the within-class variance within of the two classes separated by T
where
This is the same as maximizing the between-class variance between
22121
222
)()()()(
)()(
TTTnTn
TT withinbetween
,)( ),(1 TITn yx TITn yx ),(2 )(
18
)()()()()( 222211
2 TTnTTnTwithin def
Otsu Thresholding Explained
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Algorithm
Precompute a cumulative grayvalue histogram h.For each potential threshold T
1.) Separate the pixels into two clusters according to T2.) Look up n1, n2 in h and compute both cluster means3.) Compute
Choose ][maxarg 2 (T)T betweenT
2between
19
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
20
Imr.tif
Effects graythresh - Global image threshold using Otsu's method
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Local Binarization [Niblack86]
Estimate a local threshold within a small neighborhood window W
where k [-1,0] is a user-defined parameter.
Improved version to suppress background noise for document binarization [Sauvola00]
where R is the dynamic range of and k > 0. Typical values: R=128 for 8-bit images and k 0.5.
WWW kT
11R
kT WWW
21
TWEffect:
Useful for text,but not for larger objects
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
% Niblack'smethod- Prepare
clear all; close all% Read image, convert to graylevels and show histogram
i=imread('coins.jpg'); i=double(rgb2gray(i)); figure(1); imshow(i,[0 255])% Calculate histogram and plot figure
ih=histc(i(1:prod(size(i))),0:255); bar(ih)% Select filter size, this works fairly well
N=31;% Calculate local means and variance, this is a neat trick in Matlab
localMean= filter2(ones(N), i) / (N*N);localVar= filter2(ones(N), i.^2) /(N*N) -localMean.^2;localStd=sqrt(localVar);% Heregoesthemagic
weight=-0.8; t=localMean+weight*localStd;% Display differentresults
figure(2)imshow(t,[0 255])it = i
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Effects
23
Original image
Global threshold selection(Otsu)
Local threshold selection(Niblack)
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
24
EffectsI = imread('coins.jpg'); level = graythresh(I); BW = im2bw(I,level); Imshow(BW)
graythresh - Global image threshold using Otsu's method
Global threshold(Otsu)
Local threshold(Niblack)
Original image
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Additional Improvements
Document images often contain a smooth gradientTry to fit that gradient with a polynomial function
Source: S. Lu & C. Tan, ICDAR07
Original image
Binarized resultShading compensation
Fitted surface
25
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
LEAST SQUARES AND PROJECTIONS
We want to fit a straight line y = c+dx to the data (0, 1), (1, 4), (2, 2), (3, 5). This means we must find the c and d that satisfy the equations
c + d 0 = 1c + d 1 = 4c + d 2 = 2c + d 3 = 5
26
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Linear least-squares using normal equations
27
Ax = bA is an m n matrix with m > n.
minimizes the norm Ax b
Ax b must be a vector orthogonal to the column space of A. This means, explicitly, that Ax b is perpendicular to each of the columns of A.
AT(Axb) = 0.
(ATA)x = ATb.
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Surface Fitting
Polynomial surface of degree d
Least-squares estimation, e.g. for d=3 (m=10)
,0
( , )d
i ji j
i jf x y b x y
IAb
,
1
11
1
0
1
0
3
31
30
11
00
2
21
20
1
0
1
0
nmnnnnnn I
II
b
bb
y
yy
yx
yxyx
x
xx
y
yy
x
xx
IAAAb TT 1)(
Solution withpseudo-inverse:
28
f(x,y) = Ax3+Bx2y+Cxy2+Dy3+Ex2+Fxy+Gy2+Hx+Iy+J
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Surface Fitting
Iterative Algorithm1.) Fit parametric surface to all points in region.2.) Subtract estimated surface.3.) Apply global threshold (e.g. with Otsu method)
4.) Fit surface to all background pixels in original region.5.) Subtract estimated surface.5.) Apply global threshold (Otsu)
6.) Iterate further if needed
The first pass also takes foreground pixels into account. This is corrected in the following passes. Basic assumption here: most pixels belong to the background.
Initialguess
Refinedguess
29
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Result Comparison
Original image Global (Otsu)
Polynomial + Global
Local (Niblack)
Source: S. Lu & C. Tan, ICDAR0730
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Result Comparison
Original image Global (Otsu)
Polynomial + Global
Local (Sauvola)
Source: S. Lu & C. Tan, ICDAR0731
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Outline of Todays Lecture
Convert the image into binary form Thresholding
Clean up the thresholded image Morphological operators
Extract individual objects Connected Components Labeling
Image Source: D. Kim et al., Cytometry 35(1), 199932
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Cleaning the Binarized Results
Results of thresholding often still contain noise
Necessary cleaning operations Remove isolated points and small structures Fill holes
Morphological Operators
Image Source: D. Kim et al., Cytometry 35(1), 199933
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
34
Z2 and Z3
set in mathematic morphology represent objects in an image binary image (0 = white, 1 = black) : the element of the set is
the coordinates (x,y) of pixel belong to the object Z2
gray-scaled image : the element of the set is the coordinates (x,y) of pixel belong to the object and the gray levels Z3
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
35
Basic Set Theory
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
36
Logic Operations
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
37
Example
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Structuring element (SE)
38
small set to probe the image under studyshape and size must be adapted to geometricproperties for the objects
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Basic morphological operations
Erosion
Dilation
combine to Opening object Closening background
39
keep general shape but smooth with respect to
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Morphological Operators
Basic idea Scan the image with a structuring element Perform set operations (intersection, union)
of image content with structuring element
Two basic operations Dilation (Matlab: imdilate) Erosion (Matlab: imerode)
Several important combinations Opening (Matlab: imopen) Closing (Matlab: imclose) Boundary extraction
Image Source: R.C. Gonzales & R.E. Woods40
Matlab:>> help strel
Create morphological structuring element
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Dilation
Definition The dilation of A by B is the set
of all displacements z, such thatand A overlap by at least one
element.
( is the mirrored version of B,shifted by z)
Effects If current pixel z is foreground, set all
pixels under (B)z to foreground. Expand connected components Grow features Fill holes
.
A
B1
B2
A B1
A B2
Image Source: R.C. Gonzales & R.E. Woods
zB)(
zB)(
41
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Dilation
42
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Dilation
43
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Erosion
Definition The erosion of A by B is the set
of all displacements z, such thatis entirely contained in A.
Effects If not every pixel under (B)z is
foreground, set the current pixel zto background.
Erode connected components Shrink features Remove bridges, branches, noise
Image Source: R.C. Gonzales & R.E. Woods
A
B1
B2
1
2
zB)(
44
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Erosion
45
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Erosion
46
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
useful
erosion removal of structures of certain shape and size, given by SE
Dilation filling of holes of certain shape and size, given by SE
47
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Opening
erosion followed by dilation, denoted
eliminates protrusions breaks necks smoothes contour
48
BBABA )(
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Opening
49
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Opening
50
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Effect of Opening
Feature selection through sizeof structuring element
Original image
Opening with smallstructuring element
Thresholded
Opening with largerstructuring elementImage Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
51
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Effect of Opening
Feature selection through shapeof structuring element
Opening with circularstructuring element
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Input Image
52
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Closing
dilation followed by erosion, denoted
smooth contour fuse narrow breaks and long thin gulfs eliminate small holes fill gaps in the contour
53
BBABA )(
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Closing
54
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Closing
55
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Effect of Closing
Fill holes in thresholded image(e.g. due to specularities)
Original image Thresholded Closing with circularstructuring element
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Size of structuringelement determineswhich structures areselectively filled.
56
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Example Application: Opening + Closing
Original image ClosingOpening
Dilated imageEroded image
Structuringelement
Source: R.C. Gonzales & R.E. Woods57
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Morphological Boundary Extraction
Definition First erode A by B, then subtract
the result from the original A.
Effects If a 33 structuring element is used,
this results in a boundary that isexactly 1 pixel thick.
Source: R.C. Gonzales & R.E. Woods
)()( BAAA
58
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Morphology Operators on Grayscale Images
Dilation and erosion typically performed on binary images.
If image is grayscale: for dilation take the neighborhood max, for erosion take the min.
59
Original Dilated Eroded
Slide credit: Kristen Grauman
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Outline of Todays Lecture
Convert the image into binary form Thresholding
Clean up the thresholded image Morphological operators
Extract individual objects Connected Components Labeling
Image Source: D. Kim et al., Cytometry 35(1), 199960
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Connected Components Labeling
Goal: Identify distinct regions
Sources: Shapiro & Stockman, Chandra
Binary image Connected componentslabeling
61
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
62
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Connectedness
Which pixels are considered neighbors?
4-connected 8-connected
Source: Chaitanya Chandra63
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Application: Blob Tracking
Slide credit: K. Grauman
Absolute differences from frame to frame
64
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Slide credit: K. Grauman
Thresholding
65
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Slide credit: K. Grauman
Eroding
66
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Application: Segmentation of a Liver
Slide credit: Li Shen67
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
Summary: Binary Image Processing
Pros Fast to compute, easy to store Simple processing techniques Can be very useful for constrained scenarios
Cons Hard to get clean silhouettes Noise is common in realistic scenarios Can be too coarse a representation Cannot deal with 3D changes
68Slide credit: Kristen Grauman
-
Perc
eptu
al a
nd S
enso
ry A
ugm
ente
d Co
mpu
ting
Com
pute
Visi
onW
S 08
/09
References and Further Reading
More on morphological operators can be found in R.C. Gonzales, R.E. Woods,
Digital Image Processing.Prentice Hall, 2001
Online tutorial and Java demos available on http://homepages.inf.ed.ac.uk/rbf/HIPR2/
69