neighborhood operations objectives why are neighborhoods important? what is linear convolution?...
TRANSCRIPT
Neighborhood Neighborhood OperationsOperations
ObjectivesObjectives• Why are neighborhoods important?• What is linear convolution?
– discrete– templates, masks or filters– algorithm mechanics– graphical interpretation
• Describe non-linear operators– maximum– minimum– median
• What is tiling?
Why are neighbourhoods important?Why are neighbourhoods important?
pixel
Because…Because…
• Provide context for individual pixels.
• Relationships between neighbors determine image features.
• Neighborhood operations:– noise reduction– edge enhancement– zooming
Noise reduction Edge Enhancement
Zooming
Neighbourhood OperationsNeighbourhood Operations
• Linear convolution (*)– A*B*C*D = B*C*D*A = ….
• Non-linear operators– median, max, min, ...
Convolution versus Spectral• We learnt two methods of processing images:
– Convolution– Spectral
• We analyzed and demonstrated how to build a processor (systolic, pipelined, parallel, cellular automaton) for 1D convolution.– 1D convolution is used in speech processing and in polynomial
multiplication.
• We will use visualized animations now to show in more detail how 2D convolution works for images.
• This should convince you how important it is to do convolution quickly in modern Spectral Architectures, especially for 3D etc.
2D Convolution2D Convolution
• Consists of filtering an image A using a filter (mask, template) B.
• Mask is a small image whose pixel values are called weights.
• Weights modify relationships between pixels.
We will show more examples of convolution now, especially for 2D data
A1,1 A1,2 A1,3 A1,4
A2,1 A2,2 A2,3 A2,4
A3,1 A3,2 A3,3 A3,4
A4,1 A4,2 A4,3 A4,4
B1,1 B1,2
B2,1 B2,2
BA C
C1,1 C1,2 C1,3
C3,1
C2,1 C2,2 C2,3
C3,2 C3,3
=
Filter,mask or template
Input image
ConvolvedImage
2 2
4 43 3
A1,1 A1,2
A2,1 A2,2
A1,3 A1,4
A2,3 A2,4
A3,1 A3,2 A3,3 A3,4
A4,1 A4,2 A4,3 A4,4
B1,1 B1,2
B2,1 B2,2 A1,1B1,1
A2,1B2,1
A1,2B1,
2
A2,2B2,
2
A1,1B1,1C1,1= A1,2B1,
2
A2,1B2,1 A2,2B2,
2
A1,1 A1,2
A2,1 A2,2
A1,3 A1,4
A2,3 A2,4
A3,1 A3,2 A3,3 A3,4
A4,1 A4,2 A4,3 A4,4
B1,1 B1,2
B2,1 B2,2A1,2B1,
1
A2,2B2,1
A1,3B1,
2
A2,3B2,
2
A1,2B1,1C1,2= A1,3B1,
2
A2,2B2,1 A2,3B2,
2
A1,1 A1,2
A2,1 A2,2
A1,3 A1,4
A2,3 A2,4
A3,1 A3,2 A3,3 A3,4
A4,1 A4,2 A4,3 A4,4
B1,1 B1,2
B2,1 B2,2A1,3B1,
1
A2,3B2,1
A1,4B1,
2
A2,4B2,
2
A1,3B1,1C1,3= A1,4B1,
2
A2,3B2,1 A2,4B2,
2
A1,1 A1,2
A2,1 A2,2
A1,3 A1,4
A2,3 A2,4
A3,1 A3,2 A3,3 A3,4
A4,1 A4,2 A4,3 A4,4
B1,1 B1,2
B2,1 B2,2
A2,1B1,
1
A3,1B2,1
A2,2B1,
2
A3,2B2,
2
A2,1B1,1C2,1= A2,2B1,
2
A3,1B2,1 A3,2B2,
2
A1,1 A1,2
A2,1 A2,2
A1,3 A1,4
A2,3 A2,4
A3,1 A3,2 A3,3 A3,4
A4,1 A4,2 A4,3 A4,4
B1,1 B1,2
B2,1 B2,2B1,1 B1,2
B2,1 B2,2B1,1 B1,2
B2,1 B2,2
Mathematical NotationMathematical Notation
1 1
1,1,,
Mk
ki
Nl
ljljkijilk BAC
NMB
A1,1B1,1C1,1= A1,2B1,
2
A2,1B2,1 A2,2B2,
2
ConvolutionConvolution
4 4 7 9
4 3 8 9
3 5 9 9
3 6 10 9
-1 2
-1 2
BA C
=
Filter,mask or template
Input image
ConvolvedImage
2 2
4 43 3
6
9
16
23
26
27
21
19
17
Convolution sizeConvolution sizeImage size = M1 N1
Mask size = M2 N2
Convolution size = M1- M2 +1 N1-N2+1
N1
N2
N1-N2+1
Typical Mask sizes= 33, 5 5, 77,9 9, 1111
What is the convolved image size for a 128 128 image and 7 7 mask?
*1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
1 1 11 1 1 1 1 1
=
We convolve with 9*9 averaging filter
Nonlinear Neighbourhood Nonlinear Neighbourhood OperationsOperations
• Maximum
• Minimum
• Median
1,1,
1:,1:,
1,1,1:,1:
,
1,1,1:,1:
,
median
min
max
ljkijiNlljMkki
lk
ljkijiNlljMkki
lk
ljkijiNlljMkki
lk
BAC
BAC
BAC
We discussed already sorter architecture (three variants – pipelined, butterfly combinational and sequential controller). It can be used for all these operations, and also for other non-linear operators
61 62
57 60
59 65
63 56
59 55 58 57
49 53 55 45
1 1
1 1C1,2=
62
60
59
636359
Max and Min OperationsMax and Min Operations
63=max, 59=min
61 62
57 60
59 65
63 56
59 55 58 57
49 53 55 45
C1,2=
Median OperationMedian Operation
1 1
1 1
1
1
1 1 1
62
60
59
63
65
56
55 58 57
62
59
65
60
63
56
57
58
55
987654321
59
rank
9x9 Median
Edge Detection
• What do we mean by edge detection?
• What is an edge?
What is Edge Detection?What is Edge Detection?
• Detects large intensity transitions between pixels
• Redraws the image with only the edges showing
0 0 0 33
0 0 45 78
0 45 23 33
0 0 42 76
0 0 0 38
What is an Edge?
Edge easy to findEdge easy to find
What is an Edge?
Where is edge? Single pixel wide or multiple pixels?Where is edge? Single pixel wide or multiple pixels?
What is an Edge?
Noise: have to distinguish noise from actual edgeNoise: have to distinguish noise from actual edge
Noise Noise is hereis here
What is an Edge?
Is this one edge or two?Is this one edge or two?
What is an Edge?
Texture discontinuityTexture discontinuity
Edge Detection – so what is an Edge Detection – so what is an edge to be detected?edge to be detected?
• What is an edge– A large change in image brightness of a short
spatial distance – Edge strength = (I(x,y)-I(x+dx,y))/dx
But this general definition still allows for many theories, software implementation and hardware architectures.
Now we will Now we will discuss and discuss and illustrate various illustrate various kinds of filter kinds of filter operatorsoperators
Edge Detection FiltersEdge Detection Filters
•High - Pass Filtering Eliminates Uniform Regions (Low Frequencies)
•edge “detection” or “enhancement”
Edge Detection FiltersEdge Detection Filters
Edge Detection ContinuedEdge Detection Continued
•Sum of Kernel Coefficients = 0
•differences in signs emphasize differences in pixel values
•reduces average image intensity
•Negative pixel values in output?
Edge Detection FiltersEdge Detection Filters
vertical
horizontal diagonal
Edge DirectionEdge Direction
Directional High Directional High Pass FiltersPass Filters
Convolution Edge Convolution Edge Detection using Detection using
Sobel and similar Sobel and similar operatorsoperators
Example of Sobel Operator
Sobel OperatorSobel Operator
Sobel Edge DetectionSobel Edge Detection
Convolution Application ExamplesConvolution Application Examples
--Edge DetectionWe apply Sobel Operator
-1 -2 -10 0 01 2 1
-1 0 1
-2 0 2
-1 0 1
Column Mask
Row Mask
as mask to a sub-field of a picture
-1 2 -1
0 0 0
1 2 1
p0, p1, p2
p3, p4, p5
p6, p7, p8 = (p6-p0)+2(p7-p1)+(p8-p2)
We can learn from the result obviously •The result of the above calculation for column mask is horizontal difference•With Row Mask we will get vertical difference
*
The final step of the convolution equation, dividing by the weight , must be ignored
--Edge Detection with Sobel Operator
The weight of a mask determines the grey level of the image after convolution.
Like the weight of Sobel Mask WW= (-1) + (-2) + (-1) + 0 + 0 + 0 +1 + 2 +1= 0
The resulting image lost its “lightness” to be dark.dark.
Convolution Application ExamplesConvolution Application Examples
Sobel OperatorSobel Operator
Sobel OperatorSobel Operator
-1 -2 -1 0 0 0 1 2 1
-1 0 1-2 0 2 -1 0 1
S1= S2 =
Edge Magnitude =
Edge Direction =
22
21 SS
2
11tanS
S
SobelPrewitt
Ticbetts
Canny
Comparison of Edge Detection Comparison of Edge Detection AlgorithmsAlgorithms
Edge DirectionEdge DirectionAssymetric kernels detect edges from Assymetric kernels detect edges from specific directionsspecific directions
EastEast
1 1 -1
1 -2 -1
1 1 -1
NorthEaNorthEastst
1 -1 -1
1 -2 -1
1 1 -1
NorthNorth
-1 -1 -1
1 -2 1
1 1 1
Robinson Robinson OperatorOperator
Robinson Compass MasksRobinson Compass Masks
-1 0 1-2 0 2 -1 0 1
0 1 2-1 0 1 -2 -1 0
1 2 1 0 0 0 -1 -2 -1
2 1 0 1 0 -1 0 -1 -2
1 0 -1 2 0 -2 1 1 -1
0 -1 -2-1 0 -1 2 1 0
-1 -2 -1 0 0 0 1 2 1
-2 -1 0-1 0 1 0 1 2
Arrows show edge directions
Roberts Roberts OperatorOperator
Roberts OperatorRoberts Operator
• Does not return any information about the orientation of the edge
22 ),1()1,()1,1(),( yxIyxIyxIyxI
),1()1,()1,1(),( yxIyxIyxIyxI
or
1 00 -1
0 1-1 0+
Prewitt OperatorPrewitt Operator-1 -1 -1 0 0 0 1 1 1
-1 0 1-1 0 1 -1 0 1
P1= P2 =
Edge Magnitude =
Edge Direction =
22
21 PP
2
11tanP
P
Prewitt Row
Edge Detection FiltersEdge Detection Filters
0 1 2-1 0 1 -2 -1 0
1 2 1 0 0 0 -1 -2 -1Original
and filtered cow
Edge Detection Edge Detection (continued)(continued)
• First Order (Gradient) Kernels
• Prewitt RowPrewitt Row
1 0 -11 0 -1
1 0 -11 0 -1
1 0 -11 0 -1
• Sobel RowSobel Row
1 0 -11 0 -1
2 0 -22 0 -2
1 0 -11 0 -1
• Combine Row and Column Operators
Edge Detection Filters: compare Edge Detection Filters: compare Prewitt Prewitt and Sobeland Sobel
1D Laplacian Operator
)(xf
x
xf
)(
2
22 )(
x
xfxf
first derivative
second derivative
2D Laplacian Operator2D Laplacian Operator
2
2
2
22 ,,
),(y
yxf
x
yxfyxf
0 -1 0-1 4 -1 0 -1 0
1 -2 1-2 4 -2 1 -2 1
-1 -1 -1-1 8 -1-1 -1 -1
Convolution masks approximating a Laplacian
This is just one example of Laplacian, we can use much larger window
0 -1 0-1 4 -1 0 -1 0
Input Mask Output
Image Processing Operations for Early Vision:
Edge Edge DetectionDetection
Reminder: Effect of FiltersReminder: Effect of Filters
low
high
EdgesEdges… are the important part of images
intensity
color
edges
textures
contours
condensation...
simplest, least robust
most difficult, most robust There are many letters B occluded
by black shape here. How to find them?
Edge Detection
• Edges are curves in the image plane across which there is a “significant” change in image brightness.
• The goal of edge detection is the construction of an idealized line drawing
Image Processing OperationsImage Processing Operations
Pixels on edgesPixels on edges
Edges foundEdges found
Edge effects: rarely ideal edgesEdge effects: rarely ideal edges
Not all information is created equal...
Causes of edgesCauses of edges
• Depth discontinuity– One surface occludes another
• Surface orientation discontinuity– the edge of a block
• reflectance discontinuity– texture or color changes
• illumination discontinuity– shadows
Edges: causesEdges: causesWhat are they? Why?
four physical events that cause image edges...
What are they? Why?
discontinuities in
• surface color/intensity• surface normal• depth• lighting (specularities)
four physical events that cause image edges...
Edges: causesEdges: causes
Edges are image locations with a local maximum in image gradient in the direction of that gradient
(steepness)
Edges: causesEdges: causes
Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge
Formal Model of Edge (cont) Formal Model of Edge: Roberts Formal Model of Edge: Roberts
Formal Model of Edge (cont) Formal Model of Edge: Laplacian and Marr-Formal Model of Edge: Laplacian and Marr-Hildreth Hildreth
Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge
Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge
ThresholdsThresholds
very high thresholdoriginal image
Thresholds are important, done before or during edge detection.
Thresholds
very high thresholdoriginal image
ThresholdsThresholds
very high thresholdoriginal image
ThresholdsThresholds
very high threshold
reasonable
original image
ThresholdsThresholds
very high threshold
too low !reasonablethis all takes time...
original image
ThresholdsThresholds