![Page 1: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/1.jpg)
Linear filtering
![Page 2: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/2.jpg)
Motivation: Image denoising• How can we reduce noise in a photograph?
![Page 3: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/3.jpg)
Moving average• Let’s replace each pixel with a weighted
average of its neighborhood• The weights are called the filter kernel• What are the weights for the average of a
3 3 i hb h d?3x3 neighborhood?
111
111
111
111
“box filter”
Source: D. Lowe
![Page 4: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/4.jpg)
Defining convolution• Let f be the image and g be the kernel. The
output of convolving f with g is denoted f * g.
∑ −−=∗lk
lkglnkmfnmgf,
],[],[],)[(,
fConvention: kernel is “flipped”kernel is flipped
Source: F. Durand
• MATLAB functions: conv2, filter2, imfilter
![Page 5: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/5.jpg)
Key properties• Linearity: filter(f1 + f2) = filter(f1) + filter(f2)• Shift invariance: same behavior regardless of
pixel location: filter(shift(f)) = shift(filter(f))• Theoretical result: any linear shift-invariant
t b t d l tioperator can be represented as a convolution
![Page 6: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/6.jpg)
Properties in more detail• Commutative: a * b = b * a
• Conceptually no difference between filter and signal
A i ti * (b * ) ( * b) *• Associative: a * (b * c) = (a * b) * c• Often apply several filters one after another: (((a * b1) * b2) * b3)• This is equivalent to applying one filter: a * (b1 * b2 * b3)This is equivalent to applying one filter: a (b1 b2 b3)
• Distributes over addition: a * (b + c) = (a * b) + (a * c)• Scalars factor out: ka * b = a * kb = k (a * b)Scalars factor out: ka b a kb k (a b)• Identity: unit impulse e = […, 0, 0, 1, 0, 0, …],
a * e = a
![Page 7: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/7.jpg)
Annoying detailsWhat is the size of the output?• MATLAB: filter2(g, f, shape)
• shape = ‘full’: output size is sum of sizes of f and g• shape = ‘same’: output size is same as f• shape = ‘valid’: output size is difference of sizes of f and gshape valid : output size is difference of sizes of f and g
ggfull same valid
gg gggg
f f f
gggg gg gg
![Page 8: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/8.jpg)
Annoying detailsWhat about near the edge?
• the filter window falls off the edge of the image• need to extrapolate• need to extrapolate• methods:
– clip filter (black)d– wrap around
– copy edge– reflect across edge
Source: S. Marschner
![Page 9: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/9.jpg)
Annoying detailsWhat about near the edge?
• the filter window falls off the edge of the image• need to extrapolate• need to extrapolate• methods (MATLAB):
– clip filter (black): imfilter(f, g, 0)d i filt (f ‘ i l ’)– wrap around: imfilter(f, g, ‘circular’)
– copy edge: imfilter(f, g, ‘replicate’)– reflect across edge: imfilter(f, g, ‘symmetric’)
Source: S. Marschner
![Page 10: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/10.jpg)
Practice with linear filters
010000 ?000
OriginalOriginal
Source: D. Lowe
![Page 11: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/11.jpg)
Practice with linear filters
010000
000
Original Filt dOriginal Filtered (no change)
Source: D. Lowe
![Page 12: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/12.jpg)
Practice with linear filters
100000 ?000
OriginalOriginal
Source: D. Lowe
![Page 13: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/13.jpg)
Practice with linear filters
100000
000
Original Shift d l ftOriginal Shifted leftBy 1 pixel
Source: D. Lowe
![Page 14: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/14.jpg)
Practice with linear filters
?111111
Original
111
Original
Source: D. Lowe
![Page 15: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/15.jpg)
Practice with linear filters
111111
Original
111
Original Blur (with abox filter)
Source: D. Lowe
![Page 16: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/16.jpg)
Practice with linear filters
111111111
000020000 - ?
Original(Note that filter sums to 1)
Original
Source: D. Lowe
![Page 17: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/17.jpg)
Practice with linear filters
111111111
000020000 -
Original Sh i filtOriginal Sharpening filter- Accentuates differences with local average
Source: D. Lowe
![Page 18: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/18.jpg)
Sharpening
Source: D. Lowe
![Page 19: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/19.jpg)
SharpeningWhat does blurring take away?
– =
original smoothed (5x5) detail
Let’s add it back:
=+
sharpenedoriginal detail
![Page 20: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/20.jpg)
Smoothing with box filter revisited• What’s wrong with this picture?• What’s the solution?
Source: D. Forsyth
![Page 21: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/21.jpg)
Smoothing with box filter revisited• What’s wrong with this picture?• What’s the solution?
• To eliminate edge effects, weight contribution of neighborhood pixels according to their closeness to the centerto the center
“fuzzy blob”
![Page 22: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/22.jpg)
Gaussian Kernel
0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003
5 x 5 σ = 1
• Constant factor at front makes volume sum to 1 (can be
5 x 5, σ 1
ignored when computing the filter values, as we should renormalize weights to sum to 1 in any case)
Source: C. Rasmussen
![Page 23: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/23.jpg)
Gaussian Kernel
σ = 2 with 30 x 30 σ 5 with 30 x 30
St d d d i ti d t i t t f thi
σ = 2 with 30 x 30 kernel
σ = 5 with 30 x 30 kernel
• Standard deviation σ: determines extent of smoothing
Source: K. Grauman
![Page 24: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/24.jpg)
Choosing kernel width• The Gaussian function has infinite support,
but discrete filters use finite kernels
Source: K. Grauman
![Page 25: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/25.jpg)
Choosing kernel width• Rule of thumb: set filter half-width to about 3σ
![Page 26: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/26.jpg)
Gaussian vs. box filtering
![Page 27: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/27.jpg)
Gaussian filters• Remove “high-frequency” components from
the image (low-pass filter)• Convolution with self is another Gaussian
• So can smooth with small-σ kernel, repeat, and get same result as larger-σ kernel would haveresult as larger σ kernel would have
• Convolving two times with Gaussian kernel with std. dev. σis same as convolving once with kernel with std. dev.
Separable kernel2σ
• Separable kernel• Factors into product of two 1D Gaussians
Source: K. Grauman
![Page 28: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/28.jpg)
Separability of the Gaussian filter
Source: D. Lowe
![Page 29: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/29.jpg)
Separability example
2D convolution(center location only)( y)
The filter factorse e ac o sinto a product of 1D
filters:
* =Perform convolutionalong rows:
* =Followed by convolutionl th i i l
Source: K. Grauman
along the remaining column:
![Page 30: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/30.jpg)
Why is separability useful?• What is the complexity of filtering an n×n
image with an m×m kernel? • O(n2 m2)
• What if the kernel is separable?• O(n2 m)
![Page 31: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/31.jpg)
Noise
• Salt and pepper noise: contains random occurrencesrandom occurrences of black and white pixels
• Impulse noise: contains random occurrences of white
Original Salt and pepper noise
occurrences of white pixels
• Gaussian noise: variations invariations in intensity drawn from a Gaussian normal distributiondistribution
Gaussian noiseImpulse noise
Source: S. Seitz
![Page 32: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/32.jpg)
Gaussian noise• Mathematical model: sum of many
independent factors• Good for small standard deviations• Assumption: independent, zero-mean noise
Source: M. Hebert
![Page 33: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/33.jpg)
Reducing Gaussian noise
Smoothing with larger standard deviations suppresses noise, but also blurs the image
![Page 34: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/34.jpg)
Reducing salt-and-pepper noise
3x3 5x5 7x7
What’s wrong with the results?
![Page 35: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/35.jpg)
Alternative idea: Median filtering
• A median filter operates over a window by selecting the median intensity in the window
• Is median filtering linear?Source: K. Grauman
![Page 36: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/36.jpg)
Median filter• What advantage does median filtering have
over Gaussian filtering?R b t t tli• Robustness to outliers
Source: K. Grauman
![Page 37: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/37.jpg)
Median filterSalt-and-pepper noise Median filtered
Source: M. Hebert
MATLAB: medfilt2(image, [h w])
![Page 38: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/38.jpg)
Gaussian vs. median filtering3 3 5 5 7 73x3 5x5 7x7
Gaussian
Median
![Page 39: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/39.jpg)
Sharpening revisited
Source: D. Lowe
![Page 40: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/40.jpg)
Sharpening revisitedWhat does blurring take away?
– =
original smoothed (5x5) detail
Let’s add it back:
=+ α
sharpenedoriginal detail
![Page 41: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/41.jpg)
Unsharp mask filter
))1(()1()( gefgffgfff −+∗=∗−+=∗−+ αααα
image blurredimage
unit impulse(identity)
it i lGaussian
unit impulseLaplacian of Gaussian
![Page 42: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/42.jpg)
Application: Hybrid Images
A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006
![Page 43: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/43.jpg)
![Page 44: Linear filtering - Computer Sciencelazebnik/spring11/lec05_filter.pdf · Microsoft PowerPoint - lec05_filter Author: lazebnik Created Date: 2/1/2011 4:55:54 PM](https://reader033.vdocuments.net/reader033/viewer/2022043007/5f9275b37dc4ac6350383179/html5/thumbnails/44.jpg)
Application: Hybrid Images
Gaussian Filter
Laplacian Filter
A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006