comput ing sum of pixels in a rectangle and haar-like features by strip sum
DESCRIPTION
Comput ing Sum of Pixels in a Rectangle and Haar-Like Features by Strip Sum. Student: Wanli Ouyang (歐陽萬里) Supervisor: Prof. W.K. Cham The Chinese University of Hong Kong. Outline. Introduction The proposed algorithm Experimental result Conclusion. Outline. Introduction - PowerPoint PPT PresentationTRANSCRIPT
112/04/19 1
Computing Sum of Pixels in a Rectangle and Haar-Like Features by Strip SumStudent: Wanli Ouyang (歐陽萬里)
Supervisor: Prof. W.K. Cham
The Chinese University of Hong Kong
112/04/19 2
Outline Introduction The proposed algorithm Experimental result Conclusion
112/04/19 3
OutlineIntroduction The proposed algorithm Experimental result Conclusion
112/04/19 4
Introduction
Haar-like features and rectangle sum
Application
112/04/19 5
Haar-like features and rectangle sum
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
Simple. Any size, any position, non-orthogonal, over-complete. Fast algorithm.
+1 -1
W=H=4
W
H
W
H 001111
111100
6X
111111
111111
Extend
Sum of pixels in a rectangle: rectangle sum
24×24: 160,000 features
112/04/19 6
Extended Haar-like features
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
R. Lienhart, J. Maydt, “An extended set of Haar-like features for rapid object detection,” ICIP Vol. 1, pp.I-900 - I-903, Sept. 2002.
Feng Tang, R. Crabb, Hai Tao, “Representing Images Using Nonorthogonal Haar-Like Bases,” IEEE PAMI. Intell. Vol. 29(12), pp. 2120 – 2134, Dec. 2007.
(a) (b) (c) (d)
(e) (f)
(a) (b)
1. Edge features
2. Line features
3. Center-surround features
(g) (h)
4. Box features
(a) (b)
5. Feature in [1]
(a) (b) (c) (d)
112/04/19 7
Introduction
Haar-like features and rectangle sum
Application
112/04/19 8
Haar-like features -- application Face recognition Template matching
Box features
(a) (b)
Feng Tang, R. Crabb, Hai Tao, “Representing Images Using Nonorthogonal Haar-Like Bases,” IEEE PAMI. Intell. Vol. 29(12), pp. 2120 – 2134, Dec. 2007.
F Tang, H Tao, Fast linear discriminant analysis using binary bases Volume 28, Issue 16, 1 December 2007, Pages 2209-2218
112/04/19 9
Haar-like features -- application
From Project of Prof. Ngan and Liu Qiang
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
W
H
W
H Face detection
112/04/19 10
Haar-like features -- application
From Cui Chunhui’s seminar
H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” In The 9th ECCV, 2006
Feature matching
112/04/19 11
Pattern matching
N
N
Image
pattern
Pattern matching seeks a given pattern within an image. For each pixel, the distance between window and pattern is calculated:
The smaller the distance is, the more similar the window is to the pattern. In practice:
• • • • • • • • • • • • •• • • • • • • • •
window
Application
TXXd wN
pN ),(
22
22||||..),,(
2222
wN
pN
wN
pN XXgeXXd
==> match!
pNX
2
wNX
2
Window matches the pattern or not
112/04/19 12
Pattern Matching application
Template matching is useful in signal processing, computer vision, image and video processing
For example, image based rendering, image compression, object detection, video compression, tracking, denoising, super resolution, texture synthesis, block matching in motion estimation, road/path tracking …
112/04/19 13
pattern matching
WHT has been successfully used for full search equivalent pattern matching.
We apply orthogonal Haar transform for full search equivalent pattern matching.
Y. Hel-Or and H. Hel-Or, “Real time pattern matching using projection kernels,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 9, pp. 1430-1445, Sept. 2005.
112/04/19 14
Outline Introduction
The proposed algorithm Experimental result Conclusion
112/04/19 15
Outline Introduction
The proposed algorithm Definition of rectangle sum and
integral image Strip sum for computing rectangle sum Orthogonal Haar transform
Experimental result Conclusion
112/04/19 16
Rectangle sum - definition A rectangle in the image
X(j1, j2) is specified by:
rect = (j1, j2, N1, N2, θ), (j1, j2): upper left position
(N1, N2): size of the rectangle. θ=0°: upright rectangle θ=45°: 45° rotated rectangle.
1 111
1
22
2),()(
Nj
jv
Nj
juvuXrectRectSum
N1
N2
N1N2
N1
N2
K1
K2
Image X(j1, j2)
Upright rect45° rotated rect
j1j2
N1N 2-1 additions
Related to dc component
1 111
1
22
2),(
1 Nj
jv
Nj
juvuX
Scale Scale
rectRectSum )(
3 additions
112/04/19 17
Integral image
1
0
1
0211 2 ),(),(j
u
j
vvuXjjI
(0,0)
j2
j1
The integral image I(j1, j2) is the sum of pixels above and to the left of (j1, j2)
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
• 1: A 2: A+B• 3: A+C 4: A+B+C+D • A-D: rectangle sum
F.C. Crow. Summed-Area Tables for Texture Mapping. in Proc.11th Ann. Conf. Computer Graphics and Interactive Techniques, pp. 207-212, 1984.
A B
C D1 2
3 4
(0,0)
A B
C D1 2
3 4
(0,0)
A B
C D1 2
3 4
(0,0)
A B
C D1 2
3 4
(0,0)
A B
C D1 2
3 4
(0,0)
112/04/19 18
Integral image method
1
0
1
0211 2 ),(),(j
u
j
vvuXjjI
The integral image I(j1, j2) is the sum of pixels above and to the left of (j1, j2)
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
• 1: A 2: A+B• 3: A+C 4: A+B+C+D
).,(),(),(),(
),()(
211221212211
1 111
1
22
2
jNjINjjIjjINjNjI
vuXrectRectSumNj
ju
Nj
jv
A B
C D1 2
3 4
j2
j1 N1
N2
A B
C D1 2
3 4
j2
j1 N1
N2
A B
C D1 2
3 4
j2
j1 N1
N2
4 1 3 2
3 additions
F.C. Crow. Summed-Area Tables for Texture Mapping. in Proc.11th Ann. Conf. Computer Graphics and Interactive Techniques, pp. 207-212, 1984.
(0,0)
j2
j1
•D = 4 + 1 − 3 − 2 = 4 − 2 − (3 − 1) • A-D: rectangle sum
(0, 0)
112/04/19 19
Outline Introduction
The proposed algorithm Definition of rectangle sum and integral
image Strip sum for computing rectangle
sum Orthogonal Haar transform
Experimental result Conclusion
112/04/19 20
The proposed strip sum
1
0
1
0211 2 ),(),(j
u
j
vvuXjjI
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
• D = 4 + 1 − 2 − 3• = 4 − 2 − (3 − 1)
A B
C D1 2
3 4
j2
j1 N1
N2
HStrip= (j1, j2, N2)
).,(),(
),(),(
),() , ,(
21221
1
0
1
0
1
0
1
0
1
0
1
221
1 21 22
1 22
2
jjINjjI
vuXvuX
vuXNjjHStripSum
j
u
j
v
j
u
Nj
v
j
u
Nj
jv
RectSum(j1, j2, N1, N2, 0°)= [I(j1+ N1, j2+N2) – I(j1+ N1, j2)] – [I(j1, j2+N2) – I(j1, j2)]= HStripSum(j1+N1, j2, N2) – HStripSum(j1, j2, N2)
N2
(0,0)
j2
j1
(j1, j2, N2) (j1+N1, j2, N2)
N1
RectSum ( j1, j2, N1, N2, 0°)
+- +HstripSum:
A B
C D1 2
3 4
j2
j1 N1
N2
(j1+N1, j2)
(j1, j2)
N2
112/04/19 21
HStripSum(j1, j2, N2) = I(j1, j2+N2) – I(j1, j2)
Constructing the strip sum
N2
(0,0)
j2
j1
N1'N1
RectSum(j1, j2, N1, N2, 0°)
RectSum(j1, j2, N1', N2, 0°)
1 addition
N2
(0,0)
j2
j1
(j1, j2, N2) (j1+N1, j2, N2)
N1
RectSum ( j1, j2, N1, N2, 0°)
+- +HstripSum:
Compute 2 RectSum:•2 adds for strip sum;•6 adds for integral image.
112/04/19 22
Other strips
N1
N2
N1N2
N1
N2
K1
K2
Image X(j1, j2)
Upright rect45° rotated rect
j1j2
N2
(0,0)
j2
j1
(j1, j2, N2) (j1+N1, j2, N2)
N1
RectSum ( j1, j2, N1, N2, 0°)
+- +HstripSum:
N2
(0,0)
j2
j1
(j1, j2, N1)
(j1, j2+N2, N1)
N1
RectSum
( j1 , j2 , N1 , N
2 , 0°)
+-
+
VstripSum:
112/04/19 23
Strip int.
Computational analysis
r rectangle sums of different sizes Box: 4r Integral image: 3r+2 Strip sum:
2 + Min{ Num(width), Num(height)} + r
These r rectangles have Num(width) different widths and Num(height) different heights.
Toy case Size 1×1 to N×N (1×1, 1×2, 2×1, 2×2, 1×3, 3×1, 2×3, … , N×N) r = N2, Num(width)=Num(height)=N. Box: 4N2 Integral image: 3N2 + 2 Strip sum: 2 + N + N2
r=30
M. J. McDonnell. Box-filtering techniques. Comput. Graph. Image Process., 17: 65–70, 1981.
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
Haar-like features
D = 4 + 1 − 2 − 3
112/04/19 24
Haar-like features
P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” CVPR pp. I: 511-I: 518, 2001.
R. Lienhart, J. Maydt, “An extended set of Haar-like features for rapid object detection,” ICIP Vol. 1, pp.I-900 - I-903, Sept. 2002.
Feng Tang, R. Crabb, Hai Tao, “Representing Images Using Nonorthogonal Haar-Like Bases,” IEEE PAMI. Intell. Vol. 29(12), pp. 2120 – 2134, Dec. 2007.
(a) (b) (c) (d)
(e) (f)
(a) (b)
1. Edge features
2. Line features
3. Center-surround features
(g) (h)
4. Box features
(a) (b)
5. Feature in [1]
(a) (b) (c) (d)
112/04/19 25
Computational analysis
(4b)(1a) (1b)
(2a)
(5)
(2b)
(2c)
(2d)
(2e)
(2f)
(2g)
(2h)
(1c) (1d)
1
3
7
# Ops
3
7
15
(4a)
Int. Features
11 5
12 6
# Rects
1
2
4
3
4
Strip
Feature(1a)=Rectsum1 – Rectsum2
112/04/19 26
Experiment – Face detection
Methods Integral Image Strip Sum
Time 23.6 Secs 20.2 Secs
H. Rowley, S. Baluja, and T. Kanade. Neural network-based face detection. IEEE Patt. Anal. Mach. Intell., Vol. 20, pp. 22-38, 1998.
OpenCV uses integral image method. In our implementation, we just replace the part that is used for computing the Haar like features.
CMU frontal face test set. All of the 130 images are used for face detection.
http://sourceforge.net/projects/opencvlibrary
86% execution time
R. Lienhart, J. Maydt, “An extended set of Haar-like features for rapid object detection,” ICIP Vol. 1, pp.I-900 - I-903, Sept. 2002.
112/04/19 27
Outline Introduction
The proposed algorithm Definition of rectangle sum and
integral image Strip sum for computing rectangle sum Orthogonal Haar transform
Experimental result Conclusion
112/04/19 28
The proposed orthogonal Haar transform OHT on sliding windows. The same Haar feature has
been considered as different basis for different window positions!
Basis 2 and 3 are the same Haar feature on different window positions.
2
1 -1 0
Values for basis
0
1
4
5
2
3
6
7
8
9
12
13
10
11
14
15
332
112/04/19 29
The proposed orthogonal Haar transform Basis 2 and 3 are the same
Haar feature on different window positions.
Similarly for 4-7. Similarly for basis 8-15.
1 -1 0
Values for basis
0
1
4
5
2
3
6
7
8
9
12
13
10
11
14
15
Compute u basis: u=2: 2 Haar features. u=4: 3 Haar features. u=8: 4 Haar features. u=16: 5 Haar features. u: 1+log2u Haar features.4 + 5log4 u additions for computing u OHT 2-D basis.
112/04/19 30
OHT and WHT- example
4x4 OHT 4x4 WHT
112/04/19 31
OHT and WHT - Energy compaction ability
0 100 200 3000.2
0.4
0.6
0.8
1
% o
f energ
y e
xetr
acte
d
Number of bases(a)
0 200 400 6000.2
0.4
0.6
0.8
1
% o
f energ
y e
xetr
acte
dNumber of operations per pixel
(b)
WHT
OHT
X: number of Basis Y: Energy extracted
X: number of operations Y: Energy extracted
Y. Hel-Or and H. Hel-Or. Real time pattern matching using projection kernels. TPAMI, 27(9):1430–1445, Sept. 2005.G. Ben-Artz, H. Hel-Or, and Y. Hel-Or. The Gray-code filter kernels. TPAMI, 29(3):382– 393, Mar. 2007. W. Ouyang and W. Cham. Fast algorithm for Walsh Hadamard transform on sliding windows. TPAMI, 32(1):165–171, Jan. 2010.
112/04/19 32
Outline Introduction The proposed algorithm
Experimental result Conclusion
112/04/19 33
Datasets (1) 120 images selected from 3
different databases. MIT, medical and remote sensing.
http://people.csail.mit.edu/torralba/images. http://zulu.ssc.nasa.gov/mrsid. www.data-compression.info/corpora/lukascorpus
F. Tombari etc. Full search-equivalent pattern matching with incremental dissimilarity approximations. IEEE TPAMI, 31(1):129–141, Jan.2009. 1,8, 9
112/04/19 34
Datasets (2)
Noises: N1, N2, N3 and N4, Variances: 100, 200, 400 and 800 PSNR: 28.1, 25.1, 22.1 and 19.2
N1 N2 N3 N4
112/04/19 35
1. Evaluating algorithms on different sizes Speedup over full search
(FS) in pattern matching
F. Tombari etc. Full search-equivalent pattern matching with incremental dissimilarity approximations. IEEE TPAMI, 31(1):129–141, Jan.2009. 1,8, 9
Y. Hel-Or and H. Hel-Or, “Real time pattern matching using projection kernels,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 9, pp. 1430-1445, Sept. 2005.
G. Ben-Artzi, H. Hel-Or, and Y. Hel-Or, “The gray-code filter kernels,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp.382 – 393, Mar. 2007
TimeFS/ TimeWHT
TimeFS/ TimeGCK
TimeFS/ TimeIDA
500s/1s = 500
OHTI: OHT using integral image
OHTs: OHT using strip sum
112/04/19 36
2. Evaluating algorithms on different pattern sizes and different noise levels Speedup over FS.
F. Tombari etc. Full search-equivalent pattern matching with incremental dissimilarity approximations. IEEE TPAMI, 31(1):129–141, Jan.2009. 1,8, 9
Y. Hel-Or and H. Hel-Or, “Real time pattern matching using projection kernels,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 9, pp. 1430-1445, Sept. 2005.
G. Ben-Artzi, H. Hel-Or, and Y. Hel-Or, “The gray-code filter kernels,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp.382 – 393, Mar. 2007
Time??/ TimeOHT
112/04/19 37
2. Evaluating algorithms on different pattern sizes and different noise levels (2) At about 4-15 times the speed of IDA At about 8-10 times the speed of GCK
F. Tombari etc. Full search-equivalent pattern matching with incremental dissimilarity approximations. IEEE TPAMI, 31(1):129–141, Jan.2009. 1,8, 9G. Ben-Artzi, H. Hel-Or, and Y. Hel-Or, “The gray-code filter kernels,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp.382 – 393, Mar. 2007
112/04/19 38
Outline Introduction The proposed algorithm Experimental result
Conclusion
112/04/19 39
Conclusion A data structure (strip sum) that computes sum of pixe
ls in a rectangle by 1 addition. A transform (orthogonal Haar transform) that requires
O(logu) additions per pixel to project N1xN2 input window onto u basis vectors.
Pattern matching using OHT Find the same result as Full Search (FS).
Experimental results show that it can achieve up to 10 times speed-up over GCK in pattern matching.
112/04/19 40
Thanks !