![Page 1: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/1.jpg)
Higher-order Statistical Modeling based
Deep CNNs (Introduction) Peihua Li
Dalian University of Technology http://peihuali.org
![Page 2: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/2.jpg)
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
![Page 3: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/3.jpg)
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment
, ( )XX f x
![Page 4: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/4.jpg)
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment
, ( )XX f x
2 21( ) ii
E X xN
= ∑
1( ) ii
E X xN
= ∑i.i.d. samples
![Page 5: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/5.jpg)
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2 21( ) i
i
E X xN
= ∑2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment 1( ) i
i
E X xN
= ∑i.i.d. samples
, ( )XX f x
3rd-order moment
![Page 6: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/6.jpg)
For scalar random variable is its proability density
What is Higher-order?─Statistical Moments
2 2( ) ( )XE X x f x dx= ∫2 21( ) i
i
E X xN
= ∑2nd-order moment
( ) ( )k kXE X x f x dx= ∫
1( )k ki
i
E X xN
= ∑kth-order moment
( ) ( )XE X xf x dx= ∫1st-order moment 1( ) i
i
E X xN
= ∑i.i.d. samples
, ( )XX f x
4th-order moment
![Page 7: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/7.jpg)
1st-order moment
1( )E XN ∈
= ∑x
x
p∈Rx
3rd-order moment
3 1( )E XN ∈
= ⊗ ⊗∑x
x x x
x
x
x3pR
2nd-order moment
2 1( )
1
TE XN
N
∈
∈
=
= ⊗
∑
∑x
x
xx
x x
x
x
2pR
What is Higher-order?─Statistical Moments Random vector
Images courtesy of “Kernel Pooling for Convolutional Neural Networks”
![Page 8: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/8.jpg)
What is Higher-order?─Statistical Moments
( )Xf x
Probaiblity density is everything ( )Xf xRandom vector
![Page 9: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/9.jpg)
What is Higher-order?─Statistical Moments
( )( ) ( )jX jxX XE e f x e dxω ωω
+∞
−∞Φ = = ∫ ( )Xf x
Characteristic Function=Probability Density
The characteristic function is defined as
1( ) ( )2
jxX Xf x e dxωω
π+∞ −
−∞= Φ∫
Fouriere Transform Pair
Random vector
![Page 10: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/10.jpg)
What is Higher-order?─Statistical Moments
( )( ) ( )jX jxX XE e f x e dxω ωω
+∞
−∞Φ = = ∫ ( )Xf x
Characteristic Function=Probaiblity Density
The characteristic function is defined as
1( ) ( )2
jxX Xf x e dxωω
π+∞ −
−∞= Φ∫
Fouriere Transform Pair
Random vector
Moments matter ( )( ) jXX E e ωωΦ =
( )Xf x
Fouriere Transform Pair
0 0
22 2
( ) ( )! !
1 ( ) ( ) ( ) .2! !
k kk k
k k
kk k
j X jE E Xk k
j jjE X E X E Xk
ω ω
ω ω ω
∞ ∞
= =
= =
= + + + + +
∑ ∑
If we know characteristic function , we know everything. ( )X ωΦ
![Page 11: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/11.jpg)
What is Higher-order?─Statistical Moments
( )( ) ( )jX jxX XE e f x e dxω ωω
+∞
−∞Φ = = ∫ ( )Xf x
Characteristic Function=Probability Density
The characteristic function is defined as
1( ) ( )2
jxX Xf x e dxωω
π+∞ −
−∞= Φ∫
Fouriere Transform Pair
Random vector If we know characteristic function , we know everything.
0
( ) ( )!
kk k
Xk
j E Xk
ω ω∞
=
Φ =∑( )Xf xFouriere Transform Probability density Characteristic function
1st-order moment 1( )E XN ∈
= ∑x
x
2 1 1( ) TE XN N∈ ∈
= = ⊗∑ ∑x x
xx x x
2nd-order moment
3rd-order moment 3 1( )E XN ∈
= ⊗ ⊗∑x
x x x
( )X ωΦMoments matter
![Page 12: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/12.jpg)
What is Higher-order?─Signal Perspective
Convolution is a linear transformation
= +y b Wxx y
( )= + + ⊗ +y b Wx H x x
Multi-variable Taylor series:
1st-order term (Linear term) ( )f=y x
![Page 13: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/13.jpg)
What is Higher-order?─Signal Perspective
Convolution is a linear transformation
[ ]( , ) (0,0) T u uf u v f u v
v v
= + + +
w H
Two variable Taylor series:
1 2w u w v+ 2 211 12 222h u h uv h v+ +
Linear term
![Page 14: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/14.jpg)
What is Higher-order?─Signal Perspective
Convolution is a linear transformation
[ ]( , ) (0,0) T u uf u v f u v
v v
= + + +
w H
Two variable Taylor series:
1 2w u w v+ 2 211 12 222h u h uv h v+ +
Linear term
Higher order enhances non-linear modeling capability
![Page 15: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/15.jpg)
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
![Page 16: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/16.jpg)
Why Higher-order? Hand-crafted Features
Learned Features
0
( ) ( )!
kk k
Xk
j E Xk
ω ω∞
=
Φ =∑( )Xf xProbability density Characteristic function
![Page 17: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/17.jpg)
Why Higher-order? Hand-crafted Features
Learned Features
0
( ) ( )!
kk k
Xk
j E Xk
ω ω∞
=
Φ =∑( )Xf xProbability density Characteristic function
Higher-order moments can better characterize real-word distributions
![Page 18: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/18.jpg)
Global average pooling
Why Higher-order?
1 1 1
1
1
cov( , ) cov( , ) cov( , ) cov( , ) cov( , ) cov( , )
j
i i j
p p j
x x x x
x x x x
x x x x
channel correlation
width
height
p channel
X
jxix
2 1( ) TE XN ∈
= ∑x
xx
2nd-order moment
![Page 19: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/19.jpg)
CNN
What does each channel indicate?
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
width
height
p channel
Why Higher-order?
Body | channel 452
Head | channel 123
Hind claw | channel 448
Legs | channel 99
Tail | channel 174
Front claw | channel 333
![Page 20: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/20.jpg)
CNN
What does each channel indicate?
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
width
height
p channel
Why Higher-order?
Body | channel 452
Head | channel 123
Hind claw | channel 448
Legs | channel 99
Tail | channel 174
Front claw | channel 333
Physical Interpretation For object recogniton, 2nd-order moment capture dependency of different parts
Context of the object
![Page 21: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/21.jpg)
Why Higher-order?
What does each channel indicate?
width
height
p channel
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
Bookcase | channel 97
Drawer | channel 360
Carpet | channel 260
Plant | channel 384
Decoration | channel 44
Floor | channel 459
CNN
![Page 22: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/22.jpg)
Why Higher-order?
What does each channel indicate?
width
height
p channel
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
Bookcase | channel 97
Drawer | channel 360
Carpet | channel 260
Plant | channel 384
Decoration | channel 44
Floor | channel 459
Physical Interpretation For scene images, 2nd-order moment capture dependency of different objects
Context of the scene
![Page 23: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/23.jpg)
Why Higher-order?
What does each channel indicate?
width
height
p channel
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.
Bookcase | channel 97
Drawer | channel 360
Carpet | channel 260
Plant | channel 384
Decoration | channel 44
Floor | channel 459
3rd-order moment or direct distribution
![Page 24: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/24.jpg)
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
![Page 25: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/25.jpg)
Overview of Speaker Wangmeng Zuo received the Ph.D. degree in computer application technology from the Harbin Institute of Technology, Harbin, China, in 2007. He is currently a Professor in the School of Computer Science and Technology, Harbin Institute of Technology. His current research interests include image enhancement and restoration, object detection, visual tracking, and image classification. He has published over 70 papers in toptier academic journals and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal of Electronic Imaging, and the Guest Editor of Neurocomputing, Pattern Recognition, IEEE Transactions on Circuits and Systems for Video Technology, and IEEE Transactions on Neural Networks and Learning Systems.
![Page 26: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/26.jpg)
Overview of Speaker Qilong Wang received the Ph.D. Degree in the School of Information and Communication Engineering, Dalian University of Technology in 2018. He is currently a lecturer in the College of Intelligence and Computing, Tianjin University. His research interests include visual classification and deep probability distribution modeling. He has published several papers in top conferences and referred journals including ICCV, CVPR, ECCV, NIPS, IJCAI, TPAMI, TIP and TCSVT.
![Page 27: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/27.jpg)
Overview of Speaker Peihua Li is a professor of Dalian University of Technology. He received Ph.D degree from Harbin Institute of Technology in 2003, and then worked as a postdoctoral fellow at INRIA/IRISA, France. He achieved the honorary nomination of National Excellent Doctoral dissertation in China. He was supported by Program for New Century Excellent Talents in University of Chinese Ministry of Education. His team won 1st place in large-scale iNaturalist Challenge spanning 8000 species at FGVC5 CVPR2018, 2nd place in Alibaba Large-scale Image Search Challenge 2015. His research topics include deep learning and computer vision, focusing on image/video recognition, object detection and semantic segmentation. He has published papers in top journals such as IEEE TPAMI/TIP/TCSVT and top conferences including ICCV/CVPR/ECCV/NIPS. As a principal investigator, he receives funds from National Natural Sceince Foundation of China (NSFC), Chinese Ministry of Education and Huawei Technologies Co., Ltd.
http://peihuali.org/
![Page 28: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/28.jpg)
Outline
What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial
![Page 29: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/29.jpg)
Overview of Tutorial─Part 1
Higher-order:
Higher-order:
![Page 30: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/30.jpg)
Overview of Tutorial─Part 2
![Page 31: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/31.jpg)
Overview of Tutorial─Part 3
![Page 32: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/32.jpg)
Overview of Tutorial─Part 4
![Page 33: Higher-order Statistical Modeling based Deep CNNs ... · and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal](https://reader036.vdocuments.net/reader036/viewer/2022071001/5fbd402b5d3ee872e72f90ba/html5/thumbnails/33.jpg)