higher-order statistical modeling based deep cnns ... · and conferences. he has served as a...

Higher-order Statistical Modeling based

Deep CNNs (Introduction) Peihua Li

Dalian University of Technology http://peihuali.org

Outline

What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial

For scalar random variable is its proability density

What is Higher-order?─Statistical Moments

2 2( ) ( )XE X x f x dx= ∫2nd-order moment

( ) ( )k kXE X x f x dx= ∫

1( )k ki

i

E X xN

= ∑kth-order moment

( ) ( )XE X xf x dx= ∫1st-order moment

, ( )XX f x

For scalar random variable is its proability density

What is Higher-order?─Statistical Moments

2 2( ) ( )XE X x f x dx= ∫2nd-order moment

( ) ( )k kXE X x f x dx= ∫

1( )k ki

i

E X xN

= ∑kth-order moment

( ) ( )XE X xf x dx= ∫1st-order moment

, ( )XX f x

2 21( ) ii

E X xN

= ∑

1( ) ii

E X xN

= ∑i.i.d. samples

For scalar random variable is its proability density

What is Higher-order?─Statistical Moments

2 2( ) ( )XE X x f x dx= ∫2 21( ) i

i

E X xN

= ∑2nd-order moment

( ) ( )k kXE X x f x dx= ∫

1( )k ki

i

E X xN

= ∑kth-order moment

( ) ( )XE X xf x dx= ∫1st-order moment 1( ) i

i

E X xN

= ∑i.i.d. samples

, ( )XX f x

3rd-order moment

For scalar random variable is its proability density

What is Higher-order?─Statistical Moments

2 2( ) ( )XE X x f x dx= ∫2 21( ) i

i

E X xN

= ∑2nd-order moment

( ) ( )k kXE X x f x dx= ∫

1( )k ki

i

E X xN

= ∑kth-order moment

( ) ( )XE X xf x dx= ∫1st-order moment 1( ) i

i

E X xN

= ∑i.i.d. samples

, ( )XX f x

4th-order moment

1st-order moment

1( )E XN ∈

= ∑x

x

p∈Rx

3rd-order moment

3 1( )E XN ∈

= ⊗ ⊗∑x

x x x

x

x

x3pR

2nd-order moment

2 1( )

1

TE XN

N

∈

∈

=

= ⊗

∑

∑x

x

xx

x x

x

x

2pR

What is Higher-order?─Statistical Moments Random vector

Images courtesy of “Kernel Pooling for Convolutional Neural Networks”

What is Higher-order?─Statistical Moments

( )Xf x

Probaiblity density is everything ( )Xf xRandom vector

What is Higher-order?─Statistical Moments

( )( ) ( )jX jxX XE e f x e dxω ωω

+∞

−∞Φ = = ∫ ( )Xf x

Characteristic Function=Probability Density

The characteristic function is defined as

1( ) ( )2

jxX Xf x e dxωω

π+∞ −

−∞= Φ∫

Fouriere Transform Pair

Random vector

What is Higher-order?─Statistical Moments

( )( ) ( )jX jxX XE e f x e dxω ωω

+∞

−∞Φ = = ∫ ( )Xf x

Characteristic Function=Probaiblity Density

The characteristic function is defined as

1( ) ( )2

jxX Xf x e dxωω

π+∞ −

−∞= Φ∫

Fouriere Transform Pair

Random vector

Moments matter ( )( ) jXX E e ωωΦ =

( )Xf x

Fouriere Transform Pair

0 0

22 2

( ) ( )! !

1 ( ) ( ) ( ) .2! !

k kk k

k k

kk k

j X jE E Xk k

j jjE X E X E Xk

ω ω

ω ω ω

∞ ∞

= =

= =

= + + + + +

∑ ∑

If we know characteristic function , we know everything. ( )X ωΦ

What is Higher-order?─Statistical Moments

( )( ) ( )jX jxX XE e f x e dxω ωω

+∞

−∞Φ = = ∫ ( )Xf x

Characteristic Function=Probability Density

The characteristic function is defined as

1( ) ( )2

jxX Xf x e dxωω

π+∞ −

−∞= Φ∫

Fouriere Transform Pair

Random vector If we know characteristic function , we know everything.

0

( ) ( )!

kk k

Xk

j E Xk

ω ω∞

=

Φ =∑( )Xf xFouriere Transform Probability density Characteristic function

1st-order moment 1( )E XN ∈

= ∑x

x

2 1 1( ) TE XN N∈ ∈

= = ⊗∑ ∑x x

xx x x

2nd-order moment

3rd-order moment 3 1( )E XN ∈

= ⊗ ⊗∑x

x x x

( )X ωΦMoments matter

What is Higher-order?─Signal Perspective

Convolution is a linear transformation

= +y b Wxx y

( )= + + ⊗ +y b Wx H x x

Multi-variable Taylor series:

1st-order term (Linear term) ( )f=y x

What is Higher-order?─Signal Perspective

Convolution is a linear transformation

[ ]( , ) (0,0) T u uf u v f u v

v v

= + + +

w H

Two variable Taylor series:

1 2w u w v+ 2 211 12 222h u h uv h v+ +

Linear term

What is Higher-order?─Signal Perspective

Convolution is a linear transformation

[ ]( , ) (0,0) T u uf u v f u v

v v

= + + +

w H

Two variable Taylor series:

1 2w u w v+ 2 211 12 222h u h uv h v+ +

Linear term

Higher order enhances non-linear modeling capability

Outline

What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial

Why Higher-order? Hand-crafted Features

Learned Features

0

( ) ( )!

kk k

Xk

j E Xk

ω ω∞

=

Φ =∑( )Xf xProbability density Characteristic function

Why Higher-order? Hand-crafted Features

Learned Features

0

( ) ( )!

kk k

Xk

j E Xk

ω ω∞

=

Φ =∑( )Xf xProbability density Characteristic function

Higher-order moments can better characterize real-word distributions

Global average pooling

Why Higher-order?

1 1 1

1

1

cov( , ) cov( , ) cov( , ) cov( , ) cov( , ) cov( , )

j

i i j

p p j

x x x x

x x x x

x x x x

channel correlation

width

height

p channel

X

jxix

2 1( ) TE XN ∈

= ∑x

xx

2nd-order moment

CNN

What does each channel indicate?

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.

width

height

p channel

Why Higher-order?

Body | channel 452

Head | channel 123

Hind claw | channel 448

Legs | channel 99

Tail | channel 174

Front claw | channel 333

CNN

What does each channel indicate?

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.

width

height

p channel

Why Higher-order?

Body | channel 452

Head | channel 123

Hind claw | channel 448

Legs | channel 99

Tail | channel 174

Front claw | channel 333

Physical Interpretation For object recogniton, 2nd-order moment capture dependency of different parts

Context of the object

Why Higher-order?

What does each channel indicate?

width

height

p channel

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.

Bookcase | channel 97

Drawer | channel 360

Carpet | channel 260

Plant | channel 384

Decoration | channel 44

Floor | channel 459

CNN

Why Higher-order?

What does each channel indicate?

width

height

p channel

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.

Bookcase | channel 97

Drawer | channel 360

Carpet | channel 260

Plant | channel 384

Decoration | channel 44

Floor | channel 459

Physical Interpretation For scene images, 2nd-order moment capture dependency of different objects

Context of the scene

Why Higher-order?

What does each channel indicate?

width

height

p channel

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. Computer Vision and Pattern Recognition (CVPR), 2016.

Bookcase | channel 97

Drawer | channel 360

Carpet | channel 260

Plant | channel 384

Decoration | channel 44

Floor | channel 459

3rd-order moment or direct distribution

Outline

What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial

Overview of Speaker Wangmeng Zuo received the Ph.D. degree in computer application technology from the Harbin Institute of Technology, Harbin, China, in 2007. He is currently a Professor in the School of Computer Science and Technology, Harbin Institute of Technology. His current research interests include image enhancement and restoration, object detection, visual tracking, and image classification. He has published over 70 papers in toptier academic journals and conferences. He has served as a Tutorial Organizer in ECCV 2016, an Associate Editor of the IET Biometrics and Journal of Electronic Imaging, and the Guest Editor of Neurocomputing, Pattern Recognition, IEEE Transactions on Circuits and Systems for Video Technology, and IEEE Transactions on Neural Networks and Learning Systems.

Overview of Speaker Qilong Wang received the Ph.D. Degree in the School of Information and Communication Engineering, Dalian University of Technology in 2018. He is currently a lecturer in the College of Intelligence and Computing, Tianjin University. His research interests include visual classification and deep probability distribution modeling. He has published several papers in top conferences and referred journals including ICCV, CVPR, ECCV, NIPS, IJCAI, TPAMI, TIP and TCSVT.

Overview of Speaker Peihua Li is a professor of Dalian University of Technology. He received Ph.D degree from Harbin Institute of Technology in 2003, and then worked as a postdoctoral fellow at INRIA/IRISA, France. He achieved the honorary nomination of National Excellent Doctoral dissertation in China. He was supported by Program for New Century Excellent Talents in University of Chinese Ministry of Education. His team won 1st place in large-scale iNaturalist Challenge spanning 8000 species at FGVC5 CVPR2018, 2nd place in Alibaba Large-scale Image Search Challenge 2015. His research topics include deep learning and computer vision, focusing on image/video recognition, object detection and semantic segmentation. He has published papers in top journals such as IEEE TPAMI/TIP/TCSVT and top conferences including ICCV/CVPR/ECCV/NIPS. As a principal investigator, he receives funds from National Natural Sceince Foundation of China (NSFC), Chinese Ministry of Education and Huawei Technologies Co., Ltd.

http://peihuali.org/

Outline

What is Higher-order? Why We Study High-order Overview of Speaker Overview of Tutorial

Overview of Tutorial─Part 1

Higher-order:

Higher-order:

Overview of Tutorial─Part 2

Overview of Tutorial─Part 3

Overview of Tutorial─Part 4