tutorial on sparse coding - pami.sjtu.edu.cn... “online dictionary learning for sparse coding,”...
TRANSCRIPT
Tutorial on Sparse Coding
Long Xu
PAMI Lab, SJTU
2014.04.21
Sparse Representation
Sparse Approximation Methods
Dictionary Learning
Applications
Sparse Representation
Olshausen B A, Field D J. Sparse coding with an overcomplete basis set: A strategy
employed by V1?[J]. Vision research, 1997, 37(23): 3311-3325.
Theoretical studies suggest that primary visual cortex (area V1) uses a sparse
code to efficiently represent natural scenes.
Sparse Representation Target
Neuron
activate
Input Signal
Sparse Representation
Sparse Representation Target (Theoretically)
0min . . xs t D‖ ‖0P
1,P 1 .in . xm s t D‖ ‖ ‖ ‖
1min . . xs t D‖ ‖1P
P 12min x
D‖ ‖ +‖ ‖
Sparse Representation
Sparse Representation Target (LASSO)
1min x
D‖ ‖+ ‖ ‖
data fitting term sparsity-inducing regularization
Sparse Representation
Why does the ℓ1-norm induce sparsity?
Physical illustration
Why does the ℓ1-norm induce sparsity?
Physical illustration
Why does the ℓ1-norm induce sparsity?
The geometric explanation
Sparse Approximation Methods
Basis Pursuit
• Linear programming
Matching Pursuit
• MP/OMP
FISTA
• fast iterative shrinkage-thresholding algorithm
LARS + Homotopy
……
Chen, S. S., Donoho, D. L., & Saunders, M. A. (1998). Atomic decomposition by basis
pursuit. SIAM journal on scientific computing, 20(1), 33-61.
Beck, A., & Teboulle, M. (2009). A fast iterative shrinkage-thresholding algorithm for linear inverse
problems. SIAM Journal on Imaging Sciences,2(1), 183-202.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso.Journal of the Royal
Statistical Society. Series B (Methodological), 267-288.
Sparse Approximation Methods
Performance comparison：
Dictionary Learning
How can we get the dictionary D?
1. Predefined dictionary based on various types of wavelets
2. Pre-trained dictionary using K-Means
3. K-SVD
4. Dictionary Learning (much better results in practice)
……
Dictionary Learning
D is an overcomplete basis set (i.e. M > k)
Dictionary learning problem statement:
Dictionary Learning
Aharon M, Elad M, Bruckstein A. K-SVD: An algorithm for designing overcomplete dictionaries for sparse
representation[J]. Signal Processing, IEEE Transactions on, 2006, 54(11): 4311-4322.
Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007). Efficient sparse coding algorithms. Advances in neural
information processing systems, 19, 801.
J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online dictionary learning for sparse coding,” in Proceedings
of the 26th ICML. ACM, 2009, pp. 689–696.
Algorithms based on iterative batch procedure
iteratively solving two convex optimization problems:
• L1-regularized least squares problem
• L2-constrained least squares problem
Online dictionary learning
• Suitable for large training set
Applications
Face Recognition
Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face
recognition via sparse representation. Pattern Analysis and Machine Intelligence, IEEE
Transactions on, 31(2), 210-227.
Applications
Image Classification
Yang J, Yu K, Huang T. Supervised translation-invariant sparse coding[C]//Computer Vision
and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010: 3517-3524.
Applications
Saliency Detection
Li, Yin, et al. "Incremental sparse saliency detection." Image Processing (ICIP), 2009
16th IEEE International Conference on. IEEE, 2009.
Applications
Tracking
Zhong W, Lu H, Yang M H. Robust object tracking via sparsity-based collaborative
model[C]//Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on.
IEEE, 2012: 1838-1845.
Applications
Cong Y, Yuan J, Liu J. Sparse reconstruction cost for abnormal event detection[C]//Computer
Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011: 3449-3456.
Abnormal event detection
“Sparse coding is not a general principle for finding statistically
indipendent components in data, it only applies if the data acturally
have sparse structure!”
Matlab Codes
SPAMS:
http://spams-devel.gforge.inria.fr/faq.html
Dictionary Learning Tools for Matlab:
http://www.ux.uis.no/~karlsk/dle/index.html#sec3
Thank You!