online learning for matrix factorization and sparse coding julien mairal, francis bach, jean ponce...
TRANSCRIPT
![Page 1: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/1.jpg)
Online Learning for Matrix Factorization and Sparse Coding
Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro
Journal of Machine Learning Research 2010
![Page 2: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/2.jpg)
Introduction
• This paper focuses on the large scale matrix factorization problem, including– Dictionary learning for sparse coding– Non-negative matrix factorization (NMF)– Sparse principal component analysis (SPCA)
• Contributions of this paper:– An iterative online algorithm is proposed for large scale matrix
factorization– This algorithm is proved to converge almost surely to a stationary
point of the objective function– This algorithm is shown to be much faster than previous methods
in the experiment.
![Page 3: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/3.jpg)
Problem Statement
• Classical dictionary learning problem Given a finite training set , the objective is
to optimize the following function
where
• Online Learning
This algorithm process one sample (or a mini-batch) at a time and sequentially minimize the following function:
![Page 4: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/4.jpg)
Basic Algorithm
![Page 5: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/5.jpg)
Dictionary Update
![Page 6: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/6.jpg)
Optimizing the Algorithm
• Handling fixed-sized data sets
• Scaling the “past” data
• Mini-batch extension
![Page 7: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/7.jpg)
Proof of Convergence
• Assumptions:
• Main results
![Page 8: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/8.jpg)
Extensions to Matrix Factorization
• Non-negative matrix factorization (NMF)
• Non-negative sparse coding (NNSC)
• Sparse principal component analysis (SPCA)
![Page 9: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/9.jpg)
Data for Experiment
• 1.25 million patches from Pascal VOC’06 image database
![Page 10: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/10.jpg)
Online VS. Batch
• Training data size: 1 million• OL1:• OL2:• OL3:
![Page 11: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/11.jpg)
Comparison with NMF and NNSC• NMF
• NNSC
![Page 12: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/12.jpg)
Face Results
![Page 13: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/13.jpg)
Image Patches Results
![Page 14: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/14.jpg)
Inpainting Results
• Image size: 12-Megapixel• Dictionary with 256 elements• Training data: 7 million 12 by 12 color patches
![Page 15: Online Learning for Matrix Factorization and Sparse Coding Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro Journal of Machine Learning Research](https://reader031.vdocuments.net/reader031/viewer/2022020417/56649e4e5503460f94b44d25/html5/thumbnails/15.jpg)
Conclusion
• A new online algorithm for learning dictionaries adapted to sparse coding tasks, and proven its convergence.
• Experiments demonstrate that this algorithm is significantly faster than existing batch methods.
• This algorithm can be extended to other matrix factorization problems such as non-negative matrix factorization and sparse principal component analysis.