![Page 1: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/1.jpg)
Bayesian Methods in Machine Learning
Dmitry Vetrov
Research professor at faculty of Compter Sciences HSE
Head of Bayesian methods research group
http://bayesgroup.ru
![Page 2: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/2.jpg)
Outline
• Intro to mathematics of big data
• Bayesian framework
• Latent variable models
• Deep Bayes
![Page 3: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/3.jpg)
What is machine learning?
![Page 4: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/4.jpg)
Simple example
![Page 5: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/5.jpg)
Areas of application
![Page 6: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/6.jpg)
Milestones
![Page 7: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/7.jpg)
Entering the Age of Big Data
Data
Computationalspeed
![Page 8: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/8.jpg)
First Steps towards Mathematics of Big Data
![Page 9: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/9.jpg)
Deep learning: why now?
![Page 10: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/10.jpg)
Conditional and marginal distributions
![Page 11: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/11.jpg)
Bayesian framework
![Page 12: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/12.jpg)
Frequentist vs. Bayesian frameworks
![Page 13: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/13.jpg)
Bayesian inference
![Page 14: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/14.jpg)
Bayesian inference
![Page 15: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/15.jpg)
Bayesian inference
![Page 16: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/16.jpg)
Bayesian inference
![Page 17: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/17.jpg)
Bayesian Learning and Inference
![Page 18: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/18.jpg)
Poor man’s Bayes
![Page 19: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/19.jpg)
Regularization
Prior term
![Page 20: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/20.jpg)
Learning from incomplete data
![Page 21: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/21.jpg)
Advantages of Bayesian framework
• Regularization• Incorporates specifics of particular problem
• Extendibility• Builds complex model from simpler ones
• Latent variable modeling• Learns from incomplete data
• Ensembling• Performs weighted voting across multiple algorithms
• Scalability (new!)• Applicable to large datasets when combined with deep neural
networks
![Page 22: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/22.jpg)
Exponential class of distributions
![Page 23: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/23.jpg)
Log-concavity of exponential class
![Page 24: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/24.jpg)
Log-concavity of exponential class
![Page 25: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/25.jpg)
Example: Gaussian distribution
![Page 26: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/26.jpg)
Incomplete likelihood
![Page 27: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/27.jpg)
Variational lower bound
![Page 28: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/28.jpg)
EM-algorithm
![Page 29: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/29.jpg)
EM-algorithm
![Page 30: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/30.jpg)
EM-algorithm
![Page 31: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/31.jpg)
EM-algorithm
![Page 32: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/32.jpg)
EM-algorithm
![Page 33: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/33.jpg)
EM-algorithm
![Page 34: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/34.jpg)
EM-algorithm
![Page 35: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/35.jpg)
Discrete T
![Page 36: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/36.jpg)
Mixture of gaussians
![Page 37: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/37.jpg)
Mixture of gaussians
![Page 38: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/38.jpg)
Mixture of gaussians
![Page 39: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/39.jpg)
Mixture of gaussians
![Page 40: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/40.jpg)
Mixture of gaussians
![Page 41: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/41.jpg)
Mixture of gaussians
![Page 42: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/42.jpg)
Mixture of gaussians: formal description
![Page 43: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/43.jpg)
EM-algorithm for mixture of gaussians
![Page 44: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/44.jpg)
Continuous T
![Page 45: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/45.jpg)
Example: PCA model
![Page 46: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/46.jpg)
Advantages of EM PCA
![Page 47: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/47.jpg)
Mixture of PCA
![Page 48: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/48.jpg)
Example: Latent Dirichlet Allocation
![Page 49: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/49.jpg)
LDA: formal description
![Page 50: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/50.jpg)
General nature of EM-framework
![Page 51: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/51.jpg)
Variational inference: way to complex Bayesian models
![Page 52: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/52.jpg)
Variational inference: way to complex Bayesian models
The most difficult part
![Page 53: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/53.jpg)
Variational inference: way to complex Bayesian models
The most difficult part
![Page 54: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/54.jpg)
Dropout
![Page 55: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/55.jpg)
Variational Dropout
Diederik P. Kingma, Tim Salimans, Max Welling. Variational Dropout and the Local Reparameterization Trick. arXiv:1506.02557
![Page 56: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/56.jpg)
Variational Dropout
![Page 57: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/57.jpg)
Adjusting dropout rates
![Page 58: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/58.jpg)
Individual dropout rate
![Page 59: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/59.jpg)
Alternative view on dropout
![Page 60: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/60.jpg)
Alternative view on dropout
Data term Regularization
![Page 61: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/61.jpg)
Alternative view on dropout
. . .Deterministic
output
Gaussiannoise
Noisyoutput
![Page 62: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/62.jpg)
Sparsifying deep NNs
99.2% of zeros
![Page 63: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/63.jpg)
Sparsifying deep NNs
![Page 64: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/64.jpg)
Summer school
![Page 65: Bayesian Methods in Machine Learning”митрий Ветров... · Bayesian Methods in Machine Learning Dmitry Vetrov Research professor at faculty of Compter Sciences HSE Head](https://reader034.vdocuments.net/reader034/viewer/2022051815/6040fa5cf2016d490d30f8e5/html5/thumbnails/65.jpg)
Conclusion
• Bayesian framework is extremely powerful and extends ML tools
• We do have scalable algorithms for approximate Bayesian inference
• Bayes + Deep Learning =
• Even the first attempts of neurobayesian inference give impressive results