animashree anandkumar, electrical engineering and cs dept, uc irvine at mlconf sf - 11/13/15
TRANSCRIPT
Beating the Perils of Non-Convexity:Machine Learning using Tensor Methods
Anima Anandkumar
U.C. Irvine
Learning with Big Data
Learning is finding needle in a haystack
Learning with Big Data
Learning is finding needle in a haystack
High dimensional regime: as data grows, more variables!
Useful information: low-dimensional structures.
Learning with big data: ill-posed problem.
Learning with Big Data
Learning is finding needle in a haystack
High dimensional regime: as data grows, more variables!
Useful information: low-dimensional structures.
Learning with big data: ill-posed problem.
Learning with big data: statistically and computationally challenging!
Optimization for Learning
Most learning problems can be cast as optimization.
Unsupervised Learning
Clusteringk-means, hierarchical . . .
Maximum Likelihood EstimatorProbabilistic latent variable models
Supervised Learning
Optimizing a neural network withrespect to a loss function
Input
Neuron
Output
Convex vs. Non-convex Optimization
Progress is only tip of the iceberg..
Images taken from https://www.facebook.com/nonconvex
Convex vs. Non-convex Optimization
Progress is only tip of the iceberg.. Real world is mostly non-convex!
Images taken from https://www.facebook.com/nonconvex
Convex vs. Nonconvex Optimization
Unique optimum: global/local. Multiple local optima
Convex vs. Nonconvex Optimization
Unique optimum: global/local. Multiple local optima
In high dimensions possiblyexponential local optima
Convex vs. Nonconvex Optimization
Unique optimum: global/local. Multiple local optima
In high dimensions possiblyexponential local optima
How to deal with non-convexity?
Outline
1 Introduction
2 Spectral Methods: Matrices and Tensors
3 Applications of Tensor Methods
4 Conclusion
Matrices and Tensors as Data Structures
Modern data: Multi-modal,multi-relational data.
Matrices: pairwise relations. Tensors: higher order relations.
Multi-modal data figure from Lise Getoor slides.
Tensor Representation of Higher Order Moments
Matrix: Pairwise Moments
E[x⊗ x] ∈ Rd×d is a second order tensor.
E[x⊗ x]i1,i2 = E[xi1xi2 ].
For matrices: E[x⊗ x] = E[xx⊤].
Tensor: Higher order Moments
E[x⊗ x⊗ x] ∈ Rd×d×d is a third order tensor.
E[x⊗ x⊗ x]i1,i2,i3 = E[xi1xi2xi3 ].
Spectral Decomposition of Tensors
M2 =∑
i
λiui ⊗ vi
= + ....
Matrix M2 λ1u1 ⊗ v1 λ2u2 ⊗ v2
Spectral Decomposition of Tensors
M2 =∑
i
λiui ⊗ vi
= + ....
Matrix M2 λ1u1 ⊗ v1 λ2u2 ⊗ v2
M3 =∑
i
λiui ⊗ vi ⊗ wi
= + ....
Tensor M3 λ1u1 ⊗ v1 ⊗ w1 λ2u2 ⊗ v2 ⊗ w2
We have developed efficient methods to solve tensor decomposition.
Tensor Decomposition Methods
SVD on Pairwise Moments toobtain W
Fast SVD via RandomProjection
+ ++ +
Dimensionality Reduction ofthird order moments using W
+ ++ +
Power method on reduced tensor T : v 7→T (I, v, v)
‖T (I, v, v)‖.
Guaranteed to converge to correct solution!
Summary on Tensor Methods
= + ....
Tensor decomposition: iterative algorithm with tensor contractions.
Tensor contraction: T (W1,W2,W3), where Wi are matrices/vectors.
Strengths of tensor method
Guaranteed convergence to globally optimal solution!
Fast and accurate, orders of magnitude faster than previous methods.
Embarrassingly parallel and suited for distributed systems, e.g.Spark.
Exploit optimized BLAS operations on CPUs and GPUs.
Preliminary Results on Spark
Alternating Least Squares for Tensor Decomposition.
minw,A,B,C
∥
∥
∥
∥
∥
T −
k∑
i=1
λiA(:, i) ⊗B(:, i)⊗ C(:, i)
∥
∥
∥
∥
∥
2
F
Update Rows Independently
Tensor slices
B C
worker 1
worker 2
worker i
worker k
Results on NYtimes corpus
3 ∗ 105 documents, 108 words
Spark Map-Reduce
26mins 4 hrs
https://github.com/FurongHuang/SpectralLDA-TensorSpark
Outline
1 Introduction
2 Spectral Methods: Matrices and Tensors
3 Applications of Tensor Methods
4 Conclusion
Tensor Methods for Learning LVMs
GMM HMM
h1 h2 h3
x1 x2 x3
ICA
h1 h2 hk
x1 x2 xd
Multiview and Topic Models
Overview of Our Approach
Tensor-based Model Estimation
= + +
Unlabeled Data
Overview of Our Approach
Tensor-based Model Estimation
= + +
Unlabeled Data → Probabilistic admixture models
Overview of Our Approach
Tensor-based Model Estimation
= + +
Unlabeled Data → Probabilistic admixture models → Tensor Decomposition
Overview of Our Approach
Tensor-based Model Estimation
= + +
Unlabeled Data → Probabilistic admixture models → Tensor Decomposition → Inference
Overview of Our Approach
Tensor-based Model Estimation
= + +
Unlabeled Data → Probabilistic admixture models → Tensor Decomposition → Inference
tensor decomposition → correct model
Overview of Our Approach
Tensor-based Model Estimation
= + +
Unlabeled Data → Probabilistic admixture models → Tensor Decomposition → Inference
tensor decomposition → correct model
Highly parallelized, CPU/GPU/Cloud implementation.
Guaranteed global convergence for Tensor Decomposition.
Community Detection on Yelp and DBLP datasets
n ∼ 20, 000
Yelp
n ∼ 40, 000
DBLP
n ∼ 1 million
Tensor vs. Variational
FB YP DBLP−sub DBLP10
0
101
102
103
104
105
Datasets
Run
ning
tim
e (s
econ
ds)
VariationalTensor
Tensor Methods for Training Neural Networks
Tremendous practical impactwith deep learning.
Algorithm: backpropagation.
Highly non-convex optimization
Toy Example: Failure of Backpropagation
x1
x2
y=1y=−1
Labeled input samplesGoal: binary classification
σ(·) σ(·)
y
x1 x2x
w1 w2
Our method: guaranteed risk bounds for training neural networks
Toy Example: Failure of Backpropagation
x1
x2
y=1y=−1
Labeled input samplesGoal: binary classification
σ(·) σ(·)
y
x1 x2x
w1 w2
Our method: guaranteed risk bounds for training neural networks
Toy Example: Failure of Backpropagation
x1
x2
y=1y=−1
Labeled input samplesGoal: binary classification
σ(·) σ(·)
y
x1 x2x
w1 w2
Our method: guaranteed risk bounds for training neural networks
Backpropagation vs. Our Method
Weights w2 randomly drawn and fixed
Backprop (quadratic) loss surface
−4
−3
−2
−1
0
1
2
3
4 −4−3
−2−1
01
23
4
200
250
300
350
400
450
500
550
600
650
w1(1) w1(2)
Backpropagation vs. Our Method
Weights w2 randomly drawn and fixed
Backprop (quadratic) loss surface
−4
−3
−2
−1
0
1
2
3
4 −4−3
−2−1
01
23
4
200
250
300
350
400
450
500
550
600
650
w1(1) w1(2)
Loss surface for our method
−4
−3
−2
−1
0
1
2
3
4 −4−3
−2−1
01
23
4
0
20
40
60
80
100
120
140
160
180
200
w1(1) w1(2)
Feature Transformation for Training Neural Networks
Feature learning: Learn φ(·) from input data.
How to use φ(·) to train neural networks?x
φ(x)
y
Feature Transformation for Training Neural Networks
Feature learning: Learn φ(·) from input data.
How to use φ(·) to train neural networks?x
φ(x)
y
Invert E[φ(x)⊗ y] to Learn Neural Network Parameters
In general, E[φ(x)⊗ y] is a tensor.
What class of φ(·) useful for training neural networks?
What algorithms can invert moment tensors to obtain parameters?
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
Input:x ∈ R
d
S1(x) ∈ Rd
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
Input:x ∈ R
d
S1(x) ∈ Rd
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
Input:x ∈ R
d
S1(x) ∈ Rd
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
mth-order score function:Input:x ∈ R
d
S1(x) ∈ Rd
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
mth-order score function:
Sm(x) := (−1)m∇(m)p(x)
p(x)
Input:x ∈ R
d
S1(x) ∈ Rd
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
mth-order score function:
Sm(x) := (−1)m∇(m)p(x)
p(x)
Input:x ∈ R
d
S2(x) ∈ Rd×d
Score Function Transformations
Score function for x ∈ Rd
with pdf p(·):
S1(x) := −∇x log p(x)
mth-order score function:
Sm(x) := (−1)m∇(m)p(x)
p(x)
Input:x ∈ R
d
S3(x) ∈ Rd×d×d
NN-LiFT: Neural Network LearnIng
using Feature Tensors
Input:x ∈ R
d
Score functionS3(x) ∈ R
d×d×d
NN-LiFT: Neural Network LearnIng
using Feature Tensors
Input:x ∈ R
d
Score functionS3(x) ∈ R
d×d×d
1
n
n∑
i=1
yi ⊗ S3(xi) =1
n
n∑
i=1
yi⊗
Estimating M3 using
labeled data {(xi, yi)}
S3(xi)
Cross-
moment
NN-LiFT: Neural Network LearnIng
using Feature Tensors
Input:x ∈ R
d
Score functionS3(x) ∈ R
d×d×d
1
n
n∑
i=1
yi ⊗ S3(xi) =1
n
n∑
i=1
yi⊗
Estimating M3 using
labeled data {(xi, yi)}
S3(xi)
Cross-
moment
Rank-1 components are the estimates of first layer weights
+ + · · ·
CP tensor
decomposition
Reinforcement Learning (RL) of POMDPsPartially observable Markov decision processes.
Proposed Method
Consider memoryless policies. Episodic learning: indirect exploration.
Tensor methods: careful conditioning required for learning.
First RL method for POMDPs with logarithmic regret bounds.
xi xi+1 xi+2
yi yi+1
ri ri+1
ai ai+1
0 1000 2000 3000 4000 5000 6000 70002
2.1
2.2
2.3
2.4
2.5
2.6
2.7
Number of Trials
Ave
rage
Rew
ard
SM−UCRL−POMDPUCRL−MDPQ−learningRandom Policy
Logarithmic Regret Bounds for POMDPs using Spectral Methods by K. Azzizade, A. Lazaric, A.
, under preparation.
Convolutional Tensor Decomposition
== ∗
xx
∑
f∗i w∗
i F∗ w∗
(a)Convolutional dictionary model (b)Reformulated model
.... ....= ++
Cumulant λ1(F∗1 )
⊗3 +λ2(F∗2 )
⊗3 . . .
Efficient methods for tensor decomposition with circulant constraints.
Convolutional Dictionary Learning through Tensor Factorization by F. Huang, A. , June 2015.
Hierarchical Tensor Decomposition
= + .... = + .... = + ....
= + ....
Relevant for learning latent tree graphical models.
Propose first algorithm for integrated learning of hierarchy andcomponents of tensor decomposition.
Highly parallelized operations but with global consistency guarantees.
Scalable Latent Tree Model and its Application to Health Analytics By F. Huang, U. N.
Niranjan, J. Perros, R. Chen, J. Sun, A. , Preprint, June 2014.
Outline
1 Introduction
2 Spectral Methods: Matrices and Tensors
3 Applications of Tensor Methods
4 Conclusion
Summary and Outlook
Summary
Tensor methods: a powerful paradigm for guaranteed large-scalemachine learning.
First methods to provide provable bounds for training neuralnetworks, many latent variable models (e.g HMM, LDA), POMDPs!
Summary and Outlook
Summary
Tensor methods: a powerful paradigm for guaranteed large-scalemachine learning.
First methods to provide provable bounds for training neuralnetworks, many latent variable models (e.g HMM, LDA), POMDPs!
Outlook
Training multi-layer neural networks, models with invariances,reinforcement learning using neural networks . . .
Unified framework for tractable non-convex methods with guaranteedconvergence to global optima?
My Research Group and Resources
Podcast/lectures/papers/software available athttp://newport.eecs.uci.edu/anandkumar/