on deep multi-view representation learning

On Deep Multi-View Representation Learning

Weiran Wang [email protected]

Toyota Technological Institute at Chicago

Raman Arora [email protected]

Johns Hopkins University

Karen Livescu [email protected]

Toyota Technological Institute at Chicago

Jeff Bilmes [email protected]

University of Washington, Seattle

AbstractWe consider learning representations (features)in the setting in which we have access to mul-tiple unlabeled views of the data for representa-tion learning while only one view is available attest time. Previous work on this problem has pro-posed several techniques based on deep neuralnetworks, typically involving either autoencoder-like networks with a reconstruction objective orpaired feedforward networks with a correlation-based objective. We analyze several techniquesbased on prior work, as well as new variants, andcompare them experimentally on visual, speech,and language domains. To our knowledge thisis the first head-to-head comparison of a vari-ety of such techniques on multiple tasks. Wefind an advantage for correlation-based represen-tation learning, while the best results on mosttasks are obtained with our new variant, deepcanonically correlated autoencoders (DCCAE).

1. IntroductionIn many applications, we have access to multiple “views”of data at training time while only one view is avail-able at test time. The views can be multiple mea-surement modalities, such as simultaneously recordedaudio + video (Kidron et al., 2005; Chaudhuri et al.,2009), audio + articulation (Arora & Livescu, 2013), im-ages + text (Hardoon et al., 2004; Socher & Li, 2010;Hodosh et al., 2013), or parallel text in two lan-

Proceedings of the32nd International Conference on MachineLearning, Lille, France, 2015. JMLR: W&CP volume 37. Copy-right 2015 by the author(s).

guages (Vinokourov et al., 2003; Haghighi et al., 2008;Chandar et al., 2014; Faruqui & Dyer, 2014), but may alsobe different information extracted from the same source,such as words + context (Pennington et al., 2014) or docu-ment text + text of inbound hyperlinks (Bickel & Scheffer,2004). The presence of multiple information sourcespresents an opportunity to learn better representations (fea-tures) by analyzing multiple views simultaneously. Typicalapproaches are based on learning a feature transformationof the “primary” view (the one available at test time) thatcaptures useful information from the second view using apaired two-view training set. Under certain assumptions,theoretical results exist showing the advantages of multi-view techniques for downstream tasks (Kakade & Foster,2007; Foster et al., 2009; Chaudhuri et al., 2009).

Several recently proposed approaches for multi-viewrepresentation learning are based on deep neural net-works (DNNs), inspired by their success in typ-ical unsupervised (single-view) feature learning set-tings (Hinton & Salakhutdinov, 2006). Compared to kernelmethods, DNNs can more easily process large amounts oftraining data and, as a parameteric method, do not requirereferring to the training set at test time.

There are two main training criteria (objectives) that havebeen applied for DNN-based multi-view representationlearning. One is based on autoencoders, where the ob-jective is to learn a compact representation that bestreconstructs the inputs (Ngiam et al., 2011). The sec-ond approach is based on canonical correlation analysis(CCA,Hotelling, 1936), which learns features in two viewsthat are maximally correlated. CCA and its kernel ex-tension (Lai & Fyfe, 2000; Akaho, 2001; Bach & Jordan,2002; Hardoon et al., 2004) have long been the workhorsefor multi-view feature learning and dimensionality re-


x

Reconstructedx Reconstructedy

f

p q

x y

View 1

Vie

w2

U V

f g

x y

Reconstructedx Reconstructedy

View 1

Vie

w2

U V

f g

p q

(a) SplitAE (b) DCCA (c) DCCAE/CorrAE/DistAE

Figure 1.Schematic diagram of DNN-based multi-view representation learning models.

duction (Vinokourov et al., 2003; Kakade & Foster, 2007;Socher & Li, 2010; Dhillon et al., 2011). Multiple neu-ral network based CCA-like models have been pro-posed (Lai & Fyfe, 1999; Hsieh, 2000), but the fullDNN extension of CCA, termed deep CCA (DCCA,Andrew et al., 2013) has been developed only recently.

The contributions of this paper are as follows. Wecompare several DNN-based approaches, along withlinear and kernel CCA, in the unsupervised multi-view feature learning setting where the second viewis not available at test time. Prior work has shownthe benefit of multi-view methods on tasks such asretrieval (Vinokourov et al., 2003; Hardoon et al., 2004;Socher & Li, 2010; Hodosh et al., 2013), clustering(Blaschko & Lampert, 2008; Chaudhuri et al., 2009)and classification/recognition (Dhillon et al., 2011;Arora & Livescu, 2013; Ngiam et al., 2011). However, toour knowledge no head-to-head comparison on multipletasks has previously been done. We address this gap bycomparing approaches based on prior work, as well asnew variants developed here. Empirically, we find thatCCA-based approaches tend to outperform unconstrainedreconstruction-based approaches. One of the new methodswe propose, a DNN-based model combining CCA andautoencoder-based terms, is the consistent winner acrossseveral tasks. To facilitate future work, we release our im-plementations and a new benchmark dataset of simulatedtwo-view data based on MNIST.

2. DNN-based multiview feature learningNotations In the multi-view feature learning scenario, wehave access to paired observations from two views, de-noted(x1,y1), . . . , (xN ,yN ), whereN is the sample size,xi ∈ R

Dx andyi ∈ RDy for i = 1, . . . , N . We also denote

the data matrices for each view byX = [x1, . . . ,xN ] andY = [y1, . . . ,yN ]. We use bold-face letters, e.g.f , to de-note mappings implemented by kernel machines or DNNs,with a corresponding set of learnable parameters, denoted,e.g.,Wf . We write thef -projected (view 1) data matrixasf(X) = [f(x1), . . . , f(xN )]. The dimensionality of theprojection (feature) is denotedL.

We now describe the DNN-based multi-view feature

learning algorithms considered here, with correspondingschematic diagrams given in Fig.1.

2.1. Split autoencoders (SplitAE)Ngiam et al.(2011) propose to extract shared representa-tions by reconstructing both views from the one view thatis available at test time. In this approach, the feature extrac-tion networkf is shared while the reconstruction networksp andq are separate for each view. We refer to this modelas a split autoencoder (SplitAE), shown schematically inFig. 1 (a). The objective of this model is the sum of recon-struction errors for the two views (we omit theℓ2 weightdecay term for all models in this section):

minWf ,Wp,Wq

1

N

N∑

i=1

(‖xi − p(f(xi))‖2

+ ‖yi − q(f(xi))‖2).

The intuition for this model is that the shared representationcan be extracted from a single view, and can be used to re-construct all views.1 The autoencoder loss is the empiricalexpectation of the loss incurred at each training sample, andthus stochastic gradient descent (SGD) can be used to op-timize the objective efficiently, with the gradient estimatedfrom a small minibatch of samples.

2.2. Deep canonical correlation analysis (DCCA)Andrew et al.(2013) propose a DNN extension of CCAtermed deep CCA (DCCA; see Fig.1 (b). In DCCA, twoDNNs f andg are used to extract nonlinear features foreach view and the canonical correlation between the ex-tracted featuresf(X) andg(Y) is maximized:

maxWf ,Wg,U,V

1

Ntr

(

U⊤f(X)g(Y)⊤V)

(1)

s.t. U⊤

(

1

Nf(X)f(X)⊤ + rxI

)

U = I,

V⊤

(

1

Ng(Y)g(Y)⊤ + ryI

)

V = I,

u⊤i f(X)g(Y)⊤vj = 0, for i 6= j,

1The authors also propose a bimodal deep autoencoder com-bining DNN transformed features from both views; this model ismore natural for the multimodal fusion setting where both viewsare available at test time. Empirically,Ngiam et al.(2011) reportthat SplitAE tends to work better in the multi-view setting.


where U = [u1, . . . ,uL] and V = [v1, . . . ,vL] arethe CCA directions that project the DNN outputs and(rx, ry) > 0 are regularization parameters for samplecovariance estimation (Bie & Moor, 2003; Hardoon et al.,2004). In DCCA, U⊤f(·) is the final projection mappingused for testing. One intuition for CCA-based objectivesis that, while it may be difficult to accurately reconstructone view from the other view, it may be easier, and may besufficient, to learn a predictor of afunction (or subspace)of the second view. In addition, it should be helpful for thelearned dimensions within each view to be uncorrelated sothat they provide complementary information.

Optimization The DCCA objective couples all trainingsamples through the whitening constraints, so stochasticgradient descent (SGD) cannot be applied in a standardway. It has been observed byWang et al.(2015) thatDCCA can still be optimized efficiently as long as the gra-dient is estimated using a sufficiently large minibatch (withgradient formulas given inAndrew et al., 2013). Intu-itively, this approach works because a large minibatch con-tains enough information for estimating the covariances.

2.3. Deep canonically correlated autoencoders(DCCAE)

Inspired by both CCA and reconstruction-based objectives,we propose a new model that consists of two autoencodersand optimizes the combination of canonical correlation be-tween the learned bottleneck representations and the recon-struction errors of the autoencoders. In other words, weoptimize the following objective

minWf ,Wg,Wp,Wq,U,V

−1

Ntr

(

U⊤f(X)g(Y)⊤V)

+λ

N

N∑

i=1

(‖xi − p(f(xi))‖2

+ ‖yi − q(g(yi))‖2) (2)

s.t. the same constraints in (1),

whereλ > 0 is a trade-off parameter. Alternatively, thisapproach can be seen as adding an autoencoder regulariza-tion term to DCCA. We call this approach deep canonicallycorrelated autoencoders (DCCAE). Similarly to DCCA, weapply stochastic optimization to the DCCAE objective; thestochastic gradient is the sum of the gradient for the au-toencoder term and the gradient for the DCCA term.

Interpretations CCA maximizes the mutual informa-tion between the projected views for certain distributions(Borga, 2001), while training an autoencoder to minimizereconstruction error amounts to maximizing a lower boundon the mutual information between inputs and learned fea-tures (Vincent et al., 2010). The DCCAE objective offersa trade-off between the information captured in the (in-put, feature) mapping within each view on the one hand,and the information in the (feature, feature) relationship

across views on the other. Intuitively, this is the same prin-ciple as the information bottleneck method (Tishby et al.,1999), and indeed, in the case of Gaussian variables, theinformation bottleneck method finds the same subspaces asCCA (Chechik et al., 2005).

2.4. Correlated autoencoders (CorrAE)In the next model, we replace the CCA term in the DC-CAE objective with the sum of the scalar correlations be-tween the pairs of learned dimensions across views, whichis an alternative measure of agreement between views. Inother words, the feature dimensions within each view arenot constrained to be uncorrelated with each other. Thismodel is intended to test how important the original CCAconstraint is. We call this model correlated autoencoders(CorrAE), shown in Fig.1 (c). Its objective can be equiva-lently written in a constrained form as

minWf ,Wg,Wp,Wq,U,V

−1

Ntr

(

U⊤f(X)g(Y)⊤V)

+λ

N

N∑

i=1

(‖xi − p(f(xi))‖2

+ ‖yi − q(g(yi))‖2) (3)

s.t.u⊤i f(X)f(X)⊤ui = v⊤

i g(Y)g(Y)⊤vi = N, 1 ≤ i ≤ L.

whereλ > 0 is a trade-off parameter. It is clear that theconstraint set in (3) is a relaxed version of that of (2). Wewill demonstrate that this difference results in a large per-formance gap. We apply the same optimization strategy ofDCCAE to CorrAE.

CorrAE is similar to the model ofChandar et al.(2014),who try to learn vectorial word representations using par-allel corpora from two languages. They use DNNs in eachview (language) to predict the bag-of-words representationof the input sentences, or that of the paired sentences fromthe other view, while encouraging the learned bottlenecklayer representations to be highly correlated.

2.5. Minimum-distance autoencoders (DistAE)The CCA objective can be seen as minimzing the dis-tance between the learned projections of the two views,while satisfying the whitening constraints for the projec-tions (Hardoon et al., 2004). The constraints complicatethe optimization of CCA-based objectives, as pointed outabove. This observation motivates us to consider additionalobjectives that decompose into sums over training exam-ples, while maintaining the intuition of the CCA objectiveas a reconstruction error between two mappings. Here weconsider a variant we refer to as minimum-distance autoen-coders (DistAE) that optimizes the following objective:

minWf ,Wg,Wp,Wq

1

N

N∑

i=1

‖f(xi) − g(yi)‖2

‖f(xi)‖2

+ ‖g(yi)‖2

+λ

N

N∑

i=1

(‖xi − p(f(xi))‖2

+ ‖yi − q(g(yi))‖2) (4)


which is a weighted combination of reconstruction errorsof two autoencoders and the average discrepancy betweenthe projected sample pairs. The denominator of the discrep-ancy term is used to keep the optimization from improvingthe objective by simply scaling down the projections (al-though they can never become identically zero due to thereconstruction terms). This objective is unconstrained andis the expectation of loss incurred at each training sample,so normal SGD applies using small minibatches.

3. Related workWe focus on related work based on feed-forwardneural networks and the kernel extension of CCA.There has also been work using deep Boltzmann ma-chines (Srivastava & Salakhutdinov, 2014; Sohn et al.,2014), where several layers of restricted Boltzmann ma-chines (RBM) are stacked to represent each view, with anadditional top layer that provides the joint representation.These are probabilistic graphical models, for which themaximum likelihood objective is intractable and the train-ing procedures are more complex. Although probabilisticmodels have some advantages (e.g., dealing with missingvalues and generating samples in a natural way), DNN-based models are tractable and efficient to train.

3.1. DNN feature learning using CCA-like objectivesThere have been several approaches to multi-view repre-sentation learning using neural networks with an objectivesimilar to that of CCA. Under the assumption that the twoviews share a common cause (e.g., depth is a commoncause for adjacent patches of images),Becker & Hinton(1992) maximize a sample-based estimate of mutual infor-mation between the common signal and the average outputsof neural networks for the two views. It is less straightfor-ward, however, to develop sample-based estimates of mu-tual information in higher dimensions.

Lai & Fyfe (1999) propose to optimize the correlation(rather than canonical correlation) between the outputs ofnetworks for each view, subject to scale constraints on eachoutput dimension. Instead of directly solving this con-strained formulation, the authors apply Lagrangian relax-ation and solve the resulting unconstrained objective usingSGD. The objective in this work is different from that ofCCA, as there are no constraints that the learned dimen-sions within each view be uncorrelated.Hsieh(2000) pro-poses a neural network based model involving three mod-ules: one module for extracting a pair of maximally cor-related one-dimensional features for the two views; and asecond and third module for reconstructing the original in-puts of the two views from the learned features. In thismodel, the feature dimensions can be learned one after an-other, each learned using as input the reconstruction resid-ual from previous dimensions. The three modules are each

trained separately, so there is no unified objective.

Kim et al. (2012) propose an algorithm that first uses deepbelief networks and the autoencoder objective to extractfeatures for two languages independently, and then applieslinear CCA to the learned features (activations at the bot-tleneck layer of the autoencoders) to learn the final repre-sentation. In this two-step approach, the DNN weight pa-rameters are not updated to optimize the CCA objective.

3.2. Kernel CCAAnother nonlinear extension of CCA is kernel CCA(KCCA, Lai & Fyfe, 2000; Akaho, 2001; Melzer et al.,2001; Bach & Jordan, 2002; Hardoon et al., 2004). KCCAcorresponds to using (potentially infinite-dimensional)feature mappings induced by positive-definite kernelskx(·, ·), ky(·, ·) (e.g., Gaussian RBF kernelsk(a,b) =

e−‖a−b‖2/2s2

wheres is the kernel width), and learninga linear CCA on them with linear projections(U,V).From the representer theorem of reproducing kernel Hilbertspaces (Scholkopf & Smola, 2001), the final projection canbe written as linear combinations of kernel functions evalu-ated on the training set, i.e.,U⊤f(·) =

∑Ni=1

αikx(x,xi)whereαi ∈ R

L, i = 1, . . . , N ; one can then work withthe kernel matrices and directly solve for the linear coef-ficients{αi}

Ni=1

. KCCA involves anN × N eigenvalueproblem and so is challenging in both memory (storing thekernel matrices) and time (solving the eigenvalue systemnaıvely costsO(N3)). To alleviate these issues, variouskernel approximation techniques have been proposed, suchas random Fourier features (Lopez-Paz et al., 2014) andthe Nystrom approximation (Williams & Seeger, 2001). Inrandom Fourier features, we randomly sampleM Dx/Dy-dimensional vectors from a Gaussian distribution and mapthe original inputs toRM by computing the dot productwith the random samples followed by an elementwise co-sine; the inner products between transformed samples ap-proximate kernel similarities between original inputs. Inthe Nystrom approximation, we randomly selectM train-ing samples and construct theM × M kernel matrix forthese samples, and use its eigen-decomposition to obtaina rank-M approximation of the full kernel matrix. Bothtechniques produce rank-M approximations of the kernelmatrices with computational complexityO(M3 + M2N);but random Fourier features are data independent and moreefficient to generate. Other approximation techniques suchas incomplete Cholesky decomposition (Bach & Jordan,2002), partial Gram-Schmidt (Hardoon et al., 2004), andincremental SVD (Arora & Livescu, 2012) have also beenproposed. However, for very large training sets, as insome of our tasks below, it remains difficult and costlyto approximate KCCA well. Although iterative algorithmshave recently been introduced for very large CCA problems(Lu & Foster, 2014), they are aimed at sparse matrices anddo not have a natural out-of-sample extension.


Figure 2.Selection of view 1 images (left) and their correspond-ing view 2 images (right) from our noisy MNIST dataset.

4. ExperimentsWe compare the following methods in the multi-view learn-ing setting, focusing on several downstream tasks: noisydigit image classification, speech recognition, and wordpair semantic similarity.

DNN-based models, including SplitAE, CorrAE, DCCA,DCCAE, and DistAE.

Linear CCA (CCA), corresponding to DCCA with only alinear network without hidden layers for both views.

Kernel CCA approximations. Exact KCCA is intractablefor our tasks; we instead implement two kernel approxima-tion techniques, using Gaussian RBF kernels. The first im-plementation, denotedFKCCA, uses random Fourier fea-tures (Lopez-Paz et al., 2014) and the second implemen-tation, denotedNKCCA, uses the Nystrom approxima-tion (Williams & Seeger, 2001). As described in Sec.3.2,in FKCCA/NKCCA we transform the original inputs toan M -dimensional feature space where the inner prod-ucts between samples approximate the kernel similarities(Yang et al., 2012). We apply linear CCA to the trans-formed inputs to obtain the approximate KCCA solution.

4.1. Noisy MNIST digits

In this task, we generate two-view data using the MNISTdataset (LeCun et al., 1998), which consists of28 × 28grayscale digit images, with60K/10K images for train-ing/testing. We generate a more challenging version ofthe dataset as follows (see Fig.2 for examples). We firstrescale the pixel values to[0, 1]. We then randomly rotatethe images at angles uniformly sampled from[−π/4, π/4]and the resulting images are used as view 1 inputs. Foreach view 1 image, we randomly select an image of thesame identity (0-9) from the original dataset, add indepen-dent random noise uniformly sampled from[0, 1] to eachpixel, and truncate the pixel final values to[0, 1] to obtainthe corresponding view 2 sample. The original training setis further split into training/tuning sets of size50K/10K.

Since, given the digit identity, observing a view 2 imagedoes not provide any information about the correspondingview 1 image, a good multi-view learning algorithm should

Table 1.Performance of several representation learning methodson the test set of noisy MNIST digits. Performance measuresare clustering accuracy (ACC), normalized mutual information(NMI) of clustering, and classification error rates of a linear SVMon the projections. The selected feature dimensionalityL is givenin parentheses. Results are averaged over5 random seeds.

Method ACC (%) NMI (%) Error (%)

Baseline 47.0 50.6 13.1CCA (L = 10) 72.9 56.0 19.6SplitAE (L = 10) 64.0 69.0 11.9CorrAE (L = 10) 65.5 67.2 12.9DistAE (L = 20) 53.5 60.2 16.0FKCCA(L = 10) 94.7 87.3 5.1NKCCA(L = 10) 95.1 88.3 4.5DCCA (L = 10) 97.0 92.0 2.9DCCAE (L = 10) 97.5 93.4 2.2

be able to extract features that disregard the noise. We mea-sure the class separation in the learned feature spaces byclustering the projected view 1 inputs into10 clusters andevaluating how well the clusters agree with ground-truthlabels. We use spectral clustering (Ng et al., 2002) so asto account for possibly non-convex cluster shapes. Specifi-cally, we first build ak-nearest-neighbor graph on the pro-jected view 1 tuning/test samples with a binary weightingscheme (edges connecting neighboring samples have a con-stant weight of1), then embed these samples inR

10 usingeigenvectors of the normalized graph Laplacian, and finallyrunK-means in the embedding to obtain a hard partition ofthe samples. In the last step,K-means is run20 times withrandom initialization and the run with the bestK-meansobjective is used. The size of the neighborhood graphk isselected from{5, 10, 20, 30, 50} using the tuning set. Wemeasure clustering performance with two criteria, cluster-ing accuracy (ACC) and normalized mutual information(NMI) (Cai et al., 2005).

Each algorithm has hyperparameters that are selected us-ing the tuning set. The final dimensionalityL is selectedfrom {5, 10, 20, 30, 50}. For CCA, the regularization pa-rametersrx/ry are selected via grid search. For KC-CAs, we fix rx/ry at a small positive value of10−4 (assuggested byLopez-Paz et al.(2014), FKCCA is robustto rx/ry), do grid search for the Gaussian kernel widthfor each view at rankM = 5, 000, and then test withM = 20, 000. For DNN-based models, feature mappings(f ,g) are implemented by networks of3 hidden layers,each of1, 024 sigmoid units, and a linear output layer ofL units; reconstruction mappings(p,q) are implementedby networks of3 hidden layers, each of1, 024 sigmoidunits, and an output layer of784 sigmoid units. We fixrx = ry = 10−4 for DCCA and DCCAE. For Spli-tAE/CorrAE/DCCAE/DistAE we select the trade-off pa-rameterλ via grid search. The two networks(f ,p) are


(a) Inputs (b) LLE1234567890

(c) SplitAE (d) CorrAE

(e) DistAE (f) CCA

(g) FKCCA (h) NKCCA

(i) DCCA (j) DCCAE

Figure 3.t-SNE embedding of the projected test set of noisyMNIST digits using different methods. Each sample is denotedby a marker located at its coordinates of embedding and colorcoded by its label. Neither the feature learning algorithms nort-SNE use the class information.

pre-trained in a layerwise manner using restricted Boltz-mann machines (Hinton & Salakhutdinov, 2006) and sim-ilarly for (g,q) with inputs from the corresponding view.For DNN-based models, we use SGD for optimization withminibatch size, learning rate and momentum tuned on thetuning set. A small weight decay parameter of10−4 is usedfor all layers. We monitor the objective on the tuning setfor early stopping. For each algorithm, we select the model

with best ACC on the tuning set, and report its results onthe test set. The ACC and NMI results (in percentage) foreach algorithm are given in Table1. As a baseline, we alsocluster the original784-dimensional view 1 images.

All of the multi-view feature learning algorithms achievesome improvement over the baseline. The nonlinear CCAalgorithms perform similarly to each other and significantlybetter than SplitAE/CorrAE/DistAE. We also qualitativelyinvestigate the features by embedding the projected fea-tures in 2D usingt-SNE (van der Maaten & Hinton, 2008);the resulting visualizations are given in Fig.3. Overall, theclass separation in the visualizations qualitatively agreeswith the relative clustering performances in Table1.

In the embedding of input images (Fig.3 (a)), samples ofeach digit form an approximately one dimensional, stripe-shaped manifold, and the degree of freedom along eachmanifold corresponds roughly to the variation in rotationangle (see supplementary material for embedding with im-ages). This degree of freedom does not change the identityof the image, which is common to both views. Projectionsby SplitAE/CorrAE/DistAE do achieve somewhat betterseparation for some classes, but the unwanted rotation vari-ation is still prominent in the embeddings. On the otherhand, without using any label information and with onlypaired noisy images, the nonlinear CCA algorithms man-age to map digits of the same identity to similar locationswhile supressing the rotational variation and separating im-ages of different identities (linear CCA also approximatesthe same behavior, but fails to separate the classes, pre-sumably because the input variations are too complex to becaptured by linear mappings). Overall, DCCAE gives thecleanest embedding, with different digits pushed far apart.

The different behaviour of CCA-based methods from Spli-tAE/CorrAE/DistAE suggests two things. First, when theinputs are noisy, reconstructing the inputs faithfully maystill lead to unwanted degrees of freedom in the features(DCCAE tends to select quite small trade-off parameterλ = 10−3 or 10−2, further supporting that it is not nec-essary to minimize reconstruction error). Second, thehard CCA constraints, which enforce uncorrelatedness be-tween different feature dimensions, appear essential; theseconstraints are the difference between DCCAE and Cor-rAE/DistAE. However, the constraints without the multi-view objective are insufficient. To see this, we also vi-sualize a10-dimensional locally linear embedding (LLE,Roweis & Saul, 2000) of the test images in Fig.3 (b). LLEsatisfies the same un-correlatedness constraints as in CCA-based methods, but without access to the second view, itdoes not separate the classes as nicely.

In view of the embeddings in Fig.3, one would expectthat a simple linear classifier can achieve high accuracy onDCCA/DCCAE projections. We train one-versus-one lin-


ear SVMs (Chang & Lin, 2011) on the projected trainingset (now using the ground truth labels), and test on the pro-jected test set, while using the projected tuning set for se-lecting the SVM hyperparameter (the penalty parameter forhinge loss). Test error rates on the optimal embedding ofeach algorithm (with highest ACC) are provided in Table1(last column). These error rates agree with the clusteringresults. Multi-view feature learning makes classificationmuch easier on this task: Instead of using a heavily non-linear classifier on the original inputs, a very simple linearclassifier that can be trained efficiently on low-dimensionalprojections already achieves high accuracy.

4.2. Acoustic-articulatory data for speech recognitionWe next experiment with the Wisconsin X-Ray Micro-Beam (XRMB) corpus (Westbury, 1994) of simultaneouslyrecorded speech and articulatory measurements from 47American English speakers. Multi-view feature learningvia CCA/KCCA has previously been shown to improvephonetic recognition performance when tested on audioalone (Arora & Livescu, 2013; Wang et al., 2015).

We follow the setup ofWang et al.(2015) and use learnedfeatures for speaker-independent phonetic recognition. In-puts to multi-view feature learning are acoustic features(39D features consisting of mel frequency cepstral coeffi-cients (MFCCs) and their first and second derivatives) andarticulatory features (horizontal/vertical displacement of 8pellets attached to several parts of the vocal tract) concate-nated over a 7-frame window around each frame, giving273D acoustic inputs and 112D articulatory inputs for eachview.

We split the XRMB speakers into disjoint sets of35/8/2/2 speakers for feature learning/recognizer train-ing/tuning/testing. The 35 speakers for feature learningare fixed; the remaining 12 are used in a 6-fold experi-ment (recognizer training on 4 2-speaker folds, tuning on1 fold, and testing on the last fold). Each speaker hasroughly 50K frames, giving 1.43M multi-view trainingframes. We remove the per-speaker mean and varianceof the articulatory measurements for each training speaker.All of the learned feature types are used in a “tandem” ap-proach (Hermansky et al., 2000), i.e., they are appendedto the original 39D features and used in a standard hid-den Markov model (HMM)-based recognizer with Gauss-ian mixture observation distributions. The baseline recog-nizer uses the original MFCC features. The recognizer hasone 3-state left-to-right HMM per phone and the same lan-guage model as inWang et al.(2015)

For each fold, we select the hyperparameters based onrecognition accuracy on the tuning set. As before, modelsbased on neural networks are trained via SGD, with no pre-training and with optimization parameters tuned by grid

Table 2.Mean and standard deviations of PERs over 6 folds ob-tained by each algorithm on the XRMB test speakers.

Method Mean (std) PER (%)

Baseline 34.8 (4.5)CCA 26.7 (5.0)SplitAE 29.0 (4.7)CorrAE 30.6 (4.8)DistAE 33.2 (4.7)FKCCA 26.0 (4.4)NKCCA 26.6 (4.2)DCCA 24.8 (4.4)DCCAE 24.5 (3.9)

search. A small weight decay parameter of5×10−4 is usedfor all layers. For each algorithm, the dimensionalityL istuned over{30, 50, 70}. For DNN-based models, we usehidden layers of1 500 ReLUs. For DCCA, we tune the net-work depths (up to3 nonlinear hidden layers) and find thatin the best-performing architecture,f has3 hidden ReLUlayers followed by a linear output layer whileg has only alinear output layer. For SplitAE/CorrAE/DistAE/DCCAE,the same encoder architecture as that of DCCA performsbest, and we set the decoders to have symmetric architec-tures to the encoders (except for SplitAE which does nothave an encoderg and its decoderq is linear). We fixrx = ry = 10−4 for DCCAE (and FKCCA/NKCCA). Thetrade-off parameterλ is tuned for each algorithm by gridsearch.

For FKCCA, we find it important to use a large numberof random featuresM to get a competitive result, consis-tent with the findings ofHuang et al.(2014) when usingrandom Fourier features for speech data. We tune kernelwidths atM = 5, 000 with FKCCA, and test FKCCA withM = 30, 000 (the largestM we could afford to obtain anexact SVD solution on a workstation with 32G main mem-ory); we are not able to obtain results for NKCCA withM = 30, 000 in 48 hours with our implementation, so wereport its test performance atM = 20, 000 with the optimalFKCCA hyper-parameters. Notice that FKCCA has about14.6 million parameters (random Gaussian samples + pro-jection matrices), which is more than the number of weightparameters in the largest DCCA model, so it is slower thanDCCA for testing (cost of obtaining test features is linearin the number of parameters for both KCCA and DNNs).

Phone error rates (PERs) obtained by different featurelearning algorithms are given in Table2. We see the samepattern as on MNIST: nonlinear CCA-based algorithmsoutperform SplitAE/CorrAE/DistAE. Since the recognizernow is a nonlinear mapping (HMM), the performance ofthe linear CCA features is highly competitive. Again, DC-CAE selects a relatively smallλ = 0.01, indicating that thecanonical correlation term is more important.


4.3. Multilingual data for word embeddingsIn this task, we learn a vectorial representation of Eng-lish words from pairs of English-German word embed-dings. We follow the setup ofFaruqui & Dyer(2014) andLu et al.(2015), and use as inputs640-dimensional mono-lingual word vectors trained via latent semantic analysison the WMT 2011 monolingual news corpora and usethe same36K English-German word pairs for multi-viewlearning. The learned mappings are applied to the origi-nal English word embeddings (180K words) and the pro-jections are used for evaluation. We evaluate on the bi-gram similarity dataset ofMitchell & Lapata(2010), usingthe adjective-noun (AN) and verb-object (VN) subsets, andtuning and test splits (of size 649/1,972) for each subset (weexclude the noun-noun subset as it is observed byLu et al.(2015) that the NN human annotations often reflect “top-ical” rather than “functional” similarity). We simply addthe projections of the two words in each bigram to obtainanL-dimensional representation of the bigram, as done inprior work (Blacoe & Lapata, 2012; Lu et al., 2015). Wecompute the cosine similarity between the two vectors ofeach bigram pair, order the pairs by similarity, and reportthe Spearman’s correlation (ρ) between the model’s rank-ing and human rankings.

We fix the feature dimensionality atL = 384; other hyper-parameters are tuned as in previous experiments. DNN-based models use ReLU hidden layers of width1, 280. Asmall weight decay parameter of10−4 is used for all lay-ers. We use two ReLU hidden layers for encoders (f andg), and try both linear and nonlinear networks with twohidden layers for decoders (p andq). FKCCA/NKCCAare tested withM = 20, 000 using kernel widths tuned atM = 4, 000. We fix rx = ry = 10−4 for nonlinear CCAs.

For each algorithm, we select the model with the highestSpearman’s correlation on the 649 tuning bigram pairs, andwe report its performance on the 1,972 test pairs in Table3(our baseline and DCCA results are different from that ofLu et al.(2015) due to a different normalization and bettertuning). Unlike MNIST and XRMB, it is important for thefeatures to reconstruct the input monolingual word embed-dings well, as can be seen from the superior performance ofSplitAE over FKCCA/NKCCA/DCCA. This implies thereis useful information in the original inputs that is not cor-related across views. However, DCCAE still performs thebest on the AN task, in this case using a relatively largeλ.

5. DiscussionWe have explored several approaches in the space ofDNN-based mutli-view representation learning. We havefound that on several tasks, CCA-based models outperformautoencoder-based models (SplitAE) and models basedon between-view squared distance (DistAE) or correlation

Table 3.Spearman’s correlation (ρ) for bigram similarities.Method AN VN Avg.

Baseline 45.0 39.1 42.1CCA 46.6 37.7 42.2SplitAE 47.0 45.0 46.0CorrAE 43.0 42.0 42.5DistAE 43.6 39.4 41.5FKCCA 46.4 42.9 44.7NKCCA 44.3 39.5 41.9DCCA 48.5 42.5 45.5DCCAE 49.1 43.2 46.2

(CorrAE) instead of canonical correlation. The best overallperformer is a new DCCA extension introduced here, deepcanonically correlated autoencoders (DCCAE).

In light of the empirical results, it is interesting to consideragain the main features of each type of objective and cor-responding constraints. Autoencoder-based approaches arebased on the idea that the learned features should be ableto accurately reconstruct the inputs (in the case of multi-view learning, the inputs in both views). The CCA objec-tive, on the other hand, focuses on how well each view’srepresentation predicts the other’s, ignoring the abilitytoreconstruct each view. CCA is expected to perform wellwhen the two views are uncorrelated given the class la-bel (Chaudhuri et al., 2009). The noisy MNIST datasetused here simulates exactly this scenario, and indeed thisis the task where CCA outperforms other objectives by thelargest margins. Even in the other tasks, however, there isoften no significant advantage to being able to reconstructthe inputs faithfully.

The constraints in the various methods also have animportant effect. The performance difference betweenDCCA and CorrAE demonstrates that uncorrelatednessbetween learned dimensions is important. On the otherhand, the stronger DCCA constraint may still not besufficiently strong; an even better constraint may be torequire the learned dimensions to be independent (orapproximately so), and this is an interesting avenuefor future work. Another future direction is to com-pare DNN-based models with models based on deepBoltzmann machines (Srivastava & Salakhutdinov, 2014;Sohn et al., 2014) and noise-constrastive learning criteria(Gutmann & Hyvarinen, 2012).

AcknowledgmentsThis research was supported by NSF grant IIS-1321015.The opinions expressed in this work are those of the au-thors and do not necessarily reflect the views of the fundingagency. The Tesla K40 GPUs used for this research weredonated by NVIDIA Corporation. We thank Geoff Hintonfor helpful discussion.


References

Akaho, Shotaro. A kernel method for canonical correlationanalysis. InProceedings of the International Meeting ofthe Psychometric Society (IMPS2001), 2001.

Andrew, Galen, Arora, Raman, Bilmes, Jeff, and Livescu,Karen. Deep canonical correlation analysis. InICML,pp. 1247–1255, 2013.

Arora, Raman and Livescu, Karen. Kernel CCA formulti-view learning of acoustic features using articula-tory measurements. InSymposium on Machine Learningin Speech and Language Processing (MLSLP), 2012.

Arora, Raman and Livescu, Karen. Multi-view CCA-basedacoustic features for phonetic recognition across speak-ers and domains. InICASSP, 2013.

Bach, Francis R. and Jordan, Michael I. Kernel indepen-dent component analysis.Journal of Machine LearningResearch, 3:1–48, 2002.

Becker, Suzanna and Hinton, Geoffrey E. Self-organizingneural network that discovers surfaces in random-dotstereograms.Nature, 355:161–163, 1992.

Bickel, Steffen and Scheffer, Tobias. Multi-view cluster-ing. In Proc. of the 4th IEEE Int. Conf. Data Mining(ICDM’04), pp. 19–26, 2004.

Bie, Tijl De and Moor, Bart De. On the regularization ofcanonical correlation analysis. Int. Sympos. ICA andBSS, 2003.

Blacoe, William and Lapata, Mirella. A comparison ofvector-based representations for semantic composition.In EMNLP, pp. 546–556, 2012.

Blaschko, Mathew B. and Lampert, Christoph H. Correla-tional spectral clustering. InCVPR, pp. 1–8, 2008.

Borga, Magnus. Canonical correlation: A tutorial. 2001.

Cai, Deng, He, Xiaofei, and Han, Jiawei. Document clus-tering using Locality Preserving Indexing.IEEE Trans.Knowledge and Data Engineering, 17(12):1624–1637,2005.

Chandar, Sarath, Lauly, Stanislas, Larochelle, Hugo,Khapra, Mitesh M., Ravindran, Balaraman, Raykar,Vikas, and Saha, Amrita. An autoencoder approach tolearning bilingual word representations. InNIPS, pp.1853–1861, 2014.

Chang, Chih-Chung and Lin, Chih-Jen. LIBSVM: A li-brary for support vector machines.ACM Trans. Intelli-gent Systems and Technology, 2(3):27, 2011.

Chaudhuri, Kamalika, Kakade, Sham M., Livescu, Karen,and Sridharan, Karthik. Multi-view clustering via canon-ical correlation analysis. InICML, pp. 129–136, 2009.

Chechik, Gal, Globerson, Amir, Tishby, Naftali, andWeiss, Yair. Information bottleneck for Gaussian vari-ables. Journal of Machine Learning Research, 6:165–188, 2005.

Dhillon, Paramveer, Foster, Dean, and Ungar, Lyle. Multi-view learning of word embeddings via CCA. InNIPS,pp. 199–207, 2011.

Faruqui, Manaal and Dyer, Chris. Improving vector spaceword representations using multilingual correlation. InProceedings of European Chapter of the Association forComputational Linguistics, 2014.

Foster, Dean P., Johnson, Rie, Kakade, Sham M., andZhang, Tong. Multi-view dimensionality reduction viacanonical correlation analysis. Technical report, 2009.

Gutmann, Michael and Hyvarinen, Aapo. Noise-contrastive estimation of unnormalized statistical mod-els, with applications to natural image statistics.Journalof Machine Learning Research, 13:307–361, 2012.

Haghighi, Aria, Liang, Percy, Berg-Kirkpatrick, Taylor,and Klein, Dan. Learning bilingual lexicons from mono-lingual corpora. InProceedings of the Annual Meetingof the Association for Computational Linguistics (ACL2008), pp. 771–779, 2008.

Hardoon, David R., Szedmak, Sandor, and Shawe-Taylor,John. Canonical correlation analysis: An overview withapplication to learning methods.Neural Computation,16(12):2639–2664, 2004.

Hermansky, Hynek, Ellis, Daniel P. W., and Sharma, San-gita. Tandem connectionist feature extraction for con-ventional HMM systems. InICASSP, pp. 1635–1638,2000.

Hinton, G. E. and Salakhutdinov, R. R. Reducing the di-mensionality of data with neural networks.Science, 313(5786):504–507, 2006.

Hodosh, Micah, Young, Peter, and Hockenmaier, Julia.Framing image description as a ranking task: Data, mod-els and evaluation metrics.Journal of Artificial Intelli-gence Research, 47:853–899, 2013.

Hotelling, Harold. Relations between two sets of variates.Biometrika, 28(3/4):321–377, 1936.

Hsieh, W. W. Nonlinear canonical correlation analysis byneural networks.Neural Networks, 13(10):1095–1105,2000.


Huang, Po-Sen, Avron, Haim, Sainath, Tara, Sindhwani,Vikas, and Ramabhadran, Bhuvana. Kernel methodsmatch deep neural networks on TIMIT: Scalable learn-ing in high-dimensional random Fourier spaces. InICASSP, pp. 205–209, 2014.

Kakade, Sham M. and Foster, Dean P. Multi-view regres-sion via canonical correlation analysis. InCOLT, pp.82–96, 2007.

Kidron, Einat, Schechner, Yoav Y., and Elad, Michael. Pix-els that sound. InCVPR, pp. 88–95, 2005.

Kim, Jungi, Nam, Jinseok, and Gurevych, Iryna. Learningsemantics with deep belief network for cross-languageinformation retrieval. InCOLING, pp. 579–588, 2012.

Lai, Pei Ling and Fyfe, Colin. A neural implementation ofcanonical correlation analysis.Neural Networks, 12(10):1391–1397, 1999.

Lai, Pei Ling and Fyfe, Colin. Kernel and nonlinear canon-ical correlation analysis.Int. J. Neural Syst., 10(5):365–377, 2000.

LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner,Patrick. Gradient-based learning applied to documentrecognition.Proc. IEEE, 86(11):2278–2324, 1998.

Lopez-Paz, David, Sra, Suvrit, Smola, Alex, Ghahramani,Zoubin, and Schoelkopf, Bernhard. Randomized nonlin-ear component analysis. InICML, pp. 1359–1367, 2014.

Lu, Ang, Wang, Weiran, Bansal, Mohit, Gimpel, Kevin,and Livescu, Karen. Deep multilingual correlation forimproved word embeddings. InNAACL-HLT, 2015.

Lu, Yichao and Foster, Dean P. Large scale canonical cor-relation analysis with iterative least squares. InNIPS,pp. 91–99, 2014.

Melzer, Thomas, Reiter, Michael, and Bischof, Horst. Non-linear feature extraction using generalized canonical cor-relation analysis. InProc. of the 11th Int. Conf. ArtificialNeural Networks (ICANN’01), pp. 353–360, 2001.

Mitchell, Jeff and Lapata, Mirella. Composition in distri-butional models of semantics.Cognitive Science, 34(8):1388–1429, 2010.

Ng, Andrew Y., Jordan, Michael I., and Weiss, Yair. Onspectral clustering: Analysis and an algorithm. InNIPS,pp. 849–856, 2002.

Ngiam, Jiquan, Khosla, Aditya, Kim, Mingyu, Nam,Juhan, Lee, Honglak, and Ng, Andrew. Multimodal deeplearning. InICML, pp. 689–696, 2011.

Pennington, Jeffrey, Socher, Richard, and Manning,Christopher D. GloVe: Global vectors for word repre-sentation. InEMNLP, 2014.

Roweis, Sam T. and Saul, Lawrence K. Nonlinear dimen-sionality reduction by locally linear embedding.Science,290(5500):2323–2326, 2000.

Scholkopf, Bernhard and Smola, Alexander J.Learningwith Kernels. Support Vector Machines, Regularization,Optimization, and Beyond. MIT Press, 2001.

Socher, Richard and Li, Fei-Fei. Connecting modalities:Semi-supervised segmentation and annotation of imagesusing unaligned text corpora. InCVPR, pp. 966–973,2010.

Sohn, Kihyuk, Shang, Wenling, and Lee, Honglak. Im-proved multimodal deep learning with variation of infor-mation. InNIPS, pp. 2141–2149, 2014.

Srivastava, Nitish and Salakhutdinov, Ruslan. Multimodallearning with deep boltzmann machines.Journal of Ma-chine Learning Research, 15:2949–2980, 2014.

Tishby, Naftali, Pereira, Fernando, and Bialek, William.The information bottleneck method. InProc. 37th An-nual Allerton Conference on Communication, Control,and Computing, pp. 368–377, 1999.

van der Maaten, Laurens J. P. and Hinton, Geoffrey E. Vi-sualizing data usingt-SNE. Journal of Machine Learn-ing Research, 9:2579–2605, 2008.

Vincent, Pascal, Larochelle, Hugo, Lajoie, Isabelle, Ben-gio, Yoshua, and Manzagol, Pierre-Antoine. Stacked de-noising autoencoders: Learning useful representations ina deep network with a local denoising criterion.Journalof Machine Learning Research, 11:3371–3408, 2010.

Vinokourov, Alexei, Cristianini, Nello, and Shawe-Taylor,John. Inferring a semantic representation of text viacross-language correlation analysis. InNIPS, pp. 1497–1504, 2003.

Wang, Weiran, Arora, Raman, Livescu, Karen, and Bilmes,Jeff. Unsupervised learning of acoustic features via deepcanonical correlation analysis. InICASSP, 2015.

Westbury, John R.X-Ray Microbeam Speech ProductionDatabase User’s Handbook Version 1.0, 1994.

Williams, Christopher K. I. and Seeger, Matthias. Usingthe Nystrom method to speed up kernel machines. InNIPS, pp. 682–688, 2001.

Yang, Tianbao, Li, Yu-Feng, Mahdavi, Mehrdad, Jin,Rong, and Zhou, Zhi-Hua. Nystrom method vs randomFourier features: A theoretical and empirical compari-son. InNIPS, pp. 476–484, 2012.

on deep multi-view representation learning

Documents