Unsupervised Feature Selection via Distributed Coding for Multi-view ObjectRecognition

C. Mario Christoudias, Raquel Urtasun and Trevor DarrellUC Berkeley EECS & ICSI

MIT CSAIL

Abstract

Object recognition accuracy can be improved when in-formation from multiple views is integrated, but informa-tion in each view can often be highly redundant. We con-sider the problem of distributed object recognition or index-ing from multiple cameras, where the computational poweravailable at each camera sensor is limited and communi-cation between cameras is prohibitively expensive. In thisscenario, it is desirable to avoid sending redundant visualfeatures from multiple views. Traditional supervised featureselection approaches are inapplicable as the class label isunknown at each camera. In this paper we propose an un-supervised multi-view feature selection algorithm based ona distributed coding approach. With our method, a Gaus-sian Process model of the joint view statistics is used atthe receiver to obtain a joint encoding of the views withoutdirectly sharing information across encoders. We demon-strate our approach on recognition and indexing tasks withmulti-view image databases and show that our method com-pares favorably to an independent encoding of the featuresfrom each camera.

1. IntroductionObject recognition often benefits from integration of ob-

servations at multiple views. However, when multiple cam-era sensors exist in a bandwidth limited environment it maybe impossible to transmit all the visual features in each im-age. When the task or target class is not known a priorithere may be no obvious way to decide which features tosend from each view. If redundant features are chosen atthe expense of informative features, performance can beworse with multiple views than with a single view, givenfixed bandwidth.

We consider the problem of how to select which featuresto send in each view to achieve optimal results at a central-ized recognition or indexing module (see Figure 1). An ef-ficient encoding of the streams might be possible in theoryif a class label could be inferred at each camera, enablingthe use of supervised feature selection techniques to encodeand send only those features that are relevant of that class.

View VObject

Transmitter

Receiver

Object

Recognition

System

Visual Features

View 1

View 2

Figure 1. Distributed object recognition. Messages are only sentbetween each camera (transmitter) and the recognition module (re-ceiver). An efficient joint feature selection is achieved without di-rectly sharing information between cameras.

Partial occlusions, unknown camera viewpoint, and limitedcomputational power, however, limit the ability to reliablyestimate the image class label at each camera. Instead wepropose an unsupervised feature selection algorithm to ob-tain an efficient encoding of the feature streams.

If each camera sensor had access to the information fromall views this could trivially be accomplished by a jointcompression algorithm that could, e.g., encode the featuresof the v-th view based on the information in the previousv − 1 views. We are interested, however, in the case wherethere is no communication between cameras themselves,and messages are only sent from the cameras to the recog-nition module with a limited backchannel back to the cam-eras. In practice, many visual category recognition and in-dexing applications are bandwidth constrained (e.g., wire-less surveillance camera networks, mobile robot swarms,mobile phone cameras), and it is infeasible to broadcast im-ages across all cameras or to send the raw signal from eachcamera to the recognition module.

It is possible to achieve very efficient encoding withoutany information exchange between the cameras, by adopt-ing a distributed encoding scheme that takes advantage ofknown statistics of the environment [14, 20, 18, 3]. Wedevelop a new method for distributed encoding based ona Gaussian Process (GP) formulation, and demonstrate itsapplicability to encoding visual-word feature histograms;such representations are used in many contemporary object

978-1-4244-2243-2/08/$25.00 ©2008 IEEE

indexing and category recognition methods [19, 12, 4].Our algorithm exploits redundancy between views and

learns a statistical model of the dependency between fea-ture streams during an off-line training phase at the re-ceiver. This model is then used along with previously de-coded streams to aid feature selection at each camera. Ifthe streams are redundant, then only a few features need tobe sent. In this paper, we consider bag-of-words represen-tations [12, 4] and model the dependency between visualfeature histograms. As shown in our experiments, our algo-rithm is able to achieve an efficient joint encoding of the fea-ture histograms without explicitly sharing features acrossviews. This results in an efficient unsupervised feature se-lection algorithm that improves recognition performance inthe presence of limited network bandwidth.

We evaluate our approach using the COIL-100 multi-view image database [11] on the tasks of instance-level re-trieval and recognition from multiple views; we compareunsupervised distributed feature selection to independentstream encoding. For a two-view problem, our algorithmachieves a compression factor of over 100:1 in the secondview while preserving multi-view recognition and retrievalaccuracy. In contrast, independent encoding at the same ratedoes not improve over single-view performance.

2. Related WorkContemporary methods for object recognition use local

feature representations and perform recognition over sets oflocal features corresponding to each image [8, 12, 4, 22].Several techniques have been proposed that generalize thesemethods to include object view-point in addition to appear-ance [16, 21, 22, 17]. Rothganger et. al. [16] presentan approach that builds an explicit 3D model from localaffine-invariant image features and uses that model to per-form view-point invariant object recognition. Thomas et.al. [21] extend the Implicit Shape Model (ISM) of Leibeand Schiele [8] for single-view object recognition to multi-ple views by combining the ISM model with the recognitionapproach of Ferrari et. al. [2]. Similarly, Savarese and Li[17] present a part-based approach for multi-view recogni-tion that jointly models object view-point and appearance.

Traditionally, approaches to multi-view object recogni-tion use only a single input image at test time [16, 21, 22,17]. Recently, there has been a growing interest in applica-tion areas where multiple input views of the object or sceneare available. The presence of multiple views can lead toincreased recognition performance; however, the transmis-sion of data from multiple cameras places an additional bur-den on the network. In this paper, we propose an unsuper-vised feature selection algorithm that enables effective ob-ject recognition from multiple cameras in the presence oflimited network bandwidth.

Feature selection algorithms exploit data dependency or

redundancy to derive compact representations for classifi-cation [9]. For our problem, traditional supervised featureselection approaches are inapplicable as the class label isunknown at each camera. Many approaches have been pro-posed for unsupervised feature selection [1, 9, 13]. Penget. al. [13] define a minimum-redundancy or maximum-relevance criterion for unsupervised feature selection basedon mutual information. Dy and Brodley [1] compute rel-evant feature subsets using a clustering approach based ona maximum likelihood criterion with the expectation maxi-mization algorithm. For multiple views, it is possible to ap-ply the above unsupervised feature selection techniques in-dependently at each camera to efficiently encode and trans-mit features over the network. A better encoding of the fea-tures, however, can be achieved if features are jointly se-lected across views [20]. Under limited network bandwidtha joint encoding is not possible as communication betweencameras is often prohibitively expensive.

Distributed coding algorithms [14, 20, 18, 3] seek a jointencoding of the streams without sharing features acrossviews. These methods exploit data redundancy at a shared,common receiver to perform a distributed feature selec-tion that in many cases approaches the joint encoding rate[20]. Contemporary techniques to distributed coding in-clude the DISCUS algorithm of Pradhan and Ramchandran[14] based on data cosets, and the approach of Schonberg[18] that builds upon low-density parity check codes. Inthis paper, we present a new distributed coding algorithmfor bag-of-words image representations in multi-view ob-ject recognition with Gaussian Processes.

Gaussian Processes (GPs) [15] have become popular be-cause they are simple to implement, flexible (i.e., they cancapture complex behaviors through a simple parametriza-tion), and are fully probabilistic. The latter enables them tobe easily incorporated in more complex systems, and pro-vides an easy way of expressing and evaluating predictionuncertainty. GPs have been suggested as a replacement forsupervised neural networks in non-linear regression [15]and they generalize a range of previous techniques (e.g.krigging, splines, RBFs). As shown in our experiments,GPs are well suited for distributed feature selection as theuncertainty measure provided by GPs is correlated with dataredundancy. Our work bears similarity to that of Kapoor et.al. [5] that use GP prediction uncertainty as a criteria forexample selection in active learning.

3. Gaussian Process ReviewA Gaussian Process is a collection of random variables,

any finite number of which have consistent joint Gaussiandistributions [15]. Given a training set D = {(xi,yi), i =1, · · · , N}, composed of inputs xi and noisy outputs yi,we assume that the noise is additive, independent and Gaus-sian, such that the relationship between the (latent) function,

f(x), and the observed noisy targets, y, is given by

yi = f(xi) + εi , (1)

where εi ∼ N (0, σ2noise) and σ2

noise is the noise variance.GP regression is a Bayesian approach that assumes a GP

prior over the space of functions,

p(f |X) = N (0,K) , (2)

where f = [f1, · · · , fn] is the vector of latent function val-ues, fi = f(xi), X = [x1, · · · ,xN ], and K is a covariancematrix whose entries are given by a covariance function,Ki,j = k(xi,xj). GPs are non-parametric models and areentirely defined by their covariance function (and trainingdata); the set of possible covariance functions is defined bythe set of Mercer kernels. During training, the model hyper-parameters, β, are learned by minimizing

− ln p(X, β |Y)=D

2ln |K|+

1

2tr(

K−1YYT)

+C . (3)

where Y = [y1, · · · ,yN ], C is a constant, and D is thedimension of the output.

Inference in the GP model is straightforward, assuminga joint GP prior over training, f , and testing, f∗, latent vari-ables,

p(f , f∗) = N

(

0,

(

Kf,f K∗,f

Kf,∗ K∗,∗

))

, (4)

where ∗ is used as shorthand for f∗ and the dependency onX is omitted for clarity of presentation, Kf,f is the covari-ance of the training data, K∗,∗, the covariance of the testdata, and Kf,∗ = KT

∗,f is the cross-covariance of trainingand test data. The joint posterior p(f , f∗|Y) is Gaussian:

p(f , f∗|Y) =p(f , f∗)p(y|f)

p(y). (5)

Marginalizing the training latent variables, f , can be donein closed form and yields a Gaussian predictive distribution[15], p(f∗|y) = N (M,C), with

M = K∗,f (Kf,f + σ2noiseI)

−1Y (6)C = K∗,∗ −K∗,f (Kf,f + σ2

noiseI)−1Kf,∗ . (7)

The variance of the GP is an indicator of the predictionuncertainty. In the following section we will show how thevariance can be used to define a feature selection criteria.

4. Distributed Object RecognitionWe consider the distributed recognition problem of V

cameras transmitting information to a central common re-ceiver with no direct communication between cameras (seeFigure 1). In our problem, each camera is equipped with a

Feature

Extractor

E

v

Z

v

D

Z

v hv

Channel

I

v

Object

Recognizer

hv+1, ...,hV

Training

Data

GPCamera

Feature

Selector

h1, ...,hv-1

Backchannelξv

Z

v

Figure 2. System diagram. Image Iv is coded by encoder Ev

and decoder D. Zv are the encoded image features, hv the re-

constructed histograms, and ξv the non-redundant bin indices forviews v = 1, ..., V (see Section 4 for details).

simple encoder used to compress each signal before trans-mission. A common decoder receives the encoded signalsand performs a joint decoding of the signal streams using amodel of the joint statistics. Note that this coding schemeoff-loads the computational burden onto the decoder and al-lows for computationally in-expensive encoders. In whatfollows, we assume a noiseless channel, but our approach isalso applicable to the more general case.

Figure 2 illustrates our proposed distributed coding algo-rithm at a single camera. With our algorithm, the decoderiteratively queries each of the V cameras and specifies thedesired encoding rate the camera should use. At the v-thview, the decoder uses its model of joint statistics alongwith side information, i.e., the previously decoded streams,to decode the signal. The use of side information allows theencoder to work at a lower encoding rate than if the streamwere encoded independently. As discussed below, the de-coder selects the camera encoding rate based on the jointstream statistics and transmits this information back to theencoder. If the v-th view is highly redundant with respect tothe side information, then little-to-no information needs tobe encoded and sent to the decoder.

In this work, we consider bag-of-words models for ob-ject recognition [12, 4]. With these models, an image isrepresented using a set of local image descriptors extractedfrom the image either at a set of interest point locations(e.g., those computed using a Harris point detector [10]) oron a regular grid. In our experiments, we employ the latterfeature detection strategy in favor of simplicity at the en-coder. To perform feature coding, the local image featuresare quantized using a global vocabulary that is shared by theencoder and decoder and computed from training images.

Let Iv , v = 1, ..., V be a collection of V views of theobject or scene of interest, imaged by each camera and Zv

be the set of quantized local image features correspondingto image Iv computed by the v-th encoder,Ev. In this con-text (see Figure 2), the encoders transmit quantized featuresto the central receiver and the encoding rate is the number

of features sent.In theory, distributed coding with individual image fea-

tures (e.g., visual words) might be possible, but preliminaryexperiments have shown that distributed coding of local fea-tures does not improve over independent encoding at eachcamera. Using a joint model over quantized features onCOIL-100 with a 991 word vocabulary gave an entropy of9.4 bits, which indicates that the joint feature distributionis close to uniform (for a 991 word feature vocabulary, theuniform distribution has an entropy of 10 bits). This is ex-pected since a local image feature is a fairly weak predictorof other features in the image.

We have found, however, that distributed coding of his-tograms of local features is effective. As seen in our exper-iments, the distribution over features in one view is predic-tive of the distribution of features in other views and, there-fore, feature histograms are a useful image representationfor distributed coding.

4.1. Joint Feature Histogram ModelLet Zv be the set of encoded features of each view,

v = 1, ..., V . To facilitate a joint decoding of the fea-ture streams, the decoder first computes a feature histogram,hv = h(Zv), using the global feature vocabulary. Note, inour approach, the form of h(·) can either be a flat [19] orhierarchical histogram [12, 4]; we present a general dis-tributed coding approach applicable to any bag-of-wordstechnique. At the decoder, the joint stream statistics areexpressed over feature histograms,

p(h1,h2, ...,hV ) = p(h1)

V∏

v=2

p(hv|hv−1, ...,h1), (8)

where the conditional probabilities are learned from train-ing data as described in Section 3.

Assuming independence between the histogram bins andpair-wise dependence between histograms we write

p(hv|hv−1, ...,h1) =

v−1∏

k=1

B∏

b=1

p(hv,b|hk) (9)

where hv,b is the b-th bin of histogram hv, and B is thenumber of bins.

The joint model of Equation 8 is used to determine whichfeatures at a given camera are redundant with the side in-formation. In particular, redundant features are those thatare associated with the redundant bins of the histogram ofthe current view. Since we are ultimately interested in thefeature histograms for performing recognition, the encoderscan send either histogram bin counts or the quantized visualfeatures themselves.

We obtain a reconstruction of the feature histogram ofeach view from the view’s encoded features and its side

Algorithm 1 GP Distributed Feature SelectionLet Ev be an encoder, ξv be defined over sets of featurehistogram bin indices, Zv be defined over sets of encodedfeatures, Hv be a N ×B matrix of N training examples,v = 1, ..., V , and Rmax be the desired encoding rate.

h = ∅ξ1 = {1, ..., B}for v = 1, ..., V doZv = request(Ev, ξv)for b = 1, ..., B do

if b ∈ ξv thenhv,b = ψ(Zv,b)

elsehv,b = (kv−1,b

∗ )T (Kv−1,b)−1Hv,b

end ifend forh = (h, hv)if v < V then

for b = 1, ..., B doσv+1,b = (kv,b(h, h) − (kv,b

∗ )T(Kv,b)−1kv,b∗ )

1

2

end forξv+1 = select(σv+1, Rmax)

end ifend for

information. Let hv be the histogram of interest and hk,k = 1, ..., v − 1, its side information, where v is the currentview considered by the decoder. From Equation 9 the prob-ability of a histogram hv given its side information is foundas,

p(hv |hv−1, ...,h1) =

v−1∏

k=1

p(hv |hk) (10)

We model the above conditional probability using a GPprior over feature histograms. To make learning moretractable we assume independence between histogram bins

p(hv |hv−1, ...,h1) =

B∏

b=1

N (0,Kv−1,b) (11)

where a GP is defined over each bin with kernel matrixKv−1,b. We compute Kv−1,b with a covariance functiondefined using an exponential kernel over the side informa-tion,

kv,b(hi,hj) =

v∏

r=1

γ−vb exp

(

d(hri ,h

rj )

2

α2b

)

+ ηbδij (12)

where hi = (h1i , ..., h

vi ) and hj = (h1

j , ..., hvj ) are multi-

view histogram instances, γb, αb are the kernel hyper-parameters of bin b, which we assume to be the same across

Figure 3. Synthetic example considered below. This scenario con-sists of two overlapping views of an object, which is presumedto fill the scene. Image features are represented using a 6 wordvocabulary.

views, and ηb is a per-bin additive noise term. Given train-ing data Hv, where Hv is a N × B matrix of N trainingexamples for the v = 1, ..., V views, the kernel hyper-parameters are learned as described in Section 3. We definea different set of kernel hyper-parameters per bin since eachbin can exhibit drastically different behavior with respect tothe side information.

The variance of each GP can be used to determinewhether a bin is redundant: a small bin variance indicatesthat the GP model is confident in its prediction, and there-fore the features corresponding to that bin are likely to beredundant with respect to the side information. In our exper-iments, we found that redundant bins generally exhibit vari-ances that are small and similar in value and that these vari-ances are much smaller than those of non-redundant bins.

4.2. GP Distributed Feature SelectionDistributed feature selection is performed by the decoder

using an iterative process. The decoder begins by query-ing the first encoder to send all of its features, since in theabsence of any side information no feature is redundant.At the v-th view, the decoder requests only those featurescorresponding to the non-redundant histogram bins of thatview, whose indices are found using the bin variances out-put by each GP. At each iteration, the GPs are evaluatedusing the reconstructed histograms of previous iterations asillustrated in Algorithm 1.

Given the encoded features Zv, the decoder reconstructshistograms hv , v = 1, ..., V , such that bins that are non-redundant are those received and the redundant bins are es-timated from the GP mean prediction

hv,b =

{

h(Zv,b), b ∈ ξv

(kv−1,b∗ )T(Kv−1,b)−1Hv,b, otherwise.

(13)

where Hv,b = (hv,b1 , ..., hv,b

N )T are the bin values for viewv and bin b in the training data, and ξv are the bin indices ofthe non-redundant bins of the histogram of view v.

0 0.05 0.1 0.15 0.2 0.25

0

0.05

0.1

0.15

0.2

0.25

Ground−Truth Value

Estimated Value

Non−Redundant Bin

Redundant Bin

Bin Number

1 2 3 4 5 6

−1

0

1

2

3

x 10−4

Me

an

Va

ria

nc

e

Figure 4. GP variance is correlated with bin redundancy. The GPmean prediction for the second view is plotted vs. ground-truthvalues for both a redundant and non-redundant bin. The GP vari-ance for each of the 6 histogram bins, averaged across examplesis also shown; error bars indicate ±1 std. deviation. The varianceof non-redundant bins is noticeably higher than that of redundantbins.

The GP distributed feature selection algorithm achievesa compression rate proportional to the number of bin indicesrequested for each view. For view v the compression rate ofour algorithm in percent bins transmitted is

R =r

B=

2|ξv|

B, (14)

where B is the total number of histogram bins and r is thenumber of bins received, which is proportional to twice thenumber of non-redundant bins as a result of the decoderrequest operation. Note, however, that in the case oflarge amounts of redundancy there are few non-redundantbins encoded at each view and therefore a small encodingrate is achieved.

As mentioned above the bin indices ξv are chosen us-ing the GP prediction uncertainty. If a desired encodingrate Rmax is provided, the decoder requests the rmax/2 his-togram bins associated with the highest GP prediction un-certainty (see Equation 14). If Rmax is not known, the en-coding rate can be automatically determined by groupingthe histogram bins at each view into two groups correspond-ing to regions of high and low uncertainty; ξv is then definedusing the bins associated with the high uncertainty group.Both strategies exploit the property that prediction uncer-tainty is correlated with bin redundancy to request the non-redundant bins at each view. Many grouping algorithms areapplicable for the latter approach, e.g., conventional clus-tering. In practice, we use a simple step detection techniqueto form each group by sorting the bin variances and findingthe maximum local difference.

2 4 6 8 100

0.2

0.4

0.6

0.8

1

Num. Nearest Neighbors

Ac

cu

rac

y

Distributed Coding

Independent Coding

Figure 5. Nearest-neighbor instance-level retrieval for the two-view synthetic dataset; average retrieval accuracy is plotted overvarying neighborhood sizes. For a fixed rate, our algorithm faroutperforms the independent encoding baseline (see text for de-tails).

5. ExperimentsWe evaluate our distributed coding approach on the tasks

of object recognition and indexing from multiple views.Given hv, v = 1, .., V , multi-view recognition is performedusing a nearest-neighbor classifier over a fused distancemeasure, computed as the average distance across views

Di,j(hi,hj) =1

V

V∑

v=1

d(hvi ,h

vj ) (15)

where for flat histograms we define d(·) using the L2 norm,and with pyramid match similarity [4] for multi-resolutionhistograms1.

We use a majority vote performance metric for nearest-neighbor recognition. Under this metric a query example iscorrectly recognized if a majority (≥ k/2) of its k nearest-neighbors are of the same category or instance. We alsoexperiment with an at-least-one criterion to evaluate perfor-mance in an interactive retrieval setting: with this schemean example is correctly retrieved if one of the first k exam-ples has the true label. We compare distributed coding toindependent encoding at each view with a random featureselector that randomly selects histogram bins according toa uniform distribution, and report feature selection perfor-mance in terms of percent bins encoded, R (see Equation14).

In what follows, we first present experiments on a syn-thetic example with our approach and then discuss our re-sults on COIL-100.

5.1. Synthetic ExampleTo demonstrate our distributed feature selection ap-

proach we consider the scenario illustrated in Figure 3. Anobject is imaged with two overlapping views, and the his-tograms of each view are represented using a 6 word vocab-ulary. As shown by the figure, the images are redundant in

1Note our distributed coding algorithm is independent of the choice ofclassification method.

2 4 6 8 100

0.2

0.4

0.6

0.8

1

Num. Nearest Neighbors

Ac

cu

rac

y

Distributed Coding

Independent Coding

Figure 6. Nearest-neighbor instance-level retrieval on the two-view synthetic dataset with partial redundancy, plotted over vary-ing neighborhood sizes. Our distributed coding algorithm per-forms favorably to independent encoding even when the bins areonly partially redundant.

4 of the 6 words, as 2 of the words (i.e., diamond and plus)do not appear in the overlapping portion of each view. Al-though real-world problems are much more complex thandescribed above, we use this simple scenario to give intu-ition and motivate our approach.

We first consider the case where there is no noise be-tween the redundant features in each view and the redun-dant features appear only in the overlapping region. Werandomly generated N = 100, 6-D histograms, where eachhistogram was generated by sampling its bins from a uni-form distribution between 0 and 1, and the histograms werenormalized to sum to one. Each histogram was split intotwo views by replicating the first 4 bins in each view andrandomly splitting the other two bins. The above data wasused to form a training set of examples, where each pairof histograms corresponds to a single object instance. Toform the test set, zero mean Gaussian noise was added tothe training set with σ = 0.01 and the test set histogramswere split into two views using the same split ratios as thetraining set.

For distributed coding we trained 6 GPs, one per dimen-sion, using each view. Figure 4 displays the predicted binvalue vs. ground truth for 2 of the bins (one redundant andthe other non-redundant) evaluated on the second view ofthe test set. The GPs are able to learn the deterministic map-ping that relates the redundant bins. For the non-redundantbin, the variance of the GP’s predictions is quite large com-pared to that of the redundant bin. Also shown in Figure 4,are the mean GP variances plotted for each histogram bin.The error bars in the plot indicate the standard deviation.The GP variance is much larger for the non-redundant binsthan those of the redundant ones whose variances are smalland centered about 0. This is expected since non-redundantbins are less correlated and therefore the GPs are less cer-tain in their prediction of the value of these bins from sideinformation.

Evaluating our distributed coding algorithm on the aboveproblem gave a bin rate of R = 0.66 in the second view.Figure 5 displays the result of nearest-neighbor instance-

5 10 150.4

0.5

0.6

0.7

0.8

0.9

1

Num. Nearest Neighbors

Ac

cu

rac

y

COIL!100 instance level retrieval

one view

two views

dist. coded

indep. coded

2 4 6 8 10

0.35

0.4

0.45

0.5

0.55

0.6

0.65

Num. Nearest Neighbors

Ac

cu

rac

y

COIL!100 instance level recognition

one view

two views

dist. coded

indep. coded

Figure 7. Nearest-neighbor (top) retrieval and (bottom) recogni-tion with two-views on COIL-100. Our algorithm performs signif-icantly better over single view performance under each task whileachieving a very low encoding rate. For the retrieval task, our ap-proach performs near multi-view performance. The independentencoding baseline is also shown, where independent feature selec-tion was performed at the same rate as our algorithm. Note thatindependent encoding with two views does worse than a singleview when operating at such a low encoding rate.

level retrieval over each of the 100 instances in the trainingset for varying neighborhood sizes. The average retrievalaccuracy, averaged over 10 independent trials, is shown forboth distributed and independent coding of the second view,where for independent coding features were selected at thesame rate as distributed coding. Distributed coding far out-performs independent encoding in the above scenario.

We also considered the case of partial redundancy, wherethe redundant bins are only partially correlated as a result ofnoise. To simulate partial redundancy we added zero meanGaussian noise to the split ratios of the first 4 bins withσ = {0, 0.01, 0.05, 0.1}. Figure 6 displays the result ofnearest-neighbor recognition with distributed and indepen-dent coding of the second view. In the plot, recognition per-formance is reported, averaged across the differentσ values,along with error bars indicating the standard deviation. Forthis experiment, an average bin rate ofR = 0.78±0.23 wasachieved with our distributed feature selection algorithm.Our distributed coding algorithm can perform favorably toindependent encoding even when the bins are only partiallyredundant.

5.2. COIL-100 ExperimentsWe evaluated our distributed feature selection algorithm

using the COIL-100 multi-view object database [11] thatconsists of 72 views of 100 objects viewed from 0 to 360

degrees in 5 degree increments. A local feature representa-tion is computed for each image using 10 dimensional PCA-SIFT features [6] extracted on a regular grid using a 4 pixelspacing. We evaluate our distributed coding algorithm andperform recognition with the COIL-100 dataset using multi-resolution vocabulary-guided histograms [4] computed withLIBPMK [7]. We split the COIL-100 dataset into train andtest sets by taking alternating views of each object. We thenpaired images 50 degrees apart to form the two views of ourproblem.

Using the training image features we perform hierarchi-cal k-means clustering to compute the vocabulary used toform the multi-resolution pyramid representation. Using 4levels and a tree branch factor of 10 gave a 991 word vo-cabulary at the finest level of the hierarchy. GP distributedfeature selection is performed over the finest level of the his-togram, such that the encoders and decoder only communi-cate bins at this level. The upper levels of the tree are thenrecomputed from the bottom level when performing recog-nition. To perform GP distributed coding we used a kerneldefined using L2 distance over a coarse, flat histogram rep-resentation.

Figure 7 displays nearest-neighbor retrieval and recog-nition accuracy using one and two views. A significantperformance increase is achieved by using the second viewwhen there are no bandwidth constraints. Applying GP dis-tributed feature selection on the above dataset resulted in abin rate of R < 0.01 in the second view; this is a compres-sion rate of over 100:1. Figure 7 displays the performanceof our GP distributed feature selection algorithm. By ex-ploiting feature redundancy across views, our algorithm isable to perform significantly better than single view perfor-mance while achieving a very low encoding rate. The re-sult of independent encoding is also shown in the Figure,where independent feature selection was performed at thesame rate as our algorithm. In contrast to our approach, in-dependent encoding is not able to improve over single-viewperformance and in fact does worse at such low encodingrates.

We also tested our approach over different encodingrates, where the desired rate is provided as input to the algo-rithm. Figure 8 displays the nearest-neighbor performanceof our approach over different encoding rates. As expected,nearest-neighbor performance increases for larger encodingrates. Performance saturates at about r = 50 bins and re-mains fairly constant for larger rates. Of coarse, for r = Bone would expect to recover ground-truth performance. Theslow convergence rate of our approach to ground-truth per-formance with increasing encoding rate suggests the needfor better bin selection criteria, which we plan to inves-tigate as part of future work. The independent encodingbaseline is also shown. Recall that at rateR the baseline ap-proach transmits twice the number of bins as our approach

10 100 200 300 400 5000

0.05

0.1

0.15

0.2

# of Bins Requested (|ξ2|)

Av

era

ge

Ac

cu

rac

y

COIL!100 instance level retrieval

dist. coded

indep. coded

50

10 100 200 300 400 5000

0.05

0.1

0.15

0.2

# of Bins Requested (|ξ2|)

Ave

rag

e A

ccu

racy

COIL−100 instance level recognition

50

dist. codedindep. coded

Figure 8. Nearest-neighbor performance increases with encodingrate. Nearest-neighbor performance is shown for the tasks of (top)retrieval and (bottom) recognition. The accuracy difference be-tween our approach and ground-truth two-view performance isshown averaged over neighborhood size; error bars indicate ±1

std. deviation. The independent encoding baseline is also shown.

as a result of the request operation. Independent encodingneeds to transmit nearly the entire histogram (|ξ2| = 400)before reaching a recognition performance close to our ap-proach. Our approach achieves similar performance withonly |ξ2| = 10.

6. Conclusion and Future WorkIn this paper we presented a distributed coding method

for unsupervised distributed feature selection and showedits application to multi-view object recognition. We devel-oped a new algorithm for distributed coding with GaussianProcesses and demonstrated its effectiveness for encodingvisual word feature histograms on both synthetic and real-world datasets. For a two-view problem with COIL-100,our algorithm was able to achieve a compression rate of over100:1 in the second view, while significantly increasing ac-curacy over single-view performance. At the same cod-ing rate, independent encoding was unable to improve overrecognition with a single-view. For future work, we plan toinvestigate techniques for modeling more complex depen-dencies as well as one-to-many mappings between viewsand evaluate our approach under different bin selection cri-teria.Acknowledgements

We gratefully acknowledge John Lee for his LIBPMKlibrary [7] and help with our experiments.

References[1] J. Dy and C. Brodley. Feature subset selection and order

identification for unsupervised learning. In ICML, 2000.

[2] V. Ferrari, T. Tuytelaars, and L. V. Gool. Integrating multiplemodel views for object recognition. CVPR, 2004.

[3] B. Girod, A. M. Aaron, S. Rane, and D. Rebollo-Monedero.Distributed video coding. Proceedings of the IEEE, Jan.2005.

[4] K. Grauman and T. Darrell. The pyramid match kernel: Ef-ficient learning with sets of features. JMLR, 2007.

[5] A. Kapoor, K. Grauman, R. Urtasun, and T. Darrell. Activelearning with gaussian processes for object categorization. InICCV, 2007.

[6] Y. Ke and R. Sukthankar. Pca-sift: a more distinctive repre-sentation for local image descriptors. CVPR, 2004.

[7] J. J. Lee. LIBPMK: A pyramid match toolkit. TechnicalReport MIT-CSAIL-TR-2008-017, MIT CSAIL, 2008.

[8] B. Leibe and B. Schiele. Scale-invariant object categoriza-tion using a scale-adaptive mean-shift search. In DAGM,2004.

[9] H. Liu and L. Yu. Feature selection for data mining. Techni-cal report, Arizona State University, 2002.

[10] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman,J. Matas, F. Schaffalitzky, T. Kadir, and L. V. Gool. A com-parison of affine region detectors. IJCV, 2005.

[11] S. A. Nene, S. K. Nayar, and H. Murase. Columbia objectimage library (coil-100). Technical report, Colmubia Uni-versity, 1996.

[12] D. Nister and H. Stewenius. Scalable recognition with a vo-cabulary tree. In CVPR, 2006.

[13] H. Peng, F. Long, and C. Ding. Feature selection basedon mutual information criteria of max-dependency, max-relevance, and min-redundancy. PAMI, 2005.

[14] S. S. Pradhan and K. Ramchandran. Distributed sourcecoding using syndromes (discus): design and construction.Transactions on Information Theory, 2003.

[15] C. E. Rasmussen and C. Williams. Gaussian Processes forMachine Learning. The MIT Press, 2006.

[16] F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce. 3dobject modeling and recognition using local affine-invariantimage descriptors and multi-view spatial constraints. IJCV,2006.

[17] S. Savarese and L. Fei-Fei. 3D generic object categorization,localization and pose estimation. In ICCV, 2007.

[18] D. Schonberg. Practical Distributed Source Coding and ItsApplication to the Compression of Encrypted Data. PhD the-sis, University of California, Berkeley, 2007.

[19] J. Sivic and A. Zisserman. Toward Category-Level ObjectRecognition, chapter Video Google: Efficient Visual Searchof Videos. Springer, 2006.

[20] D. Slepian and J. Wolf. Noiseless coding of correlated infor-mation sources. Transactions on Information Theory, 1973.

[21] A. Thomas, V. Ferrari, B. Leibe, T. Tuytelaars, B. Schiel,and L. V. Gool. Towards multi-view object class detection.In CVPR, 2006.

[22] A. Torralba, K. P. Murphy, and W. T. Freeman. Sharing vi-sual features for multiclass and multiview object detection.PAMI, 2007.