learning generative visual models from few training …...learning generative visual models from few...

www.elsevier.com/locate/cviu

Computer Vision and Image Understanding 106 (2007) 59–70

Learning generative visual models from few training examples:An incremental Bayesian approach tested on 101 object categories

Li Fei-Fei a,*, Rob Fergus b, Pietro Perona c

a Princeton University, 35 Olden St., Princeton, NJ 08540, USAb Oxford University, Parks Road, Oxford OX1 3PJ, UK

c California Institute of Technology, 136-93 Mail Code, Pasadena, CA 91125, USA

Received 15 August 2005; accepted 19 September 2005Available online 9 March 2007

Communicated by Arthur E. C. Pece

Abstract

Current computational approaches to learning visual object categories require thousands of training images, are slow, cannot learn inan incremental manner and cannot incorporate prior information into the learning process. In addition, no algorithm presented in theliterature has been tested on more than a handful of object categories. We present an method for learning object categories from just afew training images. It is quick and it uses prior information in a principled way. We test it on a dataset composed of images of objectsbelonging to 101 widely varied categories. Our proposed method is based on making use of prior information, assembled from (unre-lated) object categories which were previously learnt. A generative probabilistic model is used, which represents the shape and appear-ance of a constellation of features belonging to the object. The parameters of the model are learnt incrementally in a Bayesian manner.Our incremental algorithm is compared experimentally to an earlier batch Bayesian algorithm, as well as to one based on maximum like-lihood. The incremental and batch versions have comparable classification performance on small training sets, but incremental learningis significantly faster, making real-time learning feasible. Both Bayesian methods outperform maximum likelihood on small training sets.� 2006 Elsevier Inc. All rights reserved.

Keywords: Object recognition; Categorization; Generative model; Incremental learning; Bayesian model

1. Introduction

One of the most exciting and difficult open problems ofmachine vision is enabling a machine to recognize objectsand object categories in images. Significant progress hasbeen made on the issues of representation of objects[17,18] and object categories [2,3,5–8] with a broad agree-ment for models that are composed of ‘parts’ (texturedpatches, features) and ‘geometry’ (or mutual position ofthe parts). Much work remains to be done on the issueof category learning, i.e. estimating the model parametersthat are to be associated to a given category. Three diffi-culties face us at the moment. First, a human operator

1077-3142/$ - see front matter � 2006 Elsevier Inc. All rights reserved.

doi:10.1016/j.cviu.2005.09.012

* Corresponding author.E-mail addresses: [email protected] (L. Fei-Fei), fergus@ro-

bots.ox.ac.uk (R. Fergus), [email protected] (P. Perona).

must identify explicitly each category to be learned, whileit would be desirable to let a machine identify automati-cally each category from a broad collection of images.While Weber et al. [5] have demonstrated encouragingresults on identifying automatically three categories in alimited image database, in order to reach human perfor-mance one would like to see tens of thousands of catego-ries identified automatically from possibly millions ofimages [1]. Second, most algorithms require the imageof each exemplar object to be geometrically normalizedand aligned with a prototype (e.g. clicking on eyes, noseand mouse corners on face images). This is expensiveand tedious; furthermore, it is problematic when fiducialpoints are not readily identifiable (can we find a naturalalignment for images of octopus, of cappuccino machines,of human bodies in different poses?). Progress on thistopic has been recently reported by Weber et al. [5] who

mailto:[email protected]




a

b

Fig. 1. Schematic comparison between maximum likelihood algorithm[5,7,8] and the Bayesian learning algorithm [4]. Note the different numberof training images, the different learning algorithm and recognitioncriterion.

60 L. Fei-Fei et al. / Computer Vision and Image Understanding 106 (2007) 59–70

proposed and demonstrated a method for training in clut-ter without supervision (see also the follow-up paper byFergus et al. [8] proposing an improved scale-invariantalgorithm). The third challenge has to do with the sizeof the training set that is required: as many as 104 trainingexamples for some algorithms. This is not surprising: awell-known rule-of-thumb says that the number of train-ing examples has to be 5–10 times the number of objectparameters—hence the large training sets for models con-taining hundreds of parameters. Yet, humans are oftenable to learn new categories from a much smaller trainingset (how many cell-phones did we need to see in order tolearn to recognize one?). On this issue Fei-Fei et al. [4]recently proposed a Bayesian framework to use priorsderived from previously learned classes in order to speedup learning of a new class. In a limited set of experimentson four categories they showed that 1–3 training examplesare sufficient to learn a new category. Their method isbatch, in that the training examples need to be consideredsimultaneously.

In this paper, we explore the third issue further. First,we argue that batch learning is an undesirable limitation.An organism, or a machine, exploring the environmentshould not be required to store explicitly the set of trainingimages that it has encountered so far. The current best esti-mate of a given class should be a sufficient memory of pastexperience. To this end we develop an incremental Bayes-ian algorithm and we study the performance/memorytrade-off. Second, we collected a training set of 101 catego-ries and we assess the new incremental Bayesian algorithmagainst the batch Bayesian algorithm of Fei-Fei et al. andagainst the maximum likelihood method of Fergus et al.We wish to emphasize at this point that previous workon object categories has been tested for the most part on1 or 2 categories [3] with the exception of Weber et al. [5]who tested on four and Fergus et al. [8] who tested onsix. We consider the effort to collect and test on datasetthat is 15 times larger one of the major contributions of thispaper.

In Section 2 we outline the generative framework to rep-resent object classes. Although some of the details can befound in [4], we reproduce them here for the sake of clarity.In Section 3, we show how to apply variational inference tolearn the parameters of the constellation model. In [4], abatch learning algorithm was developed for the sameframework. Here we extend this into an incrementalmethod in which the algorithm is only given a single train-ing image at a time during the learning stage, a processmuch more natural for living organisms. As an overview,in Fig. 1 we schematically compare and contrast the struc-ture of the Bayesian algorithm in learning and recognitionwith the Maximum likelihood algorithm used by [5,7,8].We then discuss in details the experimental setup, withemphasis on our dataset of 101 object categories in Section4. Finally in Section 5 we present experimental results onthe 101 object categories. We conclude this paper with adiscussion in Section 6.

2. The generative model

To illustrate the generative model, we start with a learntobject class model and its corresponding model distribu-tion p(h), where h is a set of model parameters for the dis-tribution. We are then presented with a new image and wemust decide if it contains an instance of our object class ornot. In this query image we have identified N interestingfeatures with locations X, and appearances A. We nowmake a Bayesian decision, R. For clarity, we explicitlyexpress training images through the detected feature loca-tions Xt and appearances At.

R ¼ pðObjectjX;A;Xt;AtÞpðNo ObjectjX;A;Xt;AtÞ

ð1Þ

¼ pðX;AjXt;At;ObjectÞpðObjectÞpðX;AjXt;At;NoobjectÞpðNo ObjectÞ ð2Þ

�R

pðX;Ajh;ObjectÞpðhjXt;At;ObjectÞdhRpðX;Ajhbg;No ObjectÞpðhbgjXt;At;No ObjectÞdhbg

ð3Þ

Note the ratio of pðObjectÞpðNo ObjectÞ in Eq. (2) is usually set man-

ually to 1, hence omitted in Eq. (3).

2.1. The constellation model

Our chosen representation for object categories is basedon the constellation model introduced by Burl [7] and devel-oped further by Weber et al. [5] and Fergus et al. [8]. A con-stellation model consists of a number of parts, eachencoding information on both the shape and appearance.The appearance of each part is modeled and the shape ofthe object is represented by the mutual position of the parts[8]. The entire model is generative and probabilistic, soappearance and shape are all modeled by probability den-sity functions, which are Gaussians. The model is bestexplained by first considering recognition. We have learned

Fig. 2. A graphical model representation of the constellation model. Notethat we assume only one mixture component in this representation.

L. Fei-Fei et al. / Computer Vision and Image Understanding 106 (2007) 59–70 61

a generative object model, with P parts and a posterior dis-tribution on the parameters h: pðhjXt;AtÞ where Xt and At

are the location and appearances of interesting featuresfound in the training data. We assume that all non-objectimages can also be modeled by a background with a singleset of parameters hbg which are fixed. The ratio of the pri-ors may be estimated from the training set or set manually(usually to 1). Our decision then requires the calculation ofthe ratio of the two likelihood functions. In order to dothis, the likelihoods may be factored as follows:

pðX;AjhÞ ¼Xh2H

pðX;A; hjhÞ

¼Xh2H

pðAjh; hÞ|fflfflfflfflfflffl{zfflfflfflfflfflffl}Appearance

pðXjh; hÞ|fflfflfflfflffl{zfflfflfflfflffl}Shape

ð4Þ

Since our model only has P (typically 3–7) parts butthere are N (up to 100) features in the image, we introducean indexing variable h which we call a hypothesis whichallocates each image feature either to an object or to thebackground.

2.1.1. Appearance

Each feature’s appearance is represented as a point insome appearance space. Each part p has a Gaussian density(denoted by G) within this space, with mean and precisionparameters hA

p ¼ flAp ;C

Ap g which is independent of other

parts’ densities.

2.1.2. ShapeThe shape is represented by a joint Gaussian density of

the locations of features within a hypothesis. For eachhypothesis, the coordinates of all parts are subtracted offfrom the left most part coordinates. Additionally, it is scaleis used to normalize the constellation. This enables ourmodel to achieve scale and translational invariance. Thedensity has parameters hX ¼ flX;CXg.

2.2. Model distribution

Let us consider a mixture model of constellation modelswith X components. Each component x has a mixing coef-ficient px; a mean of shape and appearance lX

x ; lAx ; a pre-

cision matrix of shape and appearance CXx ;C

Ax . The X and

A superscripts denote shape and appearance terms respec-tively. Collecting all mixture components and their corre-sponding parameters together, we obtain an overallparameter vector h ¼ fp; lX; lA;CX;CAg. Assuming wehave now learnt the model distribution pðhjXt;AtÞ froma set of training data Xt and At, we define the model dis-tribution in the following way

pðhjXt;AtÞ ¼ pðpÞYx

pðCXx ÞpðlX

x jCXxÞpðCA

x ÞpðlAx jCA

x Þ ð5Þ

where the mixing component is a symmetric Dirichlet:pðpÞ ¼ DirðkxIXÞ, the distribution over the shape preci-sions is a Wishart pðCX

xÞ ¼WðCXx jaX

x ;BXx Þ and the distribu-

tion over the shape mean conditioned on the precisionmatrix is Normal: pðlX

x jCXx Þ ¼ GðlX

x jmXx ; b

XxCX

x Þ. Togetherthe shape distribution pðlX

x ;CXx Þ is a Normal-Wishart den-

sity [9,13]. Note that the set of variables {kx,ax,Bx,mx,bx} are hyper-parameters for defining their correspondingdistributions of model parameters. Identical expressionsapply to the appearance component in Eq. (5). A graphicalmodel representation is shown in Fig. 2, where only onemixture component is used.

2.3. Bayesian decision

Recall that for the query image we wish to calculate the ratioof pðObjectjX;A;Xt;AtÞ and pðNo ObjectjX;A;Xt;AtÞ.It is reasonable to assume a fixed value for all modelparameters when the object is not present, hence thelatter term may be calculated once for all. For the formerterm, we use Bayes’s rule to obtain the likelihoodexpression: pðX;AjXt;At;ObjectÞ which expands toR

pðX;AjhÞpðhjXt;AtÞdh. Since the likelihood pðX;AjhÞcontains Gaussian densities and the parameter posterior,pðhjXt;AtÞ is its conjugate density (a Normal-Wishart)the integral has a closed form solution of a multivariateStudent’s T distribution (denoted by S):

pðX;AjXt;At;ObjectÞ ¼XX

x¼1

XjHnj

h¼1

~pxS XhjgXx ;m

Xx ;K

Xx

� �

�S AhjgAx ;m

Ax ;K

Ax

� �ð6Þ

gx ¼ ax þ 1� d

Kx ¼bx þ 1

bxgx

Bx

~px ¼kxPx0kx0

Note d is the dimensionality of the parameter vector. Ifthe ratio of posteriors, R in Eq. (3), calculated using thelikelihood expression above exceeds a pre-defined thresh-old, then the image is assumed to contain an occurrenceof the learnt object category.


3. Learning the generative model: batch vs. incremental

The task in learning is to estimate the densitypðhjXt;AtÞ. This is done using the Variational Bayes pro-cedure [9–11]. It approximates the posterior distributionpðhjXt;AtÞ by q(h,x,h). x is the mixture component labeland h is the hypothesis. Using Bayes’ rule:qðh;x; hÞ � pðhjXt;AtÞ / pðXt;AtjhÞpðhÞ. The likelihoodterms use Gaussian densities and by assuming priors of aconjugate form, in this case a Normal-Wishart, our poster-ior q-function is also a Normal-Wishart density. The vari-ational Bayes procedure is a variant of EM whichiteratively updates the hyper-parameters and latent vari-ables to monotonically reduces the Kullback-Liebler dis-tance between pðhjXt;AtÞ and q(h,x,h). Using thisapproach allows us to incorporate prior information in asystematic way and is far more powerful that a maximumlikelihood approach used in [8]. We first briefly give anoverview of the algorithm [4], based on [9], which is a batchlearning algorithm. Then we introduce the new incrementalversion of the algorithm.

3.1. Batch learning

There are two stages to learning: an E-step where theresponsibilities of the hidden variables are calculated andan M-step where we update the hyper-parameters ofq(h,x,h), H = {k,m,b,a,B}. For each image, n we calculatethe responsibilities:

~cnx;h ¼ ~px~cxðXn

hÞ~cxðAnhÞ ð7Þ

using the update rules given in [9]. The hyper-parametersare updated from these responsibilities. This is done bycomputing the sufficient statistics. While the update rulesfor the shape components are shown, they are of the sameform for the appearance terms. The sufficient statistics, formixture component x are calculated as follows:

�px ¼1

N

XN

n¼1

XjHnj

h¼1

cnx;h and �Nx ¼ N �px ð8Þ

�lXx ¼

1�Nx

XN

n¼1

XjHnj

h¼1

cnx;hXn

h and ð9Þ

RXx ¼

1�Nx

XN

n¼1

XjHnj

h¼1

cnx;hðXn

h � lXx ÞðXn

h � lXx Þ

T ð10Þ

Note that to compute these, we need the responsibilitiesfrom across all images. From these we can update thehyper-parameters (update rules are in [9]).

3.2. Incremental learning

We now give an incremental version of the update rules,based on Neal and Hinton’s adaptation of conventionalEM [16]. Let us assume that we have a model withhyper-parameters H = {k,m,b,a,B}, estimated using M

previous images (M P 0) and we have N new images(N P 1) with which we wish to update the model. Fromthe M previous images, we have retained sufficient statisticspe

x; lex;R

ex for each mixture component x. We then com-

pute the responsibilities for the new images, i.e. cnx;h for

n = 1, . . . ,N and from them, the sufficient statistics,�px; lx;Rx using Eqs. (8) and (10). In the Incremental M-step we then combine the sufficient statistics from thesenew images with the existing set of sufficient statistics fromthe previous M images. Then the overall sufficient statistics,p̂x; l̂x; R̂x are computed:

p̂x ¼Mpe

x þ N �px

M þ Nð11Þ

l̂x ¼Mle

x þ N �lx

M þ Nð12Þ

R̂x ¼MRe

x þ N �Rx

M þ Nð13Þ

From these we can then update the model hyper-pa-rameters. Note the existing sufficient statistics are notupdated within the update loop. When the model con-verges, the final value of the sufficient statistics fromthe new images are combined with the existing set, readyfor the next update: pe

x ¼ p̂x; lex ¼ l̂x;R

ex ¼ R̂x. Initially

M = 0, so pex; l

ex;R

ex drop from our equations and our

model hyper-parameters are set randomly (within somesensible range).

4. Methods

4.1. 101 object categories

We test our Bayesian algorithms (Incremental andBatch) using 101 assorted object categories. The namesof 101 categories were generated by flipping through thepages of the Webster Collegiate Dictionary [12], pickinga subset of categories that were associated with a drawing.After we generated the list of category names, we usedGoogle Image Search engine to collect as many imagesas possible for each category. Two graduate studentsnot associated with the experiment then sorted througheach category, mostly getting rid of irrelevant images(e.g. a zebra-patterned shirt for the ‘‘zebra’’ category).Fig. 9 shows examples of both the 101 foreground objectcategories as well as the background clutter category.Minimal preprocessing was performed on the categories.Categories such as motorbike, airplane, cannon, etc.where two mirror image views were present, were manuallyflipped, so all instances faced in the same direction.Additionally, categories with a predominantly verticalstructure were rotated to an arbitrary angle, as the modelparts are ordered by their x-coordinate, so have troublewith vertical structures. One could also avoid rotating theimage by choosing the y-coordinate as ordering reference.This rotation is used for the sake of programming simplic-ity. At last, images were scaled roughly to around 300 pixelswide.


4.2. Feature detection

Feature points are found using the detector of Kadirand Brady [14]. This method finds regions that are salientover both location and scale. Gray-scale images are usedas the input. The most salient regions are clustered overlocation and scale to give a reasonable number of featuresper image, each with an associated scale. The coordinatesof the center of each feature give us X. Once the regionsare identified, they are cropped from the image andrescaled to the size of a small (11 · 11) pixel patch. Eachpatch exists in a 121 dimensional space. We then reducethis dimensionality by using PCA [8]. A fixed PCA basis,pre-calculated from the background datasets, is used forthis task, which gives us the first 10 principal componentsfrom each patch. The principal components from allpatches and images form A. Note that the same set ofparameters were used for feature detection for all 101object categories. Figs. 4a and 5a show examples of featuredetection on some training images.

4.3. Prior and initialization

One critical issue is the choice of priors for the Normal-Wishart distributions. In this paper, learning is performedusing a single mixture component. The choice of prior isitself a topic worth full investigation. In order to keep theconsistency and prove the concept, we use here a single pri-or distribution for all 101 object categories. Since priorknowledge should reflect some information of the real-world object categories, we estimated this prior distributionusing well-learned maximum likelihood (ML) models offaces, spotted cats and airplanes in [8]. It is important topoint out that the idea of using prior information fromunrelated categories was proposed independently by Milleret al. [15]. The ML models are simply averaged together foreach parameter to obtain a model distribution for the

100 80 60 40 20 0 20 40 60

30

20

10

0

10

20

30

40

x position (pixel)

y po

sitio

n (p

ixel

)

a

Fig. 3. (a and b) Prior distribution for shape mean (lX) and appearance meanestimated from models learned with maximum likelihood methods, using ‘‘thepriors are displayed. All four parts of the appearance begin with the same pri

prior. Fig. 3a and b shows the prior shape and appearancemodel for each category.

Initial conditions are chosen in exactly the same way as[4]. Again, the same initialization is used for all 101 objectcategories.

4.4. Experimental setup for each category

Each experiment was carried out under identical condi-tions. For each category dataset, N training images aredrawn randomly first. Then 50 testing images are randomlydrawn from the remaining images in the dataset. For somedataset, less than 50 images are left after training imagesare drawn. In this case we use all the remaining ones astesting images. We then learn models using both Bayesianand ML approaches and evaluate their performance on thetest set. For evaluation purposes, we also use 50 imagesfrom a background dataset of assorted junk images fromthe Internet. For each category, we vary N at 0,1,3,6,15,repeating the experiments 10 times for each value (usinga randomly chosen N training images each time) to obtaina more robust estimate of performance. When N = 0, weuse the prior model alone to perform object categorizationwithout any training data. Only the Bayesian algorithm isused in this case. In addition, when N = 1, ML fails to con-verge, so we only show results for the Bayesian methods inthis case.

When evaluating the models, the decision is a simpleobject present/absent one. Under these conditions, an algo-rithm performing at random has a 50% performance rate.All performance values are quoted as area under the receiv-er-operating characteristic (ROC). ROC curve is obtainedby testing the model on 50 foreground test images and 50background images. In all the experiments, the followingparameters are used: number of parts in model = 4; num-ber of PCA dimensions for each part appearance = 10;and average number of detections of interest point for each

0.5

2

0.5

2

3

1

-1 10

-1 10

-1 10

PCA coefficient

PC

A c

oeffi

cien

t den

sity

b

(lA) for all the categories to be learned. Each prior’s hyper-parameters areother’’ datasets [8]. Only the first three PCA dimensions of the appearanceor distribution for each PCA dimension.

45 55 65 75 85 9545

55

65

75

85

95

Prior Performance (percent correct)

Per

form

ance

(pe

rcen

t cor

rect

)

Performance Comparison: Train Number = 1

Bayesian BatchBayesian Incremental

Maximum LikelihoodBayesian BatchBayesian Incremental

45 55 65 75 85 9545

55

65

75

85

95


Per

form

ance

(pe

rcen

t cor

rect

)



45 55 65 75 85 9545

55

65

75

85

95


Per

form

ance

(pe

rcen

t cor

rect

)



45 55 65 75 85 9545

55

65

75

85

95


Per

form

ance

(pe

rcen

t cor

rect

)Performance Comparison: Train Number = 15

1. trilobite2. face3. pagoda4. tick5. inlineskate6. metronome7. accordion8. yinyang9. soccerball10. spotted cat11. nautilus12. grand-piano13. crayfish14. headphone15. hawksbill16. ferry17. cougar-face

18. bass19. ketch20. lobster21. pyramid22. rooster23. laptop24. waterlilly25. wrench26. strawberry27. starfish28. ceilingfan29. seahorse30. stapler31. stop-sign32. zebra33. brontosaurus34. emu

35. snoopy36. okapi37. schooner38. binocular39. motorbike40. hedgehog41. garfield42. airplane43. umbrella44. panda45. crocodile-head46. llama47. windsor-chair48. car-side49. pizza50. minaret51. dollarbill

52. gerenuk53. sunflower54. rhino55. cougar-body56. crab57. ibis58. helicopter59. dalmatian60. scorpion61. revolver62. beaver63. saxophone64. kangaroo65. euphonium66. flamingo67. flamingo-head68. elephant

69. cellphone70. gramophone71. bonsai72. lotus73. cannon74. wheel-chair75. dolphin76. stegosaurus77. brain78. menorah79. chandelier80. camera81. ant82. scissors83. butterfly84. wildcat85. crocodile

86. barrel87. joshua-tree88. pigeon89. watch90. dragonfly91. mayfly92. cup93. ewer94. octopus95. platypus96. buddha97. chair98. anchor99. mandolin100. electric-guitar101. lamp

a b

c

e

d

Fig. 4. (a–d) Performance comparison between ML, Bayesian batch and Bayesian incremental methods for all 101 object categories at TrainingNumber = (1,3,6,15). In each plot, a category has three markers: red-circle represents Bayesian incremental method, green-plus Bayesian batch methodand blue-diamond maximum likelihood method. For each panel, the x-axis indicates Bayesian method categorization performance with only the priormodel. Note with one training image, the Bayesian incremental method and Batch method are exactly the same. The y-axis indicates categorizationperformance for each of the three methods. There is no maximum likelihood performance in the degenerate case of Training Number = 1. Moreover,Bayesian Incremental method and Batch method are also the same for Training Number = 1. (e) Category name list is sorted according to the BayesianIncremental method performance at Training Number = 15.



image = 20. All parameters remain the same for learningdifferent categories of objects.

5. Experimental results

Fig. 4 illustrates the recognition performance at Train-ing Number = 1,3,6,15 for the different algorithms: Bayes-ian incremental, Bayesian batch and maximum likelihood.The performance of each method is compared with the per-formance of the prior-model on the given category. Despitethe rudimentary prior, a strong performance gain isobserved for the majority of the 101 categories, even whenonly a few training examples are used. With one trainingexample, the Bayesian method achieves an average perfor-mance of 71%, with the best results being over 80%. AtTraining Number = 3, maximum likelihood has a perfor-mance at chance while both Bayesian methods achieve aperformance near 75%. At Training Number = 15 thatthe maximum likelihood catches up the recognition perfor-mance with the Bayesian methods. At Training Num-ber = 15, the average Bayesian incremental method is notas reliable as the Bayesian batch method. In general theincremental method is much more sensitive to the qualityof the training images and to the order in with they are pre-sented to the algorithm. Therefore it is more likely to formsuboptimal models.

While performance is a key measurement for recogni-tion algorithms, efficiency in training and learning are alsoimportant. One big advantage that we have gained fromthe Bayesian Incremental method is its fast speed in train-ing. Fig. 5 shows the comparison of average learning timeacross all 101 categories between these three methods. Allmethods show approximately linear increase in learningtime as the number of training images increases. The incre-mental method, particularly, shows a very small slope. Inour Matlab implementation, it takes approximately 6 sper image to train in Bayesian incremental method, whilethe batch and maximum likelihood methods take 6 timesas long.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

100

200

300

400

500

600

Training Number

Lear

ning

Tim

e (s

econ

ds)

Learning speed comparison among different learning methods


Fig. 5. Average learning time for ML, Bayesian batch and Bayesianincremental methods over all 101 categories.

In Figs. 6 and 7 we show in details the results from thegrand-piano and cougar-face categories, both of whichperformed well (ROC Areas 16% and 15%, respectively,for Training Number = 15). As the number of trainingexamples increases, we observe that the shape model ismore defined and structured with reducing variance. Thisis expected since the algorithm should be more and moreconfident of what is to be learned. Figs. 6c and 7c showsexamples of the part appearance that are closest to themean distribution of the appearance. Notice that criticalfeatures such as keyboards for the piano and eyes orwhiskers for the cougar-face are successfully learned bythe algorithm. Three learning methods’ performances arecompared in Figs. 6d and 7d. The Bayesian methodsclearly show a big advantage over the ML method whentraining number is small. Bayesian Incremental, however,shows more greater performance fluctuations as comparedto the Bayesian Batch method. Finally, we show someclassified test images, using an incremental model trainedfrom a single image.

It is also useful to look at the other end of the perfor-mance spectrum—those categories that have low recogni-tion performance. We give some insights into the causeof the poor performance.

Feature detection is a crucial step for both learning andrecognition. On both the crocodile and mayfly figures inFig. 8, notice that some testing images marked ‘‘INCOR-RECT’’ have few detection points on the target objectitself. When feature detection fails either in learning or rec-ognition, it affects the performance results greatly. Further-more, Fig. 8a shows that a variety of viewpoints is presentin each category. In this set of experiments we have onlyused one mixture component, hence only a single viewpointcan be accommodated. Our model is also a simplified ver-sion Burl, Weber and Fergus’ constellation model [5,7,8] asit ignores the possibility of occluded parts.

A great source of improvement could potentially comefrom prior model information. The prior model is currentlyrather weak and improvements in this area would undoubt-edly improve the models performance. However, the per-formance could be degraded if the model was toincorporate misleading information—as illustrated inFig. 8b. Our choice of prior for this paper is kept as simpleas possible to facilitate the experiments. We expect furtherexploration into this topic can help improving recognitionperformances greatly.

6. Summary and discussion

We presented a Bayesian incremental algorithm forlearning generative models of object categories from a verylimited training set. Our work is an extension of Fei-Feiet al.’s Bayesian batch method [4]. We have tested bothmethods using a prior derived from three unrelated catego-ries, alongside a simplified version of Fergus et al.’s maxi-mum likelihood method, on a large dataset of 101 objectcategories.

Examples Shape Appearance

100 50 0 50 100

50

0

50

Shape Model: Train = 1 Part 1

Part 2

Part 3

Part 4

100 50 0 50 100

50

0

50


Part 2

Part 3

Part 4

100 50 0 50 100

50

0

50


Part 2

Part 3

Part 4

100 50 0 50 100

50

0

50


Part 2

Part 3

Part 4

0 2 4 6 8 10 12 140

10

20

30

40

50

Training Number

Per

form

ance

Err

or

M.L.Bayesian Batch

Performance Comparison

Bayesian Incr

INCORRECT Correct Correct Correct

Correct Correct Correct INCORRECT

Correct Correct Correct Correct

a b c

d e

Fig. 6. Results for the ‘‘grand-piano’’ category. (a) Examples of feature detection. (b) The shape models learned at Training Number = (1,3,6,15).Similarly to Fig. 3a, the x-axis represents the x position, measured by pixels, and the y-axis represents the y position, measured by pixels. (c) Theappearance patches for the model learned at Training Number = (1,3,6,15). (d) The comparative results between ML, Bayesian batch and Bayesianincremental methods (the error bars show the variation over the 10 runs). (e) Recognition result for the incremental method at Training Number = 1. Pinkdots indicate the center of detected interest points.


First of all, it is possible to train complex models ofobject categories with a handful of images. It is clear thatBayesian methods allow category learning from small

training sets. On one, three and six training examples themaximum likelihood method is unable to discriminateany category from images containing random clutter. By

Examples Shape Appearance

100 50 0 50 100

50

0

50


Part 2

Part 3

Part 4

100 50 0 50 100

50

0

50


Part 2

Part 3

Part 4

100 50 0 50 100

50

0

50

100

Shape Model: Train = 6

100 50 0 50 100

50

0

50


Part 2

Part 3

Part 4

Part 1

Part 2

Part 3

Part 4

0 2 4 6 8 10 12 140

10

20

30

40

50

Training Number

Per

form

ance

Err

or

M.L.Bayesian Batch


Bayesian Incr



Correct Correct INCORRECT Correct

a

d e

b c

Fig. 7. Results for the ‘‘cougar face’’ category.


contrast the Bayesian methods achieve better than chanceperformance on 90 higher than 80.

Second, the desirable incremental learning feature leadsto much faster learning speed (Fig. 3), but is paid withworse recognition performance for larger training set sizes.We conjecture that this is due to the fact that less informa-tion is carried along by the incremental algorithm from onelearning epoch to the next, while the batch algorithm has

all training images available at the same time thus allowing,for example, to test a larger number of hypotheses on howforeground and clutter features should be assigned in eachtraining image.

Third, the maximum likelihood method matches theperformance of the Bayesian methods when the trainingset reaches size 15. This is surprising, given that the numberof parameters in each model is 50, and therefore a few

examples

Shape

100 50 0 50 100

50

0

50

Shape Model: Train – 1

100 50 0 50 100

50

0

50

100


100 50 0 50 100

50

0

50

100


100 50 0 50 100

50

0

50


M.L.Bayesian Batch


Bayesian Incr

Training Number

10

20

30

40

50

Per

form

ance

Err

or

INCORRECT INCORRECT Correct Correct

INCORRECT Correct INCORRECT INCORRECT


examples

Shape

100 50 0 50 100

50

0

50


100 50 0 50 100

50

0

50


100 50 0 50 100

50

0

50


100 50 0 50 100

50

0

50


0 2 4 6 8 10 12 14

0 2 4 6 8 10 12 14

Training Number

10

20

30

40

50

Per

form

ance

Err

or

M.L.Bayesian Batch


Bayesian Incr

Correct INCORRECT INCORRECT Correct

Correct Correct INCORRECT Correct

INCORRECT Correct INCORRECT INCORRECT

a

b

Fig. 8. Two categories with unsatisfactory performance. (a) Crocodile (ROC area = 26%). (b) Mayfly. (ROC area = 27%).


Fig. 9. The 101 object categories and the background clutter category. Each category contains between 45 and 400 images. Two randomly chosen samplesare shown for each category. The categories were selected prior to the experimentation, collected by operators not associated with the experiment, and nocategory was excluded from the experiments. The last row shows examples from the background dataset. This dataset is obtained by collecting imagesthrough the Google image search engine (www.google.com). The keyword ‘‘things’’ is used to obtain hundreds of random images. Note only gray-scaleinformation is used in our system. Complete datasets can be found at http://vision.caltech.edu.


http://www.google.com

http://vision.caltech.edu


hundred training examples are in principle required by amaximum likelihood method—one might have expectedthe Bayesian methods to be bettered by ML only around100 training examples. The most likely reason for thisresult is that the prior that we employ is too simple. Bayes-ian methods live and die by the quality of the prior that isused. Our prior density is derived from only three objectcategories. Given the variability of our training set, it isrealistic that we would need many more categories to traina reasonable prior.

Fourth, the good news is that the problem of recog-nizing automatically hundreds, perhaps thousands, ofobject categories does not belong to a hopelessly farfuture. We hope that the success of our method on thelarge majority of 101 very diverse and challenging cate-gories, despite the simplicity of our implementation andthe rudimentary prior we employ, will encourage othervision researchers to test their algorithms on larger andmore diverse datasets.

Acknowledgment

The authors thank Marc’Aurelio Ranzato and MarcoAndreetto for their help in image collection.

References

[1] I. Biederman, Recognition-by-components: a theory of human imageunderstanding, Psychol. Rev. 94 (1987) 115–147.

[2] Y. Amit, D. Geman, A computational model for visual selection,Neural Comput. 11 (7) (1999) 1691–1715.

[3] H. Schneiderman, T. Kanade, A statistical approach to 3D objectdetection applied to faces and cars, in: Proc. Conf. on ComputerVision Pattern and Recognition, 2000, pp. 746–751.

[4] L. Fei-Fei, R. Fergus, P. Perona, A Bayesian approach to unsuper-vised learning of object categories, in: Proc. Int. Conf. on ComputerVision, 2003, pp. 1134–1141.

[5] M. Weber, M. Welling, P. Perona, Unsupervised learning of modelsfor recognition, in: Proc. 6th Europ. Conf. on Computer Vision, vol.2, 2000, pp. 101–108.

[6] P. Viola, M. Jones, Rapid object detection using a boosted cascade ofsimple features, in: Proc. Conf. on Computer Vision Pattern andRecognition, vol. 1, 2001, pp. 511–518.

[7] M.C. Burl, M. Weber, P. Perona, A probabilistic approach to objectrecognition using local photometry and global geometry, in: Proc.Europ. Conf. on Computer Vision, pp. 628–641.

[8] R. Fergus, P. Perona, A. Zisserman, Object class recognition byunsupervised scale-invariant learning, in: Proc. Conf. on ComputerVision Pattern and Recognition, vol. 2, 2003, pp. 264–271.

[9] H. Attias, Inferring parameters and structure of latent variablemodels by variational bayes, in: 15th Conf. on Uncertainty inArtificial Intelligence, 1999, pp. 21–30.

[10] M.I. Jordan, Z. Ghahramani, T.S. Jaakkola, L.K. Saul, Anintroduction to variational methods for graphical models, Mach.Learn. 37 (1999) 183–233.

[11] S.R. Waterhouse, D.J.C. MacKay, A.J. Robinson, Bayesian Methodsfor Mixtures of Experts, NIPS, 1996.

[12] Merriam-Webster’s collegiate dictionary, 10th ed. Springfield, Mas-sachusetts, USA, 1994.

[13] W.D. Penny, Variational Bayes for d-dimensional Gaussian mixturemodels, Tech. Rep., University College London, 2001.

[14] T. Kadir, M. Brady, Scale, saliency and image description, Int. J.Comp. Vis. 45 (2) (2001) 83–105.

[15] E. Miller, N. Matsakis, P. Viola, Learning from one example throughshared densities on transforms, in: Proc. Conf. on Computer VisionPattern and Recognition, 2000, pp. 464–471.

[16] R.M. Neal, G.E. Hinton, A view of the EM algorithm that justifiesincremental, sparse and other variants, in: M.I. Jordan (Ed.),Learning in Graphical Models, Kluwer academic press, Norwell,1998, pp. 355–368.

[17] D. Lowe, Object recognition from local scale-invariant features, in:Proc. Int. Conf. on Computer Vision, 1999, pp. 1150–1157.

[18] C. Schmid, R. Mohr, Local Greyvalue Invariants for Image Retrieval,IEEE Trans. Pattern Anal. Machine Intell. 19 (5) (1997) 530–534.

learning generative visual models from few training …...learning generative visual models from few...

Documents