FlowBoost – Appearance Learning from Sparsely Annotated Video

Karim Ali1,2 David Hasler2 Francois Fleuret3

1 Ecole Polytechnique Federale de Lausanne (EPFL), CVLAB, Switzerland2 Swiss Center for Electronic and Microtechnology (CSEM), Switzerland

3 Idiap Research Institute and Ecole Polytechnique Federale de Lausanne (EPFL), Switzerlandkarim.ali@epfl.ch, david.hasler@csem.ch, francois.fleuret@idiap.ch

Abstract

We propose a new learning method which exploits tem-poral consistency to successfully learn a complex appear-ance model from a sparsely labeled training video. Ourapproach consists in iteratively improving an appearance-based model built with a Boosting procedure, and the re-construction of trajectories corresponding to the motion ofmultiple targets. We demonstrate the efficiency of our pro-cedure on pedestrian detection in videos and cell detectionin microscopy image sequences. In both cases, our methodis demonstrated to reduce the labeling requirement by oneto two orders of magnitude. We show that in some instances,our method trained with sparse labels on a video sequenceis able to outperform a standard learning procedure trainedwith the fully labeled sequence.

1. Introduction

Many classes of objects can now be successfully de-tected with machine learning techniques. Faces, cars,pedestrians and hands, have all been detected with low er-ror rates by learning their appearance in a highly genericmanner from extensive training sets. These recent advanceshave enabled the use of reliable object detection compo-nents in real systems, such as automatic face focusing func-tions on digital cameras. One key drawback of these meth-ods, and the issue addressed here, is the prohibitive require-ment that training sets contain thousands of manually anno-tated examples.

We propose to reduce the requirement for such an ex-tensive labeling by exploiting the temporal consistency oc-curring in a training video. Our method, FlowBoost, starts

This work has been supported in part by the Swiss National Sci-ence Foundation under project 200021-117997, by the Fond Quebecois derecherche sur la nature et les technologies, and by the European Commu-nity’s Seventh Framework Programme FP7 - Challenge 2 - Cognitive Sys-tems, Interaction, Robotics - under grant agreement No 247022 - MASH.

from a sparse labeling of the video, and alternates the train-ing of an appearance-based detector with and a convex,multi-target, time-based regularization. The latter relabelsthe full training video in a manner that is both consistentwith the response of the current detector, and in accordancewith physical constraints on target motions.

The performance of this approach is evaluated on pedes-trian detection in a surveillance camera setting, and on celldetection in microscopy data. Our experiments show thatour method allows for a reduction in the number of trainingframes by a factor of 15 to 60. This comes with virtually noloss in performance when compared to a standard learningprocedure trained on a fully labeled sequence. In fact, insome cases, gains in performance are observed.

2. Related Work

The use of labeled and unlabeled data to learn a clas-sification model falls under the broad category of semi-supervised learning, initially introduced in [12]. Given theprominence of machine learning techniques, the cost asso-ciated with producing annotated data and the abundance ofunlabeled data, a growing number of works have addressedthis problem.

Broadly speaking, one can view the semi-supervisedlearning problem as a constrained instance of unsupervisedlearning, where the additional constraints come in the formof labeled data. This approach was taken in [11] where agenerative classification model for document classificationis used along with Expectation-Maximization. The algo-rithm first trains a naive Bayes classifier using the labeleddata and probabilistically labels the unlabeled data. It thenretrains a classifier using the most confident labels and theprocedure is iterated.

A number of other works rely on the assumption thatthe data lies on a low-dimensional manifold. As a conse-quence, the data is represented as a graph where verticesrepresent samples and edges-weights, pairwise similarity.Various methods which propagate labels to the entire graph

1433

until convergence are proposed in [15, 13, 3, 7]. These al-gorithms are naturally transductive and hence of limited ap-plicability to the object detection setting where an inductiveclassification rule is desired.

Still other works rely on the assumption that the deci-sion boundary must lie in a low-density region. In [14],a method to maximize the margin of both labeled and unla-beled sample, termed Transductive Support Vector Machine(TSVM), is introduced. However, the corresponding prob-lem is non-convex and therefore difficult to optimize. Oneapproach presented in [6] starts with an initial SVM solutionobtained from the labeled data alone. Next, the unlabeleddata points are labeled by SVM predictions, their weightsincreased and the SVM retrained. An alternative approachderived from a bayesian discriminative framework and in-volving a so-called Null Category Noise Model with Gaus-sian Processes is presented in [10].

In this paper, we exploit temporal consistency in videosto assist in the labeling of unlabeled data and iterativelyimprove an appearance based classifier. Given a trainingvideo, we begin by annotating a small subset of frameswhile the remaining frames are not annotated. This lim-ited initial training data is used to train an appearancebased classifier which is subsequently evaluated on the en-tire video sequence. Admissible trajectories of targets areretained as positive samples while the remaining data is re-tained as negative samples and the process is iterated.

Thus, our unlabeled data is in fact highly structured. Inparticular, once all trajectories are correctly identified, allremaining samples certainly belong to the negative class.From this perspective, our work is related to the trackingapproach of [8, 9] even though the goals differ significantly.In [8, 9], a system is built with three components: a tracker,a detector and a model. The tracker and detector are alwaysrun in parallel and the hypothesis from either that minimizesthe distance to the model is kept. The model itself is updatedby growing and pruning events which generate positive andnegative samples to update the model. The system is on-line, real-time and shows promising results.

One important difference with our work is that [8, 9] re-lies on the very stringent assumption that no more than onetarget is present in any given frame. Thus, once the hypoth-esis with maximal confidence is identified in a frame, all re-maining samples are considered negative. By contrast, werely on a global offline optimization, that is integrated intothe AdaBoost algorithm while allowing for an unknown andunconstrained number of targets as is next explained.

3. Methodology

We are interested in detection by classification. That is,given an image and a feature vector xs ∈ RD for every

Table 1. Notation

T = {1, . . . , T} time stepsS = {1, . . . ,W} × {1, . . . ,H} spatial locationsxt,s ∈ RD feature vector at location s in time frame tyt,s ∈ {−1, 0, 1} ground-truth labelhm : RD → {−1, 1} weak learnersH = span{h1, . . . , hM} the combinations of weak learnersϕ(x) ∈ H a strong classifierV = T × S vertices of the motion graphE ⊂ V × V edges of the motion graphF ⊂ {0, 1}V Boolean labelings of the graph consistent withmulti-target motionsLV(ϕ;y) exponential loss summed on verticesLE(ϕ; f) exponential loss summed on edgesΘ(f) signed labels on vertices corresponding to theBoolean labeling f of the edges.

location s in the image, we want to build a classifier

ϕ : RD → R

such that the set of locations

{s : ϕ(xs) ≥ 0}

forms a good prediction of the set of locations where thetargets (faces, cars, etc.) are actually visible. The standardtechnique consists in building a hand-labeled training set

(xn, yn) ∈ RD × {−1, 1}

and training a classifier ϕ with a low empirical error rate.As described in the introduction, we want to avoid the re-

quirement for a large training set by using a training video,and by going back-and-forth between the optimization ofan appearance-based classifier ϕ, and the optimization ofthe labels of non-labeled frames. The latter can be refor-mulated as the optimization of a Boolean flow f in a graphunder physically plausible motion constraints.

Let T = {1, . . . , T} be the set of time steps, and S ={1, . . . ,W} × {1, . . . ,H} the set of locations, where Wand H are respectively the width and heights of the videoframes1. Let

x ∈(RD)T ×S

,

be the feature vectors of dimension D computed in everytime frame t and at every location s in the training sequence(see § 4.1 for details), and let

y ∈ {−1, 0, 1}T ×S ,

be the available ground-truth for every time frame and everylocation, where the value 0 stands for “not available”.

1We consider in this article only locations in the image plane, but ad-ditional parameters such as scale or rotation could be handled in a similarmanner, albeit with an increase of the computational cost.

1434

3.1. Boosting

We quickly summarize here some characteristics of Ad-aBoost relevant to the understanding of our approach. Let

hm : RD → {−1, 1}, m = 1, . . . ,M

be a family of “weak learners”, and

H = span{h1, . . . , hM}

the set of linear combinations of weak-learners. Given alabeling y and a mapping

ϕ : RD → R

we can define the exponential loss

LV(ϕ;y) =∑

(t,s)∈V

exp (−yt,sϕ(xt,s)) (1)

which is low when ϕ takes values consistent with the label-ing y. This loss ignores samples whose labels are equal to0. Boosting consists in approximating the mapping

ϕ∗ = argminϕ∈H

LV(ϕ;y)

by successively picking Q weak learners hmq and weightsωq ∈ R such that LV is reduced in a greedy fashion. In ourexperiments, we used stumps, parameterized by a featureindex d ∈ {1, . . . , D} and a threshold ρ ∈ R, as weaklearners of the form:

∀x ∈ RD, h(x) = 2 · 1{xd≥ρ} − 1.

3.2. Time-based regularization

The core idea of our approach is to automatically com-pute labels for non-labeled samples, which are as consistentas possible with an existing classifier and with a constraintof continuous motion for the targets.

Our estimation of a labeling consistent with the physi-cally possible motions is strongly inspired from the convexmulti-target tracking of [2]. Prior knowledge regarding tar-get motions is represented by a directed “motion graph”,which possesses one vertex for each (t, s) pair, where t is atime step and s a spatial location. Let

V = T × S

be this set of vertices. The edges of that graph correspondto admissible motions:

E ={(

(t, s), (t+ 1, s′)), 1 ≤ t < T, s ∈ S, s′ ∈ N (s)

},

where N (s) ⊂ S denotes the locations a target located in sat time t can reach at time t + 1. Given such a graph, weoptimize a flow

f : E → {0, 1}

which associates to every edge of the motion graph the num-ber of targets moving along it. If the flow f is equal to 0 onthe edge from (t, s) to (t + 1, s′), it means that no targetmoves from location s at time t to locations s′ at time t+ 1,and conversely if f is equal to 1 on that edge, it means thata target makes that motion at that moment. Let F stand forthe set of Boolean labeling of the edges which are physi-cally plausible and therefore obey the following constraints:(1) the flow on each edge is smaller than one and greaterthan 0, and (2) the sum of the flow on the edges arriving at acertain vertex is equal to the sum of the flows on the edgesleaving that vertex.

For clarity, for any edge e = {(t, s), (t+ 1, s′)} ∈ E , wewill use ϕ(e) as a short-cut for ϕ(xt,s), that is the value ofthe classifier at the time and location corresponding to theoriginating vertex of the edge. To select a flow consistentwith the responses of the classifier, we minimize the expo-nential loss of Equation (1), as during Boosting. This meansthat the flow f should minimize

LE(ϕ; f) =∑e∈E

exp(−(2f(e)− 1)ϕ(e)). (2)

Since f takes its values in {0, 1}, we have

f∗ = argminf∈F

LE(ϕ; f)

= argminf∈F

∑e∈E

exp(−(2f(e)− 1)ϕ(e))

= argminf∈F

∑e∈E

exp(−ϕ(e))f(e) + exp(ϕ(e))(1− f(e))

= argminf∈F

∑e∈E

(exp(−ϕ(e))− exp(ϕ(e))) f(e).

As in [2], we have to minimize a linear cost, under the linearequalities of conservation of flow at every vertex, and thelinear inequalities corresponding to an upper-bound of onetarget moving per edge, and a lower bound of zero. If werelax the binary flow constraint and let f take continuousvalues in [0, 1], this results into a convex linear program-ming system, which can be solved optimally2.

Additionally, entrance into the graph is made possibleby relaxing the constraint of conservation of flow for ver-tices corresponding to “virtual locations”, connected to theborder of the images, where targets can appear or disap-pear. Finally, the flow is forced to pass through vertices forwhich we have explicit positive labels and conversely it isprevented from passing through vertices with explicit nega-tive labels by setting the scores appropriately.

2As shown in [2], this linear programming system can equivalently besolved with the more efficient K-Shortest Paths (KSP) algorithm. Our im-plementation relies on the KSP algorithm and in the remainder of this paperthe terms KSP and linear programming system are used interchangeably.

1435

3.3. Iterative optimization

Given the Boosting procedure, and the time-based reg-ularization described above, we depict here our iterativelearning process. Let

y(0) ∈ {−1, 0, 1}T ×S

be the initial labeling provided by experts. In practice, onlyone frame in every M will be labeled, leading to labelsequal to ±1 at all the locations in these frames, and 0 ev-erywhere in other frames. Given that initial labeling, wedefine the following algorithm:

for k = 1, . . . ,K doϕ(k) ← argmin

ϕ∈HLV

(ϕ;y(k−1)

)f (k) ← argmin

f∈FLE

(ϕ(k); f

)y(k) ← Θ

(f (k)

)end for

To summarize, we train with Boosting a first classifierϕ(1) from the scarce labeling provided by experts, thencompute an optimal Boolean labeling of edges f (1), con-sistent with physical motions of targets, and from it a newsigned labeling y(1) over all frames and locations. Fromthat new labeling, we train with Boosting a second classi-fier ϕ(2), etc. The Θ operator computes the ±1 labels onvertices, from the Boolean labels on edges. Precisely:

Θ(f)t,s = 2∑

s′∈N (s)

f ((t, s), (t+ 1, s′)) − 1

Hence, both the Boosting and the KSP procedure are min-imizing a common exponential loss function. This loss fa-vors consistency between the response maps generated bythe classifier and the Boolean flows while penalizing dis-crepancy.

3.4. Implementation details

In this section, we outline specific implementation de-tails used in our our algorithm.

3.4.1 Numerical stability

The cost of Equation (2) grows quickly with a high classi-fier response ϕ(e). This naturally causes numerical stabilityissues for the convex linear programming system. For thisreason, given that, as was shown in [5],

ϕ(xt,s) '1

2log

P (ys,t = 1|xs,t)P (ys,t = −1|xs,t)

, (3)

we clip the classifier response ϕ(xt,s) at ±8.0 which cor-responds to allowing a maximum classifier confidence ofP (ys,t = ±1|xs,t) = 1− 10−7.

3.4.2 Regularization

Non-Maxima Suppression. Ideally, at every iteration, theclassifier would feed the linear programming system adense response map. Doing so, however, would result ina cluster of trajectories around each target being output bythe linear programming system. This behavior is not de-sirable from both a computational perspective, in that thelinear optimization would deploy vast resources to resolvenumerous trajectories centered on each target, and from alearning perspective in that multiple shifted copies of a tar-get would be fed back to the Boosting stage at subsequentiterations. For this reason, the linear programming systemis fed a sparse response map obtained by applying standardnon-maxima suppression to the dense response map withoutthresholding.Trajectory Filtering. The linear programming system al-lows for an unconstrained number of targets to appear anddisappear from any location along the boundary of the videoframe. This behavior is naturally desired to maintain thegenerality of the approach. A drawback of this approachhowever is that as soon as the classifier outputs a positiveresponse on a boundary point, the linear programming sys-tem will admit a trajectory at that location even if its du-ration is only one frame. To mitigate this effect, we intro-duced a simple heuristic that rejects any trajectory whichdoes not venture deep enough (10 pixels) into the middle ofthe scene.Soft Consensus. We introduced a final heuristic in order toinclude a hard confidence estimation on the output of theKSP stage and only retain the most confident samples forthe subsequent iteration. To this end, the samples along thetrajectories output by the linear programming system arenot fed as a whole to the Boosting stage of the next iter-ation. Instead, they are sorted according the response ofthe classifier of the current stage and the bottom 25% arepruned. This choice was ad hoc and was not optimized onour test sets.

4. Experimental ResultsTo evaluate the performance of our proposed learning

strategy, experiments were performed on two different datasets: video sequences of pedestrians and time-lapse mi-croscopy data containing migrating neurons. In what fol-lows, the specifics of our experimental setup are given andthe results of our experiments provided.

4.1. Image features

We describe here the image features that we used for thelearning component. A scene x is preprocessed by comput-ing and thresholding the derivatives of the image intensityto obtain an edge image. The orientation of these edges arefurther quantized into q bins, resulting in q edge maps. Let φ

1436

denote the possible orientations of an edge on Φ = [−π, π[,and let Φ = {0, 2π

q ,4πq , . . . , (q− 1) ∗ 2π

q } denote the possi-ble orientations of a quantized edge.

Now ∀e ∈ Φ, x ∈ I, s ∈ {1, . . . ,W}× {1, . . . ,H}, let

ξe(x, s) ∈ {0, 1}, (4)

denote the presence of an edge with quantized orientation eat pixel s in image x. We assume ξe(x, s) is equal to 0 ifthe location s is not in the image plane. Thus, each ξe(x, s)is simply a map of edges with quantized orientation e.

Our features, as those in [1], compute the ratio of edgesof a particular orientation within a sub-window of the de-tector’s r × r square of interest, with respect to the totalnumber of edges within the same sub-window. Let R de-note such a sub-window of random size and location con-tained in {1, . . . , r} × {1, . . . , r} plane. Our features areentirely parameterized by the sub-window R and the edgetype e and are defined as:

hR,e(x) =∑m∈R

ξe(x,m) /∑

d∈Φ,m∈R

ξd(x,m). (5)

These features give the classifier the ability to check for thepresence of outlines and textures and can be computed inconstant time using q integral images, one for each edgemap. In all our experiments, we used q = 8.

4.2. AdaBoost

The standard AdaBoost learning procedure is used. Theselection of the stump at every iteration of AdaBoost re-sults from examining 1000 of our features. The thresholdρi of the selected stumps is optimized through an exhaus-tive search. Finally, feature parameters R and e are cho-sen uniformly at random. In all our experiments, a singleAdaBoost stage is trained with the bootstrapping proceduredescribed in [4]: this allows us to avoid the difficulties as-sociated with training and tuning a cascade.

4.3. Data sets

Pedestrian data Our pedestrian data set contains two se-quences: a training sequence and a testing sequence. Thisdata was obtained by mounting a standard DV camera in aslightly elevated location over a central and relatively busycampus site. The training and testing sequences are nonoverlapping in time and both sequences contain multiplepedestrians entering and exiting the scene from all bordersof the images. Some positive patches sampled uniformly atrandom from the test sequence are shown in Figure 1.

Both sequences have a resolution of 568× 328, a framerate of 25fps, contain approximately 1000 frames and ap-proximately 3500 positive samples. The sequences are rela-tively challenging in that pedestrians often pass through thescene walking closely to each other and thereby partially

occluding each other. In addition, a number of pedestrianswalk through a relatively unlit area and a large range in posevariation is observed. The detector’s window size for thesesequences is 68× 68.

Neuron data Our neuron data set is made up of 14 se-quences of resolution 168 × 128 each containing exactly97 frames. The sequences show developing neurons in acell culture imaged with bright-field microscopy and usingGreen Fluorescent Protein staining. Each sequence is im-aged over 16 hours with an image taken every 10 minutes.The detector’s window size for these sequences is 34× 34.

Ordinarily, these developing neurons would move in aguided fashion while extending and retracting neurites seek-ing protein signals. However given that they are in a cellculture, they are moving randomly. These sequences arehighly challenging as the neurons vary greatly in appear-ance and can move almost their entire length from oneframe to the next. In addition, the microscope periodicallyloses focus causing a drastic change in appearance in theneurons.

Out of these 14 short 97 frame sequences, we build atraining sequence by randomly selecting 8 of the sequences,while reserving the remaining 6 for testing. The trainingand test sequences contain approximately 3000 neurons inmotion each. Some positive patches selected uniformly atrandom from the test sequence are shown in Figure 3.

4.4. Error rates

Error rates were computed in a conservative fashion. Adetection is a true alarm if its location is within a certaindistance from the target and a false alarms otherwise. Theconsidered distance is 0.25 times the length of the detector’ssquare window of interest for the pedestrian data and 0.4times for the neuron data set. The choice of these numbersreflect how closely two targets may appear to each other ineach data set. Two targets may still lie within the abovementioned distance. In this scenario, if only one alarm israised, a miss is counted.

4.5. Baselines and FlowBoost

In all our experiments we compare the performanceof our algorithm against two baselines. The first is anAdaBoost classifier as described above trained with 200stumps and using the full training data. The second is anAdaBoost classifier, as described above, trained with 200stumps but using the sparse labeling. We varied the label-ing rate with each experiments by annotating 1 frame in 16,32, 64 and 128.

The results are shown for the first three iterations ofFlowBoost. The first iteration trains an AdaBoost stage withonly 50 stumps, while the remaining two iterations train anAdaBoost stage with 200 stumps.

1437

4.6. Results

Experiments were run on the pedestrian sequences forthe cases where 1 frame is labeled every 32, 64 and 128frames. This corresponds to initial training sets containing114, 61 and 29 positive examples out of a population of3500. Results are shown in Figure 2. For the neuron dataset, experiments were run for the cases where 1 frame islabeled every 16, 32 and 64 frames. This corresponds toinitial training sets containing 228, 141 and 102 positiveexamples out of a population of 3000. Results are shown inFigure 4.

Results on both data sets are very good, confirming thesoundness of this approach. With the exception of the ex-periment where 1 frame in 128 is labeled for the pedestriandata set (see Figure 2 (c)), FlowBoost’s third iteration is ableto outperform a standard Boosting procedure trained withthe entire data set. This is a truly surprising result whichcan be explained by the ability of the system to register datain translation more precisely, or at the very least in a morelearning-friendly manner, than a human annotator.

Performance at very high true positive rates is still of-ten slightly worse when compared to the fully labeled base-line. It is indeed extremely difficult to handle unlabeled pos-itive outliers: trajectories passing through truly positive, yethighly unusual, samples may in fact score as bad as trajec-tories passing through the background. This, unfortunately,cannot be handled in a totally unsupervised manner withoutadditional knowledge.

5. Concluding Remarks

We presented a novel approach that propagates a sparselabeling of a training video to every frame in a manner con-sistent with the known physical constraints on target mo-tions. Experiments demonstrate that, except at very highconservative regimes, our algorithm trained with less than3% of the labels is able to outperform an equivalent Boost-ing algorithm trained with the fully labeled set.

This work admits several natural extensions. The first isto cope with geometrical poses of greater complexity. Inboth of our applications, we only considered location inthe image plane, without variations in scale or orientation.While this extension is straightforward formally, the com-putational costs could be prohibitory. Hence, it would re-quire additional development in hierarchical representationor adaptive evaluation.

The second is the explicit inclusion of the non-maximumsuppression in the iterative framework: the flow should beoptimized so that, through the non-maximum suppression,it is consistent with the classifier response. In the currentversion, this is ignored, which leads to suboptimal perfor-mance.

The third, finally, is the development of an adaptive prun-

ing of the samples to use after the flow-relabeling. The cur-rent choice of the top 75% is ad-hoc, and probably a poorman’s version of a confidence estimation based on the dis-tance to the separation boundary.

AknowledgmentThe authors thank Kevin Smith, Engin Turetken and

Jerome Berclaz for their invaluable help.

References[1] K. Ali, F. Fleuret, D. Hasler, and P. Fua. Joint Pose Estimator

and Feature Learning for Object Detection. In InternationalConference on Computer Vision, 2009.

[2] J. Berclaz, F. Fleuret, E. Turetken, and P. Fua. Multiple Ob-ject Tracking Using K-Shortest Paths Optimization. IEEETransactions on Pattern Analysis and Machine Intelligence,2011.

[3] A. Blum and S. Chawla. Learning from labeled and unla-beled data using graph mincuts. In International Conferenceon Machine Learning, pages 19–26, 2001.

[4] F. Fleuret and D. Geman. Stationary Features and Cat Detec-tion. Journal of Machine Learning Research, 9:2549–2578,2008.

[5] J. Friedman, T. Hastie, and Tibshirani. Additive Logistic Re-gression: a Statistical View of Boosting. Annals of Statistics,28:2000, 1998.

[6] T. Joachims. Transductive inference for text classificationusing support vector machines. In International Conferenceon Machine Learning, pages 200–209, 1999.

[7] T. Joachims. Transductive learning via spectral graph parti-tioning. In International Conference on Machine Learning,pages 290–297, 2003.

[8] Z. Kalal, J. Matas, and K. Mikolajczyk. Online Learningof Robust Object Detectors During Unstable Tracking. InInternational Conference on Computer Vision, 2009.

[9] Z. Kalal, J. Matas, and K. Mikolajczyk. P-N Learning: Boot-strapping Binary Classifiers from Unlabeled Data by Struc-tural Constraints. In Conference on Computer Vision andPattern Recognition, 2010.

[10] N. D. Lawrence and M. I. Jordan. Semi-supervised learningvia gaussian processes. In Advances in Neural InformationProcessing Systems, pages 753–760, 2005.

[11] K. Nigam, A. McCallum, S. Thrun, and T. Mitchell. TextClassification from Labeled and Unlabeled Documents Us-ing Em. In Machine Learning, pages 103–134, 1999.

[12] H. Scudder. Probability of Error of Some Adaptive Pattern-Recognition Machines. Information Theory, IEEE Transac-tions on, 11(3):363–371, July 1965.

[13] M. Szummer and T. Jaakkola. Partially labeled classificationwith markov random walks. In Advances in Neural Informa-tion Processing Systems, pages 945–952, 2002.

[14] V. Vapnik. Statistical Learning Theory. Wiley-Interscience,New York, 1998.

[15] X. Zhu and Z. Ghahramani. Learning from labeled and unla-beled data with label propagation. Technical report, CarnegieMellon University, 2002.

1438

Figure 1. Some examples of people from our pedestrian test video taken uniformly at random across the entire set.

10 6 10 40.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

1 Frame in 32 Labeled

AdaBoost with full labeling AdaBoost FlowBoost, iteration 1 FlowBoost, iteration 2 FlowBoost, iteration 3

(a)

10 6 10 40.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

1 Frame in 64 Labeled

AdaBoost with full labeling AdaBoost FlowBoost, iteration 1 FlowBoost, iteration 2 FlowBoost, iteration 3

(b)

10 6 10 40.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

1 Frame in 128 Labeled

AdaBoost with full labeling AdaBoost FlowBoost, iteration 1 FlowBoost, iteration 2 FlowBoost, iteration 3

(c)

1 in 1 1 in 32 1 in 64 1 in 12810 6

10 5

10 4

10 3

Frame labelling rate

Fals

e po

sitv

e ra

te

@ TP = 0.95, iteration 3 @ TP = 0.90, iteration 3 @ TP = 0.85, iteration 3

(d)Figure 2. Performance of our learning framework compared with a standard labeling for our pedestrian data set. Figures (a,b,c) displaytrue-positive rate as a function of the false alarms rate on a log scale for the case (a) where one frame in 32 is annotated, (b) where oneframe in 64 is annotated and (c) where 1 frame in 128 is annotated. Figure (d) displays the false alarm rate at various true positive rate as afunction of the number of labeled frames.

1439

Figure 3. Some examples of migrating neurons taken uniformly at random across our microscopy test data.

10 5 10 4 10 3 10 20.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

1 Frame in 16 Labeled

AdaBoost with full labeling AdaBoost FlowBoost, iteration 1 FlowBoost, iteration 2 FlowBoost, iteration 3

(a)

10 5 10 4 10 3 10 20.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

1 Frame in 32 Labeled

AdaBoost with full labeling AdaBoost FlowBoost, iteration 1 FlowBoost, iteration 2 FlowBoost, iteration 3

(b)

10 5 10 4 10 3 10 20.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

False positive rate

True

pos

itive

rate

1 Frame in 64 Labeled

AdaBoost with full labeling AdaBoost FlowBoost, iteration 1 FlowBoost, iteration 2 FlowBoost, iteration 3

(c)

1 in 1 1 in 16 1 in 32 1 in 6410 4

10 3

10 2

Frame labelling rate

Fals

e po

sitv

e ra

te

@ TP = 0.95, iteration 3 @ TP = 0.90, iteration 3 @ TP = 0.85, iteration 3

(d)Figure 4. Performance of our learning framework compared with a standard labeling for our neuron data set. Figures (a,b,c) display true-positive rate as a function of the false alarms rate on a log scale for the case (a) where one frame in 16 is annotated, (b) where one frame in32 is annotated and (c) where 1 frame in 64 is annotated. Figure (d) displays the false alarm rate at various true positive rate as a functionof the number of labeled frames.

1440