ieee transactions on medical imaging, vol. 34, no. 6, …

IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 34, NO. 6, JUNE 2015 1

Multi-Target Tracking with Time-Varying ClutterRate and Detection Profile: Application to

Time-lapse Cell Microscopy SequencesSeyed Hamid Rezatofighi, Student Member, IEEE, Stephen Gould, Member, IEEE, Ba Tuong Vo, Ba-Ngu Vo,

Katarina Mele and Richard Hartley, Fellow, IEEE

Abstract—Quantitative analysis of the dynamics of tiny cellularand sub-cellular structures, known as particles, in time-lapsecell microscopy sequences requires the development of a reliablemulti-target tracking method capable of tracking numeroussimilar targets in the presence of high levels of noise, high targetdensity, complex motion patterns and intricate interactions. Inthis paper, we propose a framework for tracking these structuresbased on the random finite set Bayesian filtering framework.We focus on challenging biological applications where imagecharacteristics such as noise and background intensity changeduring the acquisition process. Under these conditions, detectionmethods usually fail to detect all particles and are often followedby missed detections and many spurious measurements withunknown and time-varying rates. To deal with this, we proposea bootstrap filter composed of an estimator and a tracker. Theestimator adaptively estimates the required meta parameters forthe tracker such as clutter rate and the detection probability ofthe targets, while the tracker estimates the state of the targets.Our results show that the proposed approach can outperformstate-of-the-art particle trackers on both synthetic and real datain this regime.

Index Terms—Multi-target tracking, particle tracking, fluores-cence microscopy, Bayesian estimation, random finite set, CPHD,clutter rate, detection probability.

I. INTRODUCTION

THE ability to accurately monitor cellular and sub-cellularstructures in their native biological environment has enor-

mous potential in addressing open questions in cell biology. Invarious applications, one of the key steps for understanding bi-ological phenomena is to assess the motion of these structures.Recent developments in time-lapse cell microscopy imagingsystems have had a great impact on the analysis of thesedynamics. However, visual inspection of sequences acquiredby these imaging techniques requires manual tracking of many

Copyright (c) 2010 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending a request to [email protected].

H. Rezatofighi is with the School of Computer Science, TheUniversity of Adelaide, Adelaide SA 5005, Australia (e-mail:[email protected]).

S. Gould and R. Hartley are with the College of Engineering and ComputerScience, The Australian National University, Canberra ACT 2601, Australia(e-mail: [email protected], [email protected]).

B.-N. Vo and B.-T. Vo are with Department of Electrical and Com-puter Engineering, Curtin University of Technology, Perth WA 6845, Aus-tralia and are supported in part by the Australian Research Council un-der Discovery Project DP120102343 (e-mail: [email protected]; [email protected]).

K. Mele is with Computational Informatics, CSIRO, North Ryde NSW2113, Australia (e-mail: [email protected]).

(a) (b)

Fig. 1. Two images of a TIRFM sequence visualizing fluorescently taggedvesicles (bright spots) close to the plasma membrane of a pancreatic beta cell(a) before and (b) after injection of insulin. Clearly, the background intensityand noise level noticeably increase during the acquisition.

tiny structures in numerous noisy images. Thus, automatedmulti-target tracking methods have been extensively used indifferent biological applications in the last decade [1–25].

Despite significant technical advances made in automati-cally tracking moving objects, tracking microscopic structures,known as particles [26], remains a challenging task due tothe complex nature of biological sequences. The microscopicsequences are usually populated with similar tiny structureshaving intricate motion patterns and sophisticated interactionswith other structures. Moreover, the structures may enter ordisappear from the field of view or be occluded by otherobjects. In addition in some imaging techniques, i.e. fluores-cence microscopy imaging, the sequences are contaminatedwith a high degree of noise. Under these conditions, detectionmethods usually fail to detect all particles and generate manyspurious measurements (clutter) [14, 27]. To be successfullyapplied in many biological applications, multi-target trackingmethods should be able to track an unknown and time-varyingnumber of similar particles in the presence of clutter anddetection uncertainty.

To this end, many particle tracking approaches have beenproposed in literature [1–24]. Some of the most popularparticle tracking approaches are based on detection followedby a deterministic linking procedure. Here, each particle is sep-arately detected in each frame. Then, a deterministic solution,e.g. an optimization technique, links the corresponding targetsbetween frames [3–7]. The performance of these algorithms isoften sensitive to the detection algorithm and may degrade in


the presence of complex target dynamics and highly cluttereddetections resulting from very noisy sequences.

Bayesian filtering approaches are another class of trackingalgorithms that have become popular for particle trackingapplications in recent years [10–24]. These approaches betterdeal with tracking multiple particles in cluttered measurementsby incorporating prior knowledge of target dynamics and mea-surement models [14–19]. However, knowledge of the clutterrate and detection profile of the chosen detection method, areof critical importance in Bayesian multi-target tracking.

Most existing multi-target tracking solutions assume knownand fixed detection and false alarm parameters. However,these parameters cannot be computed in many practical ap-plications and worse, it is not even known whether they aretime-invariant. In biological imaging techniques, the noisecharacteristic and the background intensity of sequences maychange during the acquisition process which make detectionprofile and clutter rate time-variant. For instance, injection of astimulus such as insulin into pancreatic beta cells increases thenoise level and overall intensity of the sequences acquired bytotal internal reflection fluorescence microscopy (TIRFM) [28](Fig. 1). Thus, the ability of multi-target trackers to accom-modate unknown clutter and detection parameters is crucial intime-lapse cell microscopy since mismatches in these modelparameters inevitably result in erroneous tracking outputs.

The key contribution of this paper is to propose an effectivesolution to the problem of tracking multiple maneuvering par-ticles in unknown and time-varying false alarm and detectionrates. To the best of our knowledge, this practical problemhas not been discussed in the biological signal processingliterature so far. To address this challenging problem, we usethe recent generation of Bayesian filters based on randomfinite set theory. This framework provides an elegant math-ematical formulation for multi-target systems and allows usto deal with the aforementioned complexities. Our proposedapproach is based on the Cardinalized Probability HypothesisDensity (CPHD) filter for unknown clutter rate and detectionprofile known as the λ-pD-CPHD filter [29]. Clutter rate anddetection probabilities are estimated by the λ-pD-CPHD filterbootstrapped onto a CPHD filter that outputs target estimates.This bootstrap idea was inspired by [30] which requires knownand uniform probability of detection. Our proposed solutioncan accommodate unknown and non-uniform probability ofdetection. To cope with maneuvering motion of sub-cellularstructures [16–21], we also propose the multiple model imple-mentation of the aforementioned filters in this paper. We willshow that the proposed method is able to deal with trackingmaneuvering structures in the presence of unknown and time-varying clutter rate and detection profile.

II. BACKGROUND

The Bayesian estimation paradigm deals with the problemof inferring the state of a target from a sequence of noisymeasurements. The state vector contains all relevant dynamicinformation about the target while the measurements containwhat can be directly observed from the sequences. The notionof Bayes optimality is fundamental to the Bayesian estimation

paradigm and is well-defined for single-target tracking [31].While Bayes optimal tracking techniques such as Kalman andparticle filters are formulated for a single-target, they canbe algorithmically extended to track a time-varying numberof targets by combining them with data association [32–34]. However, the notion of Bayes optimality does not carryover. Nonetheless this approach to multi-target tracking hasbeen used in a wide range of applications, including particletracking [11–14, 18–23].

The random finite set (RFS) approach, introduced by Mahlergeneralizes the notion of Bayes optimality to multi-targetsystem using RFS theory [31]. The key difference with otherapproaches is that the collection of target states at any giventime is treated as a set-valued multi-target state. This rep-resentation provides an elegant formulation for multi-targetsystem which avoids the need for data association. The RFSapproach has generated substantial interest in recent yearswith the development of the Probability Hypothesis Density(PHD) filter [35], Cardinalized PHD (CPHD) filter [36],multi-Bernoulli filters [37, 38], labeled RFS filter [39] anda host of many applications [40] including cell and particletracking [15–17, 24].

A. RFS for multi-target tracking

In the Bayesian estimation paradigm, the state and mea-surement are treated as realizations of random variables whichare suitable for single target and single measurement system.However, in a multi-target system, the collection of states andmeasurements can be naturally represented as finite sets.

Let xk,1, ..., xk,N(k) and zk,1, ..., zk,M(k) be the states ofall N(k) targets and all M(k) measurements at time k,respectively. Over time, some of these targets may disappear,new targets may appear, and the surviving targets evolveto new states. Moreover, due to poor detector performance,only some targets are detected at each time step and manymeasurements are spurious detections (clutter). Thus, we canconveniently represent the targets and measurements at eachtime slice with two finite sets as

Xk = {xk,1, . . . , xk,N(k)},Zk = {zk,1, . . . , zk,M(k)}.

(1)

Consequently, the concept of a random finite set (RFS) isrequired to cast the multi-target estimation problem in theBayesian framework. Intuitively, an RFS is a finite-set-valuedrandom variable. What distinguishes an RFS from a randomvector is that the number of constituent variables is ran-dom and the variables themselves are random, distinct andunordered. Mahler’s Finite Set Statistics (FISST) providespowerful yet practical mathematical tools for dealing withRFSs [31, 35], based on the notion of integration and densitythat is consistent with point process theory [41]. The center-piece of the RFS approach is the so called Bayes multi-targetfilter, a generalization of the single-target (optimal) Bayesfilter to accommodate multiple targets based on multi-targetdynamic and measurement models.


1) Multi-Target Dynamical Model: Given a multi-targetstate Xk−1 at time k−1, each target xk−1 ∈ Xk−1 either con-tinues to exist at time k with probability pS,k(xk−1) and movesto a new state xk with probability density fk|k−1(xk|xk−1),or dies with probability 1−pS,k (xk−1) and takes on the value∅. Thus, given a state xk−1 ∈ Xk−1 at time k−1, its behaviorat time k is modeled by a Bernoulli RFS, Sk|k−1, which hasFISST density1 given by

pk|k−1(Sk|k−1|xk−1) ={1− pS,k(xk−1) if Sk|k−1 = ∅,pS,k(xk−1)fk|k−1(xk|xk−1) if Sk|k−1 = {xk}.

(2)

Assuming that individual targets move independently, each ofthe above Bernoulli RFSs are mutually independent condi-tioned on Xk−1. Thus the survival or death of all existingtargets from time k − 1 to time k is modeled by the multi-Bernoulli RFS

Tk|k−1(Xk−1) =⋃

xk−1∈Xk−1

Sk|k−1(xk−1). (3)

The appearance of new targets at time k is modeled by anRFS of spontaneous births Γk which is usually specified asan i.i.d. cluster RFS with intensity function γk and cardinalitydistribution ρΓ,k

2. Consequently, the RFS multi-target state Xk

at time k is given by the union

Xk = Tk|k−1(Xk−1) ∪ Γk. (4)

The likelihood of the multi-target state Xk, at timek, is described by the multi-target transition densityfk|k−1(Xk|Xk−1) which incorporates target dynamics, birthsand deaths [31].

2) Multi-Target Measurement Model: Given a multi-targetstate Xk at time k, each target xk ∈ Xk, at time k, iseither detected with probability pD,k (xk) and generates anobservation zk with likelihood gk(zk|xk), or missed withprobability 1− pD,k (xk) and generates the value ∅, i.e. eachtarget xk ∈ Xk generates a Bernoulli RFS, Dk, which hasFISST density given by

pk(Dk|xk) =

{1− pD,k(xk) if Dk = ∅,pD,k(xk)gk(zk|xk) if Dk = {zk}.

(5)

The set of measurements generated by Xk, at time k, ismodeled by the multi-Bernoulli RFS

Θk(Xk) =⋃

xk∈Xk

Dk(xk). (6)

In addition, the sensor receives a set of false/spuriousmeasurements or clutter, modeled by an RFS Kk, which isusually specified as an i.i.d. cluster RFS with intensity functionκk and cardinality distribution ρK,k [31]. Consequently, at

1For more detail on RFS density and probability density please see [31]2An i.i.d. cluster RFS can be statistically represented by its intensity and

cardinality distribution. The cardinality distribution, ρ(n), provides a discreteprobability distribution over the number of the constituent random variables,n, while the intensity distribution v(x) at the point x in the state space isinterpreted as the instantaneous expected number of the variables that existat that point [31].

time k, the multi-target measurement Zk generated by themulti-target state Xk is formed by the union

Zk = Θk(Xk) ∪Kk. (7)

The likelihood of the multi-target measurement Zk, giventhe multi-target state, at time k, is described by the multi-target measurement likelihood gk(Zk|Xk) which incorporatesdetection uncertainty and clutter.

3) Bayes Multi-Target Filter: The multi-target posteriordensity pk(Xk|Z1:k) contains all statistical information aboutthe multi-target state at time k and can be recursively com-puted by the aforementioned multi-target transition density andmeasurement likelihood using the following prediction andupdate equations.

pk|k−1(Xk|Z1:k−1) =∫fk|k−1(Xk|Xk−1)pk−1(Xk−1|Z1:k−1)δXk−1,

(8)

pk(Xk|Z1:k) ∝ gk(Zk|Xk)pk|k−1(Xk|Z1:k−1), (9)

where∫

(·)δX is the set integral [31].This filter generalizes the notion of Bayes optimality to

multi-target system using RFS theory. Further details can befound in [31].

B. The PHD filters

Although the above framework provides an elegantBayesian formulation of the multi-target filtering problem, itcan be computationally expensive for applications with largenumber of targets [35]. In [35], Mahler proposed to propagatethe probability hypothesis density (PHD), or posterior intensitydistribution, of the targets vk(xk) which is the first statisticalmoment of the probability density function pk(Xk|Z1:k). Theprimary weakness of the PHD filter is a loss of higher ordercardinality information which causes erratic estimates in thenumber of targets especially when the density of targets ishigh [16, 36, 42].

Mahler [36] subsequently proposed the Cardinalized PHD(CPHD) filter which jointly propagates the posterior intensityfunction, vk, and cardinality distribution, ρk. The propagationof the cardinality distribution makes the filter more robust inthe estimation of the number of targets [36].

As with all Bayesian filtering frameworks, this filter alsorequires some predefined models such as the single targetMarkov transition density fk|k−1(·|·) for the dynamic model,measurement likelihood gk(·|·) and probabilities of survivalpS,k(·) and detection pD,k(·). Moreover, the models for newborn targets and false alarms (clutter) are described by the birthcardinality distribution ρΓ,k and the birth intensity functionγk, and the clutter cardinality distribution ρK,k and the clutterintensity function κk, respectively [36]. Obviously, this filterrequires knowledge of the detection probability and clutterdistributions similar to other Bayesian filters.

In [29], it was shown that the CPHD filter can also bereformulated such that it jointly estimates clutter distributionsand detection probability while tracking. We refer to thisversion of the filter as the λ-pD-CPHD filter. However, the λ-pD-CPHD cannot naturally perform as well as the CPHD filter


MM-�-��-CPHD(Detection & clutter estimator)

MM-CPHD(Multi-target tracking filter)

�� ,�

��

��

��

Fig. 2. A schematic of the proposed bootstrap filter.

when exact knowledge of detection probability and clutterdensity is available.

III. THE BOOTSTRAP CPHD FILTER

To propose an effective multi-target particle tracker for prac-tical applications with unknown and time-varying clutter rateand detection profile, we borrow the bootstrap idea proposedin [30]. However, in addition to the clutter rate, we calculatethe detection probability, using the λ-pD-CPHD filter, and feedthese to a robust multi-target tracking filter such as the CPHDfilter (Fig. 2). We use the CPHD filter for target estimationbecause it is a good trade-off between accuracy and speed.Nonetheless, it can be replaced by any multi-target filter thatrequires knowledge of false alarm and detection rate. In thispaper, we assume that pD,k and the clutter distributions, ρK,kand κk, are the meta parameters to be estimated for the CPHDfilter in each time frame. However, we will show that theestimation of the mean of the clutter rate, λk, is sufficientto uniquely define the ρK,k and κk.

To deal with maneuvering dynamics, we assume that targetmotion can be characterized by multiple dynamic models.Thus, here, we propose the multiple model (or jump-Markov)implementation of both the CPHD and λ-pD-CPHD filters anduse these filters as the tracker and the parameter estimator,respectively.

A. Multiple Model CPHD filterComplex maneuvering motions can be often characterized

by multiple simpler dynamic models. The idea of tracking atarget with multiple motion models is to augment the state ofthe targets by the index of the model [43].

For notational convenience, we remove the time index k inour formulation throughout the paper. However, the randomvariables and distributions are generally time-indexed. Allrandom variables (·)k at time k and (·)k−1 at time k − 1 aresimply denoted by (·) and (·), respectively.

Let X and R = {1, 2, · · · , R} denote the kinematic statespace and the multiple models state space, respectively. Definethe augmented state space as X = X ×R, where × denotesthe Cartesian product. The underscore notation is used for theaugmented state space so that x = [x, r] ∈ X denotes x ∈ Xfor the kinematic state and r ∈ R for the model state. Theintegral of a function f : X → R is given by∫

Xf(x)dx =

R∑r=1

∫Xf(x, r)dx. (10)

The measurement likelihood and probabilities of survival anddetection for the augmented state vector can be respectivelywritten as

g(z|x) = g(z|x, r), (11)

pS

(x) = pS

(x, r), (12)

pD

(x) = pD

(x, r). (13)

Also, for the augmented transition density and birth intensity,we have the following factored forms,

f(x|x) = f(x, r|x, r) = f(x|x, r)τ(r|r), (14)

γ(x) = γ(x, r) = γ(x)π(r), (15)

where τ(r|r) is the probability transition from model r tor and π(r) is the probability of birth for model r [43]. Asdescribed in Section II, the filter also requires the knowledgeof the birth cardinality distribution ρΓ(n).

Clutter is usually assumed to be Poisson RFS and indepen-dent from the target state [31–34] with

ρK(n) = Pois(n;λ), (16)

κ(z) = λK(z), (17)

where K(z) is a known probability distribution, e.g. uniformdistribution, representing how the false alarms are distributedover the measurement space and

∫K(z)dz = 1. Therefore,

given K(z), knowledge of the parameter λ is sufficient todefine the clutter distribution.

Substituting the aforementioned augmented model terms(Eqs. 11-17) into the conventional CPHD equations [42] andusing Eq. 10 yields the recursive equations for the multiplemodel or jump Markov CPHD filter (see Appendix A).

Fig. 3 depicts a simple graphical representation of theproposed multiple model CPHD filter. At each time step k,the intensity vk and cardinality ρk distributions are predictedand updated from the intensity vk−1 and cardinality ρk−1

distributions in the previous time step k − 1 and using theaugmented model terms (Eqs. 11-17). The intensity distribu-tion provides information on the state of the targets whilethe number of targets can be estimated using the cardinalitydistribution. Specifically, the number of targets is estimatedusing the posterior mode Nk = arg max ρk(n) [42]. Notethat this filter requires the clutter rate λ and the detectionprobability pD as prior knowledge.

B. Multiple Model λ-pD-CPHD filter

We borrow a key idea from Mahler et al. [29] to developa multiple model CPHD filter with unknown clutter rate anddetection profile. To estimate clutter, the idea is to model itas a random finite set of false targets which is statisticallyindependent from the set of actual targets. These false targetsare defined by their own models such as birth, death, survivaland detection probabilities and transition model. Therefore,the multi-target state is composed of a disjoint combination ofthese two finite sets including actual targets and clutter. Thishybrid multi-target state allows us to track both actual and falsetargets simultaneously. To deal with multiple model dynamics


Prediction Update

�� |��

��

Car

din

alit

yIn

ten

sity

��|��

��, �, �, �� , ��,

Pre

def

ined

mo

del

s

Fig. 3. A schematic of the proposed MM-CPHD filter.

and unknown detection probability, the state of targets areaugmented by the index of the models and the unknowndetection probability. Finally this hybrid and augmented stateis estimated from the sequence of finite sets of measurementsgenerated by both real targets and clutter while accommo-dating multiple model dynamics and estimating the detectionprobability and clutter rate.

Let X (1) denote the state space for actual targets, X (0)

denote the state space for clutter generators and X (∆) = [0, 1],R = {1, 2, · · · , R} and Q = {1, 2, · · · , Q} denote the statespace for the unknown detection probability, and the multiplemodels for actual targets and clutter, respectively. Define thehybrid and augmented state space as

X =(X (1) ×X (∆) ×R

)](X (0) ×X (∆) ×Q

)(18)

where ] denotes a disjoint union. The double dot notationis used throughout to denote a function or variable definedon the hybrid state space and the underscore notation is usedfor the augmented state space. Therefore, x ∈ X represents ahybrid and augmented state such that x = (x, a, r) ∈ X (1) =X (1)×X (∆)×R and c = (c, b, q) ∈ X (0) = X (0)×X (∆)×Qdenotes the actual and clutter states comprising the kinematicstate and augmented components, respectively. The integral ofa function f : X → R is given by

∫Xf(x)dx =

R∑r=1

∫X (∆)

∫X (1)

f(x, a, r)dxda+

Q∑q=1

∫X (∆)

∫X (0)

f(c, b, q)dcdb. (19)

In theory, there may exist multiple model clutter generators.However, in most tracking applications, it is assumed thatthe clutter is uniformly distributed with Poisson cardinal-ity [31–34]. As a result, multiple model clutter generators withuniform-Poisson distribution can be substituted by a singlemodel clutter generator with uniform distribution and higherPoisson mean. Therefore, we continue our derivations withthe single model clutter generators by eliminating the variableq from the equations. Deriving the equations for applications

with multiple clutter generators and non-uniform distributionis straightforward.

In addition, since clutter generators are identical and thefalse targets do not follow any motion pattern, it is reasonableto ignore any functional dependence on the state of a cluttergenerator c, [29]. As a result, the probabilities of survival anddetection and the measurement likelihood for the hybrid andaugmented state vector can be respectively written as

pS

(x) =

{p

(1)S (x, r), x ∈ X (1)

p(0)S , x ∈ X (0)

(20)

pD

(x) =

{a, x ∈ X (1)

b, x ∈ X (0) (21)

g(z|x) =

{g(z|x, r), x ∈ X (1)

K(z), x ∈ X (0) (22)

where K(·) is clutter density which is often assumed to beuniform distribution in the measurement space. As the survivalprobabilities and the measurement likelihoods are independentfrom the detection probabilities, the terms a and b are removedfrom the equations. Obviously, the detection probabilities areonly dependent on a and b.

For the hybrid and augmented birth intensity and transitiondensity, we have the following factored forms,

γ(x) =

{γ(1)(x, a)π(r), x ∈ X (1)

γ(0)(b), x ∈ X (0) (23)

f(x|´x) = f (1)(x|x, r)f (∆)(a|a, r)τ(r|r), x, ´x ∈ X (1)

f (0)(c|c), x, ´x ∈ X (0)

0, otherwise,

(24)

where f (∆)(a|a, r) is the transition density for the detectionprobability a and f (0)(c|c) = f (∆)(b|b) due to the indepen-dence of false alarms from the clutter state c. Furthermore,the cardinality distribution of birth for the hybrid space isρΓ = ρ

(1)Γ ∗ ρ

(0)Γ [29].

By substituting the hybrid and augmented state space modelterms (Eqs. 20–24) into the conventional CPHD equations [42]and using Eq. 19, the recursive equations for this filter can becalculated (see Appendix B).

Fig. 4 shows a simple graphical representation of theproposed multiple model λ-pD-CPHD filter. At each time stepk, the target’s intensity v(1)

k , the clutter intensity v(0)k and the

hybrid cardinality distribution ρk are predicted and updatedfrom the target’s intensity v(1)

k−1, the clutter intensity v(0)k−1 and

the hybrid cardinality distribution ρk−1 in the previous timestep k − 1 and using the hybrid model terms (Eqs. 20–24).

The posterior cardinality ρk provides information on thetotal number of real and clutter targets. Therefore, the numberof actual targets cannot be estimated using the posteriormode Nk = arg max ρk(n), since ρk(n) includes both realtargets and clutter. In this filter, the posterior mean N

(1)k =∑

r

∫ ∫v

(1)k (x, a, r)dxda is used for estimation of the number


�� |��

��

� �

Prediction UpdateH

ybri

d

card

inal

ity

Clu

tter

Inte

nsi

ty

��|��

��

��

� �

Tar

get

s

Inte

nsi

ty

��|��

��

��, ��, �� , �� , ��

Pre

def

ined

mod

els

Fig. 4. A schematic of the proposed MM-λ-pD-CPHD filter.

of real targets [29]. Therefore, in contrast with the CPHD filter,this filter cannot benefit from propagation of the cardinalitydistribution for the estimated number of the targets. Similar tothe PHD filter, this leads the erratic estimates of the numberof targets which worsens the tracking result. To this end, weuse this filter only as the estimator in our bootstrap filter.The estimated detection and mean clutter are respectivelycalculated as follows [29].

pD,k =〈v(1)k , p(1)

D,k〉,

λk =〈v(0)k , p(0)

D,k〉,

(25)

where 〈·,·〉 is the inner product operator, p(1)D,k=a and p(0)

D,k=b.

IV. EXPERIMENTAL RESULTS

To evaluate the performance of our proposed filters, we ap-ply them to multi-target tracking in 2-D Total Internal Reflec-tion Fluorescence Microscopy (TIRFM) sequences. TIRFM isan imaging technique that enables visualization of sub-cellularstructures such as vesicles that are on or close to the plasmamembrane of cells [28]. The vesicles are very tiny sub-cellularstructures and are seen in TIRFM sequences as small particles(bright spots) moving with varying dynamics. These smallparticles appear and disappear from the field of view andcan be occluded by other structures. Due to limitations in theTIRFM acquisition process, the sequences are contaminatedwith a high level of noise. We consider the case where themain characteristics of the sequences such as noise level andthe background intensity gradually increase over time (Fig. 1).

In this specific application, we are interested in the overallmotion of the vesicles before and after injection of insulin, notmotion of a single object. In this case, the primary concernis not how well a tracking approach maintains the identity ofa tracked object over time. Instead, we are mainly concernedwith how well a tracker avoids false and missed tracks.

We compare the results of our bootstrap filter against thoseof several popular state-of-the-art particle tracking methods

including two reliable deterministic linking techniques, P-Tracker [6] and U-Tracker [4], and two robust detection basedtraditional Bayesian filtering approaches, MHT [14] and IMM-JPDA [19].3 Our results are also compared against the resultof a multiple model PHD filter [16]. Finally, the results of ourother derived filters such as the multiple-model CPHD (MM-CPHD) and λ-pD-CPHD (MM-λ-pD-CPHD) filters are alsoreported here in order to show the efficiency of our bootstrapidea.

A. Setup and Implementation Details

To fairly evaluate the performance of all tracking algo-rithms, the same detection lists were provided for all compet-ing tracking methods. We chose the detector proposed in [44]as it performs reliably in our synthetic and real sequences.

Having ground truth, we can accurately calculate the clutter(false positive) and the detection probability (true positive rate)of the chosen detector for each time frame. True positive andfalse positive were calculated based on the optimal sub-patternassignment between the ground truth and detected point sets.In this case, if the distance between two optimally assignedpoints is less than a predefined distance (2 pixel), the detectedpoint is counted as true positive, otherwise it is a false positive.

For the MHT, IMM-JPDA, MM-PHD and MM-CPHDfilters, the average number of false detections per frame λ andthe mean value of the detection probability pD (the optimalvalue for these parameters) were used as the predefined clutterrate and detection probability. However, accurate knowledgeof these values is not possible in many practical applicationsand thus, the reported results for these filters are optimistic.For the MM-λ-pD-CPHD filter and similarly for our bootstrap(B-MM-CPHD) filter, the clutter rate and detection probabilityare adaptively estimated using our proposed framework.

To ensure the validity of our experiments, for the inde-pendently implemented tracking methods such as MHT, P-Tracker and U-Tracker, we attempted to either estimate theirparameters from the ground truth or to find the values thatresulted in their best performance. Moreover, the multiple mo-tion model implementation of all competing tracking methodswas used. For the IMM-JPDA and different PHD and CPHDfilters, we chose identical state and measurement vectors anddynamic models. We modeled the state of each particle byits position, xx, xy , and velocity, xx, xy . The measurementsvector also contained the estimated position of the particles asz = (xx, xy). To model maneuvering motion of particles, twolinear dynamics including random walk and small accelerationmotion models were used [18, 19].

Since the target dynamics and measurement models in thisapplication can be properly expressed by multiple linear andGaussian terms, we used the Gaussian mixture implementa-tion [42] and the Beta-Gaussian mixture approach [29] foranalytical implementations of the MM-CPHD and MM-λ-pD-CPHD filters, respectively. The birth intensity distribution γ(·)

3The performance of the first three trackers, P-Tracker, U-Tracker and MHT,has been evaluated in a recent particle tracking challenge and their results havebeen reported in [26]. According to this article, they can be assumed as threetop performing methods for similar applications.


for all the PHD and CPHD filters was set as a Gaussiandistribution centered at the image with a very high standarddeviation [16].

As previously discussed, the PHD and CPHD filters prop-agate the intensity distribution of all targets vk(x) in eachtime frame k. By calculating the estimated number of targetsNk, the state of all targets in each time frame can beeasily extracted using vk(x) [41, 42, 45]4. However, in thisfiltering framework, the temporal correspondences betweenthe estimated states is not considered. In other words, theoutput of the PHD and CPHD filters is a set of the estimatedstates for each time frame without considering the identity oftrajectories. Thus, the dynamics of an individual target cannotbe evaluated. To deal with this, some authors combine thePHD filter with a track management technique to maintainthe identity of tracks [46, 47]. To avoid any computationalburden due to a track management step, we simply use our tagpropagation scheme [16], which only propagates the identityof the intensity distributions for all the PHD and CPHDfilters. As it only considers the previous time step to propagatethe identities [16], this is not the most reliable approach foridentity-to-track assignment. However, since we are interestedin the overall, not individual, motion of the vesicles, we usedthis approach in this application.

B. Evaluation Metric

To qualitatively assess the performance of these trackingmethods, we used a recent and popular metric based onoptimal sub-pattern assignment (OSPA) [48] followed by anextension of this metric for multi-target tracking applications,known as OSPA-T [49].

1) OSPA Metric: This metric measures the distance be-tween two sets of points, which can be the set of estimatedtracks and the ground truth tracks at each time frame, and rep-resents different aspects of multi-target tracking performancesuch as track accuracy, track truncation and missed or falsetracks by a single value.

For two arbitrary sets X = {x1, ..., xm} and Y ={y1, ..., yn}, where m ≤ n, the metric of order p is defined as

Dcp(X,Y )=

(1

n

(minθ∈Θn

m∑i=1

dcp(xi, yθi)p+ cp(n−m)

))1p

, (26)

where Θn is all feasible sets of permutations on Y and

dcp(x, y) = min(c, ‖x− y‖p), (27)

where c > 0 is the cut-off parameter. For m > n, Dcp(Y,X)

is calculated, instead.In Eq. 26, the first term, known as the location error,

measures track accuracy while the second term, the so calledcardinality error, represents the error for missed or false tracks.In this metric, the parameter p controls the sensitivity to outlierestimates that are distant from the true targets. Moreover, themetric penalizes the cardinality error by the cut-off parameter

4For example, in the Gaussian mixture implementations, the mean ofNk Gaussian terms with the highest weights in vk(x) can be used as theestimation of the state vector at each time frame [45].

c as a higher value for this parameter penalizes the false andmissing targets more. At a first glance, the selection of theparameters c and p seems to be critical for the performanceof different methods. However, it was proved that the relationbetween the performance of different methods is independentof the parameters c and p and different values for theseparameters only change the scale of the error, and do not affectthe method ranking [48].

2) OSPA-T: Although the OSPA metric measures the multi-target tracking errors such as track accuracy and missed orfalse tracks, it does not evaluate some other errors such asinconsistent labeling (label switching between the targets incrossing cases) and incorrect label initiation in the case oftrack truncation. An extension of the OSPA metric, known asOSPA-T5, considers the aforementioned errors by measuringthe distance between two sets of labeled points.

Let us assume that we have two labeled sets such thatX = {(l1, x1), ..., (lm, xm)} and Y =

{(l1, y1), ..., (ln, yn)

}.

In a multi-target tracking problem, these would be the groundtruth and estimated state of the targets with their labels ineach time frame. In this performance measure, the labelcorrespondence between the estimated and ground truth sets isfirst performed based on an optimal global assignment usingthe trajectories’ temporal information [49]. Then, dcp(x, y) inEq. 26 is substituted by the following distance.

d(c,`)p

((l, x), (l, y)

)= (28)

min

(c,(‖x− y‖pp +

(`Fδ(l, l)

)p)1/p),

where Fδ(l, l) = 0 if l and l are the corresponding labels, andFδ(i, j) = 1 otherwise. The parameter ` should be between 0and c and controls the penalty assigned to the labeling error.The case ` = 0 assigns no penalty, and ` = c assigns themaximum penalty. In this paper, we reported the results with` = c.

C. Evaluation on synthetic data

To quantitatively evaluate the tracking algorithms, they werefirst evaluated using 10 realistic synthetic movies generatedby the framework proposed in [50]. The sequences simulatedusing this framework reflect the difficulties existing in realTIRFM sequences while providing accurate ground truth. Inthese sequences, the aim is to mimic the effect of the injectionof a stimulus such as insulin into a pancreatic cell. Fig. 5 showstwo frames of the synthetic sequences using this frameworkwhich are comparable with the real TIRFM sequences shownin Fig. 1.

Each synthetic sequence was simulated with spatial res-olution of 158nm/pixel and temporal resolution 10 fps andconsists of time-varying number of targets (on average ≈ 190particles per frame) moving through 60 time frames insidea cell membrane (an estimated background) that is extractedfrom real TIRFM sequences with effective region ≈ 230×230pixels. The spots were generated in different sizes, ≈ 1.2−4.5

5Contrary to the OSPA, the OSPA-T is not a metric in the formalmathematical sense. However, it can be used as a performance measure.


(a) (b)

Fig. 5. Two frames of the synthetic TIRFM sequences generated by theframework proposed in [50].

pixels (200 − 700 nm). The dynamics of the targets weremodeled using random walk and linear movement. Further-more, targets can switch between these two dynamics. Thenumber of intersecting and touching spots in each framewas counted according to the Rayleigh resolution [51]. Theaveraged percentage of intersecting and touching spots perframe in these sequences is equal to 1.2%.

Due to the 3-D motion of the structures, their intensitychanges according to the TIRFM exponential equation [50].Therefore, they may either temporarily or permanently disap-pear from the frames. New born targets may also graduallyappear from the background. The sequences are contaminatedwith Poisson noise and the main characteristics of the se-quences such as background intensity and noise level graduallychange based on a model extracted from real sequences. Dueto spatio-temporally varying noise level and backgrounds anddynamic intensity of spots, the signal-to-noise ratio (SNR)6 ofan object cannot be constant. Instead, it varies between 1 and9.

Similar to the real TIRFM data, the spots in the syntheticsequences fade over time as noise level and backgroundintensity gradually increases. From a biological perspective, itis important to assess the dynamics of vesicles after injectionof the stimulus which leads the escalation of noise level andbackground intensity. Therefore, the threshold in the detectionmethod needs to be set low to ensure consistent detectionsof the objects in order to avoid early track termination. Tothis end, we chose a value for the threshold such that theaveraged detection probability for all sequences pD is about0.9. However, this scenario noticeably increases the clutter rateand its variation over time. Fig. 6 shows that the mean clutterrate estimated using the proposed MM-λ-pD-CPHD filter canappropriately follow the quick changes in the ground truthclutter rate.

In this experiment, the performance of the trackers was eval-uated using these time-varying and highly cluttered detectionsand their results are reported in Table I. In this Table, the errorsare averaged over the number of frames in all 10 synthetic

6SNR value is used for representing the level of noise and is calculatedusing definition of Smal et al. [22] which is the difference in intensity betweenthe object and the background, divided by the standard deviation of the objectnoise.

0 10 20 30 40 50 60

30

50

70

90

110

130

150

170

190

Time

Clu

tter

Rat

e

Ground truth rate

Estimated rate

Averaged rate

Fig. 6. The ground truth, averaged and estimated (using the proposed MM-λ-pD-CPHD filter) clutter rates for a synthetic image sequence in the highclutter scenario (pD = 0.88 and λ = 112).

sequences. The results show that our bootstrap filter benefitsfrom a reliable estimator, the MM-λ-pD-CPHD filter, whichaccurately estimates the detection probability and clutter rate.Consequently compared to the other tracker, its tracking resultscontain fewer false and missing tracks, which decreases thecardinality error significantly. Similarly, this error is also lowin the MM-λ-pD-CPHD filter. The other Bayesian trackerscannot benefit from these estimations as their parametersare fixed. Therefore, they have higher cardinality error. Incomparison with traditional Bayesian trackers such as theIMM-JPDA and MHT trackers, the RFS filters have relativelybetter cardinality error as they properly incorporate birth, deathand clutter models in their formulations. However, the higherdifference between the OSPA and OSPA-T errors of the allRFS filters represent that their results include more identityswitch errors compared to the other trackers due to the simpletag propagation scheme used.

The performance of the P-tracker as one of the deterministictrackers seems to be sensitive to cluttered measurements.Therefore, it has the highest cardinality error of the trackers.In contrast, the results show that another deterministic trackingscheme, U-tracker, can robustly track the particles whiledealing with highly cluttered detections.

The P-Tracker has the lowest location error of all trackers.Although this error reflects the accuracy of the trackers intracking particles, its lower value can be also an artifactof the tracker’s poor performance. The trackers with highercardinality error have relatively lower location error and viceversa.

The method rankings may change in other scenarios, e.g.where the clutter rate is low and its variation over time canbe ignored. In our application, the clutter rate and its varia-tion over time can be decreased by increasing the detectionthreshold value. For example, we chose a value such thatthe averaged clutter rate λ and detection probability pD arerespectively about 11 and 0.7. In this case, the detected pointsfrom a target are inconsistent and many faint particles are notdetected after initial time frames. This case is undesirable for


0 10 20 30 40 50 600.6

0.65

0.7

0.75

0.8

0.85

Time

Det

ecti

on P

robab

ilty

Ground truth pD

Estimated pD

Averaged pD

Fig. 7. The ground truth, averaged and estimated (using the proposed MM-λ-pD-CPHD filter) detection probability for a synthetic image sequence inthe low clutter scenario (pD = 0.7 and λ = 11).

our application due to many truncated tracks with short lifetime. Nevertheless, we report the tracker’s OSPA and OSPA-Terrors in this case in Table II in order to compare their differentperformance in the low and high clutter rate scenarios. Notethat both false tracks in high clutter scenario and missed ortruncated tracks in low clutter scenario increase the OSPA andOSPA-T errors.

The results show that the trackers such as the IMM-JPDAfilter and P-Tracker which have the worst performance inhighly cluttered measurements perform better for a low clutterrate with long missed detections compared to other trackers.In contrast, the U-Tracker cannot perform reliably in this caseas it has the highest cardinality error. Estimating decliningtrend in the detection probability (Fig. 7) helps our bootstraptracker to still have lower OSPA error compared to the othertrackers, but relatively high OSPA-T error due to the simpletag propagation scheme.

In a nutshell, the performance of the trackers varies ac-cording to the detector reliability. Our bootstrap approach hassuperior performance for highly cluttered detections with thetime-varying rates.

TABLE ITHE AVERAGED LOCATION, CARDINALITY, OSPA AND OSPA-T ERRORSOF THE TRACKERS IN PIXEL WITH THE METRIC PARAMETERS p = 1 ANDc = ` = 10 (PIXEL) FOR THE 10 SYNTHETIC SEQUENCES IN THE HIGH

CLUTTER SCENARIO (pD = 0.88 AND λ = 112).

Method Location Cardinality OSPA OSPA-T

B-MM-CPHD 2.52 0.56 3.08 5.41

MM-λ-pD-CPHD 2.64 0.63 3.27 5.48

MM-CPHD 2.39 1.12 3.51 5.50

MM-PHD [16] 2.71 1.30 4.01 6.55

IMM-JPDA [19] 1.54 2.68 4.22 5.72

MHT [14] 1.71 2.23 3.94 5.67

P-Tracker [6] 1.33 2.70 4.03 6.50

U-Tracker [4] 1.85 1.43 3.28 5.04

D. Evaluation on real data

The tracking methods were also tested on a real TIRFM se-quence with spatial and temporal resolution of ≈ 150nm/pixeland 10 fps respectively. The image sequences were acquiredfrom a pancreatic beta cell injected by insulin during theacquisition.

In order to prepare a ground truth, an independent expertmanually tracked all visible structures (332 trajectories from21752 spots) in a part of the cell within 500 time frames usingthe freely available software MTrackJ [52]. The annotated tra-jectories were double-checked by another biologist to ensurereliability of the ground truth.

In order to detect and track all structures, especially afterinjection of insulin, the threshold in the detection method wasset low, which noticeably increases clutter rate and its variationover time (Fig. 8). The results of the trackers for the real dataare reported in Table III. In this Table, the errors are averagedover the number of time steps in the real image sequence.

The higher clutter rate along with significantly higher num-ber of time frames and numerous faint structures existing inreal sequences cause higher values of the OSPA and OSPA-T errors of the all trackers for real sequences compared tothe synthetic data. In addition, the results confirm our argu-ments about the performance of the trackers in the syntheticsequences for the same high clutter scenario. The ability of ourB-MM-CPHD to properly track true targets while accuratelyestimating the clutter rate and detection probability (Fig. 8)leads to the lowest cardinality and also OSPA errors.

Since there were always faint structures in real sequencesthat even experienced annotators were unable to detect ordetermine whether they are real or false targets, the reliabilityof the manual ground truth cannot be completely guaranteed.In this case, the quantitative results may be biased to a specificmethod. To maximize the validity of our comparison, theresults of the tracking were also visually assessed by theexperts. This assessment also showed that our bootstrap filtercan better detect and track the real vesicles, especially faintones, while avoiding tracking false targets.

In Fig. 9 (a), the tracking results of several crossing particleswith maneuvering motions using the proposed bootstrap filter

TABLE IITHE AVERAGED LOCATION, CARDINALITY, OSPA AND OSPA-T ERRORSOF THE TRACKERS IN PIXEL WITH THE METRIC PARAMETERS p = 1 ANDc = ` = 10 (PIXEL) FOR THE 10 SYNTHETIC SEQUENCES IN THE LOW

CLUTTER SCENARIO (pD = 0.7 AND λ = 11).


B-MM-CPHD 2.71 0.68 3.39 5.39

MM-λ-pD-CPHD 2.78 0.82 3.60 5.97

MM-CPHD 2.84 0.79 3.63 5.61

MM-PHD [16] 2.90 1.08 3.98 6.01

IMM-JPDA [19] 1.37 1.67 3.04 4.21

MHT [14] 0.81 3.07 3.88 5.48

P-Tracker [6] 0.85 2.55 3.40 5.20

U-Tracker [4] 0.64 3.36 4.00 4.93


0 100 200 300 400 50070

90

110

130

150

170

Time

Clu

tter

Rat

e

Ground truth rate

Estimated rate

Averaged rate

(a)

0 100 200 300 400 500

0.85

0.9

0.95

1

Time

Det

ecti

on p

roba

bilt

y

Ground truth pD

Estimated pD

Averaged pD

(b)

Fig. 8. The ground truth, averaged and estimated (a) clutter rate and (b) detection probability using the proposed MM-λ-pD-CPHD filter for the real imagesequence.

are shown. The results show that it can properly deal withthe maneuvering motion of the targets. However, the resultsare not error free and include some missing and false tracksand switching labeling errors. Fig. 9 (b) also demonstrates theperformance of our proposed bootstrap filter in tracking twofaint particles moving through the synthetic and real sequenceswith time-varying background and noise level.

V. CONCLUSION

The estimation of clutter rate and detection probabilityhelps improve tracking of the targets in particle trackingapplications where measurements include many spurious andmissed detections with unknown and time-varying parameters.The RFS framework provides a principled solution to deal withthe estimation of these parameters which was not possiblein previous approaches. The λ-pD-CPHD filter is one of thefilters derived based on RFS theory which is able to estimatethese parameters. However, the filter cannot naturally performas well as the CPHD filter with known parameters. To this end,in this paper we proposed a bootstrap filter by combinationof the CPHD and λ-pD-CPHD filters. To accommodate themaneuvering motion, we also proposed a multiple modelimplementation of the filters. Therefore, the clutter rate anddetection probability are estimated by the multiple model λ-pD-CPHD filter bootstrapped onto a multiple model CPHDfilter that outputs target estimates.

The proposed approach was evaluated on a challengingparticle tracking application where vesicles move in noisysequences of total internal reflection fluorescence microscopywhile the noise characteristic and background intensity ofthe sequences change during the acquisition process. In thisapplication which the clutter rate and detection probability aretime-varying, we demonstrated that our bootstrap filter is ableto better track the real targets and more reliably avoid falsetracks compared to the other RFS filters such as the MM-PHD, MM-CPHD and MM-λ-pD-CPHD as well as other state-of-the-art particle tracking methods such as the IMM-JPDA,

MHT, P-Tracker and U-Tracker in both synthetic and realsequences. However, in the applications where the motion ofeach individual targets is required, the tag propagation schememay need to be changed to a track management algorithm,e.g., that proposed in [46, 47] or more generally, the trackerin the bootstrap filter can be replaced by any multi-targettracker that requires knowledge of false alarm and detectionrate and performs reliably in the label assignments. In the RFSconcept, a principled solution to the track labeling problemusing labeled RFS [39] has been recently proposed which canbe also used for this purpose.

Obviously, our approach may not be a good choice for someapplications. For example, in the cases where the particlescan be easily detected or the rates for false alarms andmissed detections are negligible, known or time-invariant,there is no point to adaptively estimate the clutter rate and thedetection profile. In addition, to estimate these time-varyingrates, the MM-λ-pD-CPHD requires some probabilistic priorsdescribing how the clutter rate and the detection probabilitychange over time [29]. Therefore, this approach requires moreparameters compared to other trackers. Moreover, the boot-strap method filters twice in each frame in order to estimatethe state of the targets. Consequently, its processing time ismore than each of its component filters, MM-λ-pD-CPHD andMM-CPHD. Our non-optimized MATLAB code was run onan ordinary PC (Intel Core 2 Quad, 2.66 GHz CPU, 8 GBRAM). The average CPU processing time per frame per targetin the highest considered clutter rate (around 280 detectedpositions per frame) for the multiple model CPHD, λ-pD-CPHD and our bootstrap filter are about 17.1ms, 8.0ms and25.1ms respectively.

Due to linearity of the dynamic and measurement modelsin our application, we only used the multiple model Gaussianlinear implementation of all filters. The recursive equationsderived in the appendices can be used for both Gaussian linearand non-Gaussian non-linear system models. In order to applythe proposed framework to applications with non-linear non-


(a) (b)

Fig. 9. Some examples of tracked vesicles using the proposed bootstrap filter (dashed line) against the ground truth (solid line) in the synthetic and realTIRFM sequences. (a) The tracking results of multiple crossing particles with maneuvering motions in the synthetic TIRFM sequences. (b) The resultingtrajectory of two faint vesicles from the synthetic (Top row) and real (Bottom row) TIRFM sequences in different time frames. For an improved visualization,the other tracks are eliminated. These particles move in different background intensity and noise level.

Gaussian system models, the sequential Monte Carlo (SMC)implementation of the filters is required. This implementationwill also allows us to further evaluate the performance of ourbootstrap filter and compare it against the SMC-PHD [41],SMC-CPHD [31] and the traditional particle [34] filters.

APPENDIX

For completeness, we now derive the recursive equationsfor both the multiple model CPHD and λ-pD-CPHD filters inthis section. The following notations are used in throughoutthis Appendix. The binomial and permutation coefficients aredenoted by Clj and Pnj , respectively. 〈·, ·〉 is the inner productoperation between two continuous or two discrete functions.The elementary symmetric function of order j defined for afinite set Z of real numbers is denoted by

ej(Z) =∑

S⊆Z,|S|=j

(∏θ∈S

θ

), (29)

where |S| is the cardinality of a set S and e0(Z) = 1 [42].

TABLE IIITHE AVERAGED LOCATION, CARDINALITY, OSPA AND OSPA-T ERRORS

OF THE TRACKERS

in pixel with the metric parameters p = 1 and c = ` = 10 (pixel) for a realimage sequence.


B-MM-CPHD 3.91 1.04 4.95 6.30

MM-λ-pD-CPHD 4.01 1.23 5.24 6.56

MM-CPHD 3.92 2.07 5.99 7.23

MM-PHD [16] 4.05 2.23 6.28 7.64

IMM-JPDA [19] 0.93 5.75 6.68 7.25

MHT [14] 1.21 4.28 5.49 6.33

P-Tracker [6] 0.55 5.45 6.00 7.32

U-Tracker [4] 1.96 3.63 5.59 6.44

A. Multiple Model CPHD Recursions

Prediction step: Suppose at time k − 1, the posteriorcardinality distribution ρk−1 and posterior intensity vk−1

are known. The predicted cardinality distribution ρk|k−1 andpredicted intensity vk|k−1 are calculated by

ρk|k−1(n) =

n∑j=0

ρΓ(n− j)Π[vk−1, ρk−1](j), (30)

vk|k−1(x, r) = γ(x)π(r)+∑r

∫pS

(x, r)f(x|x, r)τ(r|r)vk−1(x, r)dx,(31)

where Π[vk−1, ρk−1](j) =

∞∑l=j

Cljρk−1(l)〈pS, vk−1〉j〈1− pS , vk−1〉l−j

〈1, vk−1〉l−j. (32)

Note that in the multiple model approach, the inner productfunction operates on both the kinematic state and the model,i.e. 〈p

S, vk−1〉 =

∑r

∫pS(x, r)vk−1(x, r)dx.

Update step: If at time k, the predicted cardinality distribu-tion ρk|k−1, predicted intensity vk|k−1 and set of measurementZk are given, the updated cardinality distribution ρk andupdated intensity vk are calculated by

ρk(n) =Υ0[vk|k−1, Zk

](n)ρk|k−1(n)⟨

Υ0[vk|k−1, Zk

], ρk|k−1

⟩ , (33)

vk(x, r) = vk|k−1(x, r)×(1− pD

(x, r))

⟨Υ1[vk|k−1, Zk

], ρk|k−1

⟩⟨

Υ0[vk|k−1, Zk

], ρk|k−1

⟩+∑z∈Zk

ψz(x, r)

⟨Υ1[vk|k−1, Zk\z

], ρk|k−1

⟩⟨

Υ0[vk|k−1, Zk

], ρk|k−1

⟩ , (34)


where Υu[vk|k−1, Zk

](n) =

min(|Zk|,n)∑j=0

(|Zk| − j)!ρK(|Zk| − j)Pnj+u

×〈1− p

D, vk|k−1〉n−(j+u)

〈1, vk|k−1〉nej

(Ξ(vk|k−1, Zk)

) (35)

where

Ξ(vk|k−1, Zk) ={⟨vk|k−1, ψz

⟩: z ∈ Zk

}, (36)

andψz(x, r) =

〈1, κ〉κ(z)

g(z|x, r)pD

(x, r). (37)

B. Multiple Model λ-pD-CPHD Recursions

Prediction step: Suppose at time k−1, the hybrid cardinalitydistributions ρk−1 and the posterior intensity distribution foractual targets v(1)

k−1 and clutter generators v(0)k−1 are given. The

hybrid predicted cardinality distribution ρk|k−1 and predictedintensity for actual targets v(1)

k|k−1 and clutter generators v(0)k|k−1

are calculated by

ρk|k−1(n) =

n∑j=0

ρΓ(n− j)∞∑l=j

Clj ρk−1(l)(1− φ)l−jφj

(38)

where

φ =

(〈p(1)S , v

(1)k−1〉+ 〈p(0)

S , v(0)k−1〉

〈1, v(1)k−1〉+ 〈1, v(0)

k−1〉

), (39)

v(1)k|k−1(x, a, r) = γ(1)

k(x, a)π(r)+∑

r

∫ ∫ 1

0

[p

(1)S (x, r)f (1)(x|x, r)×

f (∆)(a|a, r)τ(r|r)v(1)k−1(x, a, r)dxda

], (40)

v(0)k|k−1(b) = γ(0)(b) + p

(0)S v

(0)k−1(b). (41)

Update step: If at time k, the predicted intensity for actualtargets v(1)

k|k−1, the predicted intensity for clutter generators

v(0)k|k−1, the predicted hybrid cardinality distribution ρk|k−1

and set of measurement Zk are all given and the functionΥu[vk|k−1, Zk

](n) defined as follows:

Υu[vk|k−1, Zk

](n) = 0 n < |Zk|+ u

P n|Zk|+uΦn−(|Zk|+u) n ≥ |Zk|+ u(42)

where

Φ = 1−

⟨v

(1)k|k−1, p

(1)D

⟩+⟨v

(0)k|k−1, p

(0)D

⟩⟨

1, v(1)k|k−1

⟩+⟨

1, v(0)k|k−1

⟩ , (43)

where p(1)D (x, a, r) = a and p(0)

D (b) = b.

The updated cardinality distribution ρk and the updated in-tensity distribution for actual targets v(1)

k and clutter generatorsv

(0)k are given as follows

ρk(n) =

0 n < |Zk|,Υ

0[vk|k−1, Zk

](n)ρk|k−1(n)⟨

Υ0, ρk|k−1

⟩ n ≥ |Zk|,

(44)

v(1)k (x, a, r) = v

(1)k|k−1(x, a, r) (1− a)

〈Υ1[vk|k−1,Zk],ρk|k−1〉〈Υ0[vk|k−1,Zk],ρk|k−1〉⟨

1, v(1)k|k−1

⟩+⟨

1, v(0)k|k−1

⟩ +

∑z∈Zk

a.g(z|x, r)⟨v

(0)k|k−1, p

(0)D K

⟩+⟨v

(1)k|k−1, p

(1)D g(z|·)

⟩ , (45)

v(0)k (b) = v

(0)k|k−1(b)

(1− b) 〈Υ1[vk|k−1,Zk],ρk|k−1〉

〈Υ0[vk|k−1,Zk],ρk|k−1〉⟨1, v

(1)k|k−1

⟩+⟨

1, v(0)k|k−1

⟩ +

∑z∈Zk

b.K(z)⟨v

(0)k|k−1, p

(0)D K

⟩+⟨v

(1)k|k−1, p

(1)D g(z|·)

⟩ . (46)

ACKNOWLEDGMENT

The authors would like to thank Dr. William E. Hughesand the research team from the Garvan Institute of MedicalResearch, NSW, Australia, for providing the real TIRFMsequences and their manual ground truth. The authors alsothank Dr. Anthony Dick from the University of Adelaidewhose comments and suggestions improved presentation ofthe paper.

REFERENCES

[1] A. Matov, M. M. Edvall, G. Yang, and G. Danuser,“Optimal-flow minimum-cost correspondence assign-ment in particle flow tracking,” Comput. Vis. Image Und.,vol. 115, no. 4, pp. 531–540, 2011.

[2] S. Bonneau, M. Dahan, and L. D. Cohen, “Single quan-tum dot tracking based on perceptual grouping usingminimal paths in a spatiotemporal volume,” IEEE Trans.Image Process., vol. 14, no. 9, pp. 1384–1395, 2005.

[3] G. Mashanov and J. Molloy, “Automatic detection ofsingle fluorophores in live cells,” Biophys. J., vol. 92,no. 6, pp. 2199–2211, 2007.

[4] K. Jaqaman, D. Loerke, M. Mettlen, H. Kuwata, S. Grin-stein, S. L. Schmid, and G. Danuser, “Robust single-particle tracking in live-cell time-lapse sequences,” Nat.methods, vol. 5, no. 8, pp. 695–702, 2008.

[5] M. Dewan, M. Ahmad, and M. Swamy, “Tracking biolog-ical cells in time-lapse microscopy: an adaptive technique


combining motion and topological features,” IEEE Trans.Biomed. Eng., vol. 58, no. 6, pp. 1637–1647, 2011.

[6] I. Sbalzarini and P. Koumoutsakos, “Feature point track-ing and trajectory analysis for video imaging in cellbiology,” J. Struct. Biol., vol. 151, no. 2, pp. 182–195,2005.

[7] D. Padfield, J. Rittscher, and B. Roysam, “Coupledminimum-cost flow cell tracking for high-throughputquantitative analysis,” Med. Image Anal., vol. 15, no. 4,pp. 650–668, 2011.

[8] E. Meijering, I. Smal, and G. Danuser, “Tracking inmolecular bioimaging,” IEEE Signal Proc. Mag., vol. 23,no. 3, pp. 46–53, 2006.

[9] D. House, M. Walker, Z. Wu, J. Wong, and M. Betke,“Tracking of cell populations to understand their spatio-temporal behavior in response to physical stimuli,” inProc. IEEE Conf. Comput. Vis. Mach. Intell. (CVPR2009), 2009, pp. 186–193.

[10] L. Liang, H. Shen, P. De Camilli, D. Toomre, andJ. Duncan, “An expectation maximization based methodfor subcellular particle tracking using multi-angle TIRFmicroscopy,” in Medical Imag. Comput. Computer-Assist.Interv. (MICCAI 2011), 2011, pp. 629–636.

[11] N. H. Nguyen, S. Keller, E. Norris, T. T. Huynh, M. G.Clemens, and M. C. Shin, “Tracking colliding cells invivo microscopy,” IEEE Trans. Biomed. Eng., vol. 58,no. 8, pp. 2391–2400, 2011.

[12] A. Genovesio, T. Liedl, V. Emiliani, W. Parak,M. Coppey-Moisan, and J. Olivo-Marin, “Multiple parti-cle tracking in 3-D+t microscopy: Method and applica-tion to the tracking of endocytosed quantum dots,” IEEETrans. Image Process., vol. 15, no. 5, pp. 1062–1070,2006.

[13] L. Yang, Z. Qiu, A. Greenaway, and W. Lu, “A newframework for particle detection in low-SNR fluores-cence live-cell images and its application for improvedparticle tracking,” IEEE Trans. Biomed. Eng., vol. 59,no. 7, pp. 2040–2050, 2012.

[14] N. Chenouard, I. Bloch, and J.-C. Olivo-Marin, “Mul-tiple hypothesis tracking for cluttered biological imagesequences,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 35, no. 11, pp. 2736–2750, 2013.

[15] R. Juang, A. Levchenko, and P. Burlina, “Trackingcell motion using GM-PHD,” in Proc. IEEE Int. Symp.Biomed. Imag. (ISBI 2009), 2009, pp. 1154–1157.

[16] S. H. Rezatofighi, S. Gould, B.-N. Vo, K. Mele, W. E.Hughes, and R. Hartley, “A multiple model probabilityhypothesis density tracker for time-lapse cell microscopysequences,” in Inform. Process. Medical Imag. (IPMI2013), 2013, pp. 110–122.

[17] T. Wood, C. Yates, D. Wilkinson, and G. Rosser, “Sim-plified multitarget tracking using the PHD filter formicroscopic video data,” IEEE Trans. Circ. Syst. Vid.,vol. 22, no. 5, pp. 702–713, 2012.

[18] L. Feng, Y. Xu, Y. Yang, and X. Zheng, “Multipledense particle tracking in fluorescence microscopy im-ages based on multidimensional assignment,” J. Struct.Biol., vol. 173, no. 2, pp. 219–228, 2011.

[19] S. H. Rezatofighi, S. Gould, R. Hartley, K. Mele, andW. E. Hughes, “Application of the IMM-JPDA filter tomultiple target tracking in total internal reflection fluo-rescence microscopy images,” in Medical Imag. Comput.Computer-Assist. Interv. (MICCAI 2012), 2012, pp. 357–364.

[20] K. Li, E. D. Miller, M. Chen, T. Kanade, L. E. Weiss, andP. G. Campbell, “Cell population tracking and lineageconstruction with spatiotemporal context,” Med. ImageAnal., vol. 12, no. 5, pp. 546–566, 2008.

[21] I. Smal, E. Meijering, K. Draegestein, N. Galjart, I. Grig-oriev, A. Akhmanova, M. Van Royen, A. Houtsmuller,and W. Niessen, “Multiple object tracking in molecularbioimaging by Rao-Blackwellized marginal particle fil-tering,” Med. Image Anal., vol. 12, no. 6, pp. 764–777,2008.

[22] I. Smal, K. Draegestein, N. Galjart, W. Niessen, andE. Meijering, “Particle filtering for multiple object track-ing in dynamic fluorescence microscopy images: Appli-cation to microtubule growth analysis,” IEEE Trans. Med.Imag., vol. 27, no. 6, pp. 789–804, 2008.

[23] W. Godinez, M. Lampe, S. Worz, B. Muller, R. Eils,and K. Rohr, “Deterministic and probabilistic approachesfor tracking virus particles in time-lapse fluorescence mi-croscopy image sequences,” Med. Image Anal., vol. 13,no. 2, pp. 325–342, 2009.

[24] R. Hoseinnezhad, B.-N. Vo, B.-T. Vo, and D. Suter,“Visual tracking of numerous targets via multi-Bernoullifiltering of image data,” Pattern Recogn., vol. 45, no. 10,pp. 3625–3635, 2012.

[25] L. Yuan, Y. F. Zheng, J. Zhu, L. Wang, and A. Brown,“Object tracking with particle filtering in fluorescencemicroscopy images: Application to the motion of neu-rofilaments in axons,” IEEE Trans. Med. Imag., vol. 31,no. 1, pp. 117–130, 2012.

[26] N. Chenouard, I. Smal, F. De Chaumont, M. Maska,I. F. Sbalzarini, Y. Gong, J. Cardinale, C. Carthel,S. Coraluppi, M. Winter et al., “Objective comparisonof particle tracking methods,” Nat. methods, vol. 11, pp.281–289, 2014.

[27] I. Smal, M. Loog, W. Niessen, and E. Meijering, “Quan-titative comparison of spot detection methods in fluores-cence microscopy,” IEEE Trans. Med. Imaging, vol. 29,no. 2, pp. 282–301, 2010.

[28] J. Burchfield, J. Lopez, K. Mele, P. Vallotton, andW. Hughes, “Exocytotic vesicle behaviour assessed byTIRFM,” Traffic, vol. 11, pp. 429–439, 2010.

[29] R. P. Mahler, B.-T. Vo, and B.-N. Vo, “CPHD filteringwith unknown clutter rate and detection profile,” IEEETrans. Signal Process., vol. 59, no. 8, pp. 3497–3513,2011.

[30] M. Beard, B.-T. Vo, and B.-N. Vo, “Multi-target filteringwith unknown clutter density using a bootstrap GM-CPHD filter,” IEEE Signal Proc. Let., vol. 20, no. 4, pp.323–326, 2013.

[31] R. P. Mahler, Statistical multisource-multitarget informa-tion fusion. Artech House Boston, 2007, vol. 685.

[32] Y. Bar-Shalom, Tracking and data association. Aca-


demic Press Professional, Inc., 1987.[33] S. S. Blackman, Multiple-target tracking with radar

applications. Artech House, Inc.,, 1986.[34] C. Hue, J.-P. Le Cadre, and P. Perez, “Tracking multi-

ple objects with particle filtering,” IEEE Trans. Aerosp.Electron. Syst., vol. 38, no. 3, pp. 791–812, 2002.

[35] R. Mahler, “Multitarget Bayes filtering via first-ordermultitarget moments,” IEEE Trans. Aerosp. Electron.Syst., vol. 39, no. 4, pp. 1152–1178, 2003.

[36] ——, “PHD filters of higher order in target number,”IEEE Trans. Aerosp. Electron. Syst., vol. 43, no. 4, pp.1523–1543, 2007.

[37] B.-T. Vo, B.-N. Vo, and A. Cantoni, “The cardinalitybalanced multi-target multi-Bernoulli filter and its imple-mentations,” IEEE Trans. Signal Process., vol. 57, no. 2,pp. 409–423, 2009.

[38] B.-N. Vo, B.-T. Vo, N.-T. Pham, and D. Suter, “Jointdetection and estimation of multiple objects from imageobservations,” IEEE Trans. Signal Process., vol. 58,no. 10, pp. 5129–5141, 2010.

[39] B.-T. Vo and B.-N. Vo, “Labeled random finite setsand multi-object conjugate priors,” IEEE Trans. SignalProcess., vol. 61, no. 13, pp. 3460–3475, 2013.

[40] R. P. Mahler, Advances in Statistical Multisource-Multitarget Information Fusion. Artech House Boston,2014.

[41] B.-N. Vo, S. Singh, and A. Doucet, “Sequential MonteCarlo methods for multitarget filtering with random finitesets,” IEEE Trans. Aerosp. Electron. Syst., vol. 41, no. 4,pp. 1224–1245, 2005.

[42] B.-T. Vo, B.-N. Vo, and A. Cantoni, “Analytic implemen-tations of the cardinalized probability hypothesis densityfilter,” IEEE Trans. Signal Process., vol. 55, no. 7, pp.3553–3567, 2007.

[43] S. Pasha, B.-N. Vo, H. Tuan, and W. Ma, “A Gaussianmixture PHD filter for jump Markov system models,”IEEE Trans. Aerosp. Electron. Syst., vol. 45, no. 3, pp.919–936, 2009.

[44] S. H. Rezatofighi, R. Hartley, and W. E. Hughes, “A newapproach for spot detection in total internal reflection flu-orescence microscopy,” in Proc. IEEE Int. Symp. Biomed.Imag. (ISBI 2012), 2012, pp. 860–863.

[45] B.-N. Vo and W. Ma, “The Gaussian mixture probabilityhypothesis density filter,” IEEE Trans. Signal Process.,vol. 54, no. 11, pp. 4091–4104, 2006.

[46] L. Lin, Y. Bar-Shalom, and T. Kirubarajan, “Track label-ing and PHD filter for multitarget tracking,” IEEE Trans.Aerosp. Electron. Syst., vol. 42, no. 3, pp. 778–795, 2006.

[47] K. Panta, B.-N. Vo, and S. Singh, “Novel data associationschemes for the probability hypothesis density filter,”IEEE Trans. Aerosp. Electron. Syst., vol. 43, no. 2, pp.556–570, 2007.

[48] D. Schuhmacher, B.-T. Vo, and B.-N. Vo, “A consistentmetric for performance evaluation of multi-object filters,”IEEE Trans. Signal Process., vol. 56, no. 8, pp. 3447–3457, 2008.

[49] B. Ristic, B.-N. Vo, D. Clark, and B.-T. Vo, “A metricfor performance evaluation of multi-target tracking algo-

rithms,” IEEE Trans. Signal Process., vol. 59, no. 7, pp.3452–3457, 2011.

[50] S. H. Rezatofighi, W. T. E. Pitkeathly, S. Gould, R. Hart-ley, K. Mele, W. E. Hughes, and J. G. Burchfield, “Aframework for generating realistic synthetic sequences oftotal internal reflection fluorescence microscopy images,”in Proc. IEEE Int. Symp. Biomed. Imag. (ISBI 2013),2013.

[51] L. Rayleigh, “Investigations in optics, with special refer-ence to the spectroscope,” Philos. Mag., vol. 8, no. 49,pp. 261–274, 1879.

[52] E. Meijering, “MTrackJ: A Java program formanual object tracking,” [Online].Available:http://www.imagescience.org/meijering/software/mtrackj/.

ieee transactions on medical imaging, vol. 34, no. 6, …

Documents