Relations as Context to improve Multi-Target Tracking and
Activity Recognition.
Cristina Manfredotti12, Enza Messina1, David Fleet2
1DISCo, University of Milano-Bicocca2Computer Science Dept, University of Toronto
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 2
Relations to improve tracking
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 3
Complex activity recognition
Y.Ke, R.Sukthankar, M.Hebert; Event Detection in Crowed Videos
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 4
Objectives
Goals: 1. To model relations and 2. To maintain beliefs over particular
relations between objects
In order to simultaneously:
• Improve tracking with informed predictions and
• Identify complex activities based on observations and prior knowledge
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 5
Relational Domain
Relational Domain: set of objects characterized by attributes1 and with relations1 between them
BoatIdcolorposition(t)velocity(t)direction(t)DecreasingVelocity(t)
A
SameDirection(t)distance(t)Before(t)
Boat BIdcolorposition(t)velocity(t)direction(t)DecreasingVelocity(t)SameDirection(t)distance(t)Before(t)
1Attributes and relations are predicate in FOL.
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 6
A Parenthesis:
To model uncertainty in a Relational Domain we will use
Relational (Dynamic) Bayesian Networks
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 7
BN: the Alarm example
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 8
Thanks to Mark Chavira
A large BN
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 9
The Alarm Relational Domain
Relational Domain contains a set of objects with relations between them
Objects
e.g.: Relation
• neighbor• alarm• burglar
• toCall (the howner of the house)• toHear (the alarm)
neighbor’s attributes: capacity of hearing, attention, ...
alarm’s attributes: its volume, its sensibility, ...
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 10
Alarm RBN:
Alarm.Volume
NeighborCalls
Earthquacke
Neigh.DegOfDef
Neigh.NoiseAround
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 11
Closing the parenthesis ...
• Syntax RBN:– a set of nodes, one for each variable
– a directed, acyclic graph – a conditional distribution for each node
given its parents
This distribution must take into account the actual “complexity” of the nodes!
• Syntax RBN:– a set of nodes, one for each predicate
– a directed, graph– a conditional distribution for each node
given its parents,
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 12
Relational State
The State of a Relational Domain is the set of the predicates that are true in the Domain.
r
a
s
ss
Relational state
State of attributes
State of relations
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 13
Dynamics
The State of a Relational Domain is the set of the predicates that are true in the Domain.
State evolves with time
We extend a RBN to a RDBN as we are used to extend a BN to a DBN.
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 14
Inference
Markov assumption andConditional independence of data on state.
bel(st) = ® p(zt|st)s p(st|st-1)bel(st-1)dst-1
Bayesian Filter
The problem of computing:
bel(st) = p(st|z1:t)
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 15
p(xt-1|z1:t-1) p(xt|z1:t-1) p(xt|z1:t)
Bel(xt-1) Bel(xt) Bel(xt)
Transition model
Sensor model
t = t+1
~
Transition model
Sen
sor
mo
del
Inference
Relations in the State result in correlating the State of different objects between them
p(xt-1|z1:t-1) p(xt|z1:t-1) p(xt|z1:t)
Bel(xt-1) Bel(xt) Bel(xt)
Transition model
Sensor model
t = t+1
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 16
Sensor model (1st assumption)
part of the state relative to relations, sr, not directly observable
p(zt|st) = p(zt|sa
t)
observation zt independent by the relations between objects.
Intuitively:
Travelling Together vs Being Close
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 17
Transition model: a trick
p(st|st-1) = p(sat,sr
t|sat-1, sr
t-1)
Sat-1
Srt-1
Sat
Srt
Intuitive
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 18
p(sat,sr
t|sat-1,sr
t-1)=
But srt independent by sa
t-1 given srt-1 and sa
t
p(sat,sr
t|sat-1,sr
t-1) = p(sat|sa
t-1,srt-1) p(sr
t|srt-1, sa
t)
bel(st) = p(st|z1:t) = p(sat,sr
t|z1:t)
bel(st)=αp(zt|sat,sr
t)s p(sat,sr
t|sat-1,sr
t-1)bel(st-1)dst-1
p(zt|sat,sr
t) = p(zt|sat)
Relational Inference
p(sat|sa
t-1,srt-1) p(sr
t|sat-1,sr
t-1, sat)
Transition model (2nd assumption)
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 19
Conditional Probability Distribution
FOPT: a Probabilistic Tree whose nodes are FOL formulas
CPD relationt(x,y):
relationt-1(x,y)
p(relationt(x,y))
xt, yt
CPD yt:
x, relationt-1(x,y)
p(yt|yt-1)
TF
p(yt|yt-1) p(xt|xt-1,yt-1,rt-1)
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 20
* It is a technique that implements a recursive Bayesian Filter through a Monte Carlo simulation. The key idea is to represent the posterior pdf as a set of samples (particles) paired with weights and to filter the mesurament based on these weights..
Particle Filtering* (general case)
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 21
Relational Particle Filter
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 22
RPF: extraction
Xat,(m)
Xrt,(m)
Xat,(m)
~ p(xat,(m)|sa
t-1,srt-1)
Xat,(m)
~ p(xrt,(m)|sa
t = xat,(m),sr
t-1)
Xrt,(m)
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 23
RPF: weighting
The consistency of the probability function ensures the convergence of the algorithm.
Xat,(m)
Xrt,(m)
Weight ( ) ~p(zt|xat)
The weighting step is done according to the attributes part of each particle only, the relational part follows.
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 24
Tracking AND activity Recognition
Xat,(m)
Xrt,(m)
Xat,(m)
Xrt,(m)
Xat,(m)
Xa{t,(m)}Xo{t,(m)}
Xrt,(m)
Xat+1,(m)
1° step of sampling: prediction of the state of attributes
Xat,(m)
Xa{t,(m)}Xo{t,(m)}
Xrt,(m)
Xat+1,(m)
Xa{t,(m)}Xo{t,(m)}
Xrt+1,(m)
2° step of sampling: prediction of the state of relationsOr activity prediction
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 25
Canadian Harbor: rendezvous
Same speed
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 26
Canadian Harbor: Avoidance
Constant speed
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 27
Exp: attributes’transition
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 28
Exp: relations’transition
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 29
Exp: Results
method TP ratio TN ratio Mean Tracking Error (km)
RPF 0.4545 0.7235 1.8379
PF 3.3906
random 0.4444 0.4841
C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 30
To conclude ...
• Modeling Relations “dynamically”:– To improve multi target tracking– To recognize complex activities
• Inference in Dynamic Relational Domain– In theory complex BUT
– Simplified by
• “smart decomposition” of the transition model
• “non-relational” sensor model
• Results are promising