modeling multiple-object tracking as constrained flow ...€¦ · modeling multiple-object tracking...
TRANSCRIPT
![Page 1: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/1.jpg)
Modeling Multiple-object Tracking as Constrained Flow
Optimization Problem
![Page 2: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/2.jpg)
Introduction• Multiple-object tracking
1. Detection Step: time-Independent 2. Linking Step: connect detections into most likely
trajectories: NP-Complete• Problem: Linking detections into trajectories for multiple
visually-similar objects• Solutions:– Filtering, Greedy dynamic programming etc.: don’t
ensure global optimum– Integer Linear Programming (ILP): Ensures global
optimum but NP-Complete
![Page 3: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/3.jpg)
Multi-Object Tracking as Constrained Flow optimization
• Divide scene into ! discrete locations.
• Model occupancy map over time using directed graph. "
#(!)
∀', ) *+:-∊/(+)
0+,-123 = 5-1 = *6∊/(-)
0-,61
∀!, ) *-∊/(6)
06,-1 ≤ 1
∀!, ', ) 06,-1 ≥ 0
![Page 4: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/4.jpg)
Multi-Object Tracking as Constrained Flow optimization
• Posterior probability of an object:– !: occupancy map (holding
variables!"#)– ₣: set of feasible maps !.– %&
#:random variable presenting the true value of ' in time (.
– )# :signal
*"∊,(./01234)
6./01234,"# = *
9:./;<=∊,(9)
69,./;<=#
!∗ = arg!BCD∊₣ EF |% = ! )#
HI&# = EF( J%&
# = 1 )#
– %&# as conditional independence
variable
!∗ = arg!BCD∊₣LMNO#,&
EF J%&# = ! )# = arg!BCD∊₣*
#,&
LMN EF J%&# = ! )#
![Page 5: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/5.jpg)
Integer Linear Programming (ILP) Formulation
• !"# as conditional independence variable
• ILP System:– Maximize ∑#," &'(
)*+
,-)*+ ∑.∊0(") 3",.
#
– Subject to ∀5, 6, 7 3",.# ≥ 0
∀5, 7 :
.∊0(")
3",.# ≤ 1
∀5, 7 :
.∊0 "
3",.# − :
>:"∊0(>)
3>,"#-, ≤ 0
:
>∊0 @ABCDEF
3.,># − :
":.∊0(")
3",.#-, ≤ 0
G∗ = argGMNO∊₣&'(Q
#,"
RS T!"# = G U# = argGMNO∊₣:
#,"
&'(V"#
1 − V"# G"
#
![Page 6: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/6.jpg)
From Integer to Continuous Linear Program
• Integer LP solution: NP-complete problem• Continuous LP: Polynomial time average
complexity• Relax the ‘integer’ condition to reduce
complexity• Problem: continuous LP does not usually
converge to optimal solution of original ILP!!• Solution: Total-unimodularity of constraint
matrix
![Page 7: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/7.jpg)
Total Unimodularity
• Total-Unimodular Matrix: – all square sub-matrices has determinants 1,0 or -1
or– For every subset of rows R Í {1,2…,m}, there is a
partition of rows such that R = R1 U R2, R1 ∩ R2 = Ø "j = 1,2,…n (Σi∈R1 aij - Σi∈R2 aij)∈ {-1,0,1}
• Ensures integer solution even for continuous LP
• Constraint matrix is Total-Unimodular
![Page 8: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/8.jpg)
Total-Unimodularity of Constraint Matrix
• Constraint matrix: – arrange columns in ascending order of time; each column represent one locationat one time instant
• Rows: – divided into 2 parts based on conditions
![Page 9: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/9.jpg)
Total-Unimodularity of Constraint Matrix
• Only 3 rows can be non-zero (∈ {-1,0,1} ) for one column
• Eight cases
for partitions R1 = R∩U1,R2 = R∩U2 for any R Í {1,2…,m}
• Every possibility satisfies total-unimodualritybut 3rd –Problem
• Solution: Move non-zero row of R1 to R2 for 3rd
case
![Page 10: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/10.jpg)
K-shortest paths (KSP) formulation
• Why?– Relaxed ILP solution polynomial but practically not
efficient!– Need real time efficiency for practical problems
• KSP:– Given a graph G(V,E), compute a set of k shortest
paths {p1,p2,…..,pk} such that the total cost is minimum
![Page 11: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/11.jpg)
KSP formulation– Problem constraints:
• Node disjoint• Node simple
Three node-simple shortest paths: 6,20,21Three non-simple shortest paths (allowing loops ‘aba’ & ‘ded’):6,8,10
![Page 12: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/12.jpg)
KSP formulation
• ILP to KSP
– Maximizing flow in ILP is equivalent to minimizing
path cost function in KSP (just negate the ILP
objective function)
– Any path between source and sink nodes with
arbitrary ‘k’ is in the set of feasible solutions of ILP
– The value of ‘k’ that achieves min. cost is the one
that maximizes the flow in ILP solution
![Page 13: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/13.jpg)
KSP formulation
• Optimality of ‘k’ is guaranteed due to convexity of the path cost function
• The shortest paths at each iteration are computed by Dijkstra’s algorithm– Complexity?• O(k(m+nlogn))
![Page 14: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/14.jpg)
Applications
• 2D segmentation to 3D segmentation:
– 2D: shortest path problem; Dijkstra’s algorithm
– 3D: minimal weight surface problem
• can be presented as instance of ILP with totally
unimodular constraint matrix
• Optimization by LP:
– LP provides an upper/ lower bound for original ILP
– Branch-and-bound: optimization of ILP through recursive
LP solution.
• Tracking multiple humans in surveillance videos
• Min-Cost flow problems: e.g. efficient network routing
![Page 15: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/15.jpg)
Question??
• How does Total-Unimodulairty ensure Integer solution even for continuous linear program??
![Page 16: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/16.jpg)
Answer: Total-Unimodularity -> Integer Solution
• Quadratic system: Cx = y
• Cramer’s rule: xj = |Cyj|/|C|
• Cyj = C with jth column replaced with y, (C1, C2,
..Cj-1 ,y, Cj+1 ,..Cm)
• |Cyj| = Σi (-1)i+j yi |Cij|
• Cij = C with ith row, jth column deleted
• If |C| ∈ {-1,1} and Cij ∈ {-1,0,1} for all I and j, then x is integer => Total Unimodularity, Integer solution
![Page 17: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/17.jpg)
Question??
• Why is the path cost function of KSP convex?
![Page 18: Modeling Multiple-object Tracking as Constrained Flow ...€¦ · Modeling Multiple-object Tracking as Constrained Flow Optimization Problem. Introduction •Multiple-object tracking](https://reader034.vdocuments.net/reader034/viewer/2022050603/5faaee4bb56180469021d702/html5/thumbnails/18.jpg)
Answer: Edge costs can be negative and total path cost function is summation for successive shortest path
costs that are monotonically increasing!
• At each iteration for ‘k’, we have:
• Therefore, the path cost function over ‘k’ is convex