![Page 1: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/1.jpg)
Learning to Track: Online Multi-Object Tracking by Decision Making
Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2
1University of Washington, 2Stanford University
ICCV 2015
1
![Page 2: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/2.jpg)
Multi-Object Tracking
2
Autonomous driving
Visual surveillance
Sport Analysis
Robot navigation
![Page 3: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/3.jpg)
Batch Mode vs. Online Mode
• Batch Mode
• Online Mode
t-2 ttime axis
t-1 t+1 t+2
t-2 ttime axis
t-1 t+1 t+2
3
![Page 4: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/4.jpg)
Tracking by Detection
4
![Page 5: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/5.jpg)
Data Association
Tracks at time t-1 Detections at time t
time axis
?
5
![Page 6: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/6.jpg)
Challenges
Noisy detection: false alarms and missing detections6
![Page 7: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/7.jpg)
Challenges
Occlusion
7
![Page 8: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/8.jpg)
Similarity Function for Data Association
Tracks at time t-1 Detections at time ttime axis
0.2
0.8
0.3
0.1
8
• Zhang et al., CVPR’08• Berclaz et al., TPAMI’11• Breitenstein et al., TPAMI’11• Pirsiavash et al., CVPR’11• Butt & Collins, CVPR’13• Milan et al., TPAMI’14Etc.
Simple Powerfulsimilarity measure optimization+Ours
![Page 9: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/9.jpg)
Learning to Track
𝜙1( ),Similarity 𝑤1= + ⋯ 𝜙𝑛( ),𝑤𝑛+
Different features/cues between targets and detections
Weights to combine different cues(to be learned)
9
• Appearance• Location• MotionEtc.
![Page 10: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/10.jpg)
Offline-learning vs. Online-learning
10
![Page 11: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/11.jpg)
Offline-learning vs. Online-learning
Offline-learning
Online-learning
Training time Before Tracking
DuringTracking
With supervision✓
Use history of the target ✓
11
✓
✓
✓
…
…
• Li et al., CVPR’09• Kim et al., ACCV’12Etc.
![Page 12: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/12.jpg)
Offline-learning vs. Online-learning
12
• Song et al., ECCV’08• Kuo et al., CVPR’10• Bae et al., CVPR’14Etc.
Offline-learning
Online-learning
Training time Before Tracking
DuringTracking
With supervision✓
Use history of the target ✓
![Page 13: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/13.jpg)
The target is tracked
The target is occluded
The target is tracked again
Our Solution: Tracking by Decision Making
13
![Page 14: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/14.jpg)
Inverse Reinforcement Learning
14
tracked lost tracked
Ground truth trajectory
Tracked Lost Tracked
MarkovDecisionProcess(MDP)
Supervision
![Page 15: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/15.jpg)
Comparison between Different Learning Strategies
15
Offline-learning
Online-learning
Ours
Training time Before Tracking
DuringTracking
Before Tracking
With supervision ✓ ✓
Use history of the target ✓ ✓
![Page 16: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/16.jpg)
Comparison between Different Learning Strategies
16
Offline-learning
Online-learning
Ours
Training time Before Tracking
DuringTracking
Before Tracking
With supervision ✓ ✓
Use history of the target ✓ ✓
![Page 17: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/17.jpg)
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
17
![Page 18: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/18.jpg)
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
18
![Page 19: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/19.jpg)
Active
Tracked
Inactive
Lost
objectdetection
19
Markov Decision Process for a Single Target
![Page 20: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/20.jpg)
Active
Tracked
Inactive
Lost
objectdetection
20
Markov Decision Process for a Single Target
![Page 21: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/21.jpg)
Active
Tracked
Inactive
Lost
objectdetection
21
Markov Decision Process for a Single Target
![Page 22: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/22.jpg)
Active
Tracked
Inactive
objectdetection
Markov Decision Process for a Single Target
22
![Page 23: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/23.jpg)
Markov Decision Process for a Single Target
TLD Tracker. Z. Kalal, K. Mikolajczyk, and J. Matas. Tracking-learning-detection. TPAMI, 34(7):1409–1422, 2012.23
Active
Tracked
Inactive
Lost
objectdetection Single object tracking
![Page 24: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/24.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 51
24
![Page 25: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/25.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 51
25
![Page 26: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/26.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 51
26
![Page 27: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/27.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 51
Tracked
27
![Page 28: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/28.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 57
28
![Page 29: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/29.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 57
29
![Page 30: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/30.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 57
30
![Page 31: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/31.jpg)
Template Tracking in Tracked StatesFrame 50 Frame 57
Tracked
Lost
31
![Page 32: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/32.jpg)
Active
Tracked
Inactive
Lost
objectdetection
Markov Decision Process for a Single Target
If lost for more than T frames
32
![Page 33: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/33.jpg)
Data Association in Lost States
t-2 t
time axis
t-1
tracked lost
?
33
![Page 34: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/34.jpg)
Learning the Similarity Function
𝜙1( ),Similarity 𝑤1= + ⋯ 𝜙𝑛( ),𝑤𝑛+ + 𝑏
34
( ), 1
( ), 2
…( ), M
Hard positive examples
( ), 1
( ), 2
( ), N
…
Hard negative examples
Inverse reinforcement learning: tracking objects in training videos!
![Page 35: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/35.jpg)
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lost
35
Ground truth trajectory
Supervision
![Page 36: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/36.jpg)
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lost
36
Ground truth trajectory
Supervision
![Page 37: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/37.jpg)
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lost
Wrong decision!Update your weights!
37
Ground truth trajectory
Supervision
( ), 1
Negative example
![Page 38: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/38.jpg)
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lostWrong decision!Association to this one!Update your weights!
No association
Try it again
38
Ground truth trajectory
Supervision
( ),Positive example
2
![Page 39: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/39.jpg)
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lostGood job!Keep going!No update of the weights
Try it again
39
Ground truth trajectory
Supervision
![Page 40: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/40.jpg)
Active
Tracked
Inactive
Lost
objectdetection
Markov Decision Process for a Single Target
40
![Page 41: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/41.jpg)
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
41
![Page 42: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/42.jpg)
Ensemble MDPs for Online Multi-Object Tracking
t-2 t
time axis
MDP1
MDP2
MDP3
t-1
42
![Page 43: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/43.jpg)
Step 1: Process tracked targets
t
time axis
MDP1
MDP2
MDP3
t-2 t-1
43
![Page 44: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/44.jpg)
Step 2: Process lost targets
t
time axis
MDP1
MDP2
MDP3
Hungarian algorithm for lost targets
t-2 t-1
44
![Page 45: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/45.jpg)
Step 3: Initialize new targets
t
time axis
MDP1
MDP2
MDP3
Initialize new targets
t-2 t-1
45
Terminate detection
![Page 46: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/46.jpg)
Tracked Lost Tracked
Tracked Lost Tracked
Tracked Tracked Tracked
MDP1
MDP2
MDP3
Online Multi-Object Tracking with MDPs
46
![Page 47: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/47.jpg)
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
47
![Page 48: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/48.jpg)
Experiments: Dataset
•Multiple Object Tracking Benchmark [1]• 11 training sequences• 11 test sequences• Object detections from the ACF detector [2]
[1] L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler. MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking. arXiv:1504.01942 [cs], 2015.[2] P. Dollár, R. Appel, S. Belongie, and P. Perona. Fast feature pyramids for object detection. TPAMI, 36(8):1532–1545, 2014. 48
![Page 49: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/49.jpg)
Experiments: Analysis on Validation Set
• Contribution of different components
49
![Page 50: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/50.jpg)
Experiments: Analysis on Validation Set
• Contribution of different components
Active
Tracked
Inactive
Lost
objectdetection
50
MOTA: multiple object tracking accuracy
![Page 51: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/51.jpg)
Experiments: Analysis on Validation Set
• Contribution of different components
Active
Tracked
Inactive
Lost
objectdetection
51
MOTA: multiple object tracking accuracy
![Page 52: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/52.jpg)
Experiments: Analysis on Validation Set
• Contribution of different components
Active
Tracked
Inactive
Lost
objectdetection
52
MOTA: multiple object tracking accuracy
![Page 53: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/53.jpg)
Experiments: Analysis on Validation Set
• Contribution of different components
53
𝜙1( ),Similarity 𝑤1=+⋯
𝜙𝑛( ),𝑤𝑛
+
+
𝑏MOTA: multiple object tracking accuracy
![Page 54: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/54.jpg)
Experiments: Analysis on Validation Set
• Contribution of different components
54
𝜙1( ),Similarity 𝑤1=+⋯
𝜙𝑛( ),𝑤𝑛
+
+
𝑏MOTA: multiple object tracking accuracy
![Page 55: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/55.jpg)
Experiments: Analysis on Validation Set• Cross-domain tracking
55
MOTA: multiple object tracking accuracy
TUD-Stadtmitte
ETH-Bahnhof
ADL-Rundle-6
KITTI-13
PETS09-S2L1
Trai
nin
g Se
qu
ence
s
Testing sequences
![Page 56: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/56.jpg)
Experiments: Analysis on Validation Set• Cross-domain tracking
56
TUD-Stadtmitte
ETH-Bahnhof
ADL-Rundle-6
KITTI-13
PETS09-S2L1
MOTA: multiple object tracking accuracy
Trai
nin
g Se
qu
ence
s
Testing sequences
![Page 57: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/57.jpg)
Experiments: Analysis on Validation Set• Cross-domain tracking
57
TUD-Stadtmitte
ETH-Bahnhof
ADL-Rundle-6
KITTI-13
PETS09-S2L1
MOTA: multiple object tracking accuracy
Trai
nin
g Se
qu
ence
s
Testing sequences
![Page 58: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/58.jpg)
Experiments: Evaluation on Test SetTracker Tracking Learning MOTA
DP_NMS [1] Batch N/A 14.5
TC_ODAL [2] Online Online 15.1
TBD [3] Batch Offline 15.9
SMOT [4] Batch N/A 18.2
RMOT [5] Online N/A 18.6
CEM [6] Online N/A 19.3
SegTrack [7] Batch Offline 22.5
MotiCon [8] Batch Offline 23.1
MDP (Ours) Online Online 30.3
[1] Pirsiavash et al., CVPR’ 11[2] Bae et al., CVPR’14[3] Geiger et al., TPAMI’14
58
MOTA: multiple object tracking accuracy
[4] Dicle et al., ICCV’13[5] Yoon et al., WACV’15[6] Milan et al., TPAMI’14
[7] Milan et al., CVPR’15[8] Leal-Taixé et al., CVPR’14
![Page 59: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/59.jpg)
Tracking Results
59
![Page 60: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/60.jpg)
60
MDP [Ours] MotiCon [Leal-Taixé et al., CVPR’14]
![Page 61: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/61.jpg)
61
MDP [Ours] MotiCon [Leal-Taixé et al., CVPR’14]
![Page 62: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/62.jpg)
62
MDP [Ours] MotiCon [Leal-Taixé et al., CVPR’14]
![Page 63: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/63.jpg)
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
63
![Page 64: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/64.jpg)
Active
Tracked
Inactive
Lost
Conclusion
Object Detection
Single Object Tracking
Data AssociationTarget Re-identification
64
![Page 65: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/65.jpg)
Code
65
![Page 66: Learning to Track: Online Multi-Object Tracking by Decision Making · Learning to Track: Online Multi-Object Tracking by Decision Making Yu Xiang1, Alexandre Alahi2, and Silvio Savarese2](https://reader033.vdocuments.net/reader033/viewer/2022050515/5f9f54d983a279505e56f4f4/html5/thumbnails/66.jpg)
Active
Tracked
Inactive
Lost
Object Detection
Single Object Tracking
Data AssociationTarget Re-identification
66
Thank you!