![Page 1: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/1.jpg)
ComputerVisionLabSeoul National University
Keyframe-Based Real-Time Camera Tracking
Young Ki BAIK
Vision seminar : Mar. 5. 2010
Computer Vision Lab.Computer Vision Lab.
![Page 2: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/2.jpg)
ComputerVisionLabSeoul National University
Reference
Keyframe-Based Real-Time Camera Tracking
- ICCV 2009- Zilong Dong, Guofeng Zhang, Jiaya Jia Hujun Bao
![Page 3: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/3.jpg)
ComputerVisionLabSeoul National University
Outline
Introduction
System overview
Offline map building
Online camera tracking
Demonstration
Conclusion
Opitimal keyframe selection
Keyframe recognition using vocabulary tree
How to use multiple thread
![Page 4: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/4.jpg)
ComputerVisionLabSeoul National University
What is the purpose?Real-time camera tracking
Known internal camera parameters The map of environment is given…
6-DOF camera pose estimation!!
![Page 5: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/5.jpg)
ComputerVisionLabSeoul National University
System overviewReal-time camera tracking
Novel keyframe selection algorithm
How to build a map
How to estimate the camera pose
Offline mapping using the conventional MVG method
Online camera tracking
Efficient key frame recognition method
Parallel computing framework
![Page 6: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/6.jpg)
ComputerVisionLabSeoul National University
Offline mapping stage
reference images
Image Acquisition Video data for the target environment
![Page 7: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/7.jpg)
ComputerVisionLabSeoul National University
Offline mapping stage
Image Acquisition
Feature Extraction and Matching
SIFT descriptors
![Page 8: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/8.jpg)
ComputerVisionLabSeoul National University
Offline mapping stage
Image Acquisition Conventional MVG
Feature Extraction and Matching
3D Map Building
![Page 9: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/9.jpg)
ComputerVisionLabSeoul National University
Offline mapping stage
Keyframe set
Image Acquisition Keyframe selection
Feature Extraction and Matching
3D Map Building
Optimal Keyframe Selection For efficiency of map management
![Page 10: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/10.jpg)
ComputerVisionLabSeoul National University
Offline mapping stage
Image AcquisitionVocabulary tree
Feature Extraction and Matching
3D Map Building
Optimal Keyframe Selection
Vocabulary Tree Construction
For fast keyframe selection
in tracking stage…
Contribution!!
![Page 11: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/11.jpg)
ComputerVisionLabSeoul National University
Why keyframe?
Which one is good for a map?
To solve the scalability problem of the map
All references Keyframe
A small number of …
![Page 12: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/12.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe Selection
select manually…
using heuristic method…
select optimal keyframe
Containing many salient features as possible
Reducing non-distinctiveness in matching
Features should be distributed evenly in keyframes
![Page 13: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/13.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe Selection
Problem formulation… to find optimal
keyframes
Completeness Term Redundancy Term
weight
Saliency of features
Distribution of features
Distinctiveness
![Page 14: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/14.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Definition
Reference images
Keyframes
![Page 15: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/15.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Definition
Feature clusters
Reference image set
2D image position
![Page 16: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/16.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Definition
Set of superior features
~Long-term surviving (or representative) features in reference
images
![Page 17: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/17.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Saliency
~
The larger s(χ) means the higher saliency!
DoG map
![Page 18: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/18.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
Distribution
The smaller d (χ)→ Features are well distributed !!
Feature density
31x31 window
![Page 19: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/19.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionCompleteness Term
=3 : controls the sensitivity of feature density…: the superior feature set in the keyframe
set F
![Page 20: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/20.jpg)
ComputerVisionLabSeoul National University
Optimal Keyframe SelectionRedundancy Term
NormalizationThe number of
keyframes which contain feature χ
Small Er(F) indicates low redundancy!
![Page 21: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/21.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
Let = 0Initialization
~
![Page 22: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/22.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
![Page 23: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/23.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
![Page 24: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/24.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Check
![Page 25: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/25.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
![Page 26: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/26.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Add Keyframe
![Page 27: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/27.jpg)
ComputerVisionLabSeoul National University
Keyframe selection
Algorithm
~
Check
![Page 28: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/28.jpg)
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
For fast keyframe recognition…
Root node
List
Branch = 8, Tree depth = 5
All FMean patch
All feature descriptor
![Page 29: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/29.jpg)
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
To build each node…
Root node
List
Clustering by K-means method
Branch = 8, Tree depth = 5
![Page 30: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/30.jpg)
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
To build each node…
Root node
List
Clustering by K-means method
List List
Computing mean patchBuilding keyframe list
Branch = 8, Tree depth = 5
![Page 31: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/31.jpg)
ComputerVisionLabSeoul National University
Vocabulary Tree Construction
Finally…
Root node
List
Clustering by K-means method
List List
Computing mean patchBuilding keyframe list
List List List List
Branch = 8, Tree depth = 5
![Page 32: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/32.jpg)
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction
SIFT feature
Input image
![Page 33: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/33.jpg)
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction Vocabulary tree
Keyframe Recognition
![Page 34: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/34.jpg)
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction
Keyframe Recognition
Keyframe-based matching
Input image Selected F
![Page 35: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/35.jpg)
ComputerVisionLabSeoul National University
Online tracking stage
SIFT Feature Extraction
Keyframe Recognition
Keyframe-based matching
Camera Pose Estimation
3-point algorithm
3-D map
2-Dpoint
![Page 36: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/36.jpg)
ComputerVisionLabSeoul National University
Keyframe recognition
With SIFT patches of input image…
Root node
List List
List List List List
Matching input patch & mean
patches
Vocabulary tree Key frame voting memory
![Page 37: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/37.jpg)
ComputerVisionLabSeoul National University
Keyframe recognition
With SIFT patches of input image…
Root node
List List
List List List List
Key frame voting memory
Move patch And Update voting memory
![Page 38: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/38.jpg)
ComputerVisionLabSeoul National University
Keyframe recognition
With SIFT patches of input image…
Root node
List List
List List List List
Key frame voting memory
Select keyframe with largest voting value
![Page 39: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/39.jpg)
ComputerVisionLabSeoul National University
Processing time
Process time per frame with a single thread…
![Page 40: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/40.jpg)
ComputerVisionLabSeoul National University
Processing time
Using multiple-thread with pipeline
SIFT Feature
Extraction
Time axis
Select Keyframe
Keyframe
Matching
SIFT Feature
ExtractionPose
Estimation
Thread#1
Thread#2
Thread#3
Thread#4
Select Keyframe
Keyframe
Matching Pose Estimation
20 fps
First frame istime consuming!
![Page 41: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/41.jpg)
ComputerVisionLabSeoul National University
Demonstration
![Page 42: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/42.jpg)
ComputerVisionLabSeoul National University
Conclusion
Real-time camera tracking is presented !!
The optimal keyframe selection algorithm
Real-time tracking is presented!
Minimizing the energy formulation!
Fast keyframe recognition using vocabulary tree
Multiple thread with pipeline structure
LimitationIncremental updating of VT is not considered…
Proposed process is only available for known environment.
![Page 43: Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar. 5. 2010 Computer Vision Lab](https://reader036.vdocuments.net/reader036/viewer/2022062422/56649ef05503460f94c012e2/html5/thumbnails/43.jpg)
ComputerVisionLabSeoul National University
Q & A