computer vision lab seoul national university keyframe-based real-time camera tracking young ki baik...

ComputerVisionLabSeoul National University

Keyframe-Based Real-Time Camera Tracking

Young Ki BAIK

Vision seminar : Mar. 5. 2010

Computer Vision Lab.Computer Vision Lab.


Reference

Keyframe-Based Real-Time Camera Tracking

- ICCV 2009- Zilong Dong, Guofeng Zhang, Jiaya Jia Hujun Bao


Outline

Introduction

System overview

Offline map building

Online camera tracking

Demonstration

Conclusion

Opitimal keyframe selection

Keyframe recognition using vocabulary tree

How to use multiple thread


What is the purpose?Real-time camera tracking

Known internal camera parameters The map of environment is given…

6-DOF camera pose estimation!!


System overviewReal-time camera tracking

Novel keyframe selection algorithm

How to build a map

How to estimate the camera pose

Offline mapping using the conventional MVG method

Online camera tracking

Efficient key frame recognition method

Parallel computing framework


Offline mapping stage

reference images

Image Acquisition Video data for the target environment



Image Acquisition

Feature Extraction and Matching

SIFT descriptors



Image Acquisition Conventional MVG


3D Map Building



Keyframe set

Image Acquisition Keyframe selection


3D Map Building

Optimal Keyframe Selection For efficiency of map management



Image AcquisitionVocabulary tree


3D Map Building

Optimal Keyframe Selection

Vocabulary Tree Construction

For fast keyframe selection

in tracking stage…

Contribution!!


Why keyframe?

Which one is good for a map?

To solve the scalability problem of the map

All references Keyframe

A small number of …



select manually…

using heuristic method…

select optimal keyframe

Containing many salient features as possible

Reducing non-distinctiveness in matching

Features should be distributed evenly in keyframes



Problem formulation… to find optimal

keyframes

Completeness Term Redundancy Term

weight

Saliency of features

Distribution of features

Distinctiveness


Optimal Keyframe SelectionCompleteness Term

Definition

Reference images

Keyframes



Definition

Feature clusters

Reference image set

2D image position



Definition

Set of superior features

~Long-term surviving (or representative) features in reference

images



Saliency

~

The larger s(χ) means the higher saliency!

DoG map



Distribution

The smaller d (χ)→ Features are well distributed !!

Feature density

31x31 window



=3 : controls the sensitivity of feature density…: the superior feature set in the keyframe

set F


Optimal Keyframe SelectionRedundancy Term

NormalizationThe number of

keyframes which contain feature χ

Small Er(F) indicates low redundancy!


Keyframe selection

Algorithm

Let = 0Initialization

~


Keyframe selection

Algorithm

~

Add Keyframe


Keyframe selection

Algorithm

~

Check


Keyframe selection

Algorithm

~

Add Keyframe


Keyframe selection

Algorithm

~

Check



For fast keyframe recognition…

Root node

List

Branch = 8, Tree depth = 5

All FMean patch

All feature descriptor



To build each node…

Root node

List

Clustering by K-means method




To build each node…

Root node

List


List List

Computing mean patchBuilding keyframe list




Finally…

Root node

List


List List

Computing mean patchBuilding keyframe list

List List List List



Online tracking stage

SIFT Feature Extraction

SIFT feature

Input image



SIFT Feature Extraction Vocabulary tree

Keyframe Recognition





Keyframe-based matching

Input image Selected F





Keyframe-based matching

Camera Pose Estimation

3-point algorithm

3-D map

2-Dpoint


Keyframe recognition

With SIFT patches of input image…

Root node

List List

List List List List

Matching input patch & mean

patches

Vocabulary tree Key frame voting memory




Root node

List List

List List List List

Key frame voting memory

Move patch And Update voting memory




Root node

List List

List List List List

Key frame voting memory

Select keyframe with largest voting value


Processing time

Process time per frame with a single thread…


Processing time

Using multiple-thread with pipeline

SIFT Feature

Extraction

Time axis

Select Keyframe

Keyframe

Matching

SIFT Feature

ExtractionPose

Estimation

Thread#1

Thread#2

Thread#3

Thread#4

Select Keyframe

Keyframe

Matching Pose Estimation

20 fps

First frame istime consuming!


Demonstration


Conclusion

Real-time camera tracking is presented !!

The optimal keyframe selection algorithm

Real-time tracking is presented!

Minimizing the energy formulation!

Fast keyframe recognition using vocabulary tree

Multiple thread with pipeline structure

LimitationIncremental updating of VT is not considered…

Proposed process is only available for known environment.


Q & A

computer vision lab seoul national university keyframe-based real-time camera tracking young ki baik...

Documents