![Page 1: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/1.jpg)
Jürgen Wolf1 Wolfram Burgard2 Hans Burkhardt2
Robust Vision-based Localizationfor Mobile Robots
Using an Image Retrieval SystemBased on Invariant Features
1University of HamburgDepartment of Computer Science
22527 HamburgGermany
2University of FreiburgDepartment of Computer Science
79110 FreiburgGermany
![Page 2: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/2.jpg)
Position tracking (bounded uncertainty) Global localization (unbounded uncertainty) Kidnapping (recovery from failure)
The Localization Problem
Ingemar Cox (1991):
“Using sensory information to locate the robot in its environment is the most fundamental problem to provide a mobile robot with autonomous capabilities.”
![Page 3: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/3.jpg)
Vision-based Localization
Cameras are low-cost sensors
that provide a huge amount of information.
Cameras are passive sensors that do not suffer from interferences.
Populated environments are full of visual clues that support localization (for their inhabitants).
![Page 4: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/4.jpg)
Related Work in Vision-based Robot Localization
Sophisticated matching techniques without filtering[Basri & Rivlin, 95], [Dudek & Sim, 99], [Dudek & Zhang, 96], [Kortenkamp & Weymouth, 94], [Paletta et al., 01], [Winters et al., 00], [Lowe & Little, 01]
Image retrieval techniques without filtering[Kröse & Bunschoten, 99], [Matsumo et al., 99], [Ulrich & Nourbakhsh, 00]
Monte-Carlo localization with ceiling mosaics[Dellaert et al., 99]
Monte-Carlo localization with pre-defined landmarks[Lenser & Veloso, 00]
![Page 5: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/5.jpg)
Key Idea
Use standard techniques from image retrieval for computing the similarity between query images and reference images. No assumptions about the structure of the environment
Use Monte-Carlo localization to integrate information over time. Robustness
![Page 6: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/6.jpg)
Image Retrieval
Given: Query image q and image database d.
Goal: Find the images in d that are “most similar” to q.
![Page 7: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/7.jpg)
Key Ideas of the System Used
Features that are invariant wrt. rotation, translation, and limited scale.
Each feature consists of a histogram of local features.
[Siggelkow & Burkhardt, 98]
![Page 8: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/8.jpg)
Example of Image Retrieval
[Siggelkow & Burkhardt, 98]
![Page 9: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/9.jpg)
Another Example
[Siggelkow & Burkhardt, 98]
![Page 10: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/10.jpg)
Image Matrices
Let f(M) be an arbitrary complex-valued function over pixel values.
We compute an image matrix
1
01100
10
2,,
1
),()]([P
p
MP
pxtxtgfP
xxMfT
![Page 11: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/11.jpg)
Computing an Image Matrixusing
Image M: :)]([ MfT
) 0, 4( ) 3, 0(M M f
)]([ MfTf
![Page 12: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/12.jpg)
Computing Global Features
:)]([ MfT Histogram F(M):
The global feature F(M) consists of the multi-dimensional histograms computed for all T[f](M).
F(M)
![Page 13: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/13.jpg)
Functions f(M) with a local support preserve information about neighboring pixels.
The histograms F(M) are invariant wrt. image translations, rotations, and limited scale.
They are robust against distortions and overlap.
Observations
… ideal for mobile robot localization.
![Page 14: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/14.jpg)
Computing Similarity
To compute the similarity between a database image d and a query image q we use the normalized intersection operator:
norm
mk mkkk
mkkk
dq
dq
dq
}1,...,0{ }1,...,0{
}1,...,0{
,min
),min(
),(
Advantage: matching of partial views.
![Page 15: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/15.jpg)
Typical Results for Robot Data
Query image:
Most similar images:
81.67% 80.18% 77.49%
77.44% 77.43% 77.19%
![Page 16: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/16.jpg)
Integrating Retrieval Results and Monte-Carlo Localization
Extract visibility area for each reference image.
Weigh the samples in a visibility area proportional to the similarity measure.
![Page 17: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/17.jpg)
Visibility Regions
![Page 18: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/18.jpg)
Experiments
936 Images, 4MB, .6secs/imageTrajectory of the robot:
![Page 19: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/19.jpg)
Odometry Information
![Page 20: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/20.jpg)
Image Sequence
![Page 21: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/21.jpg)
Resulting Trajectories
Position tracking:
![Page 22: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/22.jpg)
Resulting Trajectories
Global localization:
![Page 23: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/23.jpg)
Global Localization
![Page 24: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/24.jpg)
Kidnapping the Robot
![Page 25: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/25.jpg)
Localization Error
![Page 26: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/26.jpg)
Robustness against Noise
Artificiallydistorted trajectory:
Estimatedrobot position:
![Page 27: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/27.jpg)
Validation
The retrieval results are essential!
Exploiting SimilarityConstraints only
In principle, the constraints imposed by the visibility regions can be sufficient for robot localization. [Burgard et al. 96]
![Page 28: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/28.jpg)
Summary
New approach to vision-based robot localization.
It uses an image retrieval-system for comparing images to reference images.
The features used are invariant to translations, rotations and limited scale.
Combination with Monte-Carlo localization allows the integration of measurements over time.
The system is able to robustly estimate the position of the robot and to recover from localization failures.
It can deal with dynamic environments and works under serious noise in the odometry.
![Page 29: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/29.jpg)
Future Work
Learning the optimal features for the retrieval process.
Better exploitation of the visibility areas.
Identifying important image regions.
…
![Page 30: Jürgen Wolf 1 Wolfram Burgard 2 Hans Burkhardt 2 Robust Vision-based Localization for Mobile Robots Using an Image Retrieval System Based on Invariant](https://reader030.vdocuments.net/reader030/viewer/2022032702/56649cf35503460f949c117c/html5/thumbnails/30.jpg)
Thanks ...
... and goodbye!