CENTER FOR

MACHINE PERCEPTION

CZECH TECHNICAL

UNIVERSITY IN PRAGUE

RESEARCH

REPO

RT

ISSN

1213

-236

5StreetView Dataset Calibration

Analysis(Version 1.1)

Jana Kostliva

kostliva@cmp.felk.cvut.cz

CTU–CMP–2009–14

August 7, 2009

Available atftp://cmp.felk.cvut.cz/pub/cmp/articles/kostliva/Kostliva-TR-2009-14.pdf

The work was supported by U.S. government project 08-255.

Research Reports of CMP, Czech Technical University in Prague, No. 14, 2009

Published by

Center for Machine Perception, Department of CyberneticsFaculty of Electrical Engineering, Czech Technical University

Technicka 2, 166 27 Prague 6, Czech Republicfax +420 2 2435 7385, phone +420 2 2435 7637, www: http://cmp.felk.cvut.cz

StreetView Dataset Calibration Analysis

Jana Kostliva

August 7, 2009

Abstract

In this report, we have studied the quality of calibration of StreetView dataset fromPittsburgh. Although the calibration on sparse correspondences seems to be precise,our analysis showed that the accuracy is not sufficient enough for applying dense recon-struction algorithms. The report identifies the problems and summarizes the possibleimprovements.

We have tested the calibration quality of the StreetView dataset. The input for ouranalysis were images and corresponding projection matrices, which have been transformedfrom original omni data and calibration (by unknown procedure for us). The input data areat: /datagrid/personal/jancom1/google/calib/dataset20090701

There are 6 cameras capturing the scene in different directions, this sixsome captures thescene in a video sequence. The images of different viewing directions do not overlap.

1 Processing + Analysis Description

We have tested the calibration precision by running a dense stereo algorithm. Standarddense stereo methods require image rectification, where the corresponding epipolar lines aretransformed to corresponding image rows. We have a stereo method (written by J. Cech)which is able to compute not just horizontal disparity, but also a vertical disparity of thecorrespondences. In an ideal case, the vertical disparity should be zero, higher the verticaldisparity, bigger the calibration error.

The calibrations were tested independently for each camera, pairs are created by twosubsequent images in the sequence. The results are stored in:/datagrid/SterIm/Work/kostliva/Street view.Each direction (fl,fm, fr, rl, rr, tm) has its own directory, the stereo results are saved asfollows (e.g. for images 00001 fl.png, 00002 fl.png):

• rectified images: in sub-directory rect images as:rect 00001-00002 fl l.png, and rect 00001-00002 fl r.png (Fig. 1).

• resulting disparity maps: individually horizontal and vertical component, the verticalcomponent is saved 3 times in different disparity interval to be easily visible its structure(Fig. 2):

– 00001-00002 fl h.png: horizontal disparity

1

03559-03560 rr 00285-00286 fr

Figure 1: Two rectified image pairs: rect 03559-03569 rr l.png,rect 03559-03560 rr r.png, and rect 00285-00286 fr l.png,rect 00285-00286 fr r.png, where the calibration is very imprecise (up to 65px invertical difference).

– 00001-00002 fl v.png: vertical disparity in the whole range it was computed, thebar on the right shows colour mapping, green (in the middle) corresponds to zerovertical disparity

– 00001-00002 fl v -10-10.png: vertical disparity in the range (−10, 10), the barshow colour mapping: green is zero vertical disparity, colours towards dark blueshow distribution of negative disparities up to −10, lower disparities than −10 arein dark blue; colours from the green towards the dark red show distribution ofpositive values up to 10, higher disparities than 10 are in dark red.

– 00001-00002 fl v -1-1.png: vertical disparity in the range (−1, 1), the bar againshows the colour mapping, where due to tiny interval only zero disparity is in green,negative disparities are in dark blue and positive disparities in dark red.

First, we tried to match all pairs in all 6 direction sequences. Some pairs are not matched,and hence their results are missing in corresponding directories, since the epipole was in theimage or very close to the image border (mainly this happens for directions fm, rl, and rr).

Then, we tried to iteratively improve the calibrations on selected pairs only. We have analgorithm, which re-estimates fundamental matrix based on preliminary fundamental matrixestimate and computed correspondences (with horizontal and vertical disparities). The resultsare saved in sub-directory corrected new2 of each direction directory. The way of saving theresults is the same, the iteration number is given in the name.

2 Results and Observations

The manual inspection showed that results of the stereo method are mostly correct. Thereare a few errors on boundaries, lines or repetitive patters, which are immediately and clearlyvisible in the saved results. Hence, it can be trusted.

We have computed average vertical disparity error for each pair, which is shown in Fig. 4.The average error is computed as mean of absolute values of vertical disparities on the pair.Since the manual inspection did not show dramatical errors in the stereo algorithm, all validdisparities are included. The missing values (mainly in directions fm, rl, and rr) are not dueto no error, but due to inability to rectify the image (epipole in the image or very close to

2

03559-03560 rr

horizontal vertical ∈ (−100, 100) vertical ∈ (−10, 10) vertical ∈ (−1, 1)

00285-00286 fr

horizontal vertical ∈ (−100, 100) vertical ∈ (−10, 10) vertical ∈ (−1, 1)

Figure 2: Disparity maps for two different pairs. ”Horizontal” shows the horizontal componentof the disparity, ”vertical” the vertical component of the disparity. The vertical disparityshould be zero (if correctly calibrated). The bar on the right of each vertical disparity imageshows colour mapping to vertical disparity values: zero is green, negative values up to the givenlimit correspond to green-to-blue colours, lower disparities are in dark blue; positive verticaldisparity values up to the given limit correspond to green-to-red colours, higher disparitiesare in dark red.

3

iter

0it

er1

iter

2

left right horizontal vertical v ∈ (−10, 10) v ∈ (−1, 1)

Figure 3: Iterative calibration improvement procedure of pair 03559-03560 rr. The colourchange in horizontal disparity is not of interest. It can be observed that no significant im-provement has been achieved.

Table 1: Mean and standard deviation of vertical disparity error in one direction; number ofpairs (last row) gives number of disparity maps used for the statistics (maximum number ofpairs is 4798).

fl fm fr rl rr tmmean 3.0350 5.1782 3.3583 6.0286 6.7356 3.1940std 2.4908 0.4021 2.6709 4.5368 4.1459 3.1565

#pairs 4778 3 4769 26 34 4782

the image). The error means and standard deviations computed over all image pairs for eachdirection independently, hence for each camera independently, are given in Tab. 1.

Our analysis discovered some weird pairs, or a set of pairs, which would deserve detailedinspection. We divide them to capturing direction for better readability:

FL 249-250-fl has error up to about 15px (mainly at the bottom)

sequence cca 257-258-fl to 278-279-fl is calibrated to clouds, the rest of the scenehas error about 15-20px.

552-553-fl calibrated to specularities

FM not analysed, almost all pairs have epipole in images

FR 249-250-fr has error up to about 40px (towards top right corner)

sequence around pair 264-265-fr has error about 20px, it is not clear what were theinliers

4

fl fm fr

rl rr tm

Figure 4: Average vertical disparity error for each image pair. Note, that each plot hasdifferent scale in y-coordinate (average error in pixels). The missing measurements, in fm, rl,rr mainly, are due to inability to rectify the pair.

285-286-fr has error up to 65px

RL most of the image pairs have epipole in the image

pairs with image numbers between 200 and 300 have correspondences on clouds

RR similar to RL

pairs 3559-3560-rr and 354-355-rr have errors up to 66px, and 50px resp.

TM as in other directions, around 250 the inliers seems to be on clouds

some of the rectified image pairs are mirrored, which was caused by our rectificationmethod and is already corrected.

The experiments with homography improvement did not show significant success (cf. Fig. 3).The reason is probably that the preliminary calibration is so bad, that it cannot be repairedby such simple procedure. It also can be caused by “projective” image creation procedure,where homography cannot help to improve the calibration.

5

3 Conclusions

In general, the calibration precision is varying within the sequence, there are well calibratedimages up to very badly calibration. It can be observed, that the tendency of vertical disparityworsening is from the (more or less) middle of the image to image borders, mainly to imagebottom, since on the sky it can be hardly detected or measured.

We conclude, that in some image sub-sequences the calibration corresponds to clouds,shadows, in some pairs it is hard to guess what were the inliers. Shadows are not a bigproblem in subsequent pairs (the time difference is so small that they can be considered asstatic features), but in cross-verification after returning to the same place after some loop,they may cause problems. Some images are calibrated to specularities of the scene in windows,not the scene itself. The scene in these mentioned configurations has a vertical disparity errorup to 20px, in average. There are pairs, where the difference is even about 65px. Anotherproblems are caused by motion blur, trees by the street and also buildings are in some imagesblurred, thus it is hard to find some discriminative features for calibration. Moreover, mostof the images are affected by dominant plane/s.

At the moment, the calibration accuracy of the dataset is far from running dense matching.It is necessary to look directly into the calibration and study, what are the inliers and why itfails. The suspicious pairs mentioned in the previous section could be a starting point.

6