peretroukhin et al. dpc-net: deep pose correction for visual localization · peretroukhin et al.:...

PERETROUKHIN et al.: DPC-NET: DEEP POSE CORRECTION FOR VISUAL LOCALIZATION 1

DPC-Net: Deep Pose Correction for VisualLocalization

Valentin Peretroukhin1, and Jonathan Kelly1

Abstract—We present a novel method to fuse the power ofdeep networks with the computational efficiency of geometricand probabilistic localization algorithms. In contrast to othermethods that completely replace a classical visual estimator witha deep network, we propose an approach that uses a convolutionalneural network to learn difficult-to-model corrections to theestimator from ground-truth training data. To this end, we derivea novel loss function for learning SE(3) corrections based ona matrix Lie groups approach, with a natural formulation forbalancing translation and rotation errors. We use this loss totrain a Deep Pose Correction network (DPC-Net) that predictscorrections for a particular estimator, sensor and environment.Using the KITTI odometry dataset, we demonstrate significantimprovements to the accuracy of a computationally-efficientsparse stereo visual odometry pipeline, that render it as accurateas a modern computationally-intensive dense estimator. Further,we show how DPC-Net can be used to mitigate the effect ofpoorly calibrated lens distortion parameters.

Index Terms—Deep Learning in Robotics and Automation,Localization

I. INTRODUCTION

DEEP convolutional neural networks (CNNs) are at thecore of many state-of-the-art classification and segmenta-

tion algorithms in computer vision [1]. These CNN-based tech-niques achieve accuracies previously unattainable by classicalmethods. In mobile robotics and state estimation, however,it remains unclear to what extent these deep architecturescan obviate the need for classical geometric modelling. Inthis work, we focus on visual odometry (VO): the task ofcomputing the egomotion of a camera through an unknownenvironment with no external positioning sources. Visual local-ization algorithms like VO can suffer from several systematicerror sources that include estimator biases [2], poor calibration,and environmental factors (e.g., a lack of scene texture).While machine learning approaches can be used to bettermodel specific subsystems of a localization pipeline (e.g., theheteroscedastic feature track error covariance modelling of [3],[4]), much recent work [5]–[9] has been devoted to completelyreplacing the estimator with a CNN-based system.

We contend that this type of complete replacement placesan unnecessary burden on the CNN. Not only must it learnprojective geometry, but it must also understand the environ-ment, and account for sensor calibration and noise. Instead, wetake inspiration from recent results [10] that demonstrate that

Manuscript received: September, 10, 2017; Revised November 24, 2017;Accepted November, 8, 2017.

1Both authors are with the Space & Terrestrial AutonomousRobotic Systems (STARS) laboratory at the University ofToronto Institute for Aerospace Studies (UTIAS), Canada.<firstname>.<lastname>@robotics.utias.utoronto.ca

Tracking

Outlier Rejection

Non-Linear Optimization

Tracking

Visual Odometry DPC-Net

Pose Graph Relaxation

cnn-layercnn-layercnn-layer

cnn-layercnn-layer

⇠ 2 R6


SE(3) Correction

SE(3) Corrected Pose

exp�⇠^

�

SE(3) Estimate

cnn-layer


Convolution

PReLU

Dropoutcnn-layer

Stereo Images

Fig. 1. We propose a Deep Pose Correction network (DPC-Net) that learnsSE(3) corrections to classical visual localizers.

CNNs can infer difficult-to-model geometric quantities (e.g.,the direction of the sun) to improve an existing localizationestimate. In a similar vein, we propose a system (Figure 1) thattakes as its starting point an efficient, classical localizationalgorithm that computes high-rate pose estimates. To it, weadd a Deep Pose Correction Network (DPC-Net) that learnslow-rate, ‘small’ corrections from training data that we thenfuse with the original estimates. DPC-Net does not requireany modification to an existing localization pipeline, and canlearn to correct multi-faceted errors from estimator bias, sensormis-calibration or environmental effects.

Although in this work we focus on visual data, the DPC-Netarchitecture can be readily modified to learn SE(3) correctionsfor estimators that operate with other sensor modalities (e.g.,lidar). For this general task, we derive a pose regression lossfunction and a closed-form analytic expression for its Jacobian.Our loss permits a network to learn an unconstrained Liealgebra coordinate vector, but derives its Jacobian with respectto SE(3) geodesic distance.

In summary, the main contributions of this work are1) the formulation of a novel deep corrective approach to

egomotion estimation,2) a novel cost function for deep SE(3) regression that

naturally balances translation and rotation errors, and3) an open-source implementation of DPC-Net in

arX

iv:1

709.

0312

8v3

[cs

.CV

] 1

0 Se

p 20

18

2 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED NOVEMBER, 2017

PyTorch1.

II. RELATED WORK

Visual state estimation has a rich history in mobile robotics.We direct the reader to [11] and [12] for detailed surveys ofwhat we call classical, geometric approaches to visual odom-etry (VO) and visual simultaneous localization and mapping(SLAM).

In the past decade, much of machine learning and its sub-disciplines has been revolutionized by carefully constructeddeep neural networks (DNNs) [1]. For tasks like imagesegmentation, classification, and natural language processing,most prior state-of-the-art algorithms have been replaced bytheir DNN alternatives.

In mobile robotics, deep neural networks have ushered ina new paradigm of end-to-end training of visuomotor policies[13] and significantly impacted the related field of reinforce-ment learning [14]. In state estimation, however, most suc-cessful applications of deep networks have aimed at replacinga specific sub-system of a localization and mapping pipeline(for example, object detection [15], place recognition [16], orbespoke discriminative observation functions [17]).

Nevertheless, a number of recent approaches have presentedconvolutional neural network architectures that purport toobviate the need for classical visual localization algorithms.For example, Kendall et al. [7], [18] presented extensive workon PoseNet, a CNN-based camera re-localization approach thatregresses the 6-DOF pose of a camera within a previouslyexplored environment. Building upon PoseNet, Melekhov [8]applied a similar CNN learning paradigm to relative cameramotion. In related work, Costante et al. [5] presented a CNN-based VO technique that uses pre-processed dense optical flowimages. By focusing on RGB-D odometry, Handa et al. [19],detailed an approach to learning relative poses with 3D SpatialTransformer modules and a dense photometric loss. Finally,Oliviera et al. [9] and Clark et al. [6] described techniquesfor more general sensor fusion and mapping. The formerwork outlined a DNN-based topometric localization pipelinewith separate VO and place recognition modules while thelatter presented VINet, a CNN paired with a recurrent neuralnetwork for visual-inertial sensor fusion and online calibration.

With this recent surge of work in end-to-end learning forvisual localization, one may be tempted to think this is theonly way forward. It is important to note, however, thatthese deep CNN-based approaches do not yet report state-of-the-art localization accuracy, focusing instead on proof-of-concept validation. Indeed, at the time of writing, the mostaccurate visual odometry approach on the KITTI odometrybenchmark leaderboard2 remains a variant of a sparse feature-based pipeline with carefully hand-tuned optimization [20].

Taking inspiration from recent results that show that CNNscan be used to inject global orientation information into avisual localization pipeline [10], and leveraging ideas fromrecent approaches to trajectory tracking in the field of con-trols [21], [22], we formulate a system that learns pose

1See https://github.com/utiasSTARS/dpc-net.2See http://www.cvlibs.net/datasets/kitti/eval_odometry.php.

ti+�pti

2, 3x3, 256, 256

2, 3x3, 256, 512

2, 3x3, 512, 1024

2, 3x3, 1024, 4096

2, 3x3, 4096, 4096

2, 3x3, 4096, 6

2, 1x2, 6, 6

⇠ 2 R6

Stereo RGB (6xHxW)

1, 3x3, 64, 128

Convolution

PReLU

Dropout

L(⇠,T⇤)SE(3) Cost

T⇤

Target SE(3) Correction

2, 3x3, 6, 642, 3x3, 64, 641, 3x3, 64, 128

2, 3x3, 6, 642, 3x3, 64, 64

Fig. 2. The Deep Pose Correction network with stereo RGB image data.The network learns a map from two stereo pairs to a vector of Lie algebracoordinates. Each darker blue block consists of a convolution, a PReLU non-linearity, and a dropout layer. We opt to not use MaxPooling in the network,following [19]. The labels correspond to the stride, kernel size, and numberof input and output channels of for each convolution.

corrections to an existing estimator, instead of learning theentire localization problem ab initio.

III. SYSTEM OVERVIEW: DEEP POSE CORRECTION

We base our network structure on that of [19] but learnSE(3) corrections from stereo images and require no pre-training. Similar to [19], we use primarily 3x3 convolutionsand avoid the use of max pooling to preserve spatial infor-mation. We achieve downsampling by setting the stride to 2where appropriate (see Figure 2 for a full description of thenetwork structure). We derive a novel loss function, unlikethat used in [7]–[9], based on SE(3) geodesic distance. Ourloss naturally balances translation and rotation error withoutrequiring a hand-tuned scalar hyper-parameter. Similar to [5],we test our final system on the KITTI odometry benchmark,and evaluate how it copes with degraded visual data.

Given two coordinate frames F−→i, F−→i+∆pthat represent a

camera’s pose at time ti and ti+∆p(where ∆p is an integer

hyper-parameter that allows DPC-Net to learn correctionsacross multiple temporally consecutive poses), we assume thatour visual localizer gives us an estimate3, Ti,i+∆p

, of the truetransform Ti,i+∆p

∈ SE(3) between the two frames. We aimto learn a target correction,

T∗i = Ti,i+∆pT−1

i,i+∆p, (1)

from two pairs of stereo images (collectively referred toas Iti,ti+∆p

) captured at ti and ti+∆p4. Given a dataset,

3We use (·) to denote an estimated quantity throughout the paper.4Note that the visual estimator does not necessarily compute Ti,i+∆p

directly. Ti,i+∆p may be compounded from several estimates.

https://github.com/utiasSTARS/dpc-net

http://www.cvlibs.net/datasets/kitti/eval_odometry.php


{T∗i , Iti,ti+∆p}Ni=1, we now turn to the problem of selecting

an appropriate loss function for learning SE(3) corrections.

A. Loss Function: Correcting SE(3) Estimates

One possible approach to learning T∗ is to break it intoconstituent parts and then compose a loss function as theweighted sum of translational and rotational error norms (asdone in [7]–[9]). This however does not account for thepossible correlation between the two losses, and requires thecareful tuning of a scalar weight.

In this work, we instead choose to parametrize our correc-tion prediction as T = exp

(ξ∧), where ξ ∈ R6, a vector of

Lie algebra coordinates, is the output of our network (similarto [19]). We define a loss for ξ as

L(ξ) =1

2g(ξ)TΣ−1g(ξ), (2)

whereg(ξ) , ln

(exp

(ξ∧)T∗−1

)∨. (3)

Here, Σ is the covariance of our estimator (expressed usingunconstrained Lie algebra coordinates), and (·)∧, (·)∨ convertvectors of Lie algebra coordinates to matrix vectorspace quan-tities and back, respectively. Given two stereo image pairs,we use the output of DPC-Net, ξ, to correct our estimator asfollows:

Tcorr

= exp(ξ∧)T, (4)

where we have dropped subscripts for clarity.

B. Loss Function: SE(3) Covariance

Since we are learning estimator corrections, we can com-pute an empirical covariance over the training set as

Σ =1

N − 1

N∑

i=1

(ξ∗i − ξ∗

)(ξ∗i − ξ∗

)T, (5)

where

ξ∗i , ln (T∗i )∨, ξ∗ ,

1

N

N∑

i=1

ξ∗i . (6)

The term Σ balances the rotational and translation lossterms based on their magnitudes in the training set, andaccounts for potential correlations. We stress that if we werelearning poses directly, the pose targets and their associatedmean would be trajectory dependent and would render thistype of covariance estimation meaningless. Further, we findthat, in our experiments, Σ weights translational and rotationalerrors similarly to that presented in [7] based on the diagonalcomponents, but contains relatively large off-diagonal terms.

C. Loss Function: SE(3) Jacobians

In order to use Equation (2) to train DPC-Net with back-propagation, we need to compute its Jacobian with respect toour network output, ξ. Applying the chain rule, we begin withthe expression

∂L(ξ)

∂ξ= g(ξ)TΣ−1 ∂g(ξ)

∂ξ. (7)

The term ∂g(ξ)∂ξ is of importance. We can derive it in two ways.

To start, note two important identities [23]. First,

exp((ξ + δξ)

∧) ≈ exp((J δξ)

∧) exp(ξ∧), (8)

where J , J (ξ) is the left SE(3) Jacobian. Second, if T1 ,exp

(ξ1∧) and T2 , exp

(ξ2∧), then

ln (T1T2)∨

= ln(exp

(ξ1∧) exp

(ξ2∧))∨

≈{

J (ξ2)−1

ξ1 + ξ2 if ξ1 smallξ1 + J (−ξ1)

−1ξ2 if ξ2 small.

}(9)

See [23] for a detailed treatment of matrix Lie groups andtheir use in state estimation.

1) Deriving ∂g(ξ)∂ξ , Method I: If we assume that only ξ is

‘small’, we can apply Equation (9) directly to define

∂g(ξ)

∂ξ= J (−ξ∗)−1

, (10)

with ξ∗ , ln (T∗)∨. Although attractively compact, note thatthis expression for ∂g(ξ)

∂ξ assumes that ξ is small, and may beinaccurate for ‘larger’ T∗ (since we will therefore require ξto be commensurately ‘large’).

2) Deriving ∂g(ξ)∂ξ , Method II: Alternatively, we can lin-

earize Equation (3) about ξ, by considering a small changeδξ and applying Equation (8):

g(ξ + δξ) = ln(

exp((ξ + δξ)

∧)T∗−1

)∨(11)

≈ ln(

exp((J δξ)

∧) exp(ξ∧)T∗−1

)∨. (12)

Now, assuming that J δξ is ‘small’, and using Equation (3),Equation (9) gives:

g(ξ + δξ) ≈ J (g(ξ))−1J (ξ)δξ + g(ξ). (13)

Comparing this to the first order Taylor expansion:g(ξ + δξ) ≈ g(ξ) + ∂g(ξ)

∂ξ δξ, we see that

∂g(ξ)

∂ξ= J (g(ξ))

−1J (ξ). (14)

Although slightly more computationally expensive, this ex-pression makes no assumptions about the ‘magnitude‘ of ourcorrection and works reliably for any target. Note further thatif ξ is small, then J (ξ) ≈ 1 and exp

(ξ∧)≈ 1. Thus,

g(ξ) ≈ ln(T∗−1

)∨= −ξ∗, (15)

and Equation (14) becomes

∂g(ξ)

∂ξ= J (−ξ∗)−1

, (16)

which matches Method I. To summarize, to apply back-propagation to Equation (2), we use Equation (7) and Equa-tion (14).


Ti,i+1

Tcorr

i,i+�p

Fig. 3. We apply pose graph relaxation to fuse high-rate visual localizationestimates (Ti,i+1) with low-rate deep pose corrections (T

corri,i+∆p

).

D. Loss Function: Correcting SO(3) Estimates

Our framework can be easily modified to learn SO(3)corrections only. We can parametrize a similar objective forφ ∈ R3,

L(φ,C∗) =1

2f(φ)TΣ−1f(φ), (17)

wheref(φ) , ln

(exp

(φ∧)C∗−1)∨. (18)

Equations (8) and (9) have analogous SO(3) formulations:

exp((φ + δφ)

∧) ≈ exp((Jδφ)

∧) exp(φ∧), (19)

and

ln (C1C2)∨

= ln(exp

(φ1∧) exp

(φ2∧))∨

≈{

J(φ2)−1

φ1 + φ2 if φ1 smallφ1 + J(−φ1)

−1φ2 if φ2 small,

(20)

where J , J(φ) is the left SO(3) Jacobian. Accordingly, thefinal loss Jacobians are identical in structure to Equation (7)and Equation (14), with the necessary SO(3) replacements.

E. Pose Graph Relaxation

In practice, we find that using camera poses several framesapart (i.e. ∆p > 1) often improves test accuracy and reducesoverfitting. As a result, we turn to pose graph relaxation to fuselow-rate corrections with higher-rate visual pose estimates. Fora particular window of ∆p + 1 poses (see Figure 3), we solvethe non linear minimization problem

{Tti,n}∆p

i=0 = argmin{Tti,n

}∆pi=0∈SE(3)

Ot, (21)

where n refers to a common navigation frame, and where wedefine the total cost, Ot, as a sum of visual estimation andcorrection components:

Ot = Ov +Oc. (22)

The former cost sums over each estimated transform,

Ov ,∆p−1∑

i=0

eTti,ti+1

Σ−1v eti,ti+1

, (23)

while the latter incorporates a single pose correction,

Oc , eTt0,t∆p

Σ−1c et0,t∆p

, (24)

with the pose error defined as

e1,2 = ln(T1,2T2T

−11

)∨. (25)

We refer the reader to [23] for a detailed treatment of pose-graph relaxation.

IV. EXPERIMENTS

To assess the power of our proposed deep correctiveparadigm, we trained DPC-Net on visual data with localiza-tion estimates from a sparse stereo visual odometry (S-VO)estimator. In addition to training the full SE(3) DPC-Net,we modified the loss function and input data to learn simplerSO(3) rotation corrections, and simpler still, yaw angle correc-tions. For reference, we compared S-VO with different DPC-Net corrections to a state-of-the-art dense estimator. Finally,we trained DPC-Net on visual data and localization estimatesfrom radially-distorted and cropped images.

A. Training & Testing

For all experiments, we used the KITTI odometry bench-mark training set [24]. Specifically, our data consisted of theeight sequences 00,02 and 05-10 (we removed sequences 01,03, 04 to ensure that all data originated from the ‘residential’category for training and test consistency).

For testing and validation, we selected the first three se-quences (00,02, and 05). For each test sequence, we selectedthe subsequent sequence for validation (i.e., for test sequence00 we validated with 02, for 02 with 05, etc.) and usedthe remaining sequences for training. We note that by design,we train DPC-Net to predict corrections for a specific sensorand estimator pair. A pre-trained DPC-Net may further serveas a useful starting point to fine-tune new models for othersensors and estimators. In this work, however, we focus onthe aforementioned KITTI sequences and leave a thoroughinvestigation of generalization for future work.

To collect training samples, {T∗i , Iti,ti+∆p}Ni=0, we used a

stereo visual odometry estimator and GPS-INS ground-truthfrom the KITTI odometry dataset5. We resized all imagesto [400, 120] pixels, approximately preserving their originalaspect ratio6. For non-distorted data, we use ∆p ∈ [3, 4, 5] fortraining, and test with ∆p = 4. For distorted data, we reducethis to ∆p ∈ [2, 3, 4] and ∆p = 3, respectively, to compensatefor the larger estimation errors.

Our training datasets contained between 35,000 and 52,000training samples7 depending on test sequence. We trained allmodels for 30 epochs using the Adam optimizer, and selectedthe best epoch based on the lowest validation loss.

1) Rotation: To train rotation-only corrections, we ex-tracted the SO(3) component of T∗i and trained our networkusing Equation (17). Further, owing to the fact that rotationinformation can be extracted from monocular images, wereplaced the input stereo pairs in DPC-Net with monocularimages from the left camera8.

5We used RGB stereo images to train DPC-Net but grayscale images forthe estimator.

6Because our network is fully convolutional, it can, in principle, operateon different image resolutions with no modifications, though we do notinvestigate this ability in this work.

7If a sequence has M poses, we collect M−∆p training samples for each∆p.

8The stereo VO estimator remained unchanged.


2) Yaw: To further simplify the corrections, we extracted asingle-degree-of-freedom yaw rotation correction angle9 fromT∗i , and trained DPC-Net with monocular images and a meansquared loss.

B. Estimators

1) Sparse Visual Odometry: To collect T∗i , we first used aframe-to-frame sparse visual odometry pipeline similar to thatpresented in [3]. We briefly outline the pipeline here.

Using the open-source libviso2 package [25], we detectand track sparse stereo image key-points, yl,ti+1

and yl,ti ,between stereo image pairs (assumed to be undistorted andrectified). We model reprojection errors (due to sensor noiseand quantization) as zero-mean Gaussians with a knowncovariance, Σy,

el,ti = yl,ti+1− π(Tti+1,tiπ

−1(yl,ti)) (26)

∼ N (0,Σy) , (27)

where π(·) is the stereo camera projection function. To gen-erate an initial guess and to reject outliers, we use three pointRandom Sample Consensus (RANSAC) based on stereo re-projection error. Finally, we solve for the maximum likelihoodtransform, T∗t+1,t, through a Gauss-Newton minimization:

T∗ti+1,ti = argminTti+1,ti

∈SE(3)

Nti∑

l=1

eTl,ti Σy

−1el,ti . (28)

2) Sparse Visual Odometry with Radial Distortion: Similarto [5], we modified our input images to test our network’sability to correct estimators that compute poses from degradedvisual data. Unlike [5], who darken and blur their images, wechose to simulate a poorly calibrated lens model by applyingradial distortion to the (rectified) KITTI dataset using a plumb-bob distortion model. The model computes radially-distortedimage coordinates, xd, yd, from the normalized coordinatesxn, yn as

[xdyd

]=(1 + κ1r

2 + κ2r4 + κ3r

6) [xnyn

], (29)

where r =√x2n + y2

n. We set the distortion coefficients, κ1,κ2, and κ3 to −0.3, 0.2, 0.01 respectively, to approximatelymatch the KITTI radial distortion parameters. We solvedEquation (29) iteratively and used bilinear interpolation tocompute the distorted images for every stereo pair in asequence. Finally, we cropped each distorted image to removeany whitespace. Figure 4 illustrates this process.

With this distorted dataset, we computed S-VO localizationestimates and then trained DPC-Net to correct for the effectsof the radial distortion and effective intrinsic parameter shiftdue to the cropping process.

9We define yaw in the camera frame as the rotation about the camera’svertical y axis.

3) Dense Visual Odometry: Finally, we present localizationestimates from a computationally-intensive keyframe-baseddense, direct visual localization pipeline [26] that computesrelative camera poses by minimizing photometric error withrespect to a keyframe image. To compute the photometricerror, the pipeline relies on an inverse compositional approachto map the image coordinates of a tracking image to theimage coordinates of the reference depth image. As the cameramoves through an environment, a new keyframe depth imageis computed and stored when the camera field-of-view differssufficiently from the last keyframe.

We used this dense, direct estimator as our benchmark for astate-of-the-art visual localizer, and compared its accuracy tothat of a much less computationally expensive sparse estimatorpaired with DPC-Net.

C. Evaluation Metrics

To evaluate the performance of DPC-Net, we use three errormetrics: mean absolute trajectory error, cumulative absolutetrajectory error, and mean segment error. For clarity, wedescribe each of these three metrics explicitly and stress theimportance of carefully selecting and defining error metricswhen comparing relative localization estimates, as results canbe subtly deceiving.

1) Mean Absolute Trajectory Error (m-ATE): The meanabsolute trajectory error averages the magnitude of the rota-tional or translational error10 of estimated poses with respectto a ground truth trajectory defined within the same navigationframe. Concretely, em-ATE is defined as

em-ATE ,1

N

N∑

p=1

∥∥∥∥ln(T−1

p,0Tp,0

)∨∥∥∥∥ . (30)

Although widely used, m-ATE can be deceiving because asingle poor relative transform can significantly affect the finalstatistic.

2) Cumulative Absolute Trajectory Error (c-ATE): Cumu-lative absolute trajectory error sums rotational or translationalem-ATE up to a given point in a trajectory. It is defined as

ec-ATE(q) ,q∑

p=1

∥∥∥∥ln(T−1

p,0Tp,0

)∨∥∥∥∥ . (31)

c-ATE can show clearer trends than m-ATE (because it is lessaffected by fortunate trajectory overlaps), but it still suffersfrom the same susceptibility to poor (but isolated) relativetransforms.

3) Segment Error: Our final metric, segment error, averagesthe end-point error for all the possible segments of a givenlength within a trajectory, and then normalizes by the segmentlength. Since it considers multiple starting points within atrajectory, segment error is much less sensitive to isolateddegradations. Concretely, eseg(s) is defined as

eseg(s) ,1

sNs

Ns∑

p=1

∥∥∥∥ln(T−1

p+sp,pTp+sp,p

)∨∥∥∥∥ , (32)

10For brevity, the notation ln (·)∨ returns rotational or translational com-ponents depending on context.


DistortedOriginal Distorted & Cropped

Fig. 4. Illustration of our image radial distortion procedure. Left: rectified RGB image (frame 280 from KITTI odometry sequence 05). Middle: the sameimage with radial distortion applied. Right: distorted, cropped, and scaled image.

0 1000 2000 3000 4000

Timestep

0

50000

100000

150000

200000

250000

Cu

mu

lati

veE

rr.

Nor

m.

(m) Translational Cumulative Err. Norm.

0 1000 2000 3000 4000

Timestep

0

20000

40000

60000

80000

Cu

mu

lati

veE

rr.

Nor

m.

(deg

) Rotational Cumulative Err. Norm.

S-VO

S-VO + DPC (yaw)

S-VO + DPC (rot)

S-VO + DPC (pose)

Dense

(a) Sequence 00: Cumulative Errors.

200 400 600 800

Segment length (m)

1.5

2.0

2.5

3.0

3.5

Ave

rage

erro

r(%

)

Translational error

200 400 600 800

Segment length (m)

0.004

0.006

0.008

0.010

0.012

0.014

0.016

Ave

rage

erro

r(d

eg/m

)

Rotational error

S-VO

S-VO + DPC (yaw)

S-VO + DPC (rot)

S-VO + DPC (pose)

Dense

(b) Sequence 00: Segment Errors.

0 1000 2000 3000 4000

Timestep

0

100000

200000

300000

400000

Cu

mu

lati

veE

rr.

Nor

m.


0 1000 2000 3000 4000

Timestep

0

20000

40000

60000

Cu

mu

lati

veE

rr.

Nor

m.

(deg


S-VO

S-VO + DPC (yaw)

S-VO + DPC (rot)

S-VO + DPC (pose)

Dense

(c) Sequence 02: Cumulative Errors.

200 400 600 800

Segment length (m)

1.0

1.5

2.0

2.5

Ave

rage

erro

r(%

)

Translational error

200 400 600 800

Segment length (m)

0.004

0.006

0.008

0.010

Ave

rage

erro

r(d

eg/m

)

Rotational error

S-VO

S-VO + DPC (yaw)

S-VO + DPC (rot)

S-VO + DPC (pose)

Dense

(d) Sequence 02: Segment Errors.

0 500 1000 1500 2000 2500

Timestep

0

20000

40000

60000

80000

Cu

mu

lati

veE

rr.

Nor

m.


0 500 1000 1500 2000 2500

Timestep

0

5000

10000

15000

20000

25000

Cu

mu

lati

veE

rr.

Nor

m.

(deg


S-VO

S-VO + DPC (yaw)

S-VO + DPC (rot)

S-VO + DPC (pose)

Dense

(e) Sequence 05: Cumulative Errors.

200 400 600 800

Segment length (m)

1.0

1.5

2.0

2.5

Ave

rage

erro

r(%

)

Translational error

200 400 600 800

Segment length (m)

0.002

0.004

0.006

0.008

0.010

0.012

Ave

rage

erro

r(d

eg/m

)

Rotational error

S-VO

S-VO + DPC (yaw)

S-VO + DPC (rot)

S-VO + DPC (pose)

Dense

(f) Sequence 05: Segment Errors.

Fig. 5. c-ATE and mean segment errors for S-VO with and without DPC-Net.

−200 0 200 400 600

Easting (m)

−100

0

100

200

300

400

Nor

thin

g(m

)

Trajectory

Ground Truth

S-VO

S-VO + DPC (pose)

Dense

(a) Sequence 00.

−250 0 250 500 750 1000

Easting (m)

−200

0

200

400

600

Nor

thin

g(m

)

Trajectory

Ground Truth

S-VO

S-VO + DPC (pose)

Dense

(b) Sequence 02.

−400 −200 0 200

Easting (m)

−100

0

100

200

300

Nor

thin

g(m

)

Trajectory

Ground Truth

S-VO

S-VO + DPC (pose)

Dense

(c) Sequence 05.

Fig. 6. Top down projections for S-VO with and without DPC-Net.

where Ns and sp (the number of segments of a given length,and the number of poses in each segment, respectively) arecomputed based on the selected segment length s. In this work,we follow the KITTI benchmark and report the mean segmenterror norms for all s ∈ [100, 200, 300, ..., 800] (m).

V. RESULTS & DISCUSSION

A. Correcting Sparse Visual Odometry

Figure 5 plots c-ATE and mean segment errors for testsequences 00, 02 and 05 for three different DPC-Net models

REFERENCES 7

200 400 600 800

Segment length (m)

4

6

8

10

12

14

16

Ave

rage

erro

r(%

)Translational error

200 400 600 800

Segment length (m)

0.02

0.03

0.04

0.05

0.06

0.07

Ave

rage

erro

r(d

eg/m

)

Rotational error

S-VO

S-VO + DPC (rot)

S-VO + DPC (pose)

(a) Sequence 00 (Distorted): Segment Errors.

200 400 600 800

Segment length (m)

4

6

8

10

12

14

Ave

rage

erro

r(%

)

Translational error

200 400 600 800

Segment length (m)

0.02

0.03

0.04

0.05

Ave

rage

erro

r(d

eg/m

)Rotational error

S-VO

S-VO + DPC (rot)

S-VO + DPC (pose)

(b) Sequence 02 (Distorted): Segment Errors.

200 400 600 800

Segment length (m)

6

8

10

12

14

16

Ave

rage

erro

r(%

)

Translational error

200 400 600 800

Segment length (m)

0.02

0.03

0.04

0.05

0.06

0.07

Ave

rage

erro

r(d

eg/m

)

Rotational error

S-VO

S-VO + DPC (rot)

S-VO + DPC (pose)

(c) Sequence 05 (Distorted): Segment Errors.

Fig. 7. c-ATE and segment errors for S-VO with radially distorted imageswith and without DPC-Net.

paired with our S-VO pipeline. Table I summarize the resultsquantitatively, while Figure 6 plots the North-East projectionof each trajectory. On average, DPC-Net trained with the fullSE(3) loss reduced translational m-ATE by 72%, rotationalm-ATE by 75%, translational mean segment errors by 40%and rotational mean segment errors by 44% (relative to theuncorrected estimator). Mean segment errors of the sparseestimator with DPC approached those observed from thedense estimator on sequence 00, and outperformed the denseestimator on 02. Sequence 05 produced two notable results:(1) although DPC-Net significantly reduced S-VO errors, thedense estimator still outperformed it in all statistics and (2)the full SE(3) corrections performed slightly worse than theirSO(3) counterparts. We suspect the latter effect is a result ofmotion estimates with predominantly rotational errors whichare easier to learn with an SO(3) loss.

In general, coupling DPC-Net with a simple frame-to-framesparse visual localizer yielded a final localization pipeline withaccuracy similar to that of a dense pipeline while requiringsignificantly less visual data (recall that DPC-Net uses resizedimages).

B. Distorted Images

Figure 7 plots mean segment errors for the radially dis-torted dataset. On average, DPC-Net trained with the full

SE(3) loss reduced translational mean segment errors by50% and rotational mean segment errors by 35% (relativeto the uncorrected sparse estimator, see Table II). The yaw-only DPC-Net corrections did not produce consistent improve-ments (we suspect due to the presence of large errors in theremaining degrees of freedom as a result of the distortionprocedure). Nevertheless, DPC-Net trained with SE(3) andSO(3) losses was able to significantly mitigate the effect of apoorly calibrated camera model. We are actively working onmodifications to the network that would allow the correctedresults to approach those of the undistorted case.

VI. CONCLUSIONS

In this work, we presented DPC-Net, a novel way of fusingthe power of deep, convolutional networks with classicalgeometric localization pipelines. Using a novel loss functionbased on matrix Lie groups, DPC-Net learns SE(3) posecorrections to improve a baseline estimator and mitigates theeffect of estimator bias, environmental factors, and poor sensorcalibrations. We demonstrated how DPC-Net can render asparse stereo visual odometry pipeline as accurate as a state-of-the-art dense estimator, and significantly improve estimatescomputed with a poorly calibrated lens distortion model.In future work, we plan to investigate the addition of amemory state (through recurrent neural network structures),extend DPC-Net to other sensor modalities (e.g., lidar), andincorporate prediction uncertainty through the use of modernprobabilistic approaches to deep learning [27].

REFERENCES

[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol.521, no. 7553, pp. 436–444, 2015.

[2] V Peretroukhin, J Kelly, and T. D. Barfoot, “Optimizing cameraperspective for stereo visual odometry,” in Canadian Conference onComp. and Robot Vision, May 2014, pp. 1–7.

[3] V Peretroukhin, W Vega-Brown, N Roy, and J Kelly, “PROBE-GK:Predictive robust estimation using generalized kernels,” in Proc. IEEEInt. Conf. Robot. Automat. (ICRA), May 2016, pp. 817–824.

[4] V Peretroukhin, L Clement, M Giamou, and J Kelly, “PROBE:Predictive robust estimation for visual-inertial navigation,” in Proc.IEEE/RSJ Int. Conf. Intelligent Robots and Syst. (IROS), 2015,pp. 3668–3675.

[5] G Costante, M Mancini, P Valigi, and T. A. Ciarfuglia, “Exploringrepresentation learning with CNNs for Frame-to-Frame Ego-Motionestimation,” IEEE Robot. Autom. Letters, vol. 1, no. 1, pp. 18–25,Jan. 2016, ISSN: 2377-3766.

[6] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni, “VINet:Visual-Inertial odometry as a Sequence-to-Sequence learning prob-lem,” 2017. arXiv: 1701.08376 [cs.CV].

[7] A. Kendall and R. Cipolla, “Geometric loss functions for camera poseregression with deep learning,” 2017. arXiv: 1704.00390 [cs.CV].

[8] I. Melekhov, J. Ylioinas, J. Kannala, and E. Rahtu, “Relative camerapose estimation using convolutional neural networks,” 2017. arXiv:1702.01381 [cs.CV].

[9] G. L. Oliveira, N. Radwan, W. Burgard, and T. Brox, “Topometriclocalization with deep learning,” 2017. arXiv: 1706.08775 [cs.CV].

[10] V. Peretroukhin, L. Clement, and J. Kelly, “Reducing drift in visualodometry by inferring sun direction using a bayesian convolutionalneural network,” in Proc. IEEE Int. Conf. Robot. Automat. (ICRA),May 2017.

[11] D Scaramuzza and F Fraundorfer, “Visual odometry [tutorial],” IEEERobot. Autom. Mag., vol. 18, no. 4, pp. 80–92, Dec. 2011.

[12] C Cadena, L Carlone, H Carrillo, Y Latif, D Scaramuzza, J Neira,I Reid, and J. J. Leonard, “Past, present, and future of simultaneouslocalization and mapping: Toward the Robust-Perception age,” IEEETrans. Rob., vol. 32, no. 6, pp. 1309–1332, Dec. 2016.

http://arxiv.org/abs/1701.08376





TABLE IM-ATE AND MEAN SEGMENT ERRORS FOR VO RESULTS WITH AND WITHOUT DPC-NET.

m-ATE Mean Segment Errors

Sequence (Length) Estimator Corr. Type Translation (m) Rotation (deg) Translation (%) Rotation (millideg / m)

00 (3.7 km)1 S-VO — 60.22 18.25 2.88 11.18Dense — 12.41 2.45 1.28 5.42S-VO + DPC-Net Pose 15.68 3.07 1.62 5.59

Rotation 26.67 7.41 1.70 6.14Yaw 29.27 8.32 1.94 7.47


Rotation 20.66 3.10 1.21 4.28Yaw 49.07 10.17 1.53 6.56


Rotation 9.67 2.53 1.10 4.68Yaw 18.37 6.15 1.37 6.90

1 Training sequences 05,06,07,08,09,10. Validation sequence 02.2 Training sequences 00,06,07,08,09,10. Validation sequence 05.3 Training sequences 00,02,07,08,09,10. Validation sequence 06.4 All models trained for 30 epochs. The final model is selected based on the epoch with the lowest validation error.

TABLE IIM-ATE AND MEAN SEGMENT ERRORS FOR VO RESULTS WITH AND WITHOUT DPC-NET FOR DISTORTED IMAGES.

m-ATE Mean Segment Errors

Sequence (Length) Estimator Corr. Type Translation (m) Rotation (deg) Translation (%) Rotation (millideg / m)

00-distorted (3.7 km) S-VO — 168.27 37.15 14.52 46.43S-VO + DPC Pose 114.35 28.64 6.73 29.93

Rotation 84.54 21.90 9.58 25.28


Rotation 269.90 53.11 9.25 25.99


Rotation 71.42 13.10 8.14 23.56

[13] S Levine, C Finn, T Darrell, and P Abbeel, “End-to-end training ofdeep visuomotor policies,” J. Mach. Learn. Res., 2016.

[14] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel, “Bench-marking deep reinforcement learning for continuous control,” in Proc.Int. Conf. on Machine Learning, ser. ICML’16, 2016, pp. 1329–1338.

[15] F. Yang, W. Choi, and Y. Lin, “Exploit all the layers: Fast and accuratecnn object detector with scale dependent pooling and cascadedrejection classifiers,” in Proc. IEEE Int. Conf. Comp. Vision andPattern Recognition (CVPR), 2016, pp. 2129–2137.

[16] N. Sunderhauf, S. Shirazi, A. Jacobson, F. Dayoub, E. Pepperell, B.Upcroft, and M. Milford, “Place recognition with ConvNet landmarks:Viewpoint-robust, condition-robust, training-free,” in Proc. Robotics:Science and Systems XII, Jul. 2015.

[17] T. Haarnoja, A. Ajay, S. Levine, and P. Abbeel, “Backprop KF: Learn-ing discriminative deterministic state estimators,” in Proceedings ofNeural Information Processing Systems (NIPS), 2016.

[18] A. Kendall, K. Alex, G. Matthew, and C. Roberto, “PoseNet: Aconvolutional network for Real-Time 6-DOF camera relocalization,”in Proc. of IEEE Int. Conf. on Computer Vision (ICCV), 2015.

[19] A. Handa, M. Bloesch, V. Patraucean, S. Stent, J. McCormac, andA. Davison, “Gvnn: Neural network library for geometric computervision,” in Computer Vision – ECCV 2016 Workshops, Springer,Cham, 2016, pp. 67–82.

[20] I Cvišic and I Petrovic, “Stereo odometry based on careful featureselection and tracking,” in Proc. European Conf. on Mobile Robots,2015, pp. 1–6.

[21] Q. Li, J. Qian, Z. Zhu, X. Bao, M. K. Helwa, and A. P. Schoellig,“Deep neural networks for improved, impromptu trajectory trackingof quadrotors,” in Proc. IEEE Int. Conf. Robot. Automat. (ICRA),2017, pp. 5183–5189.

[22] A Punjani and P Abbeel, “Deep learning helicopter dynamics mod-els,” in Proc. IEEE Int. Conf. Robot. Automat. (ICRA), May 2015,pp. 3223–3230.

[23] T. D. Barfoot, State estimation for robotics. Cambridge UniversityPress, Jul. 2017.

[24] A Geiger, P Lenz, C Stiller, and R Urtasun, “Vision meets robotics:The KITTI dataset,” Int. J. Rob. Res., vol. 32, no. 11, pp. 1231–1237,2013.

[25] A Geiger, J Ziegler, and C Stiller, “StereoScan: Dense 3D reconstruc-tion in real-time,” in Proc. Intelligent Vehicles Symp. (IV), IEEE, Jun.2011, pp. 963–968.

[26] L. Clement and J. Kelly, “How to train a CAT: Learning canonicalappearance transformations for robust direct localization under illumi-nation change,” 2017. arXiv-preprint: arXiv:submit/2002132 (cs.CV).

[27] A. Kendall and Y. Gal, “What uncertainties do we need in bayesiandeep learning for computer vision?,” 2017. arXiv: 1703 . 04977[cs.CV].

arXiv:submit/2002132



peretroukhin et al. dpc-net: deep pose correction for visual localization · peretroukhin et al.:...

Documents