Self-supervising Fine-grained Region Similaritiesfor Large-scale Image Localization

Yixiao Ge1, Haibo Wang3, Feng Zhu2, Rui Zhao2, and Hongsheng Li1

1 The Chinese University of Hong Kong2 SenseTime Research

3 China University of Mining and Technologyyxge@link,hsli@ee.cuhk.edu.hk

zhufeng,zhaorui@sensetime.com haibo@cumt.edu.cn

Abstract. The task of large-scale retrieval-based image localization isto estimate the geographical location of a query image by recognizing itsnearest reference images from a city-scale dataset. However, the generalpublic benchmarks only provide noisy GPS labels associated with thetraining images, which act as weak supervisions for learning image-to-image similarities. Such label noise prevents deep neural networks fromlearning discriminative features for accurate localization. To tackle thischallenge, we propose to self-supervise image-to-region similarities inorder to fully explore the potential of difficult positive images along-side their sub-regions. The estimated image-to-region similarities canserve as extra training supervision for improving the network in gener-ations, which could in turn gradually refine the fine-grained similaritiesto achieve optimal performance. Our proposed self-enhanced image-to-region similarity labels effectively deal with the training bottleneck inthe state-of-the-art pipelines without any additional parameters or man-ual annotations in both training and inference. Our method outperformsstate-of-the-arts on the standard localization benchmarks by noticeablemargins and shows excellent generalization capability on multiple imageretrieval datasets.†

1 Introduction

Image-based localization (IBL) aims at estimating the location of a given imageby identifying reference images captured at the same places from a geo-taggeddatabase. The task of IBL has long been studied since the era of hand-craftedfeatures [29, 35, 2, 21, 22, 34] and has been attracting increasing attention withthe advances of convolutional neural networks (CNN) [46], motivated by its wideapplications in SLAM [30, 17] and virtual/augmented reality [5]. Previous workshave been trying to tackle IBL as image retrieval [1, 28, 23], 2D-3D structurematching [27, 38] or geographical position classification [43, 44, 19] tasks. In thispaper, we treat the problem as an image retrieval task, given its effectivenessand feasibility in large-scale and long-term localization.

†Code of this work is available at https://github.com/yxgeee/SFRS.

2 Y. Ge et al.

6LPLODULW\/DEHO

4XHU\ 4XHU\ )XOO,PDJH +DOI5HJLRQV

6/ 6/

6/

6/

+DOI5

HJLRQV

4XDUWHU5HJLRQV 6/ 6/

6/ 6/ 6LPLODULW\/DEHO

)XOO,PDJH

4XHU\

6LPLODULW\/DEHO

4XHU\723(DVLHVW 723(DVLHVW 3RWHQWLDO'LIILFXOW3RVLWLYHV

6LPLODULW\/DEHO 6/ 6/ 6/ 6/

D E

ED

Fig. 1. (a) To mitigate the noise with weak GPS labels, existing works [1, 28] onlyutilized the easiest top-1 image of the query for training. (b) We propose to adopt self-enhanced similarities as soft training supervisions to effectively explore the potentialof difficult positives

6LPLODULW\/DEHO

4XHU\ 4XHU\ )XOO,PDJH +DOI5HJLRQV

6/ 6/

6/

6/

+DOI5HJLRQV

4XDUWHU5HJLRQV 6/ 6/

6/ 6/ 6LPLODULW\/DEHO

)XOO,PDJH

4XHU\

6LPLODULW\/DEHO

4XHU\723(DVLHVW 723(DVLHVW 3RWHQWLDO'LIILFXOW3RVLWLYHV

6LPLODULW\/DEHO 6/ 6/ 6/ 6/

D E

ED

Fig. 2. (a) Even with true positive pairs, the image-level supervision provides mislead-ing information for non-overlapping regions. (b) We further estimate fine-grained softlabels to refine the supervision by measuring query-to-region similarities

The fundamental challenge faced by image retrieval-based methods [1, 28,23] is to learn image representations that are discriminative enough to tell apartrepetitive and similar-looking locations in GPS-tagged datasets. It is cast as aweakly-supervised task because geographically close-by images may not depictthe same scene when facing different directions.

To avoid being misled by noisy and weak GPS labels, most works [1, 28]utilized on-the-fly first-ranking images of the queries as their positive trainingsamples, i.e. to force the queries to be closer to their already nearest neighbors(Fig. 1(a)). Consequently, a paradox arises when only the most confident, or inother words, the easiest on-the-fly positives are utilized in the training process,but these images in turn result in a lack of robustness to varying conditionsas the first-ranking images for queries might be too simple to provide enoughsupervisions for learning robust feature representations. To tackle the issue, weargue that difficult positives are needed for providing informative supervisions.

Identifying the true difficult positives, however, is challenging with only ge-ographical tags provided from the image database. Therefore, the network iseasy to collapse when naıvely trained with lower-ranking positive samples, whichmight not have overlapping regions with the queries at all. Such false positivesmight deteriorate the feature learning. Kim et al.[23] attempted to mine truepositive images by verifying their geometric relations, but it is limited by theaccuracy of off-the-shelf geometric techniques [18, 9]. In addition, even if pairs ofimages are indeed positive pairs, they might still have non-corresponding regions.The correct image-level labels might not necessarily be the correct region-levellabels. Therefore, the ideal supervisions between paired positive images shouldprovide region-level correspondences for more accurately supervising the learningof local features.

Self-supervising Fine-grained Region Similarities 3

To tackle the above mentioned challenge, we propose to estimate informativesoft image-to-region supervisions for training image features with noisy pos-itive samples in a self-supervised manner. In the proposed system, an imageretrieval network is trained in generations, which gradually improves itself withself-supervised image-to-region similarities. We train the network for the firstgeneration following the existing pipelines [1, 28], after which the network canbe assumed to successfully capture most feature distribution of the trainingdata. For each query image, however, its k-nearest gallery images accordingto the learned features might still contain noisy (false) positive samples. Di-rectly utilizing them as difficult true positives in existing pipelines might worsenthe learned features. We therefore propose to utilize their previous generation’squery-gallery similarities to serve as the soft supervision for training the networkin a new generation. The generated soft supervisions are gradually refined andsharpened as the network generation progresses. In this way, the system is nolonger limited to learning only from the on-the-fly easiest samples, but can fullyexplore the potential of the difficult positives and mitigate their label noise withrefined soft confidences (Fig. 1(b)).

However, utilizing only the image-level soft supervisions for paired positiveimages simply forces features from all their spatial regions to approach the sametarget scores (Fig. 2(a)). Such an operation would hurt the network’s capabilityof learning discriminative local features. To mitigate this problem, we furtherdecompose the matching gallery images into multiple sub-regions of differentsizes. The query-to-region similarities estimated from the previous generation’snetwork can serve as the refined soft supervisions for providing more fine-grainedguidance for feature learning (Fig. 2(b)). The above process can iterate and trainthe network for multiple generations to progressively provide more accurate andfine-grained image-to-region similarities for improving the features.

Our contributions can be summarised as three-fold. (1) We propose to es-timate and self-enhance the image similarities of top-ranking gallery images tofully explore the potential of difficult positive samples in image-based localization(IBL). The self-supervised similarities serve as refined training supervisions toimprove the network in generations, which in turn, generates more accurate im-age similarities. (2) We further propose to estimate image-to-region similaritiesto provide region-level supervisions for enhancing the learning of local features.(3) The proposed system outperforms state-of-the-art methods on standard lo-calization benchmarks by noticeable margins, and shows excellent generalizationcapability on both IBL and standard image retrieval datasets.

2 Related Work

Image-based Localization (IBL). Existing works on image-based localizationcan be grouped into image retrieval-based [1, 28, 23], 2D-3D registration-based[27, 38] and per-position classification-based [43, 44, 19] methods. Our work aimsat solving the training bottleneck of the weakly-supervised image retrieval-basedIBL problem. We therefore mainly discuss related solutions that cast IBL as an

4 Y. Ge et al.

image retrieval task. Retrieval-based IBL is mostly related to the traditionalplace recognition task [3, 6, 24, 39, 41], which has long been studied since the eraof hand-engineered image descriptors, e.g. SIFT [29], BoW [35], VLAD [2] andFisher Vector [34]. Thanks to the development of deep learning [26], NetVLAD[1] successfully transformed dense CNN features for localization by proposing alearnable VLAD layer to effectively aggregate local descriptors with learnable se-mantic centers. Adopting the backbone of NetVLAD, later works further lookedinto multi-scale contextual information [23] or effective metric learning [28] toachieve better performance. Our work has the similar motivation with [23], i.e.to mine difficult positive samples for more effective local feature learning. How-ever, [23] was only able to roughly mine positive images by adopting off-the-shelfgeometric verification techniques [18, 9] and required noticeable more parame-ters for both training and inference. In contrast, our method self-enhances fine-grained supervisions by gradually estimating and refining image-to-region softlabels without any additional parameters.Self-supervised label estimation has been widely studied in self-supervisedlearning methods [16, 25, 8, 32, 33], where the network mostly learns to predicta collective label set by properly utilizing the capability of the network itself.Some works [4, 11, 12, 14] proposed to create task-relative pseudo labels, e.g. [4]generated image-level pseudo labels for classification by clustering features on-the-fly. The others [31, 15, 13, 48] attempted to optimize the network by dealingwith pretext tasks whose labels are more easily to create, e.g. [31] solved jigsawpuzzles and [15] predicted the rotation of transformed images. Inspired by theself-supervised learning methods, we propose a self-supervised image-to-regionlabel creation scheme for training the network in generations, which effectivelymitigates the image-to-image label noise caused by weak geo-tagged labels.Self-distillation. Training in generations via self-predicted soft labels has beeninvestigated in self-distillation methods [10, 47, 45]. However, soft labels in thesemethods are not applicable to our task, as they focus on the classification prob-lem with predefined classes. We successfully generalize self-distillation to theweakly-supervised IBL problem by proposing to generate soft labels for bothquery-to-gallery and query-to-region similarities in the retrieval task.

3 Method

We propose to self-supervise image-to-region similarities for tackling the problemof noisy pairwise image labels in image-based localization (IBL). In major pub-lic benchmarks, each training image is generally associated with a GPS locationtag. However, images geographically close to each other do not necessarily shareoverlapping regions. This is because the images might be captured from oppo-site viewing directions. The GPS tags could only help to geographically discoverpotential positive images with much label noise. Such a limitation causes bottle-neck for effectively training the neural networks to recognize challenging positivepairs. There were previous methods [1, 28, 23] tackling this problem. However,they either ignored the potential of difficult but informative positive images,

Self-supervising Fine-grained Region Similarities 5

(QKDQFHG,PDJHWRUHJLRQ6LPLODULW\/DEHOV

(DVLHVW

3RVLWLYH

4XHU\

'LIILFXOW3RVLWLYHV

:HDN*36/DEHOV

4XHU\DQG(DVLHVW3RVLWLYHV

(VWLPDWHG,PDJHWRUHJLRQ6LPLODULW\/DEHOV

'LIILFXOW3RVLWLYHV

«

1(7 1(7 1(7 1(7

(QKDQFHG,PDJHWRUHJLRQ6LPLODULW\/DEHOV

'LIILFXOW3RVLWLYHV 'LIILFXOW3RVLWLYHV

«

«

D

6HFRQG*HQHUDWLRQ)LUVW*HQHUDWLRQ 7KLUG*HQHUDWLRQ )RXUWK*HQHUDWLRQ

6LPLODULW\

/DEHO

)XOO,PDJH +DOI5HJLRQV +DOI5HJLRQV 4XDUWHU5HJLRQV4XHU\

E

Fig. 3. Illustration of our proposed self-supervised image-to-region similarities for im-age localization, where the image-to-region similarities are gradually refined via trainingthe network in generations as shown in (a). The sidebars for each difficult positive imagedemonstrate the soft similarity labels for the full image, left half, right half, top half,bottom half, top-left quarter, top-right quarter, bottom-left quarter and bottom-rightquarter sub-regions respectively, detailed in (b), whereas the easiest positives only haveone bar indicating the similarity label for the full image. Note that the most difficultnegative samples utilized for joint training are not shown in the figure for saving space

or required off-the-shelf time-consuming techniques with limited precision forfiltering positives.

As illustrated in Fig. 3, the key of our proposed framework is to gradually en-hance image-to-region similarities as the training proceeds, which can in turn actas soft and informative training supervisions for iteratively refining the networkitself. In this way, the potential of difficult positive samples can be fully exploredand the network trained with such similarities can encode more discriminativefeatures for accurate localization.

3.1 Retrieval-based IBL Methods Revisit

State-of-the-art retrieval-based IBL methods [1, 23, 28] adopt the following gen-eral pipeline with different small modifications. They generally train the networkwith learnable parameters θ by feeding triplets, each of which consists of onequery image q, its easiest positive image p∗ and its multiple most difficult nega-

6 Y. Ge et al.

tive images nj |Nj=1. Such triplets are sampled according to the image featuresfθ encoded by the current network on-the-fly. The easiest positive image p∗ isthe top-1 ranking gallery image within 10 meters of the query q, and the mostdifficult negative images nj |Nj=1 are randomly sampled from top-1000 galleryimages that are more than 25 meters away from q.

The network usually consists of a backbone encoder and a VLAD layer [1],where the encoder embeds the image into a dense feature map and the VLADlayer further aggregates the feature map into a compact feature vector. Thenetwork can be optimized with a triplet ranking loss [1, 23] or a contrastiveloss [37]. More recently, Liu et al.[28] proposed a softmax-based loss with dotproducts for better maximizing the ratio between the query-positive pair againstmultiple query-negative pairs, which is formulated as

Lhard(θ) = −N∑j=1

logexp〈fqθ , f

p∗

θ 〉exp〈fqθ , f

p∗

θ 〉+ exp〈fqθ , fnj

θ 〉, (1)

where θ is the network parameters, fqθ is the query image features encoded by thecurrent network, and 〈·, ·〉 denotes dot product between two vectors. Trained bythis pipeline, the network can capture most feature distributions and generateacceptable localization results. Such a network acts as the baseline in our paper.

However, an obvious problem arises: the network trained with the easiest pos-itive images alone cannot well adapt to challenging positive pairs, as there gener-ally exists variations in viewpoints, camera poses and focal lengths in real-worldIBL datasets. In addition, the image-level weak supervisions provide misleadinginformation for learning local features. There are non-overlapping regions even intrue positive pairs. The network trained with image-level supervisions might pullthe non-overlapping regions to be close to each other in the feature space. Suchincorrect image-level supervisions impede the feature learning process. To tacklesuch challenges, we propose to utilize self-enhanced fine-grained supervisions forsupervising both image-to-image and image-to-region similarities. In this way,difficult positive images as well as their sub-regions can be fully exploited toimprove features for localization.

3.2 Self-supervising Query-gallery Similarities

A naıve way to utilize difficult positive images is to directly train the networkwith lower-ranking positive images (e.g. , top-k gallery images) as positive sam-ples with “hard” (one-hot) labels in Eq. (1). However, this may make the trainingprocess deteriorate, because the network cannot effectively tell the true positivesapart from the false ones when trained with the existing pipeline in Sec. 3.1. Tomitigate the label noise and make full use of the difficult positive samples, wepropose to self-supervise query-gallery similarities of top-ranking gallery imageswhich also act as the refined soft supervisions, and to gradually improve thenetwork with self-enhanced similarity labels in generations.

We use θω to indicate the network parameters in different generations, de-noted as θω|Ωω=1, where Ω is the total number of generations. In each genera-

Self-supervising Fine-grained Region Similarities 7

tion, the network is initialized by the same ImageNet-pretrained [7] parametersand is trained again with new supervision set until convergence. We adopt thesame training process for each generation, except for the first initial generation,for which we train the network parameter θ1 following the general pipeline in Sec.3.1. Before proceeding to training the 2nd-generation network, the query-galleryfeature distances are estimated by the 1st-generation network. For each queryimage q, its k-reciprocal nearest neighbors p1, · · · , pk according to Euclideandistances are first obtained from the gallery image set. The query-gallery simi-larities can be measured by dot products and normalized by a softmax functionwith temperature τ1 over the top-k images,

Sθ1(q, p1, · · · , pk; τ1) = softmax([〈fqθ1 , f

p1θ1〉/τ1, · · · , 〈fqθ1 , f

pkθ1〉/τ1

]>), (2)

where fpiθ1 is the encoded feature representations of the ith gallery image by the1st-generation network, and τ1 is temperature hyper-parameter that makes thesimilarity vector sharper or smoother.

Rather than naıvely treating all top-k ranking images as the true positivesamples for training the next-generation network, we propose to utilize the rela-tive similarity vector Sθ1 as extra training supervisions. There are two advantagesof adopting such supervisions: (1) We are able to use the top-ranking gallery im-ages as candidate positive samples. Those plausible positive images are moredifficult and thus more informative than the on-the-fly easiest positive samplesfor learning scene features. (2) To mitigate the inevitable label noise from theplausible positive samples, we propose to train the next-generation network toapproach the relative similarity vectors Sθ1 , which “softly” measures the rela-tive similarities between (q, p1), · · · , (q, pk) pairs. At earlier generations, we setthe temperature parameter τω to be large, as the relative similarities are lessaccurate. The large temperature makes the similarity vector Sθω more equallydistributed. At later generations, the network becomes more accurate. The rel-ative similarity vector Sθω ’s maximal response is more accurate to identify thetrue positive samples. Lower temperatures are used to make Sθω concentrate ona small number of gallery images.

The relative similarity vectors Sθ1 estimated by the 1st-generation networkare used to supervise the 2nd-generation network via a “soft” cross-entropy loss,

Lsoft(θ2) = `ce (Sθ2(q, p1, · · · , pk; 1),Sθ1(q, p1, · · · , pk; τ1)) , (3)

where `ce(y, y) = −∑i y(i) log(y(i)) denotes the cross-entropy loss. Note that

only the learning target Sθ1 adopts the temperature hyper-parameter to con-trol its sharpness. In our experiments, the performance of the trained networkgenerally increases with generations and saturates around the 4th generation,with temperatures set as 0.07, 0.06, 0.05, respectively. In each iteration, both the“soft” cross-entropy loss and the original hard loss (Eq. (1)) are jointly adoptedas L(θ) = Lhard(θ) + λLsoft(θ) for supervising the feature learning.

By self-supervising query-gallery similarities, we could fully utilize the dif-ficult positives obtained from the network of previous generations. The “soft”

8 Y. Ge et al.

supervisions can effectively mitigate the label noise. However, even with the re-fined labels, paired positive images’ local features from all their spatial regionsare trained to approach the same similarity scores, which provide misleading su-pervision for those non-overlapping regions. The network’s capability on learningdiscriminative local features is limited given only such image-level supervisions.

3.3 Self-supervising Fine-grained Image-to-region Similarities

To tackle the above challenge, we propose to fine-grain the image-to-image sim-ilarities into image-to-region similarities to generate more detailed supervisions.Given the top-k ranking images p1, · · · , pk for each query q, instead of directlycalculating image-level relative similarity vector Sθω , we can further decomposeeach plausible positive image’s feature maps into 4 half regions (top, bottom, left,right halves) and 4 quarter regions (top-left, top-right, bottom-left, bottom-rightquarters). Specifically, The gallery image pi’s feature maps mi are first obtainedby the network backbone before the VLAD layer. We split mi into 4 half regionsand 4 quarter regions, and then feed them into the aggregation VLAD respec-

tively for obtaining 8 region feature vectors fr1i

θ , fr2iθ , · · · , f

r8iθ corresponding to

the 8 image sub-regions r1i , r2i , · · · , r8i , each of which depicts the appearanceof one sub-region of the gallery scene. Thus, the query-gallery supervisions arefurther fine-grained by measuring the relative query-to-region similarities,

Srθω (q, p1, · · · , pk;τω) = softmax([〈fqθω , f

p1θω〉/τω, 〈fqθω , f

r11θω〉/τω, · · · , 〈fqθω , f

r81θω〉/τω,

· · · , 〈fqθω , fpkθω〉/τω, 〈fqθω , f

r1kθω〉/τω, · · · , 〈fqθω , f

r8kθω〉/τω

]). (4)

The “soft” cross-entropy loss in Eq. (3) can be extended such that it can learnfrom the fine-grained similarities in Eq. (4) to encode more discriminative localfeatures,

Lrsoft(θω) = `ce

(Srθω (q, p1, · · · , pk; 1),Srθω−1

(q, p1, · · · , pk; τω−1)). (5)

The image-to-region similarities can also be used for mining the most difficultnegative gallery-image regions for each query q. The mined negative regions canbe used in the hard loss in Eq. (1). For each query q, the accurate negativeimages could be easily identified by the geographical GPS labels. We proposeto further mine the most difficult region n∗j for each of the negative images njon-the-fly by measuring query-to-region similarities with the current network.The image-level softmax-based loss in Eq. (1) could be refined with the mostdifficult negative regions as

Lrhard(θω) = −N∑j=1

logexp〈fqθω , f

p∗

θω〉

exp〈fqθω , fp∗

θω〉+ exp〈fqθω , f

n∗j

θω〉. (6)

Note that the image-to-region similarities for selecting the most difficult nega-tive regions are measured by the network on-the-fly, while those for supervisingdifficult positives in Eq. (5) are measured by the previous generation’s network.

Self-supervising Fine-grained Region Similarities 9

In our proposed framework, the network is trained with extended triplets,each of which consists of one query image q, the easiest positive image p∗, thedifficult positive images pi|ki=1, and the most difficult regions in the negativeimages n∗j |Nj=1. The overall objective function of generation ω is formulated as

L(θω) = Lrhard(θω) + λLrsoft(θω), (7)

where λ is the loss weighting factor. The multi-generation training can be per-formed similarly to that mentioned in Sec. 3.2 to self-supervise and to graduallyrefine the image-to-region similarities.

3.4 Discussions

Why not decompose queries for self-supervising region-to-region sim-ilarities? We observe that region-to-region similarities might contain many su-perficially easy cases that are not informative enough to provide discriminativetraining supervisions for the network. For instance, the similarity between thesky regions of two overlapping images might be too easy to serve as effectivetraining samples. Image-to-region similarities could well balance the robustnessand granularity of the generated supervisions, as region-to-region similaritieshave more risk to be superficial.Why still require Lrhard? We observe that the top-1 gallery images obtainedfrom the k-reciprocal nearest neighbors can almost always act as true positivesfor training with hard supervisions in Eq. (6). The hard loss could stabilize thetraining process to avoid error amplification.

4 Experiments

4.1 Implementation Details

Datasets. We utilize the Pittsburgh benchmark dataset [42] for optimizing ourimage retrieval-based localization network following the experimental settings ofstate-of-the-art methods [28, 1]. Pittsburgh consists of a large scale of panoramicimages captured at different times and are associated with noisy GPS locations.Each panoramic image is projected to create 24 perspective images. For faircomparison, we use the subset Pitts30k for training and select the best modelthat achieves the optimal performance on the val-set of Pitts30k. The Pitts30k-train contains 7,416 queries and 10,000 gallery images, and the Pitts30k-valconsists of 7,608 queries and 10,000 gallery images. We obtain the final retrievalresults by ranking images in the large-scale Pitts250k-test, which contains 8,280probes and 83,952 database images.

To verify the generalization ability of our method on different IBL tasks, wedirectly evaluate our models trained on Pitts30k-train on the Tokyo 24/7 [41]dataset, which is quite challenging since the queries were taken in varying con-ditions. In addition, we evaluate the model’s generalization ability on standardimage retrieval datasets, e.g. the Oxford 5k [35], Paris 6k [36], and Holidays [20].

10 Y. Ge et al.

Table 1. Comparison with state-of-the-arts on image-based localization benchmarks.Note that the network is only trained on Pitts30k-train and directly evaluated on bothTokyo 24/7 and Pitts250k-test datasets

MethodTokyo 24/7 [41] Pitts250k-test [42]

R@1 R@5 R@10 R@1 R@5 R@10

NetVLAD [1] (CVPR’16) 73.3 82.9 86.0 86.0 93.2 95.1CRN [23] (CVPR’17) 75.2 83.8 87.3 85.5 93.5 95.5SARE [28] (ICCV’19) 79.7 86.7 90.5 89.0 95.5 96.8

Ours 85.4 91.1 93.3 90.7 96.4 97.6

Evaluation. During inference, we perform PCA whitening whose parametersare learnt on the Pitts30k-train, reducing the feature dimension to 4,096. Wefollow the same evaluation metric of [28, 1], where the top-k recall is measuredon the localization datasets, Pitts250k-test [42] and Tokyo 24/7 [41]. The queryimage is determined to be successfully retrieved from top-k if at least one of thetop k retrieved reference images locates within d = 25 meters from the queryimage. As for the image-retrieval datasets, Oxford 5k [35], Paris 6k [36], andHolidays [20], the mean average precision (mAP) is adopted for evaluation.Architecture. For fair comparison, we adopt the same architecture used in [28,1], which comprises of a VGG-16 [40] backbone and a VLAD layer [1] for encodingand aggregating feature representations. We use the ImageNet-pretrained [7]VGG-16 up to the last convolutional layer (i.e. conv5) before ReLU, as thebackbone. Following [28], the whole backbone except the last convolutional block(i.e. conv5) is frozen when trained on image-based localization datasets.Training details. For data organization, each mini-batch contains 4 triplets,each of which consists of one query image, one easiest positive image, top-10difficult positive images and 10 negative images. The negative images are sam-pled following the same strategy in [28, 1]. Our model is trained by 4 genera-tions, with 5 epochs in each generation. We empirically set the hyper-parametersλ = 0.5, τ1 = 0.07, τ2 = 0.06, τ3 = 0.05 in all experiments. The stochastic gra-dient descent (SGD) algorithm is utilized to optimize the loss function, withmomentum 0.9, weight decay 0.001, and a constant learning rate = 0.001.

4.2 Comparison with State-of-the-arts

We compare with state-of-the-art image localization methods NetVLAD [1],CRN [23] and SARE [28] on localization datasets Pitts250k-test [42] and Tokyo24/7 [41] in this experiment. Our model is only trained on Pitts30k-train with-out any Tokyo 24/7 images. CRN (Contextual Reweighting Network) improvesNetVLAD by proposing a contextual feature reweighting module for selectingmost discriminative local features for aggregation. While a Stochastic Attractionand Repulsion Embedding (SARE) loss function is proposed on top of VLAD-aggregated feature embeddings in [28]. None of the above methods have wellhandled the bottleneck of weakly-supervised training for image localization.

Self-supervising Fine-grained Region Similarities 11

0 5 10 15 20 25N-Number of top retrieved database images

75

80

85

90

95

Rec

all@

N(%

)

OursOurs w/o quarter regionsOurs w/o all sub-regionsOurs w/o positive sub-regionsOurs w/o negative sub-regionsOurs w/o top-k ranking refineryOurs w/o softmax temperature annealingBaselineBaseline w/ top-k positives

5 10 15 20 25N-Number of top retrieved database images

88

90

92

94

96

98

Rec

all@

N(%

)

OursOurs w/o quarter regionsOurs w/o all sub-regionsOurs w/o positive sub-regionsOurs w/o negative sub-regionsOurs w/o top-k ranking refineryOurs w/o softmax temperature annealingBaselineBaseline w/ top-k positives

(a) Tokyo 24/7 [41] (b) Pitts250k-test [42]

Fig. 4. Quantitative results of ablation studies on our proposed method in terms oftop-k recall on Tokyo 24/7 and Pitts250k-test datasets. The models are only trainedon Pitts30k-train set

Experimental results are shown in Tab. 1. The proposed method achieves90.7% rank-1 recall on Pitts250k-test, outperforming the second best 89.0% ob-tained by SARE, with an improvement of 1.7%. The feature embeddings learnedby our method show very strong generalization capability on the challengingTokyo 24/7 dataset where the rank-1 recall is significantly boosted to 85.4%, upto 5.7% performance improvement against SARE. The superior performancesvalidate the effectiveness of our self-enhanced image-to-region similarities inlearning discriminative features for image-based localization.

4.3 Ablation Studies

We perform ablation studies on Tokyo 24/7 [41] and Pitts250k-test [42] to anal-yse the effectiveness of the proposed method and shed light on the importanceof supervisions from self-enhanced fine-grained image-to-region similarities. Weillustrate quantitative results in Fig. 4, and show the detailed ablation exper-iments in Tab. 2. “Baseline” is our re-implementation of SARE [28], which istrained with only the best-matching positive image using Lhard in Eq. (1).Directly training with noisy difficult positives in the existing pipeline.Existing image-based localization benchmarks only provide geo-tagged imagesfor learning image-to-image similarities. These GPS labels are weak and noisyin finding true positive images for given query images. SARE [28] and previ-ous works only choose the easiest positive for training, i.e. the most similar oneto the query in the feature space. Our proposed approach can effectively learninformative features from multiple difficult positives (top-k positive images) byintroducing self-enhanced image-region similarities as extra supervisions, thusboost the rank-1 recall to 90.7% and 85.4% on Pitts250k-test and Tokyo 24/7respectively. To check whether such difficult (top-k) positive images can be easilyused without our proposed self-enhanced image-to-region similarities to improvethe final performance, we conduct an ablation experiment by training the “Base-line” with extra difficult positive images. This model is denoted as “Baseline w/

12 Y. Ge et al.

Table 2. Ablation studies for our proposed method on individual components. Themodels are only trained on Pitts30k-train set

MethodTokyo 24/7 [41]

R@1 R@5 R@10

Baseline 80.6 87.6 90.8Baseline w/ top-k positives 76.2 88.6 90.8Baseline w/ regions 79.8 86.9 90.4

Ours w/o all sub-regions 80.6 88.0 90.9Ours w/o positive sub-regions 80.8 88.6 91.1Ours w/o negative sub-regions 82.2 89.2 92.7Ours w/o quarter regions 84.4 90.5 92.1

Ours w/o top-k ranking refinery 83.8 90.2 92.4Ours w/o softmax temperature annealing 83.5 88.6 90.8

Ours 85.4 91.1 93.3

top-k positives” in Tab. 2. It shows that naıvely adding more difficult positiveimages for training causes drastic performance drop, where the rank-1 recall de-creases from 80.6% (“Baseline”) to 76.2% on Tokyo 24/7. The trend is similar onPitts250k-test. This phenomenon validates the effectiveness of our self-enhancedimage-to-region similarities from difficult positive samples on the IBL tasks.

Effectiveness of fine-grained image-to-region similarities. As describedin Sec. 3.3, our proposed framework explores image-to-region similarities by di-viding a full image into 4 half regions and 4 quarter regions. Such design iscritical in our framework as query and positive images are usually only par-tially overlapped due to variations in camera poses. Simply forcing query andpositive images to be as close as possible in feature embedding space regardlessof non-overlapping regions would mislead feature learning, resulting in inferiorperformance. As shown in Tab. 2, on Tokyo 24/7, the rank-1 recall drops from85.4% to 84.4% (“Ours w/o quarter regions”) when the 4 quarter regions areexcluded from training, and further drastically drops to 80.6% (“Ours w/o allsub-regions”) when no sub-regions are used. The effectiveness of sub-regions inpositive and negative images are compared by “Ours w/o positive sub-regions”and “Ours w/o negative sub-regions” in Tab. 2. It shows that in both cases therank-1 recall is harmed, and the performance drop is even more significant whenpositive sub-regions are ignored. The above observations indicate that difficultpositives are critical for feature learning in IBL tasks, which have not been wellinvestigated in previous works, while our fine-grained image-to-region similaritieseffectively help learn discriminative features from these difficult positives.

Effectiveness of self-enhanced similarities as soft supervisions. Whencomparing “Ours w/o all sub-regions” and “Baseline w/ top-k positives”, onemay find that the only difference between these two methods is Lsoft in Eq.(3). By adding the extra objective Lsoft to “Baseline w/ top-k positives”, therank-1 recall is significantly improved from 76.1% to 80.6% (the same with “Base-line”) on Tokyo 24/7. “Ours w/o all sub-regions” even outperforms “Baseline”

Self-supervising Fine-grained Region Similarities 13

(a) Query (b) Our heatmap (c) [28]’s heatmap (d) Our top-1 (e) [28]’s top-1

Fig. 5. Retrieved examples of our method and state-of-the-art SARE [28] on Tokyo24/7 dataset [41]. The regions highlighted by red boxes illustrates the main differences.

on Pitts250k-test as shown in Fig. 4. The above comparisons demonstrate the ef-fectiveness of using our self-enhanced similarities as soft supervisions for learningfrom difficult positive images even at the full image level.

More importantly, soft supervisions serve as a premise for image-to-regionsimilarities to work. We evaluate the effects of soft supervisions at the regionlevel, dubbed as “Baseline w/ regions”, where the dataset is augmented withdecomposed regions and only the noisy GPS labels are used. The result in Tab.2 is even worse than “Baseline” since sub-regions with only GPS labels providemany too easy positives that are not informative enough for feature learning.Benefit from top-k ranking refinery. We propose to find k-reciprocal near-est neighbors of positive images for recovering more accurate difficult positivesimages for training in Sec. 3.2. Although our variant (“Ours w/o top-k rank-ing refinery”) without using k-reciprocal nearest neighbors refining top-k imagesalready outperforms “Baseline” by a large margin, ranking with k-reciprocalnearest neighbor further boosts the rank-1 recall by 1.6% on Tokyo 24/7. Thesuperior performance demonstrates that k-reciprocal nearest neighbor rankingcan more accurately identify true positive images.Benefits from softmax temperature annealing. We validate the effective-ness of the proposed temperature annealing strategy (Sec. 3.2) in this experimentby setting a constant temperature τ = 0.07 in all generations. Comparisons be-tween “Ours” and “Ours w/o softmax temperature annealing” in Fig. 4 and Tab.2 show that temperature annealing is beneficial for learning informative featureswith our self-enhanced soft supervisions.

4.4 Qualitative Evaluation

To better understand the superior performance of our method on the IBL tasks,we visualize the learned feature maps before VLAD aggregation as heatmaps,

14 Y. Ge et al.

Table 3. Evaluation on standard image retrieval datasets in terms of mAP (%) tovalidate the generalization ability of the networks

MethodOxford 5k [35] Paris 6k [36]

Holidays [20]full crop full crop

NetVLAD [1] (CVPR’16) 69.1 71.6 78.5 79.7 83.1CRN [23] (CVPR’17) 69.2 - - - -SARE [28] (ICCV’19) 71.7 75.5 82.0 81.1 80.7

Ours 73.9 76.7 82.5 82.4 80.5

shown in Fig. 5. In the first example, our method pays more attention on thediscriminative shop signs than SARE, which provide valuable information for lo-calization in city-scale street scenarios. In the second example, SARE incorrectlyfocuses on the trees, while our method learns to ignore such misleading regionsby iteratively refining supervisions from fine-grained image-to-region similarities.Although both methods fail in the third example, our retrieved top-1 image ismore reasonable with wall patterns similar to the query image.

4.5 Generalization on Image Retrieval Datasets

In this experiment, we evaluate the generalization capability of learned featureembeddings on standard image retrieval datasets by directly testing trained mod-els without fine-tuning. The experimental results are listed in Tab. 3. “Full”means the feature embeddings are extracted from the whole image, while onlycropped landmark regions are used in the “crop” setting. Our method showsgood generalization ability on standard image retrieval tasks, and outperformsother competitors on most datasets and test settings. All compared methods donot perform well on Holidays due to the the fact that Holidays contains lots ofnatural sceneries, which are difficult to find in our street-view training set.

5 Conclusion

This paper focuses on the image-based localization (IBL) task which aims toestimate the geographical location of a query image by recognizing its nearestreference images from a city-scale dataset. We propose to tackle the problem ofweak and noisy supervisions by self-supervising image-to-region similarities. Ourmethod outperforms state-of-the-arts on the standard localization benchmarksby noticeable margins.

Acknowledgements

This work is supported in part by SenseTime Group Limited, in part by theGeneral Research Fund through the Research Grants Council of Hong Kongunder Grants CUHK 14202217 / 14203118 / 14205615 / 14207814 / 14213616 /14208417 / 14239816, in part by CUHK Direct Grant.

Self-supervising Fine-grained Region Similarities 15

References

1. Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., Sivic, J.: Netvlad: Cnn ar-chitecture for weakly supervised place recognition. In: Proceedings of the IEEEconference on computer vision and pattern recognition. pp. 5297–5307 (2016)

2. Arandjelovic, R., Zisserman, A.: All about vlad. In: Proceedings of the IEEE con-ference on Computer Vision and Pattern Recognition. pp. 1578–1585 (2013)

3. Arandjelovic, R., Zisserman, A.: Dislocation: Scalable descriptor distinctivenessfor location recognition. In: Asian Conference on Computer Vision. pp. 188–204.Springer (2014)

4. Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsuper-vised learning of visual features. In: Proceedings of the European Conference onComputer Vision. pp. 132–149 (2018)

5. Castle, R., Klein, G., Murray, D.W.: Video-rate localization in multiple maps forwearable augmented reality. In: 2008 12th IEEE International Symposium on Wear-able Computers. pp. 15–22. IEEE (2008)

6. Chen, D.M., Baatz, G., Koser, K., Tsai, S.S., Vedantham, R., Pylvanainen, T.,Roimela, K., Chen, X., Bach, J., Pollefeys, M., et al.: City-scale landmark identi-fication on mobile devices. In: CVPR 2011. pp. 737–744. IEEE (2011)

7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer visionand pattern recognition. pp. 248–255. Ieee (2009)

8. Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learningby context prediction. In: Proceedings of the IEEE International Conference onComputer Vision. pp. 1422–1430 (2015)

9. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for modelfitting with applications to image analysis and automated cartography. Communi-cations of the ACM 24(6), 381–395 (1981)

10. Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., Anandkumar, A.: Born againneural networks. International Conference on Machine Learning (2018)

11. Ge, Y., Chen, D., Li, H.: Mutual mean-teaching: Pseudo label refinery for unsuper-vised domain adaptation on person re-identification. In: International Conferenceon Learning Representations (2020)

12. Ge, Y., Chen, D., Zhu, F., Zhao, R., Li, H.: Self-paced contrastive learning withhybrid memory for domain adaptive object re-id (2020)

13. Ge, Y., Li, Z., Zhao, H., Yin, G., Yi, S., Wang, X., Li, H.: Fd-gan: Pose-guidedfeature distilling gan for robust person re-identification. In: Advances in NeuralInformation Processing Systems. pp. 1229–1240 (2018)

14. Ge, Y., Zhu, F., Zhao, R., Li, H.: Structured domain adaptation with online relationregularization for unsupervised person re-id (2020)

15. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning bypredicting image rotations. arXiv preprint arXiv:1803.07728 (2018)

16. Goyal, P., Mahajan, D., Gupta, A., Misra, I.: Scaling and benchmarking self-supervised visual representation learning. In: Proceedings of the IEEE Interna-tional Conference on Computer Vision. pp. 6391–6400 (2019)

17. Hane, C., Heng, L., Lee, G.H., Fraundorfer, F., Furgale, P., Sattler, T., Pollefeys,M.: 3d visual perception for self-driving cars using a multi-camera system: Calibra-tion, mapping, localization, and obstacle detection. Image and Vision Computing68, 14–27 (2017)

16 Y. Ge et al.

18. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridgeuniversity press (2003)

19. Hongsuck Seo, P., Weyand, T., Sim, J., Han, B.: Cplanet: Enhancing image geolo-calization by combinatorial partitioning of maps. In: Proceedings of the EuropeanConference on Computer Vision. pp. 536–551 (2018)

20. Jegou, H., Douze, M., Schmid, C.: Hamming embedding and weak geometric con-sistency for large scale image search. In: European conference on computer vision.pp. 304–317. Springer (2008)

21. Jegou, H., Douze, M., Schmid, C., Perez, P.: Aggregating local descriptors intoa compact image representation. In: 2010 IEEE computer society conference oncomputer vision and pattern recognition. pp. 3304–3311. IEEE (2010)

22. Jegou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., Schmid, C.: Aggre-gating local image descriptors into compact codes. IEEE transactions on patternanalysis and machine intelligence 34(9), 1704–1716 (2011)

23. Kim, H.J., Dunn, E., Frahm, J.M.: Learned contextual feature reweighting forimage geo-localization. In: 2017 IEEE Conference on Computer Vision and PatternRecognition. pp. 3251–3260. IEEE (2017)

24. Knopp, J., Sivic, J., Pajdla, T.: Avoiding confusing features in place recognition.In: European Conference on Computer Vision. pp. 748–761. Springer (2010)

25. Kolesnikov, A., Zhai, X., Beyer, L.: Revisiting self-supervised visual representationlearning. In: Proceedings of the IEEE conference on Computer Vision and PatternRecognition. pp. 1920–1929 (2019)

26. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444(2015)

27. Liu, L., Li, H., Dai, Y.: Efficient global 2d-3d matching for camera localizationin a large-scale 3d map. In: Proceedings of the IEEE International Conference onComputer Vision. pp. 2372–2381 (2017)

28. Liu, L., Li, H., Dai, Y.: Stochastic attraction-repulsion embedding for large scaleimage localization. In: Proceedings of the IEEE International Conference on Com-puter Vision. pp. 2570–2579 (2019)

29. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Interna-tional journal of computer vision 60(2), 91–110 (2004)

30. Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: a versatile and accuratemonocular slam system. IEEE transactions on robotics 31(5), 1147–1163 (2015)

31. Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solvingjigsaw puzzles. In: European Conference on Computer Vision. pp. 69–84. Springer(2016)

32. Noroozi, M., Pirsiavash, H., Favaro, P.: Representation learning by learning tocount. In: Proceedings of the IEEE International Conference on Computer Vision.pp. 5898–5906 (2017)

33. Paulin, M., Douze, M., Harchaoui, Z., Mairal, J., Perronin, F., Schmid, C.: Localconvolutional features with unsupervised training for image retrieval. In: Proceed-ings of the IEEE international conference on computer vision. pp. 91–99 (2015)

34. Perronnin, F., Liu, Y., Sanchez, J., Poirier, H.: Large-scale image retrieval withcompressed fisher vectors. In: 2010 IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition. pp. 3384–3391. IEEE (2010)

35. Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval withlarge vocabularies and fast spatial matching. In: 2007 IEEE conference on computervision and pattern recognition. pp. 1–8. IEEE (2007)

Self-supervising Fine-grained Region Similarities 17

36. Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Lost in quantization:Improving particular object retrieval in large scale image databases. In: IEEEConference on Computer Vision and Pattern Recognition (2008)

37. Radenovic, F., Tolias, G., Chum, O.: Cnn image retrieval learns from bow: Un-supervised fine-tuning with hard examples. In: European conference on computervision. pp. 3–20. Springer (2016)

38. Sattler, T., Leibe, B., Kobbelt, L.: Fast image-based localization using direct 2d-to-3d matching. In: 2011 International Conference on Computer Vision. pp. 667–674.IEEE (2011)

39. Schindler, G., Brown, M., Szeliski, R.: City-scale location recognition. In: 2007IEEE Conference on Computer Vision and Pattern Recognition. pp. 1–7. IEEE(2007)

40. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scaleimage recognition. arXiv preprint arXiv:1409.1556 (2014)

41. Torii, A., Arandjelovic, R., Sivic, J., Okutomi, M., Pajdla, T.: 24/7 place recog-nition by view synthesis. In: Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition. pp. 1808–1817 (2015)

42. Torii, A., Sivic, J., Pajdla, T., Okutomi, M.: Visual place recognition with repetitivestructures. In: Proceedings of the IEEE conference on computer vision and patternrecognition. pp. 883–890 (2013)

43. Vo, N., Jacobs, N., Hays, J.: Revisiting im2gps in the deep learning era. In: Pro-ceedings of the IEEE International Conference on Computer Vision. pp. 2621–2630(2017)

44. Weyand, T., Kostrikov, I., Philbin, J.: Planet-photo geolocation with convolutionalneural networks. In: European Conference on Computer Vision. pp. 37–55. Springer(2016)

45. Xie, Q., Hovy, E., Luong, M.T., Le, Q.V.: Self-training with noisy student improvesimagenet classification. Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition (2020)

46. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks.In: European conference on computer vision. pp. 818–833. Springer (2014)

47. Zhang, M., Song, G., Zhou, H., Liu, Y.: Discriminability distillation in group rep-resentation learning. In: European Conference on Computer Vision (2020)

48. Zhou, H., Liu, J., Liu, Z., Liu, Y., Wang, X.: Rotate-and-render: Unsupervised pho-torealistic face rotation from single-view images. In: Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition. pp. 5911–5920 (2020)