RRT_GLD

RerankingTransformers (RRTs): Experiments on Google Landmarks v2, Revisited Oxford/Paris

Updates

• [04/19/2023] Thank Pavel Šuma for bringing to our attention a bug in dataset.py

  RerankingTransformer/RRT_GLD/utils/data/dataset.py

  Line 57 in c198e7e

  scale_inds[:seq_len] = torch.as_tensor([bisect.bisect_right(self.scales, s) for s in scales[:seq_len]]).long() - 1

  , which caused the number of scales to be reduced from 7 to 4. However, Pavel Šuma's experiments showed that fixing the bug resulted in worse performance. Therefore, we have decided not to make any changes to the code. Once again, we appreciate Pavel Šuma for his valuable contribution.

About

This folder contains the code for training/evaluating RRTs using the pretrained DELG descriptors.

The code is built on top of the metric learning framework provided by @jeromerony.

---

Preparing the descriptors

@@ Please create a separate python virtual environment for this task. @@

Extraction scripts wrap-up

Backbone	Desc. version	Google Landmarks v2	Revisited Oxford5k	Revisited Paris6k
ResNet-50	v1	script	script	script
ResNet-50	v2-clean	script	script	script
ResNet-101	v1	script	script	script
ResNet-101	v2-clean	script	script	script

Install the DELG package

Please follow the instruction to install the DELG library.

All the instructions below assume the DELG package is installed in DELG_ROOT.

Extract the features of Revisited Oxford/Paris

cd $(DELG_ROOT)/delf/python/delg

Please follow the instruction to extract the features of Revisited Oxford/Paris. The table above also provides example scripts to help extract the features. The scripts will not work out-of-the-box, you will still need to set the paths of the input/output directories properly. Please refer to the instruction for more details.

Extract the features of Google Landmarks v2, if you'd like to train the model from scratch

cd $(DELG_ROOT)/delf/python/delg

Download the training set of Google Landmarks v2 from CVDF, or Kaggle. We call the directory of the downloaded dataset as GLDv2_ROOT.

Copy-paste the file of the training split train.txt to GLDv2_ROOT, and the python script extract_features_gld.py to DELG_ROOT/delf/python/delg.

Run the scripts shown in the table above. Again, the scripts may not work out-of-the-box, you may still need to set the paths of the input/output directories properly.

Dataset structure

The code assumes datasets in the structure described below. Note that, we don't need the image data for our experiments.

GLDv2¶

      data/gldv2/
          train.txt/
          delg_r50_gldv1/
          delg_r50_gldv2/
          delg_r101_gldv1/
          delg_r101_gldv2/
      

Revisited Oxford¶

      data/oxford5k/
          test_query.txt/
          test_gallery.txt/
          gnd_roxford5k.pkl/
          delg_r50_gldv1/
          delg_r50_gldv2/
          delg_r101_gldv1/
          delg_r101_gldv2/
      

Revisited Paris¶

      data/paris6k/
          test_query.txt/
          test_gallery.txt/
          gnd_rparis6k.pkl/
          delg_r50_gldv1/
          delg_r50_gldv2/
          delg_r101_gldv1/
          delg_r101_gldv2/
      

Here, the gnd_roxford5k.pkl and gnd_rparis6k.pkl files are already included in the repo, they can also be downloaded from the Revisiting Oxford and Paris page. The test_query.txt and test_gallery.txt files are also included in the repo. They were generated by running (you do need the image data to run this):

python tools/prepare_data.py

---

Experiments

	Backbone	Desc. version	mAP medium / hard	Evaluation	Training	Log
Global retrieval	ResNet-50	v1	ROxf: 69.7 / 45.1 RPar: 81.6 / 63.4	ROxf: script RPar: script	N.A.
Reranking	ResNet-50	v1	ROxf: 75.5 / 56.4 RPar: 82.7 / 68.6	ROxf: script RPar: script	script	log (retrained)
Global retrieval	ResNet-50	v2-clean	ROxf: 73.6 / 51.0 RPar: 85.7 / 71.5	ROxf: script RPar: script	N.A.
Reranking	ResNet-50	v2-clean	ROxf: 78.1 / 60.2 RPar: 86.7 / 75.1	ROxf: script RPar: script	script
Global retrieval	ResNet-101	v1	ROxf: 73.2 / 51.2 RPar: 82.4 / 64.7	ROxf: script RPar: script	N.A.
Reranking	ResNet-101	v1	ROxf: 78.8 / 62.5 RPar: 83.2 / 68.4	ROxf: script RPar: script	script
Global retrieval	ResNet-101	v2-clean	ROxf: 76.3 / 55.6 RPar: 86.6 / 72.4	ROxf: script RPar: script	N.A.
Reranking	ResNet-101	v2-clean	ROxf: 79.9 / 64.1 RPar: 87.6 / 76.1	ROxf: script RPar: script	script	log

---

Evaluation

global retrieval

python tools/prepare_topk_revisited.py with dataset_name=[oxford5k|paris6k] \
    feature_name=[r50_gldv1 | r50_gldv2 | r101_gldv1 | r101_gldv2] \
    gnd_name=[gnd_roxford5k.pkl | gnd_rparis6k.pkl]

Please specify the dataset_name, feature_name, and ground-truth filename gnd_name accordingly.

This command will generate the nearest neighbor file to the dataset folder.

You can also check the specific command included in the table above.

Reranking

python evaluate_revisited.py with model.RRT
    dataset.[roxford|rparis]_[r50|r101]_[gldv1|gldv2] \
    resume=rrt_gld_ckpts/[r50|r101]_[gldv1|gldv2].pt

Please specify the dataset, desc. version, and the checkpoint accordingly.

All the pretrained weights are included in the repo.

Note that reranking requires the nearest neighbor file generated from global retrieval, so please run the global retrieval script first.

You can also check the specific command included in the table above.

---

Training

In order to train RRTs, we need the top100 nearest neighbors for each training image.

You can generate them by running (it may take hours):

python tools/prepare_topk_gldv2.py with feature_name=[r50|r101]_[gldv1|gldv2]

Run the training:

python experiment.py with dataset.gldv2_roxford_[r50|r101]_[gldv1|gldv2] model.RRT max_norm=[0.0|0.1]

The training scripts for the specific models are shown in the table above.