We evaluated NVIDIA's mapnet model under challenging conditions using the AachenDayNight dataset. We use the percentage formulation that is also used here.
Furthermore, we tested the influence of two augmentation techniques:
- We augmented the dataset by artificial night images generated by a CycleGAN network.
- We forced the model to also learn a semantic segmentation of the input images as additional targets. The semantic segmentation was generated using this repository.
 |
|---|
| Concept of multitask model |
| Artificial night images | Semantic targets | Day-time results | Night-time results |
|---|---|---|---|
| no | no | 0 / 0 / 8.4 | 0 / 0 / 0 |
| yes | no | 0 / 0.1 / 16.1 | 0 / 0 / 4.1 |
| no | yes | 0 / 0.1 / 17.4 | 0 / 0 / 5.1 |
| yes | yes | 0 / 0 / 21.6 | 0 / 0 / 9.1 |
Modifications to NVIDIA repository
Modifications by Leonhard Feiner and Alexander Ziller:
- Support for DeepLoc dataset
- Development of Dual-Input model (additional semantics as input to model)
- Development of Multitask model (additional semantics as output to model)
Modifications by Alexander Ziller:
- Support for AachenDayNight and Cambridge Landmarks dataset
- Including augmentations for AachenDayNight dataset (using CycleGANs)
Setup
Unlike the original implementation we use Python 3.
MapNet uses a Conda environment that makes it easy to install all dependencies.
- Install Anaconda
- Create the
mapnetConda environment:conda env create -f environment_py3.yml. - Activate the environment:
conda activate mapnet_release.
To use a dataset
- Create in data/deepslam_data a directory with the corresponding name e.g. AachenDayNight
- Download data into this directory
- Go to scripts directory and run dataset_mean.py and calc_pose_stats.py (also good to verify data structure is correct)
Available models
The following models are available
- Posenet: Standard visuallocalization network by Alex Kendall
- Mapnet: Base where this repository is forked from
- Mapnet++: Mapnet with additional visual odometry and GPS data
- Multitask: Mapnet with additional semantics as output (learning target)
Training
The executable script is scripts/train.py. See python train.py -h for options.
Example usage:
python train.py --dataset AachenDayNight --model mapnet --config configs/mapnet.ini --learn_beta --learn_gamma
If you have visdom = yes in the config file, you will need to start a Visdom
server for logging the training progress:
python -m visdom.server -env_path=scripts/logs/.
Demo/Inference
The inference script is scripts/eval.py. See python eval.py -h for options. Please store downloaded models in scripts/logs.and go to the scripts folder to run the commands.
Mapnet model weights trained on the Aachen Day Night dataset can be downloaded here
Visual Explanations of Model
We included code to calculate maps as described in the GradCam++ paper.
Example usage:
python show_gradcampp.py --dataset DeepLoc --model multitask --val --n_activation_maps 3 --layer_name layer1,layer2 --config_file configs/uncertainty-criterion.ini --weights logs/DeepLoc__multitask_multitask-new-criterion_learn_beta_learn_gamma_learn_sigma_seed13/epoch_300.pth.tar
This repository is a fork of NVIDIA's mapnet repository. Look there for more background and additional information.
Leonhard Feiner contributed equally to include support for learning semantic labels until commit 7e5c754.
Copyright (C) 2018 NVIDIA Corporation. All rights reserved. Licensed under the CC BY-NC-SA 4.0 license (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode).
The original CVPR 2018 paper can be found at
My master's thesis that discusses those results can be found here
Citation for original MapNet:
@inproceedings{mapnet2018,
title={Geometry-Aware Learning of Maps for Camera Localization},
author={Samarth Brahmbhatt and Jinwei Gu and Kihwan Kim and James Hays and Jan Kautz},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2018}
}