Inconsistent evaluation results for same model and dataset

[ppurwar]

Command: python train_net.py --num-gpus 1 --config-file configs/faster_rcnn_V_39_FPN_3x.yaml --resume --eval-only MODEL.WEIGHTS checkpoints/FRCN-V2-39-3x_1/model_final.pth MODEL.ROI_HEADS.SCORE_THRESH_TEST 0.5

The content of config file:
BASE: "Base-RCNN-VoVNet-FPN.yaml"
MODEL:
WEIGHTS: "https://www.dropbox.com/s/q98pypf96rhtd8y/vovnet39_ese_detectron2.pth?dl=1"
MASK_ON: False
VOVNET:
CONV_BODY: "V-39-eSE"
ROI_HEADS:
NUM_CLASSES: 30
SOLVER:
STEPS: (210000, 250000)
MAX_ITER: 200000
IMS_PER_BATCH: 8
BASE_LR: 0.0001
CHECKPOINT_PERIOD: 1000
DATASETS:
TRAIN: ("train_dataset_leafi",)
TEST: ("train_dataset_leafi","val_dataset_leafi")
OUTPUT_DIR: "checkpoints/FRCN-V2-39-3x_crops/"
DATALOADER:
NUM_WORKERS: 4
TEST:
EVAL_PERIOD: 1000

The evaluation results (for validation dataset) running the command:

• Run 1:

[06/17 10:07:33 d2.evaluation.coco_evaluation]: 'val_dataset_leafi' is not registered by register_coco_instances. Therefore trying to convert it to COCO format ...
[06/17 10:07:33 d2.evaluation.evaluator]: Start inference on 66 images
[06/17 10:07:35 d2.evaluation.fast_eval_api]: Evaluate annotation type bbox
[06/17 10:07:35 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 0.01 seconds.
[06/17 10:07:35 d2.evaluation.fast_eval_api]: Accumulating evaluation results...
[06/17 10:07:35 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 0.04 seconds.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.086
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.209
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.052
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.070
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.137
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.093
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.093
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.093
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.104
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.085
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.144
[06/17 10:07:35 d2.evaluation.coco_evaluation]: Evaluation results for bbox:

AP	AP50	AP75	APs	APm	APl	AR1	AR10	AR100	ARs	ARm	ARl
8.572	20.874	5.198	10.182	6.964	13.746	9.268	9.326	9.326	10.449	8.500	14.444

• Run 2:

[06/17 10:11:20 d2.evaluation.coco_evaluation]: 'val_dataset_leafi' is not registered by register_coco_instances. Therefore trying to convert it to COCO format ...
WARNING [06/17 10:11:20 d2.data.datasets.coco]: Using previously cached COCO format annotations at 'checkpoints/FRCN-V2-39-3x_crops/inference/val_dataset_leafi_coco_format.json'. You need to clear the cache file if your dataset has been modified.
[06/17 10:11:20 d2.evaluation.evaluator]: Start inference on 66 images
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.094
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.242
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.043
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.110
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.196
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.116
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.116
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.116
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.142
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.233
[06/17 10:11:22 d2.evaluation.coco_evaluation]: Evaluation results for bbox:

AP	AP50	AP75	APs	APm	APl	AR1	AR10	AR100	ARs	ARm	ARl
9.403	24.244	4.297	0.042	10.984	19.574	11.642	11.642	11.642	0.353	14.227	23.333

This seems to be issue similar to facebookresearch/detectron2#739.