Did the method of calculating the eval metrics on training(such a Recall etc) change between version 1.0.1 and 1.1.1 ?

[ys201810]

Hi!

When we performed training with RecBole (v1.0.1) and RecBole-TRN (using RecBole v1.1.1) with the same data set(own dataset) and the same algorithm(SINE), the Recall and MRR values of the validation data displayed during training were much higher for RecBole v1.0.1.

Is there any significant change in the calculation method of Recall etc. between v1.0.1 and v1.1.1?
I would be glad to get your advice!

thx!

The config's are below.

v1.0.1

# general
gpu_id: 0
use_gpu: True
seed: 2020
state: INFO
reproducibility: True
data_path: 'dataset/path/'
checkpoint_dir: 'saved/path/'
show_progress: True
save_dataset: True
save_dataloaders: True
# dataloaders_save_path: 'saved/dataloaders.pth'
# dataset_save_path: 'saved/dataset.pth'

# Atomic File Format
field_separator: "\t"
seq_separator: "@" 

# Common Features
USER_ID_FIELD: user_id
ITEM_ID_FIELD: item_id
RATING_FIELD: ~ 
TIME_FIELD: timestamp

load_col:
    inter: [user_id, item_id, timestamp]
    user: [user_id]
    item: [item_id]
unused_col:
    inter: [timestamp]

# Training and evaluation config
epochs: 120
stopping_step: 120
train_batch_size: 512
eval_batch_size: 512
neg_sampling:
    uniform: 1
eval_args:
    group_by: user
    order: TO
    split: {'RS': [0.8,0.1,0.1]}
    mode: pop100  # full
metrics: ['Recall', 'MRR', 'NDCG', 'Hit', 'Precision']
topk: [1, 5, 10, 20, 30]  # 20
valid_metric: Recall@20
metric_decimal_place: 4
eval_step : 10
user_inter_num_interval: '[5,inf)'
item_inter_num_interval: '[0,inf)'

v1.1.1

# general
gpu_id: 0
use_gpu: True
seed: 2020
state: INFO
reproducibility: True
data_path: '../data/path'
checkpoint_dir: '../saved/path'
show_progress: True
save_dataset: True
dataset_save_path: '../saved/path/a'
save_dataloaders: True
dataloaders_save_path: '../saved/path/a'
log_wandb: False
wandb_project: 'recbole'

# training settings
epochs: 160
train_batch_size: 1024
learner: adam
learning_rate: 0.001
neg_sampling:
  uniform: 1
eval_step: 10
stopping_step: 20
clip_grad_norm: ~
# clip_grad_norm:  {'max_norm': 5, 'norm_type': 2}
weight_decay: 0.0
loss_decimal_place: 4
require_pow: False

# evaluation settings
eval_args: 
  split: {'RS':[0.8,0.1,0.1]}
  group_by: user
  order: TO  # RO(ランダム) TO(Time Order)
  mode: pop100  # full
repeatable: False
metrics: ["Recall","MRR","NDCG","Hit","Precision"]
topk: [30]  # [10]
valid_metric: Recall@30 # MRR@10
valid_metric_bigger: True
eval_batch_size: 1024 # 2048
metric_decimal_place: 4

Eval Metrics output

v1.0.1

recall@1 : 0.3289    recall@5 : 0.6206    recall@10 : 0.7316    recall@20 : 0.827    mrr@1 : 0.3908    mrr@5 : 0.5093    mrr@10 : 0.5234    mrr@20 : 0.5291    ndcg@1 : 0.3908    ndcg@5 : 0.5143    ndcg@10 : 0.5526    ndcg@20 : 0.5805    hit@1 : 0.3908    hit@5 : 0.7041    hit@10 : 0.8079    hit@20 : 0.8888    precision@1 : 0.3908    precision@5 : 0.1682    precision@10 : 0.1053    precision@20 : 0.0628
recall@1 : 0.3309    recall@5 : 0.615    recall@10 : 0.7253    recall@20 : 0.8206    mrr@1 : 0.3952    mrr@5 : 0.5105    mrr@10 : 0.5245    mrr@20 : 0.5302    ndcg@1 : 0.3952    ndcg@5 : 0.5129    ndcg@10 : 0.5509    ndcg@20 : 0.5786    hit@1 : 0.3952    hit@5 : 0.6993    hit@10 : 0.8024    hit@20 : 0.8831    precision@1 : 0.3952    precision@5 : 0.1673    precision@10 : 0.1046    precision@20 : 0.0623
15 Jul 17:21    INFO  test result: OrderedDict([('recall@1', 0.3161), ('recall@5', 0.603), ('recall@10', 0.7157), ('recall@20', 0.8116), ('mrr@1', 0.3731), ('mrr@5', 0.4907), ('mrr@10', 0.5054), ('mrr@20', 0.5112), ('ndcg@1', 0.3731), ('ndcg@5', 0.4963), ('ndcg@10', 0.5352), ('ndcg@20', 0.563), ('hit@1', 0.3731), ('hit@5', 0.6859), ('hit@10', 0.7948), ('hit@20', 0.8777), ('precision@1', 0.3731), ('precision@5', 0.1617), ('precision@10', 0.1016), ('precision@20', 0.0607)])
recall@1 : 0.2321    recall@5 : 0.5117    recall@10 : 0.6343    recall@20 : 0.7454    recall@30 : 0.8027    mrr@1 : 0.2737    mrr@5 : 0.3919    mrr@10 : 0.4091    mrr@20 : 0.4163    mrr@30 : 0.4183    ndcg@1 : 0.2737    ndcg@5 : 0.401    ndcg@10 : 0.4438    ndcg@20 : 0.476    ndcg@30 : 0.4904    hit@1 : 0.2737    hit@5 : 0.594    hit@10 : 0.7213    hit@20 : 0.8237    hit@30 : 0.8719    precision@1 : 0.2737    precision@5 : 0.1353    precision@10 : 0.0892    precision@20 : 0.0556    precision@30 : 0.0412

v 1.1.1

recall@30 : 0.2328    mrr@30 : 0.0554    ndcg@30 : 0.0925    hit@30 : 0.2328    precision@30 : 0.0078
recall@30 : 0.2778    mrr@30 : 0.066    ndcg@30 : 0.1103    hit@30 : 0.2778    precision@30 : 0.0093
recall@30 : 0.2954    mrr@30 : 0.0707    ndcg@30 : 0.1178    hit@30 : 0.2954    precision@30 : 0.0098
recall@30 : 0.3052    mrr@30 : 0.0725    ndcg@30 : 0.1213    hit@30 : 0.3052    precision@30 : 0.0102
recall@30 : 0.3089    mrr@30 : 0.0728    ndcg@30 : 0.1224    hit@30 : 0.3089    precision@30 : 0.0103
recall@30 : 0.3111    mrr@30 : 0.0731    ndcg@30 : 0.1231    hit@30 : 0.3111    precision@30 : 0.0104

[ys201810]

I found difference between v1.0.1 and v1.1.1

I used RecBole-TRM and eval_args config overwritten by propaties/quick_start_config/sequential.yml.

propaties/quick_start_config/sequential.yml

eval_args:
  split: {'LS': 'valid_and_test'}
  order: TO
  mode: full
repeatable: True

So eval_args changed , eval metrics change a lots.