What is actually train loss and train accuracy?

[AaronBlare]

Hi! I wanted to test how the model would work if I use train data as test data.
For this I used your example with MNIST dataset.
I changed the file lightning-hydra-template/src/datamodules/mnist_datamodule.py in the following way:

def test_dataloader(self):
    return DataLoader(
        dataset=self.data_train,   # Here originally was self.data_test
        batch_size=self.hparams.batch_size,
        num_workers=self.hparams.num_workers,
        pin_memory=self.hparams.pin_memory,
        shuffle=False,
    )

In this case, test accuracy is not the same as train accuracy, test loss is not the same as train loss.
Could you please help me explain why is this happening?

[AlexHarn]

Hi! I would not expect them to be the same. I would expect the test metrics to be better. I see two main reasons for differences in test vs train metrics when using the same data:

1. The train metrics are averaged over the epoch while the test metrics are calculated with the weights at the end of the epoch. Since the model is being trained and improving with each batch during an epoch, it is performing better at the end of the epoch. Therefore the average over the training epoch (train metrics) will be worse.

2. Things like dropout and batch normalization behave differently during training and testing. The example net in this repository does not use dropout, but it uses batch norm. The parameters of those batch norm layers are frozen for inference (testing), while they highly depend on the individual batches during training.

For an easy problem with very little data (like the MNIST example), I expect the first reason to be the main factor. For large models running on very large data, the second reason generally has a larger effect (it's even possible to observe "overtraining" of batch normalization parameters to individual batches in some cases).

[AaronBlare]

Thanks a lot!