RuntimeError: CUDA error: device-side assert triggered

[jS5t3r]

/home/user/.conda/envs/bm/lib/python3.7/site-packages/torch/jit/_trace.py:152: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  if a.grad is not None:
checkpoint_path:  checkpoint/resnet50/128x128_16_0.1_Hair_Color_Wednesday_07_April_2021_23h_21m_26s
Training Epoch: 1 [16/13670]    Loss: 1.5864    Acc: 1.0000     LR: 0.000000
epoch 1 training time consumed: 241.82s, Elapsed time: 0:04:01.824466
/opt/conda/conda-bld/pytorch_1607370128159/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/opt/conda/conda-bld/pytorch_1607370128159/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "train.py", line 289, in <module>
    acc, total_time = eval_training(epoch, total_time)
  File "/home/user/.conda/envs/bm/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "train.py", line 124, in eval_training
    test_loss += loss.item()
RuntimeError: CUDA error: device-side assert triggered