THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function

[nrose171]

Hello, I am wondering if anyone else has gotten this THCudaCheck Fail error while running this. I am running cuda 12.2v with an NVIDIA RTX A3000 Laptop GPU. Please let me know if you figure anything out.

Here is my full output after running:

07-19 23:21:43.075 train fine cities: ['train/aachen', 'train/bochum', 'train/bremen', 'train/cologne', 'train/darmstadt', 'train/dusseldorf', 'train/erfurt', 'train/hamburg', 'train/hanover', 'train/jena', 'train/krefeld', 'train/monchengladbach', 'train/strasbourg', 'train/stuttgart', 'train/tubingen', 'train/ulm', 'train/weimar', 'train/zurich']
07-19 23:21:43.083 Cityscapes-train: 2975 images
07-19 23:21:43.083 val fine cities: ['val/frankfurt', 'val/lindau', 'val/munster']
07-19 23:21:43.085 Cityscapes-val: 500 images
07-19 23:21:43.085 Using Per Image based weighted loss
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/loss.py:216: UserWarning: NLLLoss2d has been deprecated. Please use NLLLoss instead as a drop-in replacement and see https://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss for more details.
warnings.warn("NLLLoss2d has been deprecated. "
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
warnings.warn(warning.format(ret))
07-19 23:21:43.085 Using Cross Entropy Loss
Creating wide resnet
/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/encoding/nn/syncbn.py:149: EncodingDeprecationWarning: encoding.nn.BatchNorm2d is now deprecated in favor of encoding.nn.SyncBatchNorm.
.format('BatchNorm2d', SyncBatchNorm.name), EncodingDeprecationWarning)
/home/nrose171/Documents/GitHub/GSCNN/network/mynn.py:29: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal.
nn.init.kaiming_normal(module.weight)
07-19 23:21:44.878 Model params = 32.2M
07-19 23:21:44.895 Loading weights from model checkpoints/best_cityscapes_checkpoint.pth
07-19 23:21:45.211 Load Compelete

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function
Traceback (most recent call last):
File "train.py", line 383, in
main()
File "train.py", line 142, in main
optim, default_eval_epoch, writer)
File "train.py", line 302, in validate
seg_out, edge_out = net(input) # output = (1, 19, 713, 713)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/Documents/GitHub/GSCNN/network/gscnn.py", line 272, in forward
m1 = self.mod1(inp)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (8) : invalid device function at /pytorch/aten/src/THC/THCGeneral.cpp:383