RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

[nyanmn]

Using resnet50_4x16f_baseline.yaml and resnet101_8x8f_baseline.yaml, I have the following error as

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2, 1024]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

That error happened at /home/itc/anaconda3/envs/alphaction/lib/python3.7/site-packages/torch/autograd/__init__.py(156)

Variable._execution_engine.run_backward(
        tensors, grad_tensors_, retain_graph, create_graph, inputs,
        allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag

How to solve error?

My libs are

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.10.1
[pip3] torchvision==0.11.2
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.21.2           py37h20f2e39_0  
[conda] numpy-base                1.21.2           py37h79a1101_0  
[conda] pytorch                   1.10.1          py3.7_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchvision               0.11.2               py37_cu102    pytorch

[yelantf]

please try rebuilding the whole environment with earlier version Pytorch.

conda install pytorch torchvision cudatoolkit=$cuda_version -c pytorch=1.4.0

Because this project was not fully tested on later versions.

[nyanmn]

Yes it works with pytorch==1.4. Video needs good resolution and it works well.