RuntimeError: expected scalar type Half but found Float

[Jessejx]

Thanks for your great work. I met this problem when I use resnet as backbone.

Traceback (most recent call last):
File "train.py", line 527, in
train(hyp, opt, device, tb_writer, wandb)
File "train.py", line 293, in train
pred = model(imgs) # forward
File "/home/jx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jx/Workings/flexible-yolov5-main/od/models/model.py", line 67, in forward
out = self.backbone(x)
File "/home/jx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jx/Workings/flexible-yolov5-main/od/models/backbone/resnet.py", line 208, in forward
x2 = self.layer2(x1)
File "/home/jx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/home/jx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jx/Workings/flexible-yolov5-main/od/models/backbone/resnet.py", line 119, in forward
out = self.conv2(out, offset)
File "/home/jx/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jx/anaconda3/lib/python3.8/site-packages/torchvision/ops/deform_conv.py", line 143, in forward
return deform_conv2d(input, offset, self.weight, self.bias, stride=self.stride,
File "/home/jx/anaconda3/lib/python3.8/site-packages/torchvision/ops/deform_conv.py", line 76, in deform_conv2d
return torch.ops.torchvision.deform_conv2d(
RuntimeError: expected scalar type Half but found Float

[Bobo-y]

@Jessejx hi, this problem is caused by resnet with dcn. I have updated the configuration and code set dcn as False. so, you can use resnet without dcn. please update your code.
And I also try to solve this problem so that the network can be trained with DCN,

if __name__ == '__main__': import torch args = {'version': '50', 'dcn': True} x = torch.zeros(2, 3, 640, 640).cuda() net = resnet(pretrained=False, **args).cuda() y = net(x) for u in y: print(u.shape) print(net.out_channels)

only test resnet, there is no error, but when i train total network, the error is same as you. And id have check their weights:

weights of DeformConv2d

weights of norml conv2d

Although they are all float32, we can see that the former should be in half format. But I don't know why. Therefore, don't use the DCN for now. I will solve it when I have time. thanks!

[Bobo-y]

I think this is caused by the mixed precision training of apex, and the DCN support for this is not very good. I train resnet with dcn for DBNet, There was no problem. @Jessejx

[Jessejx]

I think this is caused by the mixed precision training of apex, and the DCN support for this is not very good. I train resnet with dcn for DBNet, There was no problem. @Jessejx

Thanks again.

[Bobo-y]

ultralytics/yolov5#2981

[ratom]

I tried to train today, still this issue is not fixed. I also set half=False, still the issue is not fixed. I got this issue while training, not during eval.

[ratom]

Moreover when I complete the training, then it only gave the bounding box but not the corresponding name of the detected object

[Bobo-y]

@ratom pleasse give me more your configs info, next week i will check it