Train with pretrained model after changing backbone in_channels

[Jaisme]

I am training my custom dataset (multispectral images with 5 channels) with the fasterrcnn and backbone resnet 50. What i did to adapt the model to my dataset is simply changed the in_channels of the backbone to 5. I'm glad it worked without changing any code, but meanwhile i'm just wondering why it can also work with a pretrained model with in_channels=3, if i changed the channels of the first layer of backbone?

Thanks in advance for any explanation!

[BIGWangYuDong]

If you simply change your in_channels from 3 to 5, this will not occur a running error but some potential problems may occur:

• The pre-trained backbone is trained by RGB-image from ImageNet, if you simply change one of the layers, this may affect the output of the subsequent layers, which may get a lower accuracy.
• It is default to frozen the first two layers while training the detector, if you change the first layer, this also may occur a lower accuracy.

[ZwwWayne]

When changing in_channels from 3 to 5, the model's first layer is changed to have 5 channels to accept the input. This does not match the pre-trained model, so the pre-trained weight of the first layer will not be successfully loaded. This will cause warnings in the log as you will find mismatch warning, but the model could still train/test. However, you will meet the issue mentioned by @BIGWangYuDong and the accuracy may be low.

[Jaisme]

Is there a better way to improve accuracy of using 5-channel images? E.g.

• unfreezing the first two layers while tranining the detector
• pretrain the changed first layer (can i do that with mmdetection?)
• or anything else so that i can still use the pretrained model?

Any suggestion would be really appreciated!

[ZwwWayne]

1. You can use the pretrained detector, only the input layer will not be loaded but the rest of the weights will be loaded normally and should improve the performance.
2. Then you can try some new data augmentations and training longer.