Multi-class transfer learning

[bw4sz]

Alot of users have questions about how to implement a multi-class model starting from the baseline. This is a good spot to discuss. The key first thing to understand is that the single class model needs to have the last layers removed, which correspond only to 'Tree'. From there we can retrain specific sets of classes. See the example here:

https://deepforest.readthedocs.io/en/latest/getting_started.html#multi-class-models

@aleksandraradecka1 let's discuss here, so that others may benefit. I'll try to turn this into a video tutorial if it progresses that far.

[aleksandraradecka1]

Thank you @bw4sz for opening a discussion on this topic.

I in fact managed to successfully implement the multi-class model with my data. I used the code you provided in the link above.

For me the problem is a bit different:
I would like to understood what happens underneath these provided functions (I am not fully proficient in PyTorch to be sure I read the source code correctly).
More precisely, I would like to understand which parts of both single and multi-class models are being frozen and which are being retrained using my external data (#325).

[ethanwhite]

In the approach described in the docs we start with both the pre-trained backbone and pre-trained regression head. These are the parts of the model that are specific to tree detection. We use the default (untrained) classification head, so that the model can learn a different number of classes.

Breaking down the code from the docs:

1. Create a new model with the appropriate number of classes for your use case

m = main.deepforest(num_classes=2, label_dict={"Alive":0,"Dead":0})

2. Get a copy of the pretrained model so that we can use parts of it in our new model

deepforest_release_model = main.deepforest()
deepforest_release_model.use_release()

3. Replace the backbone of our new model with the pretrained backbone.

m.model.backbone.load_state_dict(deepforest_release_model.model.backbone.state_dict())

4. Replace the regression head of the new model with the pretrained regression head.

m.model.head.regression_head.load_state_dict(deepforest_release_model.model.head.regression_head.state_dict())

Nothing is frozen in this process, so when we train the new model (m) everything will be trained. In our use cases we consider this good because it allows the model to adjust the features used by the classification and regression heads to match the system being trained on. It takes longer to train, but we expect better results.

[ethanwhite]

1. It is the same for both the single-class and multi-class models. However, we're rechecking a detail on the defaults and having looked a little more carefully I think that actually the first two layers of the backbone are frozen upstream from us (see trainable_backbone_layers in the Retinanet50 docs). So I think that for both cases the first two of the five backbone layers are frozen, so retraining is happening on the last 3 layers of the backbone and the regression head (and the classification head for the single-class model). In other words, all of the layers that we trained in the release model are being trained, but not all of the layers in the underlying Retinanet. We'll get this documented so it's easier for all of us to remember.
2. Yes, there are definitely cases like the one you're describing where freezing deeper layers or the whole backbone can be beneficial. To do this you need to set requires_grad to False to prevent the relevant layers from being trained. In this case if you wanted to freeze the backbone you would do something like:

m.model.backbone.requires_grad_(False)

If freezing the backbone you may also want to freeze the feature pyramid network

m.model.fpn.requires_grad_(False)

but I'd want to hear @bw4sz's thoughts on that.

[aleksandraradecka1]

Thanks again for the clear explanation. If @bw4sz agreed with us upon this being an interesting path to take, I would be happy to do some quick experiments and share what has changed.

[jyo94]

When we labeling both alive and dead as zero, then how it is a multiclass problem? should we change the same in the annotation file?

As in my case, I have 2 labels (cassava and weed). In my annotation file, there are these two labels, before model building should I change it as numeric labels ? Should I label both as zero?

[aleksandraradecka1]

Hi,

what you should do is as follows:

1. create the model:
   m = main.deepforest(num_classes=2, label_dict={"cassava":0,"weed":1})

2. use words cassava and weed in you reference .csv file (in the "label" column)

[jyo94]

Hi,
Thanks for your response.
Yes I have given in the similar fashion, but I'm getting sanity check error. Not sure of what the issue is
More details of the issue: #571 (comment)