Support for Knowledge Distillation

[thangld201]

Hi, I'm looking for a way to distil from ViT to MobilenetV2, given the checkpoints trained using mmclassification. This would involve using 2 dataloaders & models at the same time with a distillation loss (e.g. KLDivergence). I was looking for a way to initialize dataloader and model separately, without using Runner (since this is not supported), such that model's forward outputs include logits given a batch drawn from dataloader, but haven't figured out how to do so. Does anyone have any ideas ? I'm new to mmclassification.

[thangld201]

@mzr1996 Can you help me ?

[tonysy]

You can refer MMRazor for some distillation example

[mzr1996]

We are planning to introduce some simple knowledge methods to mmcls in the future. But now, you can try to use MMRazor, it's a professional model compression tool for OpenMMLab.