Tutorial about using Lightning and Fastai callbacks for model checkpoint by metric

[medphisiker]

📓 New <Tutorial/Example>

Is this a request for a tutorial or for an example?
I think it will be very great to see example about using Lightning and Fastai callbacks for model check point by metric.
For example, we train some model like in tutorial https://airctic.com/0.7.0/getting_started_object_detection/.
And we have a great result by COCO metric on 10th epoch, and bad on 25th epoch.
If we use callback by metric we will be able to save checkpoints on great metric values.
I think it will be great to add code example in this tutorial https://airctic.com/0.7.0/getting_started_object_detection/.

I use them in pure Lightning. To do it we need to create special metric loggers in Lightning model class like train_on_epoch() and ect.
But in Ice Vision framework we get predefined class.
Unfortunately, I didn't understand how to write methods for collecting metrics and using callbacks for Ice Vision Lightning Callbacks.
Or Lightning function for finding learning rate. Example with fastai backend show how to do that.

What is the task?
First of all I am intresting in Object detection. But it looks like it is a general opportunity for all ice vision tasks.
As I understand ice vision wokrs with fastai and Lightning backend for all of its CV tasks.

Is this example for a specific model?
No, it is a general example. I think for all models in tutorial https://airctic.com/0.7.0/getting_started_object_detection/

Is this example for a specific dataset?
No.

---

Don't remove
Main issue for examples: #39

[m-ali-awan]

Hi @medphisiker
You can use in fastai as follows:

from icevision.all import *
from fastai.callback.tracker import EarlyStoppingCallback,ReduceLROnPlateau,SaveModelCallback

learn = model_type.fastai.learner(dls=[train_dl, valid_dl], model=model,
                                    metrics=metrics,model_dir = icevision_models_pth)

learn.fine_tune(5, 1e-3, freeze_epochs = 1,
                  cbs=[EarlyStoppingCallback(min_delta = 0.01,patience = 3),
                       ReduceLROnPlateau(factor = 5),
                       SaveModelCallback(fname='only-scratch-with-3tasks-data')])