WandB callback does not record enough information

[lyangfan]

Hi:
I want to get the train log, train loss plot, val loss plot and lr plot using WandB callback. But it seems that I just can get PART of training log.
I just follow the tutorial code:

        callbacks = []
        wandb_cb = WandbCallback()  
        wandb_cb.setup(
            training_config=training_config,
            model_config=model_config,
            project_name=args.wandb_project_name,
        )
        pipeline(train_data=train_ds,
                eval_data=val_ds,
                callbacks=callbacks)

and no online loss plot:

Looking forward to your reply

Yangfan
2023.12.7

[lyangfan]

PLUS: Sometimes I want to monitor the reconstruction loss and KL Divergence separately, not just total loss. So I need to finish my own callback?

[clementchadebec]

Hi @lyangfan ,

Thanks for the question, I think it is due to the fact that the logs are provided to the callback at the end of an epoch. This is indeed suboptimal and it would be better to define a batch_frequency variable instead to log throughout a training step. I will try to improve this soon.

[clementchadebec]

As to the custom metrics you have indeed to provide your own callback following the template in pythae.trainers.training_callbacks since the code does not support custom metric selection for now.

[lyangfan]

As to the custom metrics you have indeed to provide your own callback following the template in pythae.trainers.training_callbacks since the code does not support custom metric selection for now.

Thank you for your patient answer and look forward to your improvement. I will try the solution you give