GPUMonitor

[Adel-Moumen]

What does this PR do?

Draft. Do not review. Subject to many changes. The core idea of this PR is to add the possibility to monitor GPU utilization within SpeechBrain.

Inspiration

https://github.com/CompVis/stable-diffusion/blob/21f890f9da3cfbeaba8e2ac3c425ee9e998d5229/main.py#L395
https://github.com/SeanNaren/minGPT/blob/master/mingpt/callback.py#L8
https://github.com/pytorch/torchtitan/blob/main/torchtitan/metrics.py#L32

To add

• clear_cuda_mem: https://github.com/Lightning-AI/pytorch-lightning/blob/b9680a364da4e875b237ec3c03e67a9c32ef475b/src/lightning/fabric/accelerators/cuda.py#L376

Before submitting

• Did you read the contributor guideline?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Does your code adhere to project-specific code style and conventions?

PR review

Reviewer checklist

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified
• Confirm that the changes adhere to compatibility requirements (e.g., Python version, platform)
• Review the self-review checklist to ensure the code is ready for review