💡[feat] Track each task GPU utilization (and other information) and display it on the WebUI

[tshu-w]

Describe the problem

It would be very convenient for cluster managers and users if the WebUI could display system information for each task.

Here are screenshots from wandb:

Describe the solution you'd like

Users can view these chart on WebUI

Describe alternatives you've considered

No response

Additional context

cc @luyaojie

[ioga]

Hello, please check out our profiling feature:

• https://docs.determined.ai/latest/reference/reference-training/experiment-config-reference.html#profiling
• https://docs.determined.ai/latest/training/dtrain-introduction.html#how-to-profiling

[tshu-w]

Hi @ioga, If I understand correctly, do the system metrics and profiling only work with experiments that use PyTorchTrial or other wrappers? To help our colleagues start quickly, we currently run tasks mainly through commands or notebook. Do the SYSTEM METRICS be tracked in this case? I cannot find it on the WebUI.

[ioga]

@tshu-w correct, today it needs to be a PyTorchTrial or TFKerasTrial. There's no metrics collections for commands or notebooks, but I can totally see this being useful as a future feature.

[vishnu2kmohan]

@tshu-w Note: You can setup Prometheus and Grafana to monitor the usage of all Determined workloads: https://docs.determined.ai/latest/integrations/prometheus/prometheus.html

[tshu-w]

Thanks, @vishnu2kmohan, is it possible to monitor each GPU and know which task is running on it intuitively with Prometheus and Grafana? Our intent is to monitor GPU utilization per task to ensure that resources are not wasted.

[huangfuhuijie]

Thanks, @vishnu2kmohan, is it possible to monitor each GPU and know which task is running on it intuitively with Prometheus and Grafana? Our intent is to monitor GPU utilization per task to ensure that resources are not wasted.

Hi, I also encountered this situation. So have you figured out is it works to use Prometheus and Grafana to monitor GPU utilization on every task?

[azhou-determined]

See https://docs.determined.ai/latest/integrations/prometheus/prometheus.html.

We provide a pre-configured Grafana panel for monitoring hardware metrics including GPU utilization. We currently only provide preset filters for tags and resource-pool.

However, we do surface various container/task mappings through a Prometheus API endpoint (prom/det-state-metrics), so with some Grafana/PromQL fiddling, it's possible to modify our Grafana panel to add custom queries/filters (on task ID for example).