Reintroduce GPU Usage & Efficiency

[thomasvn]

What does this PR change?

• This PR reintroduces GPU Usage/Efficiency to the Allocation API. It also adds GPURequestAverage and GPUUsageAverage to bingen.
• It requires deploying the dcgm-exporter and configuring Prometheus to scrape the exporter. Specifically, this PR leverages the DCGM_FI_DEV_GPU_UTIL metric.

TODO. I think there is some future work required to further validate the usage/querying of the DCGM_FI_DEV_GPU_UTIL metric.

• Works correctly for both controlled and uncontrolled pods?
• Works correctly for a job which only uses the GPU for a short amount of time, then ending in a "Completed" status?
• Works correctly when pods are waiting in the queue to use the GPU?

Does this PR relate to any other PRs?

• Reintroduces GPU Usage/Efficiency (previously Support Allocation GPU utilization/efficiency through integration with Nvidia GPU Operator/DCGM. #944)

How will this PR impact users?

• The next release will include three new fields to the Allocation API. gpuRequestAverage, gpuUsageAverage, and gpuEfficiency.

Does this PR address any GitHub or Zendesk issues?

• Closes Reintroduce gpuRequestAverage and gpuUsageAverage to the Allocation API Schema kubecost/cost-analyzer-helm-chart#1787

How was this PR tested?

Setup

kubectl port-forward svc/prometheus-server 9080:80

rm -rf /tmp/localrun/default
export CLUSTER_ID="cluster-localrun-default"
export CONFIG_PATH="/tmp/localrun/default"
export PROMETHEUS_SERVER_ENDPOINT="http://127.0.0.1:9080"
export KUBERNETES_PORT="helloworld"
export CLOUD_PROVIDER_API_KEY="REDACTED"
mkdir -p $CONFIG_PATH

go run cmd/costmodel/main.go

1. Setup GPU Node + Prometheus
2. Run OpenCost pointed at that Prometheus
3. Deploy a sample workload to request usage of the GPU
4. http://localhost:9003/metrics. Check to see that metrics are updated.
5. http://locahost:9003/allocation?window=1d. Validate my dcgmproftester deployment has new gpuRequestAverage and gpuUsageAverage fields. Example result here allocation.json.
6. http://locahost:9003/allocation/summary?window=1d. Validate GPU fields. Example result here allocationsummary.json.

Does this PR require changes to documentation?

• TODO. Document how to deploy the DCGM Exporter.

Have you labeled this PR and its corresponding Issue as "next release" if it should be part of the next OpenCost release? If not, why not?

• v2.4

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
opencost	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2024 0:30am

[thomasvn]

I added these new fields to the bottom of the struct in accordance with https://github.com/opencost/opencost/blob/develop/core/pkg/opencost/bingen.go.

[thomasvn]

I've noticed that the codecs performs "field version checks". Which leads me to believe that we don't have to add new fields to the end of the struct. If possible, I'd like to group these new fields alongside the other GPU fields.

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud