way to get resource consumption of individual jobs

[gowrisankar22]

Hello All,

Is there any way to get the resource utilization(CPU, memory) of individual jobs? Whether concourse emits this metrics?

BRs, Gowrisankar

[evanchaoli]

@vito This is also needed by us, as we talked in last week's meeting.

[jamieklassen]

I've been thinking about this a bit this morning, and I'm particularly interested in finding solutions that require no code/configuration changes to Concourse itself.

https://docs.docker.com/config/containers/runmetrics/#metrics-from-cgroups-memory-cpu-block-io contains some good discussion on getting things like CPU and memory consumption per-container in docker. It turns out that garden creates cgroupfs mounts in a structured way very similar to docker, at paths like /sys/fs/cgroup/memory/garden/<container handle>/memory.stat. If you wanted, you could periodically probe these files and report them to your timeseries with something like the telegraf cgroup plugin -- but your data will be labelled with raw garden container handles, which is a bit clunky.

Let's suppose I want to see how many CPU shares are in use by the pipeline concourse on the team main. I could start by finding all the containers scoped to that pipeline with

% fly -t ci curl '/api/v1/teams/main/containers?pipeline_name=concourse' -- -s | jq -r '.[] | "\(.worker_name)\t\(.id)"' | sort -k1

where I have also sorted the containers by worker, to get output a bit like

c184f9d5-a073-4de7-8ded-e2b78055cc82	423ba054-59cf-41d3-7c2c-716136901944
ci-worker-0	1763b972-46e7-47d5-4202-d2f44c1fa2bc
ci-worker-0	204a61f8-0c8d-4937-439f-c1202f4c6f19
ci-worker-0	3c066742-3b4c-4766-64e3-22ddfb2cab30
ci-worker-0	5a675006-d920-40cc-4e56-2af087a4c6ae
ci-worker-0	c1953e49-4fcd-4cbf-5e58-1acd8177174a
ci-worker-0	f8e44787-9436-4c62-5f5c-1bba18eac2f6
ci-worker-1	bf7276ac-ea1a-4550-4d53-4e73e87a476a
ci-worker-2	aa59b3b7-2b90-47a6-4744-dd2901753915
ci-worker-2	c901b472-9029-48e4-6e81-fe6037e34b54
ci-worker-3	b12d5d25-3640-45d2-479a-4264cb401d86
ci-worker-4	936ee011-7e6b-4de0-5ce3-3d21238e4232
ci-worker-5	2abb9669-8278-4572-7e02-a4b7674427a5
ci-worker-5	4ed2d351-d9ce-4cf2-43ec-2156085a5c33
ci-worker-5	aaa5ff4d-f7e6-4ce9-4226-4d36f6871b1a
ci-worker-5	c4828a9b-1449-42fe-565e-9402ed0131b3
ci-worker-5	e069293a-618d-4254-5cd4-19dcf5cb73b8
ci-worker-6	8cd78b4a-0cb2-4f3f-7427-e3bd9e70549a
ci-worker-6	fc46ae84-0e64-4835-5548-1b5600d32c5c
ci-worker-7	4a73f82f-a449-43c0-40ca-8f0daac99c54
ci-worker-7	83bc2067-b8a9-42dd-4c32-ddf3ead9f585
ci-worker-7	903f20e8-23e9-4729-74c7-cc70cecf5f1c
ci-worker-7	9ad8ac11-cf35-4d0c-57aa-c10eba536a97
darwin-worker	f0b071f5-91c0-4c51-5a5a-181c3aed7036

I could then read the resource consumption a few different ways:

• I could use the bulk metrics endpoint on the garden server on each worker, i.e. /containers/bulk_metrics?handles=8cd78b4a-0cb2-4f3f-7427-e3bd9e70549a,fc46ae84-0e64-4835-5548-1b5600d32c5c and add up all the results.
• I could execute a bunch of /var/gdn/assets/runc events --stats <container handle> commands on each worker (this is the crux of guardian's implementation anyway)
• I could read each of the /sys/fs/cgroup/cpuacct/garden/<container handle>/cpuacct.stat files on each worker, which is probably the crux of how runc events --stats works.

Something that feels reasonable would be to follow the example of something like kube-state-metrics or flight_recorder and create a configurable prometheus exporter that can do this basic joining/munging/aggregating. Maybe it could expose a prometheus endpoint with aggregate CPU data per-pipeline (and maybe aggregate CPU data for all check containers as well?):

cpu_shares[team="main",pipeline="concourse"]	109337212
cpu_shares_checks	9877237718

Maybe it would be deployed as a Deployment in the same namespace as concourse, and therefore be able to use the forwarded addresses of workers. Then it could serve prometheus data by making a series of RPC calls:

1. get all the workers with a ListWorkers request to the web service (like this), and remember the name->garden address mapping.
2. get the grand list of containers -- potentially by making a super-admin ListContainers request to the web service (like with this API), and grouping them appropriately. This could also be done in the fashion of flight_recorder by reaching directly into the database.
3. Once for each worker, make a forwarded /containers/bulk_metrics request to get all metrics for all containers. Use an api like this.
4. Join the above two datasets together on their container ID, and bucket the CPU by team/pipeline with one extra bucket for check containers.

[jamieklassen]

The ideas I just presented don't work particularly well for collecting resource consumption down to individual builds, because that is potentially very high-cardinality data. That data is so specific and abundant that it probably makes more sense to model as a series of events, e.g. that might be sent to a logging system.

[jamieklassen]

Something I could imagine is, when distributed tracing is enabled, for an agent process on the worker to periodically collect cgroup data from all the running garden containers and add events to the span context specified by the container's TRACEPARENT env var.