How do we find the root cause of "no suitable workers found, waiting for worker..."

[jhohiii]

Question: How do we find the root cause constraint for the message (and corresponding wait) above?

We have a pretty large concourse deployment with ~40 workers.

Our workers did not have their disks full and were nowhere near the number of containers (max was < 80 and most were check, get, and put containers)

container_placement_strategies: [limit-active-containers, limit-active-volumes, volume-locality, fewest-build-containers]
enable_rerun_when_worker_disappears: true
streaming_artifacts_compression: raw
max_active_volumes_per_worker: 200
max_active_containers_per_worker: 200

I checked the stdout/stderr logs on the web nodes and could not find anything pointing to an error.

[taylorsilva]

Sadly the logging in the placement strategies leaves a lot to be desired :(
https://github.com/concourse/concourse/blob/master/atc/worker/placement.go

That said, I would guess that your active volumes per worker is what is limiting you in this case. I would probably make the number of volumes 2-3 times higher than number of containers, as a starting point at least. You usually have a lot more volumes than you do containers.

You can see this in the table output from fly workers, but if you add --json you'll see a lot more details are available, like how many volumes are on each worker!

On the Concourse CI system I can see the workers generally have around 7 times the number of volumes compared to containers:

┌─[master][taylor-macbook][~/workspace/concourse/concourse]
└─▪ fly -t ci workers --json | jq '.[] | {name, active_containers, active_volumes}'
{
  "name": "ci-workers-worker-2",
  "active_containers": 84,
  "active_volumes": 512
}
{
  "name": "windows-worker-ci",
  "active_containers": 2,
  "active_volumes": 26
}
{
  "name": "ci-pr-worker-0",
  "active_containers": 0,
  "active_volumes": 0
}
{
  "name": "topgun-worker-worker-0",
  "active_containers": 0,
  "active_volumes": 0
}
{
  "name": "855c7ffd-a80e-4224-90fa-e4bc74e26ed4",
  "active_containers": 1,
  "active_volumes": 186
}
{
  "name": "ci-workers-worker-4",
  "active_containers": 84,
  "active_volumes": 480
}
{
  "name": "administrators-Mac-mini.local",
  "active_containers": 1,
  "active_volumes": 20
}
{
  "name": "ci-workers-worker-0",
  "active_containers": 64,
  "active_volumes": 368
}
{
  "name": "ci-workers-worker-3",
  "active_containers": 90,
  "active_volumes": 547
}
{
  "name": "ci-workers-worker-1",
  "active_containers": 68,
  "active_volumes": 488
}