Builds are not starting in concourse v6.1.0

[shyamz-22]

I think this is related to the issue here: #3682

In one of our pipelines, the job is not started and stuck like you can see in the image below:

I have tried to:

• restart the web node as suggested in the issue
• restart the worker node
• delete the pipeline and set it again

None seems to start the job. The resource the job needs to trigger is a maven resource

Please find the stack dump here:

1. web
2. Worker

Concourse Version: 6.1.0

[jamieklassen]

just to level on terminology -- "triggering" refers to when a job has trigger: true on one of its get steps. a job "triggers successfully" when a pending build appears after a new resource version is discovered via checking. Your screenshot shows a pending build, so is it fair to say that "triggering" is not the problem, but rather "starting" is?

[shyamz-22]

yes, the jobs get triggered when a new resource is available, but does not start.

[jamieklassen]

nice! so the component responsible for starting builds is, unsurprisingly, called the buildstarter. Depending on your log level (ideally set it to debug), concourse should be emitting events related to the try-start-next-pending-build procedure: https://github.com/concourse/concourse/blob/v6.1.0/atc/scheduler/buildstarter.go#L148-L151. Those events will be labeled with the build id, so hopefully you can track them down in your log system (fly builds can surface the real ID for builds -- that #3 in your screenshot is called the build 'name' which is not globally unique). This should give you a bit more observability into what's going on

[shyamz-22]

Thank you for the swift response. I will have a look at it. In the mean while I also see such error below in the web node,

{"timestamp":"2020-06-17T15:20:26.312312151Z","level":"error","source":"atc","message":"atc.scheduler.pipeline.job.failed-to-schedule-job","data":{"error":"schedule job: get build inputs: adopt inputs and pipes: sql: no rows in result set","job":"deploy","pipeline":"x-pipeline","session":"21.708.1"}}

We recently upgraded the concourse from 5.8.0 - 6.1.0, could this be causing some issues related to database migration?

[vito]

Sounds like this issue: #5477 - which was resolved in v6.3.0.

[shyamz-22]

Thank you for your timely response. I will update our team to 6.3.0 and give you feedback if we see the same behavior again.