Managing Sequential Job Dependencies and Scheduling in Concourse Pipeline

[srbhklkrn]

I am configuring a pipeline with three interdependent jobs that need to process sequentially and use the latest Git commit made by the previous job. Additionally, I want to trigger this chain of jobs on a schedule. Here’s the specific challenge I'm facing:

Job A processes and commits changes, moving the Git HEAD from commit 101 to 102.
Job B and Job C are dependent on Job A and should ideally use the latest commit (102) made by Job A. However, they currently use the commit which was HEAD when Job A started (commit 101).
I want Job B and Job C to always use the latest commit from Job A, and I need all jobs to run automatically in a sequence based on a scheduled trigger.

Here’s the basic structure of my pipeline with the scheduled trigger included:

resources:
- name: every-30-mins
  type: time
  source:
    interval: 30m

jobs:
- name: job-a
  plan:
  - get: every-30-mins
    trigger: true
  - get: repo
  - task: make-changes
    file: repo/ci/tasks/make-changes.yml

- name: job-b
  plan:
  - get: repo
    passed: [job-a]
  - task: use-changes
    file: repo/ci/tasks/use-changes.yml

- name: job-c
  plan:
  - get: repo
    passed: [job-b]
  - task: deploy-changes
    file: repo/ci/tasks/deploy-changes.yml

Is there a way to adjust this configuration to ensure that Job B and Job C always use the commit created by Job A and that the entire job chain triggers automatically according to a schedule? Any suggestions or examples would be greatly appreciated.

Thank you!

[taylorsilva]

I think this will work if you use the put step to push whatever commit is made by your task.

resources:
  - name: every-30-mins
    type: time
    source:
      interval: 30m

jobs:
  - name: job-a
    plan:
    - get: every-30-mins
      trigger: true
    - get: repo
    - task: make-changes
      file: repo/ci/tasks/make-changes.yml
+   - put: repo
+     params:
+       repository: repo

  - name: job-b
    plan:
    - get: repo
      passed: [job-a]
    - task: use-changes
      file: repo/ci/tasks/use-changes.yml

  - name: job-c
    plan:
    - get: repo
      passed: [job-b]
    - task: deploy-changes
      file: repo/ci/tasks/deploy-changes.yml

Then Concourse will correctly understand the constraints between your jobs. Pushing the commit from your task results in Concourse not knowing where this new commit came from. Using the put step Concourse will then be aware of this new commit and pass it along to the next job(s) in your pipeline.

Your make-changes task should also specify repo as an input and output of the task for this to work. All that task should do is create the commit. Do not push the commit in this task. Use the put step to push the commit back to your remote git repo.

[srbhklkrn]

@taylorsilva thanks for replying, actually I push changes using my script it's part of the script that these jobs are running. Adding put requires a significant refactoring to my job script.

[taylorsilva]

Understood. Sadly using the put step is the only way to make the commit pass correctly from one job to the next 😞

[srbhklkrn]

That's fine, I will work on refactoring. Another question related to this: if the pipeline is triggered on schedule and all jobs are linked to each other as shown in the example, will all the jobs in the pipeline run one after another in sequence?