AZP and Zuul on pushes to main / stable-x branches

[felixfontein]

SUMMARY

When releasing community.general 1.3.1 from the stable-1 branch of https://github.com/ansible-collections/community.general/ I noticed some (potential) problems with AZP and Zuul checks.

How I proceeded:

1. I created a commit with the release summary, pushed it (to trigger CI);
2. I ran antsibull-changelog release to create the release commit, and pushed it (to trigger CI for that commit as well);
3. Later I bumped the version in galaxy.yml, commited that, and pushed it as well.

While Shippable runs for the latest commit after a push (i.e. for all the three above commits), the behavior for Zuul (third-party check) and AZP is different:

1. The Zuul checks (third-party checks) do not run at all for commits directly pushed to main or stable-1.
2. When pushing the first two commits directly after each other, AZP was still running for a PR merged shortly before.
   • For both commits, only Shippable run.
   • Once AZP finished for the commit for the merged PR, AZP started for the latest commit in stable-1, which was the second commit I pushed.
   • Later (while AZP still ran for the second commit), I pushed the third commit. Again, only Shippable ran.
   • When AZP finished for the collection release commit, AZP also ran for the version bump commit. If I would have pushed another commit in the meantime, AZP probably would not have run for the version bump commit.

In this case, nothing really bad happened - the third-party check is not that important, and at least AZP ran for the release commit (though not for the commit before it). If I would have pushed the version bump commit earlier though, AZP would have skipped the release commit!

Shippable was behaving the same way as GitHub Actions are behaving, I think.

I have no idea why AZP and Zuul have different behavior than Shippable and GHA. I think the expectance that CI runs for the latest commit that is pushed is widespread, so this is probably something that should be investigated.

CC @gundalow @mattclay (you might have an idea about the AZP behavior).

ISSUE TYPE

• Bug Report

[mattclay]

@felixfontein The behavior you're seeing on Azure Pipelines is due to use of the batch: true setting in the configuration:

https://github.com/ansible-collections/community.general/blob/ddaad1e650a45c83d591a7f5b3ae6a34d9405d63/.azure-pipelines/azure-pipelines.yml#L2

This limits concurrent merge runs for a single branch to one -- and cancels all but the latest pending run for each branch.

Shippable does not have a commit batching feature. If it did, we would have enabled it. During times of heavy usage we often manually cancelled pending runs in the queue, which had a similar effect.

AFAIK Zuul is only configured for pull requests, not merges.

[felixfontein]

@mattclay thanks! The batch: true setting makes sense, it's just a bit annoying when doing the last pushes before a release :) I guess as long as everyone is aware of it, it should be fine.

(If anyone else is interested in this, search for Batching CI runs or batch: true on https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops)

[felixfontein]

It might make sense to also specify tags triggers:

trigger:
  tags:
    include:
    - '*'

Since collections only use tags for releases, this should ensure that CI is run for all releases. This might run CI twice for release commits, but it's probably better than accidentally not running it at all (after all releases are usually seldom compared to commits, at least for collections with a lot of commits where this matters). What do you think?

[mattclay]

@felixfontein What benefit is there to running tests on release tags? Unless someone is going to wait for the tests to finish after tagging, and then take some corrective action upon failure, it seems like there is not much point in having the tests run.

[felixfontein]

@mattclay there's https://github.com/ansible-collections/overview/blob/main/collection_requirements.rst#ci-testing, "All CI tests MUST pass for the commit that releases the collection."

If they fail for the release tag, it's a bit late though :) That's why I'd prefer if they run for the commit before I push the tag. Then I can decide whether to push the tag when everything's green.

[mattclay]

@felixfontein Isn't it enough then to push the commit, and wait for CI to finish before proceeding with the release?

[felixfontein]

@mattclay yep, that's the obvious solution for this.

[Andersson007]

The repo is getting archived, closing, thanks everyone!