Disabling Pipeline auto-run from specific git materials #9858
-
Feature proposal / Bug Report?SummaryI need a pipeline that will autorun from one material, but never auto run from a git material. I thought disabling polling from this material would work, but other pipelines 'discover' the commit which triggers the pipeline from this git material. I don't know if this is a bug with disabled polling. The wording seems to indicate it simply wont check for new commits, but to me that sounds like if a new commit was discovered (somehow, i.e. another pipeline was run) and the pipeline has auto_run enabled, then the pipeline will trigger. Consider this example: I want to trigger A1 after Client1 git repo commits automatically, and B2 should flow on automatically. I want B2 to always use the latest commit of PublishScripts. My issue is that when this example is expanded with Client2, A2, B2, ect. Committing to PublishScripts is triggering other pipelines. PublishScripts has polling disabled, as it's a general purpose git repo for maintaining scripts used to publish to our network infrastructure - powershell scripts, ect. Under no circumstances do we want PublishScripts to trigger any pipeline automatically after a commit. Our pipeline should Pipeline B1.. BN should only ever trigger Is there any option I am missing? There is an option for disabling this for pipeline materials (ignore_for_scheduling in yaml) yet not for git materials. |
Beta Was this translation helpful? Give feedback.
Replies: 9 comments
-
|
As to whether it is a bug or not, I don't think I have enough context to comment yet it certainly feels un-intuitive. One thing I came across recently in GoCD's own pipelines is leaving the regular polling turned on, but setting a blanket denylist of I believe the intent, and effect here is to achieve the same thing as you are wanting - so perhaps you can try this as a workaround and see if it gives you the behaviour you expect. As to the root of the problem, is My theory here is that the way materials are de-duplicated across pipelines perhaps is causing the "should I poll this repo?" setting to be read from a different logical material and overridden during de-duplication. When determining uniqueness, really only the material type, url and branch are included by default. By contrast the allow/deny options are not stored with the material itself, as they belong to its connection to a given pipeline. This challenge with determining "uniqueness of materials" is actually the motivation for the gocd-git-path-material-plugin since the triggering inconsistency can cause fan-in issues with conflicting allow/denylists for the same repo used by different pipelines. |
Beta Was this translation helpful? Give feedback.
-
|
On a side note, thanks for the great description of your problem, and what you are trying to achieve. 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick reply!
I might give this a shot and see how it goes after I try reproducing my current issue.
No, all usages are the same in my use case - no client pipelines should ever trigger from PublishScripts at all, and instances of this git material have polling disabled. After playing around just now, I can reproduce the behaviour easily. In my earlier example with Client1, pipelines A1, B1, Client 2, A2, B2... ect...
Although polling is disabled, gocd will still 'discover' this new commit after I manually ran B2. This then schedules all dependent pipelines. I've now added the blacklist filter to all B* pipelines like so: Sequence of events:
So this blacklist approach is a good workaround, however I think this is a good reason to introduce a new parameter (ignore for scheduling?) which can be configured for each material. Ideally all material types and not just git The material 'de-duplication' does seem like it might get in the way though... |
Beta Was this translation helpful? Give feedback.
-
|
OK, that helps me understand. To clarify further, when you earlier said
..did you actually mean committing, and then triggering a manual pipeline run for 1 of the dependent pipelines? Because if polling is disabled on all instances of that material and there are no webhooks, I can't think of a way committing on its own would trigger anything. However, by triggering the manual start you are triggering a "manual check" of the material to find its latest revision. At that point GoCD says "well, this wasn't a scheduled poll, and I have new information about this repo/branch's latest revision, so I should still check which pipelines are dependent on this material and trigger them if their allow/denylists say it should". I am also contemplating how this could be documented/displayed differently to make it "feel" more intuitive. I think part of the challenge is the way things are expressed in YAML can give a different impression to the UI, e.g
|
Beta Was this translation helpful? Give feedback.
-
You are correct, committing did not schedule any pipelines. My second post I've tried to observe the exact behaviour and no commits are scheduling anything.
Yep, this is the behaviour I am seeing. Manual start triggers a manual check for latest material revision, and then all dependant pipelines are scheduled.
From today experience for me this behaves as I'd expect, but the lack of "ignore for scheduling" option per material adds to the confusion as it feels like a fundamental option that should exist. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, possibly, or some better documentation about how denylists should be used to prevent scheduling/triggering, since it has exactly the same effect as desired. If you want to have a look, this approach is used on GoCD itself for its "codesigning" material (can login as guest) on stable releases used to sign artifacts, commits etc and provide proof of their origin/authenticity. This is basically the same use case as yours. If we make changes to the logic/scripts/code for how we sign/produce new releases we don't want to actually trigger and produce a new stable release with all its signed artifacts. However the same material is also used earlier in the pipeline to produce experimental/trial releases on every tested commit, and in these cases we do want it to trigger and validate the signing logic using the latest scripting/code. Denylists are used throughout the "produce stable release" pipelines to prevent these earlier runs with "latest codesigning scripting" triggering a re-run of a stable release.
|
Beta Was this translation helpful? Give feedback.
-
For pipeline materials there appears to be this option which my interpretation is the desired option I'd want on the git material: Perhaps it's worth implementing ignoreForScheduling and having this functionality standardized across all material types. Yaml pipelines will also be less confusing with a common property |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your help. Should I close this? |
Beta Was this translation helpful? Give feedback.
-
|
I converted this to a discussion, and created a summary issue at #9859 to help discovery if others have the same issue. |
Beta Was this translation helpful? Give feedback.
As to whether it is a bug or not, I don't think I have enough context to comment yet it certainly feels un-intuitive.
One thing I came across recently in GoCD's own pipelines is leaving the regular polling turned on, but setting a blanket denylist of
**/*on the git material ofPublishScripts.I believe the intent, and effect here is to achieve the same thing as you are wanting - so perhaps you can try this as a workaround and see if it gives you the behaviour you expect.
As to the root of the problem, is
PublishScriptsas a material used by any other pipelines where you do want it to poll, or where it is configured to do so? Put a different way, if you find the material in theMaterials…