Feedback thread for set_pipeline: self

[vito]

This thread is to gather feedback for the use of set_pipeline: self.

(I'll circle back here and add some example questions/prompts, this is just a placeholder for now so we can link to it from the PR that adds it.)

[fnaranjo-vmw]

I find this functionality very useful.
Currently, I'm using it to implement self-updating pipelines (keeping the pipeline automatically in sync with the repo).

It was pretty straightforward to implement thanks to this step.
I created a small demo for a self-updating pipeline here: https://github.com/fejnartal/concourse-selfupdate-pipeline

[username-is-already-taken2]

Hi there,
Thank you for this, I've been using this to help keep things updated, especially when I (or others) don't have fly to hand.

I was using set_pipeline: self for a while, I did manage to shot myself in the foot a couple of times which resulted in the pipeline deleting itself (you did say it was experimental :) I think this could be down to the fact that the update job changed? either way I've settles with a model of managing pipeline update jobs as a separate pipeline.

Thanks for your help, really liked this feature.

[RikuXan]

I have been using set_pipeline: self for the same use case as the other commenters, however I'm still facing one issue with it: When actually updating the pipeline in my repository, the pipeline starts running, updates itself as intended, but then finishes its run with the old pipeline definition.

While this is very understandable - updating the contents of a running pipeline would probably do more harm than good - I would like to inquire whether there is a possibility of "resetting" such a build, or cancelling it and triggering a re-run with the same inputs.

[evanchaoli]

While this is very understandable - updating the contents of a running pipeline would probably do more harm than good - I would like to inquire whether there is a possibility of "resetting" such a build, or cancelling it and triggering a re-run with the same inputs.

Our pipelines are widely using set_pipeline: self which is super convenient especially when we use a template to roll out a lot of pipelines.

In our practice, we use set_pipeline: self in a separate job, the job is very simple: get a git resource, then set_pipeline self. With this logic, I guess it doesn't have the problem you mentioned (job ran with old pipeline definition), because pipeline code is standalone from product code, and set_pipeline is in a separate job.

Triggered a re-run with the same inputs after set_pipeline: self is an interesting topic. I guess it's technically feasible, but it may also have pros and cons. We may investigate more on that.

[pete-om]

I use this on the couple of pipelines that I manage and I love that it takes away some of the uncertainty around maintaining them. There's no longer any possibility of checking in a definition change to a repo where the pipeline accidentally wasn't updated, likewise pipeline changes don't get made without the definitions being checked in (nor without explanation via commit messages).

It's now just: if you want to change a pipeline, change the definition and check it in. One source of truth, etc. And everyone gets visibility of it in the repo

[siennathesane]

I very much like this feature and find it useful, so I would like to keep it around for sure. ❤️

[spire-allyjweir]

This has been a game-changer for reducing the maintainance burden when it comes to updating pipelines. I've also been able to implement some orchestration pipelines to configure downstream pipelines which has been good. Thank you very much! ❤️

Something that I use when using fly set-pipeline is the --check-creds flag. It is a nice way to ensure any changes regarding credential variables are correct.

Would it be possible to extend the set_pipeline step to allow this flag to be optionally added?

[ranger6]

Plus 1. I've been working on the oneflow release process and set_pipeline fits well: when a pipeline is triggered by a new release manifest being published, it creates the release branch and then set's itself to complete the release process. So, the pattern is sort of "fork/exec" or maybe even closer to a tail recursive function. Used as part of such controlled and well known idioms, set_pipeline is clean and powerful.

[jwglista-dsg]

Just wanted to say that we use this on all my group's pipelines at our organization and I agree with others that this is a game-changer. Please do not remove this feature because it will send us back to the stone ages where conflicting pipeline overwrites were commonplace.

[lrstanley]

@vito or others, could we possibly get an idea of how "concrete" this experimental feature is? It's still widely warned in the UI.

I'm not looking for exacts, but more of a "we don't expect it to be removed, but maybe changed" or "we have plans to deprecate due to X potential feature", etc. I also understand that things can change and that there are no promises, but I'd like to find out roughly where things are heading..

Reason I ask is, although this is experimental, the benefit of using set_pipeline: self (among other functionality), will help us tremendously, since we have over ~3800 pipelines with over ~16000 total jobs, Getting everyone to stay up to date, consistent, with little drift, has been VERY hard over the past few years. We may also be looking to add another 2000-3000 pipelines over the next year due to some internal migrations.

[taylorsilva]

We're probably gonna keep it. There doesn't seem to be any foot-gun'ing going on so I personally don't see any reason to remove it.

[Etone]

I really like this feature, it helps keeping the manual work at a minimum. One thig that I stepped into which is not documented (but clear for most users i think) is the fact, that the set_pipeline: self step should be in its own job at the begining of the pipeline. Or more clear, that only jobs AFTER the set_pipeline: self step are updated. Not much of a problem but I walked right into it by only updating the pipeline in the beginning of a job and then wondering why the job itself did not update.

[mpb10]

As others have said, this feature is truly a game changer.

We have a git resource that watches the pipeline configuration file and pipeline variable files for commits. Upon a new commit to any of them, the pipeline immediately updates itself. This also allows us to have very dynamic, rapidly changing pipelines when paired with Jinja2 templating for the pipeline configuration file. Upon new commits to the pipeline's files, the pipeline configuration file is templated using Jinja2 in a task, and then it is passed to the set_pipeline: self step as an input.

[Bengreen]

Would like to find out if it is possible to use this feature and optionally mark a set_pipeline task so that it will not be culled when it is not updated.
The reason I would like this feature is that we use a scanner resource across all of our git repos that checks for updates to a concourse pipeline config file in any repo. If a repo adds a new concourse config file then this scanner will spawn a new pipeline for that repo. BUT if the old pipelines are not refreshed in the same job then they are culled off.

Thus we have a single scanner for each team to scan all repos in our organisation for a 'desire' to create a pipeline.

once created each repo will then maintain their own pipelines using set_pipeline (in this part fo the use-case the set_pipeline and culling works well).

[Bengreen]

ie my request is an option flag to disable the auto delete of the pipeline that has been generated with set_pipeline

[tomasgareau]

We've been experimenting with the parent pipeline --> child pipeline pattern described by @taylorsilva in this blog post and have been loving it. It was particularly useful while we experimented with how we were hosting Concourse since all we had to do was set a single pipeline and, like a phoenix from the ashes, our entire configuration would get re-created in a new Concourse cluster.

However, if both parent & child are triggered by the same resource (e.g., some git repo) then the child can kick off a set of builds and might get updated halfway through. What I'd want is for the child pipeline to first get updated, THEN for it to run.

We've managed to address this by adding a set-pipeline: self job as the first job in the child but this feels a bit duplicated (since both parent & child are now watching the same git repo to run the same set-pipeline job). Curious to see if anyone knows of any alternatives here.

[multimac]

We use this pattern too and it works to avoid the child pipeline running with out-of-date configuration.

Though there seems to be some check in place to avoid two set-pipeline steps from updating the same pipeline, which causes our child pipelines to occasionally error with pipeline set by a newer build.

Not sure how we work around that error. Would using an attempts modifier help, allowing it to retry the set-pipeline step?

[marcinkubica]

Using this in every pipeline. Please don't remove.

[rdrgmnzs]

Use the feature extensively, however I feel like there needs to be something similar to the --dry-run flag in order to let users verify the changes they are making before they land a change. This is specially needed for situations where you use tons of rendered pipelines.

[blchelle]

Agreed, a dry-run flag would be extremely useful for validating changes.

[ericgravelle]

Loving this feature. In my use case, I have 50+ pipelines that I have to manage in an "Phoenix" style environment where Concourse needs bootstrapping of the current pipelines and takes over the commissioning process of an environment. Also using this functionality to treat the pipelines as attached to specific feature commits. Incredibly useful feature.

[pratikbin]

Also using this functionality to treat the pipelines as attached to specific feature commits. Incredibly useful feature.

@ericgravelle If you can a little more regarding this particular use case that would be great, I tried but couldn't figure it out.

[tomasgareau]

Here's a resource that's useful for this use-case (creating a pipeline per PR): https://github.com/aoldershaw/github-pr-resource#example

We've been using the pattern shown in the example there to run unit tests on PRs and it's worked mint for us so far!

[Anonymous-Coward]

It should get a parameter to tell whether the pipeline should be created/updated paused, and there should be another step that should allow pausing/unpausing. Reason: sometimes you don't want to just create a pipeline from a template, you want to create a pipeline and set up things around the pipeline so the new pipeline can run. If you create pipelines unpaused, they start running right away, before you can create what else they need to run properly. If you do the setup first, on update, the previous versions of the pipelines start running before you are able to create the new versions.

[cleve-fauna]

Please do not remove this feature it is great!

[mmlanger]

Essential for me! Enabled me to be not dependent on GitHub/Gitea/GitLab/ForgeJo Actions anymore!
Also brings me very near for a GitOps workflow for my projects.

[ianatarewe]

How do people use this? Do you have to manually re-run the task? What if you don't grant users more than viewer privileges (to enforce configuration as code)? I'm not sure how to make it so when I run it against an SCM revision it deterministically brings it into the requested state.

If I make any changes to the pipeline they do not actually occur until the second revision after that change, even if I put it in a separate job and mark them both serial.

I'm starting to think its best not to update pipelines in pipelines at all because of how poor the execution model supports it, I'm thinking maybe have a pipeline trigger a remote bazel job that updates the pipelines, that way no pipeline is self updating except for the bootstrap pipeline itself which does not execute anything after updating itself.

Maybe another workaround is to make another resource that just acts like a cron job to subvert concourse's usual execution model so that it's eventually consistent.

[ianatarewe]

Figured it out by reading: https://blog.concourse-ci.org/2020/08/24/gitops-for-pipelines.html

It seems the one way to actually re-run a pipeline on the new version is by using the "passed" semantic.

Can we get some discussion of this on the docs for set_pipeline and/or the docs for get? It doesn't seem obvious.