Automating the Github Actions workflow provisioning in the provider repositories

[ringods]

Pulumi has a lot of the Github Actions Workflow management automated in repo ci-mgmt. Their process is as follows:

• generate Typescript files from the Github Actions JSON Schema file
• write all the workflows & steps they need in Typescript
• generate the workflows, Makefile & other standard files in the ci-mgmt repo
• manually trigger a workflow to copy over the generated files from the ci-mgmt repo to the provider repo
• create a pull request in the provider repo when the new workflow files changed

Pulumi has a central providers team, hence they implemented a push based system from ci-mgmt to all the providers under their control. Pulumiverse is different in that we have separate maintainers for each of the providers so we decided to implement a pull based setup.

This PR implements the generation process for the following files:

• Makefile: standardized build file for a bridged provider
• .github/workflows/main.yml: a workflow run when a new commit is merged to the main branch
• .github/workflows/run-regenerate-workflows.yml: a workflow which can be manually triggered to request regeneration of the workflows.

Maintainers should be able to choose themselves when they want to regenerate the workflows in their provider after we publish changes to the standard setup. Our process will be different from Pulumi's one.

Initial setup

The first time we need to set up the workflows, we check out the infra and pulumi-<providername> repositories side-by-side, then execute the following commands:

$ cd infra/ci-mgmt
infra/ci-mgmt$ npm ci
infra/ci-mgmt$ npm run gen -- --name <providername> --path ../.. --debug

> @pulumiverse/[email protected] gen
> ts-node cmd/generate.ts "--name" "<providername>" "--path" "../.." "--debug"

Writing ../../pulumi-<providername>/.github/workflows/run-regenerate-workflows.yml
Writing ../../pulumi-<providername>/.github/workflows/main.yml
Writing ../../pulumi-<providername>/Makefile

Create a PR for these changed files and manually process these. After this initial setup, the run-regenerate-workflows.yml workflow file is available in the repo.

Subsequent regeneration by maintainers

Any subsequent action can be triggered via the Github CLI:

$ cd pulumi-<providername>
pulumi-<providername>$ gh workflow run run-regenerate-workflows.yml

This should trigger the workflow on Github Actions. The workflow will clone both repositories, run the npm script and create a pull request automatically when there are changed files.

[tenwit]

This is a huge PR. I don't know if governance board members are the right people for this sort of review; maybe we should be reviewing the design, more than the implementation. We could find a technical board, or a sub-group of technical volunteers, for these sorts of large developments.

That said, I have a few points:

• This many changes shouldn't be in a single PR. There's too much to grok. Smaller groups of changes, even if they're not yet functional (and not wired up) will make managing reviews much easier.
• There's no design, no tests, no documentation of that the generated workflows will do. Even with the best of intentions and large amounts of available time, I wouldn't be able to review this against anything. What problem is being solved? Why is this solution better than others? Why is this implementation the best for the chosen solution?
• It looks like the workflow YAML has been translated to a fairly full implementation. Is that implementation original? Or copied from GitHub or Pulumi code? Can we use the original code, instead of copying it? Is all of this implementation used in generating the specific workflows that are generated by this code? That is, has too much code been written, that could have been avoided with a test-first approach to design?

[tenwit]

Also: it looks like there's no issue open for this work. Why is this work a higher priority then the existing issues?

[ringods]

The reason I started this PR, even without an open ticket, is that I was the one currently setting up new provider repositories. The current process, although we don't have that many providers yet, is clunky and hard to update when changes need to be made to the build system. I wanted to find a way to keep in sync much easier.

It is indeed quite a bulky PR, but most of the code was copied from Pulumi ci-mgmt repository. The only changes I made were to switch the model from push to pull.

Because of this PR, I had a chat with one of the internal engineers. There is no need to go forward with this PR in its current form. The team is reworking the code in ci-mgmt so it becomes a consumable artefact, not only for Pulumi itself, but also for thirdparties like Pulumiverse or vendor companies managing their own provider.

For now, I'm going to close this PR. I have filed #40 as a replacement.