Our new release workflow

[laurenceisla]

Was going to comment on #3082, it's kinda related but out of the scope of that issue.

Problem

When doing a release from a rel-X.Y branch, then the bump to vA.B commit is only present in the rel-A.B branch. That's why we have to manually create a CHANGELOG commit in main, but not bump the postgrest.cabal version. This causes 2 things:

• The next patch release needs to cherry pick the commits (although this is OK for back patches)
• Makes the release process more painful. If the main branch has all the commits for the release, then it should be more straightforward to bump the version.
• The postgrest.cabal version in main is permanently outdated (right now it's 11.2.0 when 12.0.1 is the latest release)

Proposal

Before the release is done, verify if the rel-A.B and main branches are pointing to the same commit ( the same git rev-parse <branch>), if so, then the bump commit can be merged into main and pushed too (pushes rel-A.B, main and vA.B.C). This could be done automatically or ask if it's OK to merge.

If the branches are not pointing to the same commit, then automatically add only the CHANGELOG update to main, or ask if it's OK, or just inform to do it (I almost forgot to do that for v12.0.1). This will keep the version in main behind of the patch releases, but I think it's OK since the commits diverge until the next minor/major release fixes this.

[steve-chavez]

Q: The CHANGELOG can be the same on the main branch and the rel branch right? Doesn't matter if we have some unreleased additions on main and if we do a patch version. After all users will only see the main branch CHANGELOG.

---

Another issue with the CHANGELOG is that it's no longer good for major or minor releases, because I have to edit it manually to link the features to the docs as on https://github.com/PostgREST/postgrest/releases/tag/v12.0.0

---

Maybe we should no longer update the CHANGELOG manually on PRs (this is easy to forget too). Whenever a postgrest-release is done, only then the CHANGELOG should be updated automatically. Then this commit can go on the main branch and on the release branch.

For this we can stick to a commit convention (similar to the one Wolfgang suggested before), but that also includes a link to the latest docs. For this #2814 is a must.

Not sure about the cabal version, I think for now we can just duplicate it. Although this would matter more if #2275 was done.

Perhaps the correct thing for main would be to have the highest version plus the date as the cabal version, like we did on the nightlies. So right now it'd be 12.0.1.20231215.

[wolfgangwalther]

Not sure about the cabal version, I think for now we can just duplicate it. Although this would matter more if #2275 was done.

Perhaps the correct thing for main would be to have the highest version plus the date as the cabal version, like we did on the nightlies. So right now it'd be 12.0.1.20231215.

But then the version number would again not reflect which kind of changes have happened since 12.0.1.

What about the following?

• Once we branch for rel-12.0, main is immediately bumped to 12.1.0. This makes it clear, that we are currently developing this version on main. It also makes it unnecessary to create a bump commit before release, you just need to branch. That means all commits on the rel branch will also be on main. It will also allow to just merge main into the release branch if only bug fix commits have happened - no need to cherry-pick in that case, keeping the history cleaner.
• When we build from main (i.e. docker build after each commit / "nightly"), we change the version number to 0.2023.12.15, i.e. the version number is the date only, but prefixed with a 0 to avoid being taken as a "new major release" by tooling. The major=0 also has nice pre-release semantics. It would also allow someone to follow the nightly releases easily, e.g. via renovate, by putting down this 0.2023.12.15 version and then preventing "major" version upgrades. This would only bump the latest nightly all the time.
• We only release to hackage from rel-X.Y branches, but from all of them (once Automate Hackage release #2275 (comment) is solved), so that all patch releases will be on hackage.

Maybe we should no longer update the CHANGELOG manually on PRs (this is easy to forget too). Whenever a postgrest-release is done, only then the CHANGELOG should be updated automatically. Then this commit can go on the main branch and on the release branch.

Yes, we should do that. We could do it as follows:

• Create a changelogs/ directory and store the changelog for each version in a separate file.
• Have a small tool that will just take all the files in this directory and concat them into the main changelog file in the right order of versions.
• Whenever the changelog tool is run, it creates the new file in changelogs/ from the commit history since the last release.
• This file can then be adjusted manually in a PR, in case the commit messages were not perfect to begin with ;).
• The release tool will not automatically tag and push the branches - but instead will create a PR with the changelog and the version bump.
• Once the version-bump-PR is merged, CI will tag it automatically - and then kick off the release machinery.
• The file in changelog/ for the current release can be used for the release notes in the "Releases" section.

Additionally, once the docs are in the main repo, the changelog/ -> changelog.md job could also make sure to sync the release notes to the docs folder, so that they show up there, too.

---

btw. regarding back-patching on release branches, I think we should do the following:

• rel-X.Y branches still need to be manually release for bump of the patch version, and then tagged. Releases will actually only be made from tags, not branches. (I think this is what we do right now anyway)
• Bugfixes are immediately back-patched, not later when trying to make a release (this reminds of the whole "when do I update the docs?" story..). We might be able to even automate this via "back patch bot", which would automatically do so when no conflicts arise, but open an Issue when there are conflicts or tests fail etc.

[wolfgangwalther]

Once we branch for rel-12.0, main is immediately bumped to 12.1.0. This makes it clear, that we are currently developing this version on main. It also makes it unnecessary to create a bump commit before release, you just need to branch. That means all commits on the rel branch will also be on main. It will also allow to just merge main into the release branch if only bug fix commits have happened - no need to cherry-pick in that case, keeping the history cleaner.

Hm, the downside of this is, that we still don't reflect what's really happening on main.

I researched a lot of other projects, to see what they do. Here's a different idea now.

First, some key goals:

• We want semantic versions (semver), or at least close to it: Major version for breaking changes, minor version for features, patch version for bugfixes.
• When building postgrest on the latest main branch, it should have a version number reflecting this. Currently the latest release is 12.0.2, but main is on 11.2.0. Bad. But it would also be bad if main was on 12.0.0 or 12.0.2. This just doesn't reflect the fact that we have added features already.
• We should automate releases to hackage for and thus our version should always be reflected in postgrest.cabal. Because of that we are mostly restricted to "numbers only" in the version number.
• We want an updated pre-release after each commit on main to the github release page and also push a docker image each time.
• We should make it easy to track updates to PostgREST's docker images with dependabot and renovate. Those should only update to other stable versions, when you're targeting a stable version - and should always update to the latest nightly, when you are targeting nightlies.

The best way to achieve this, is to do what GHC and node.js are doing: They have uneven version numbers as pre-releases/development, and even version as real releases/stable. Node is doing it on the major, GHC on the minor. I think we already discussed this a while back, but haven't done so, yet (we still have 11.1 and 11.2 released).

My proposal is this:

• main is always on an uneven minor, reflecting "development". So our last release was 12.0.x, so main should now be on 12.1. Note that it should always be on <major>.<minor> only, so only two components, not three.
• When we make a new release, we make two commits: On main, we bump to 12.3 and on the new rel-12.2 branch we bump to 12.2.0 (now with patch version 0).
• After each commit to main, we make a pre-relase and docker release with <major>.<minor> in postgrest.cabal (so 12.1 or 12.3 etc.), but on github releases and dockerhub we tag this as <major>.<minor>-nightly. The nightly suffix will prevent dependabot and renovate from updating to/from this - they will always want to keep the suffix or avoid adding it. We are going to replace the existing pre-release / nightly tag.
• The nightly image on dockerhub will also be tagged as -nightly. This is just to avoid the image being deleted when replaced, so that pins to a specific digest still work. At the same time we should think about tagging our regular releases with more version numbers, i.e. instead of only v12.0.2, we do v12.0 and v12, too.
• The version number for nightlies reported by --version and the openapi output will then be <major>.<minor> (<commit-hash>), which gives us a clear indication that this was/is a pre-release, e.g. for bug-reports.
• No dates anywhere in version number anymore. Those just cause trouble, because they always compare "bigger" than regular version numbers, even if in fact older.

I think this could work pretty well.

[laurenceisla]

I think this could work pretty well.

@wolfgangwalther Agree, I like the idea. That way main would be always 1 step ahead of the latest release instead of perpetually stuck in an older version.

No dates anywhere in version number anymore. Those just cause trouble, because they always compare "bigger" than regular version numbers, even if in fact older.

+1 they gave more headaches than anything.

[steve-chavez]

We want an updated pre-release after each commit on main to the github release page and also push a docker image each time.

After each commit to main, we make a pre-relase and docker release with . in postgrest.cabal (so 12.1 or 12.3 etc.), but on github releases and dockerhub we tag this as .-nightly. The nightly suffix will prevent dependabot and renovate from updating to/from this - they will always want to keep the suffix or avoid adding it. We are going to replace the existing pre-release / nightly tag.

Hm, will we do that for pure docs commits too? (it doesn't seem right) That should happen once #3120 is merged. For now I'm thinking that nightlies should also be manual. Our postgrest-release command works pretty well already, we just need to extend it.

Maybe we should finish #3120 before settling on a new release model. It will simplify a lot of things.

[wolfgangwalther]

Hm, will we do that for pure docs commits too? (it doesn't seem right) That should happen once #3120 is merged.

We already do docs releases on each commit. The latest tag on the docs is the version from the postgrest-docs main branch. That should stay the same in the future. We just need to also make nightly releases on every commit, so that docs and latest nightly are actually in sync.

For the release branches, whether to release on each commit as we do right now, or only after tagging a new release would be up for debate, I guess.

For now I'm thinking that nightlies should also be manual.

Why? That will just bring us back to where we were before - and I don't want to go there. Currently, it's much harder than we need it to be to get the latest main branch for a quick test in an actual project repo. If we build a nightly image after each commit, all I need to do is to open a PR in my project, where I change the postgrest version to the latest nightly. Then I can test everything we currently have on main and provide feedback before we actually make a new release. Hopefully this would cut down on the number of bugs we have to fix after each release.

Maybe we should finish #3120 before settling on a new release model. It will simplify a lot of things.

The opposite! We need to first have a clear idea on how we want to do releases, including the docs, before we should actually move the docs over. We need to get the docs migration right from the beginning. And we need to know what to merge into which branches. But for that we need to know how we want to proceed later on.

---

Extending the proposal from #3113 (comment) with docs:

• The main branch will build nightlies and the latest docs - that's easy, see above.
• Each release branch will have the docs for that release.
• We only develop features on main, not on release branches, and will commit docs and feature together in the future.
• That means the only changes to docs on release branches will be corrections and stuff like sponsors etc.
• We can stick with "publish docs on each commit on release branches" as we currently do, and will never risk having something published in the docs that is not available via release, yet. Both docs and releases always go hand-in-hand.

I don't see any problem integrating the docs here. I do find it a bit strange, that we currently list all minor releases on readthedocs:

Maybe we can think about only having one version of the docs online for each major version? The latest minor release of that, ofc. I think that should be enough - after all, the idea is that upgrading through minors should always be easily possible, because it's not breaking. So there should be no "this feature is only available on 11.2, but I am still on 11.1". No need to stay there.

---

Taking the "publish docs on each commit on release branches" idea one step further: I think we should do the same for bugfixes. Here's how:

• We only push bugfixes to existing release branches anyway - features are not backported.
• We can immediately release a new patch version after every bugfix. There is really no problem at all in having the patch number in our version number actually get used more often. Compare that with renovate, which truly releases on every commit to main. 5000+ releases on GitHub :)
• Users will get bugfixes ASAP.
• Ultimately, we can get a bot to work for backpatching. We can then work on a bugfix on main and have the bot backpatch it. If the cherry-pick applies cleanly, the bot will create a new PR and auto-merge it if all pipelines pass. If the patch does not apply, it will open an issue for us, to manually backpatch. If the patch applies, but the pipelines fail, the PR will stay open for us.

This would leave us with only minor and major releases which we need to do manually via postgrest-release. Much better workflow, our users get their bugfixes and features much quicker.

WDYT?

[steve-chavez]

We already do docs releases on each commit. The latest tag on the docs is the version from the postgrest-docs main branch. That should stay the same in the future. We just need to also make nightly releases on every commit, so that docs and latest nightly are actually in sync.

Right, forgot about that - docs do get a release for each commit on main.

Maybe we can think about only having one version of the docs online for each major version? The latest minor release of that, ofc.

Agree.

Taking the "publish docs on each commit on release branches" idea one step further: I think we should do the same for bugfixes.
Ultimately, we can get a bot to work for backpatching

That sounds perfect/ideal.. though I don't know the effort it takes. It seems considerable with the bot and all.

Compare that with renovate, which truly releases on every commit to main. 5000+ releases on GitHub :)

I think we should avoid that. Having too many releases will create notification fatigue and we'll lose watchers on the repo.

The way we did nightlies before would be good for this https://github.com/PostgREST/postgrest/releases/tag/nightly. (Related #2997)

This would leave us with only minor and major releases which we need to do manually via postgrest-release. Much better workflow, our users get their bugfixes and features much quicker.
WDYT?

Agree overall 👍

[wolfgangwalther]

I think we should avoid that. Having too many releases will create notification fatigue and we'll lose watchers on the repo.

We'll be far away from that, if we just release every bugfix immediately.

The way we did nightlies before would be good for this https://github.com/PostgREST/postgrest/releases/tag/nightly.

Well, yes. I was not planning to create a new release for every nightly, but instead update the existing one.

But bugfixes should get their own release, for sure.

[wolfgangwalther]

Agree overall 👍

Ok, so now that we agree on the idea, here are the steps I think we should take to actually implement this:

Cleanup / Preparation

• Bump main to v12.1 immediately.
• Hide 11.1, 11.0 and 10.0 on readthedocs.

Migrate the docs repo

• Finish open PRs to postgrest-docs
• Archive the docs repo first to prevent more edits
• Merge docs/main into postgrest/main
• Merge docs/v12.0 into postgrest/rel-12.0
• Merge docs/v11.2 into postgrest/rel-11.2
• Merge docs/v10.2 into postgrest/rel-10.2
• Merge docs/v9.0 into postgrest/rel-9.0 Disabled v9 on readthedocs
• Make docs-CI work on all 4 branches
• Configure readthedocs to build from the main repo
• Integrate docs-CI+nix into core-CI / nix tooling properly only on the main branch
• Decide how to deal with open issues in the docs repo (migrate or not?)

Prepare the next release

• Update postgrest-release to support major/minor on main branch, branching off automatically, as discussed above. nix: Adjust postgrest-release to new release workflow #3500
• Add "pg version last supported by pgrst version X" and maybe "support for pg version Y will be removed in next version of pgrst" hints on startup as discussed in feat: drop support for pg 9.6 #2052 (comment).
• Write docs for new features.
• Release v12.2.0
• Enable stable build on readthedocs after release / remove temporary redirect

Further improvements

• Revive nightly releases.
• Automate bugfix releases on backbranches.
• Certainly last: Automate backporting of bugfixes via bot.

@laurenceisla @steve-chavez, anything I forgot?

[wolfgangwalther]

Bump main to v12.1 immediately.

Hide 11.1, 11.0 and 10.0 on readthedocs.

Did both of that already.

[laurenceisla]

@wolfgangwalther

Migrate the docs repo

Is it necessary to migrate the docs to older branches? Will they not stay as they are right now in RTD?

Otherwise, everything LGTM.

[wolfgangwalther]

Is it necessary to migrate the docs to older branches? Will they not stay as they are right now in RTD?

If we do that, then we can't archive the docs repo. And then we risk running out of sync. The cleanest solution is to clearly move everything over.

[wolfgangwalther]

Finish open PRs to postgrest-docs

I dealt with all open PRs to the postgrest-docs repo.

[wolfgangwalther]

Archive the docs repo first to prevent more edits

I archived the docs repo to allow working on #3120 etc. without getting out of sync again.

[wolfgangwalther]

I added "Write docs for new features." to the section of preparing a new release. The changelog tells me we might need docs for those items?

### Added
 - #2887, Add Preference `max-affected` to limit affected resources - @taimoorzaeem
 - #3171, Add an ability to dump config via admin API - @skywriter
 - #3061, Apply all function settings as transaction-scoped settings - @taimoorzaeem
 - #3171, Log schema cache stats to stderr - @steve-chavez
...
### Deprecated
 - `Prefer: params=single-object` is deprecated. Use [a function with a single unnamed JSON parameter](https://postgrest.org/en/latest/references/api/stored_procedures.html#s-proc-single-json) instead. - @steve-chavez

Not sure if those have been written already.

[steve-chavez]

"Write docs for new features."

True. Those are missing.

[wolfgangwalther]

Merge docs/v9.0 into postgrest/rel-9.0

While experimenting with different strategies to merge the two repos, I realized that the v9 branches in both repos don't pass CI anymore - it seems like the nix-related github actions are too old by now and thus our nix tooling doesn't work anymore. I didn't dig deeper - but instead decided to just remove the v9 branch from the docs. Given that we haven' touched that branch for more than 1 1/2 years, I think it's ok to not show it anymore going forward. IIRC, the last time we discussed this, our policy was to say we show the current latest major and two more older majors - so that would mean we are going back to v10 only.

[wolfgangwalther]

Merge docs/main into postgrest/main
Merge docs/v12.0 into postgrest/rel-12.0
Merge docs/v11.2 into postgrest/rel-11.2
Merge docs/v10.2 into postgrest/rel-10.2
Make docs-CI work on all 4 branches

All done.

[wolfgangwalther]

Currently, it's much harder than we need it to be to get the latest main branch for a quick test in an actual project repo. If we build a nightly image after each commit, all I need to do is to open a PR in my project, where I change the postgrest version to the latest nightly. Then I can test everything we currently have on main and provide feedback before we actually make a new release. Hopefully this would cut down on the number of bugs we have to fix after each release.

Prime example here: I had to use a latest development snapshot of postgrest to kickstart development of a new feature in our project, while the next PostgREST version is not released, yet. This took more effort than I wanted it to take. When I finally gave it a test-run in our project's CI, I identified the following bugs:

• 16bc853
• --dump-schema does not dump media handlers #3237
• Regression / breaking change for RPCs with SET #3242

With the current manual workflow, I won't repeat this often. But with a nightly... that would be much easier.

[wolfgangwalther]

I am currently working on the release tooling and there is one question that comes up for me around nightly releases:

Do we want to persist nightly releases forever or do we want to replace them with every new nightly release?

In the past, we had kind of a mix:

• GitHub release: The old nightly release was replaced.
• GitHub release: Later pre-releases were kept, e.g. v9.0.0.20220130
• git tags: same as GitHub releases
• docker: always kept, not replaced, e.g. postgrest/postgrest:nightly-2021-07-06-11-19-0961a58

So the old nightly releases were not really consistent, the pre-releases were consistent and kept everything.

I don't like keeping the old prereleases, though. This makes it basically impossible to use the release page as kind of a changelog to the project, because all the pre-releases keep repeating the same changes over and over again. Just have a look at page two: https://github.com/PostgREST/postgrest/releases?page=2

Ultimately the question is: Why do we want to provide nightly releases? Imho, we should provide them to allow users to try out new features before an official release is made - but not to depend long-term on those pre-releases. So once the new release is made, I don't think we need those pre-releases anymore.

Thus, I suggest:

• Always replace the same GitHub pre-release on a new commit to main.
• Don't tag pre-relases anymore - we're basing this off of main.
• Delete the old nightly release and all old pre-releases from GitHub Releases and their respective git tags.
• Always replace the same docker tag. So not 12.1-nightly or 12.1-devel - but just nightly or devel.
• Remove all the old nightly and pre-relase docker tags from docker hub.

At first I thought it would be helpful to have the version in the docker tag like 12.1-devel to be able to tell which changes you are actually looking at, e.g. for bug reports. But unless we put everything down to the commit hash in the tag (like we did for nightly tags in the past), it won't actually help. So we can just as well remove all the version numbers from it and always replace the same tag here.

@steve-chavez any objections?

[wolfgangwalther]

Don't tag pre-relases anymore - we're basing this off of main.

Uhm, that won't work - all releases need a tag. So we'd do a nightly or devel tag after each commit to main automatically. I like devel more, because we already use that as a version in the latest docs: https://postgrest.org/en/latest/.

[steve-chavez]

I don't like keeping the old prereleases, though. This makes it basically impossible to use the release page as kind of a changelog to the project, because all the pre-releases keep repeating the same changes over and over again. Just have a look at page two: https://github.com/PostgREST/postgrest/releases?page=2

Yeah, don't like that too. Too much noise.

@steve-chavez any objections?

So we'd do a nightly or devel tag after each commit to main automatically. I like devel more, because we already use that as a version in the latest docs:

No objections.. devel looks good 👍

[wolfgangwalther]

Delete the old nightly release and all old pre-releases from GitHub Releases and their respective git tags.

Remove all the old nightly and pre-relase docker tags from docker hub.

Did both of that.

[wolfgangwalther]

While going through the docker images, I noticed that we have the newer releases all with an additional -arm tag.

Our docker-publish.sh has this to say:

postgrest/.github/scripts/arm/docker-publish.sh

Lines 36 to 39 in e5aee49

	# Push final images to Docker hub
	# NOTE: This command publishes a separate ARM image because the builds cannot
	# be added to the manifest if they are not in the registry beforehand.
	# This image must be manually deleted from Docker Hub at the end of the process.

But we don't delete the tag, so far. We could easily do so, the actual image is part of the multi-layer image under the regular version tag.

@laurenceisla since you wrote this script, can you shine light on why we still keep the -arm tag? Was there a specific reason or was it just not done/forgotten? If so - I would fix that once I come around to make the new GitHub Runner on ARM work nicely - and then delete all the old -arm tags, too, given that the underlying images will still be there. WDYT?

[laurenceisla]

Was there a specific reason or was it just not done/forgotten?

The latter.

I would fix that once I come around to make the new GitHub Runner on ARM work nicely

Awesome!

and then delete all the old -arm tags, too, given that the underlying images will still be there. WDYT?

Yeah, no need to keep those -arm tags. Go ahead!