Will future migrations for minor and bug fix versions avoid creating downtime for clients and workers?

[shawnstephens]

Hello! We deploy various binaries in a given application repo to production automatically after a merge and successful CI build. Typically, for custom migrations in our apps, we make sure changes are backwards compatible to avoid any downtime.

So for instance in certain cases when adding a field that should be "not null" we will:

1. Add a DB field to an actively written table with a migration and merge/deploy.
2. Once deployed, we add code to populate the the new field and merge/deploy.
3. Next we add code to backfill null values and merge/deploy.
4. Finally, we add a migration to make the column "not null".

I'm curious on the River team's methodology here for preventing required downtime for migrations, understanding that it may be necessary with semantic versioning and major versions?

[brandur]

Hi @shawnstephens,

Both Blake and I have quite a lot of operational experience with Postgres and are well aware of its various DDL footguns. I can't give you a precise answer about how we'll handle every future migration yet, but we're already assuming that users have large River jobs tables, and will write future migrations so they'll work safely with such. We'll probably take an approach of doing only safe migrations, but in case that's not possible, we'll put together specific guides for how users should approach them.

We're not quite sure about how this will fit into versioning on a permanent basis. Right now we're reserving the right to add migrations during minor version bumps, although as noted above, they'll be safe ones. This may change in the future.

BTW, just in case you missed it — these days you can get away with adding a NOT NULL without the safety steps you noted above as long as it has a DEFAULT statement that's non-volatile [1].

[1] https://brandur.org/postgres-default

[bgentry]

@shawnstephens Thanks for the question and I will definitely echo what @brandur said here. We care deeply about making upgrades as smooth as possible and we each have a lot of experience doing this kind of thing in our careers.

We already had some conversations about this following a mistake we made in rushing out migration 3 (#64) shortly after launch. In particular we came to realize that there is going to be some extra care required with future migrations since we aren't going to be in a position to do a multi-part deploy strategy given we aren't the ones running all the apps using River. If we do find ourselves in a situation where a migration requires multiple steps to be done safely and without production impact, we're going to be as clear about that as we can.

[shawnstephens]

@brandur and @bgentry thanks so much for taking the time to explain your methodology here. It's great insight to have as we evaluate River in our first deployment.

@brandur also I greatly appreciate the link regarding the NOT NULL with defaults!