Snapshots, Retention for short lived replications

[namelessjon]

I'm interested in using litestream for syncing the state of a process which runs every half hour. Currently the process copies the sqlite db down from S3 at the start, writes data into it and then copies the database back up at the end. This is fine when the process takes a few seconds (as it normally does). It has the potential to take a lot longer though, if a large swathe of data needs reprocessing. In that case any failure could throw away hours of work. Enter litestream, and replicating the changes off to S3 every second (probably every minute in reality) via running the process with -exec. Which would be quite nice.

• If litestream is only running for a minute or two, will it ever assess the replicated data for snapshots, retention?
• If not, is there any way of saying "take a snapshot just before you shut down after the supervised process exits?"

[benbjohnson]

If litestream is only running for a minute or two, will it ever assess the replicated data for snapshots, retention?

No, it probably won't. It only checks periodically based on the retention-check-interval in the config.

If not, is there any way of saying "take a snapshot just before you shut down after the supervised process exits?"

It would be a good idea to have it enforce on exit or even have a litestream enforce-retention command but that doesn't currently exist.

[namelessjon]

Thanks for the reply @benbjohnson . In the end, I went with "background thread which periodically backs up the DB, gzips it and posts it back to S3". It's not ideal in that I'm copying more data than I need to, but it works which is the main thing.

Having such an enforce-retention command, option, whatever would be nice though.