Bugfix & Feature Wishlist zrepl 0.6

[problame]

With zrepl 0.5 pretty much done, I'd like to take the opportunity to ask users about which bug fixes and features should be tackled next.
It doesn't matter whether there is already an issue for it.
The purpose of this discussion is to gauge relevance, and also to figure out the big-picture direction for zrepl.
I hope I'll find time to post some ideas myself over the holidays.

[schulzh]

I would like some easier way of getting notified of certain error conditions. I know there is the ability to setup Prometheus and Grafana and have it send Notifications when some metric exceeds a threshold, but that is quite a lot of extra effort. I think such notifications should be easy to configure and not require lots of additional software setup.

That being said, the configuration of these notifications and their thresholds should be flexible enough. E.g. I have a replication over a quite unstable connection, so replications frequently fail for various reasons, but overall they replicate fine eventually.

Maybe this could be implemented using the existing Prometheus coding and just building some form of configuring and sending notifications.

Additionally, some more metrics might be needed. I think this this issue in particular is quite important, it would be a good basis for notifying about "bigger" replication issues and naturally filter out smaller, unimportant errors.

[Freeaqingme]

On my setup, I'm using a set of custom icinga2 commands to check the age of the most recent snapshot on the receiver.

@problame is this something you could share? I found this issue by searching for how to monitor zrepl using Icinga and there doesn't seem to be too much other ideas about this thus far.

[papamoose]

Definitely being able to easily send a notification to slack/discord/generic webhook would be nice. I would be happy with the ability to setup a hook post send that would have access to a few environment variables to help determine if a send succeeded.
Obviously, using Prometheus is always going to be preferred in an enterprise environment however a simple method to get notifications would be appreciated if you're not running Prometheus.

My current method involves a python script which queries the Prometheus metrics (zrepl_replication_last_successful).

Discord has a Slack compatibility mode. zfs-zed has Slack notification option. Zed's pattern seems reasonable to follow.

Also +1 to gotify support (in addition to, not instead of the above).

[papamoose]

shoutrrr might also be a possibility.

[skirmess]

I don't like edge-triggered stuff. What if the script fails? Should zrepl retry? How often? How frequently?

I'm using Uptime Kuma to monitor my services. With Kuma you can define a push trigger and get URLs like these:

https://kuma.example.net/api/push/abcdef123?status=up&msg=OK&ping=
https://kuma.example.net/api/push/abcdef123?status=down&msg=Could%20not%20create%20snapshot&ping=

The nice thing is that you can also define a timeout and if the webhook wasn't called for the timeouts duration, it also fails the service.

could simply use zrepl status + jq in a cronjob.

How would I get the "healthiness" of zrepl from the status?

[tbleiker]

@skirmess And where do you call these urls? I had a look "Pre- and Post-Snapshot Hooks" in the docs, but I couldn't figure out how to detect a failure during the job itself in a post-snapshot hook...

[budachst]

I'd really like this feature to become available: introduce conflict_resolution.initial_replication.replicate_all_snapshots #470

Being able to initially sync a complete ZFS to the destination, or re-sync a long standing zrepl source should be easy to do and not require to have to manually sync the entire zpool beforehand.

And just to stress this a bit more… I just found out that ZFS currently only handles up to 65k Snapshots for send/recv. Having a ZPOOL with 375 ZFS and 215k Snapshots, there's currently no way of running a simple zfs send -R from the root of the ZPOOL, but you'll have to send each of the 375 ZFS by themselves.

[problame]

Thanks for this feedback. Do you have opinions on syntax? The underlying issue is that I'd like to ensure that the syntax we pick is compatible with future conflict resolution stuff, such as, automatic handling of the case where the source side destroys a filesystem and re-creates one with the same name.

[problame]

I am finishing @grahamc's work on this in #592 . Upvoters, please consider giving the code a spin before I merge it!

[khenderick]

I would like to add #554 to this list.

In summary, I would like to be able to schedule taking a snapshot (and subsequent operations such as replicating). This might be possible by using a newly introduced cron type in the snapshotting section. It could be considered a more advanced version of the periodic type.

My own use-case is using a hook to take a backup of a non-zfs system, downloading it to a zfs dataset and then snapshot/replicate it. I want this to happen at a for the non-zfs system suitable time (off hours).

[problame]

@lapo-luchini There is a long-standing issue to use ZFS channel programs for consistent snapshots ( #46 ). Basically we'd need code that compiles a zrepl filesystems filter to a channel program. The only weak spot with channel programs is that they're only atomic, but not transactional. I.e., if we run over the amount of allowed lua instructions / mem limit, only some snapshots will have been taken. We'd need to detect that error condition in zrepl and then clean-up, and potentially retry with a higher limit.

The alternative is to do a recursive snapshot and then delete snaps that don't match filesystems, but I don't like that approach.

Perhaps this should be a separate wishlist item, so it doesn't get lost here?

[mdlayher]

I don't have a need for scheduling snapshots, but I do have a need to schedule replication. NixOS ships with some hooks to set up zfs-auto-snapshot pretty easily, so I complemented that with a quick systemd unit and timer to trigger replication:

[matt@servnerr-3:~]$ systemctl cat zrepl-signal-jobs.timer
# /etc/systemd/system/zrepl-signal-jobs.timer
[Unit]
PartOf=zrepl-signal-jobs.service

[Timer]
OnCalendar=hourly
Unit=zrepl-signal-jobs.service

[matt@servnerr-3:~]$ systemctl cat zrepl-signal-jobs.service
# /etc/systemd/system/zrepl-signal-jobs.service
[Unit]

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/vxxqn7g4ds1vlhdcbpi1m2hw4xqm80r9-glibc-locales-2.33-78/lib/locale/locale-archive"
Environment="PATH=/nix/store/yyr35ka8sqq6mym9r20djvsjl8rzg9ah-zrepl-0.5.0/bin:/nix/store/5wch96kji9zlffxjqpjdrszjzp4i7m3a-coreutils-9.0/bin:/nix/store/0hj8idivq7434gbsi0jx3dfhd7bsgb3a-findutils->
Environment="TZDIR=/nix/store/6cc9lmldp0w0vvgr5k1h1gr8i6hn791b-tzdata-2021c/share/zoneinfo"

ExecStart=/nix/store/livfw3wv30jf3azv3wvrxx6gwgwr85np-unit-script-zrepl-signal-jobs-start/bin/zrepl-signal-jobs-start 
Type=oneshot

[matt@servnerr-3:~]$ cat /nix/store/livfw3wv30jf3azv3wvrxx6gwgwr85np-unit-script-zrepl-signal-jobs-start/bin/zrepl-signal-jobs-start
#!/nix/store/kq5si67lhbidwkw444kc3lz3x3l7nxy6-bash-5.1-p8/bin/bash -e
zrepl signal wakeup primary_to_secondary
zrepl signal wakeup primary_to_backup0
zrepl signal wakeup primary_to_backup1

This does the job and my zfs-auto-snapshot snapshots all get replicated hourly. I was curious if there'd be a way to trigger a periodic replication without prior snapshotting in the config, but have no strong feelings about whether or not that should be included since I am making use of two different systems here.

[problame]

if there'd be a way to trigger a periodic replication without prior snapshotting in the config,

There's a WIP PR for this, but I preferred a different design. Started implementation but never finished.
#462

@mdlayher could you turn this into an independent entry on this wishlist?

[mdlayher]

Can do!

[spi43984]

I would like to add #554 to this list.

In summary, I would like to be able to schedule taking a snapshot (and subsequent operations such as replicating). This might be possible by using a newly introduced cron type in the snapshotting section. It could be considered a more advanced version of the periodic type.

My own use-case is using a hook to take a backup of a non-zfs system, downloading it to a zfs dataset and then snapshot/replicate it. I want this to happen at a for the non-zfs system suitable time (off hours).

I second this idea. I also would like to create snapshots at a particular time and run the replication at off hours rather then at periodic time intervals.

Something else I would like to add to the list is some kind of benchmark support to see the throughput (like for different servers and different transports, as e.g. scp is doing). Sorry if there is something and I've just missed it but the only thing I could find is some live view in zrepl status which isn't of much help. I'd like to see how much data was transmitted in what amount of time for each filesystem and in total.

[TheCasualObserver]

I would kindly ask you to have a look at the bug report #490 for the next release.

[lpulley]

Is this "push to B, pull from B" a recommended way to handle A>B>C replication? I'm trying to figure out how to do something just like that.

[problame]

Push vs pull doesn't matter as long as all jobs involved have disjoint names, and you don't need fan-out (for fan-out, you'll need something like #380 (comment) )

[lpulley]

I decided A>B>C isn't ideal for me because I still want A>C if B is down. I suppose A>B>C isn't exactly right, since my A and B are on the same machine, with C remote. I'm thinking of setting up fan-out with systemd timer wakeups for the push jobs -- unless you think source/pull is more stable than push/sink for fan-out?

Also, are there any changes in 0.6.0 that should affect fan-out?

[problame]

There are no changes that affect fan-out in 0.6

It works with push jobs because each push job has a different name and can hence maintain different replication cursors.
To make it work with pull jobs, you need a separate source job per pull job so that the replication cursor bookmark names don’t conflict.

In general, I would advise to diversify the backup strategy instead of going all in on zfs send/recv.

[lpulley]

The plan is to have

• [zrepl] push A to local B (keeping a fading history)
• [zrepl] push A to remote C (keeping a fading history)
• [restic] backup A to remote D (keeping a fading history)

So I do have a non-ZFS (file-based) backup included in this plan. Don't worry, I don't trust ZFS completely. 😆

[eoli3n]

Ability to disable jobs : #333

[problame]

@eoli3n I responded there, and I'm interested in your thoughts. Please comment there.

[papamoose]

Ability to specify the destination dataset per source dataset.

e.g.
tank/foo => tank/foo
tank/bar => tank/containers/bar

Additionally if and some other value you set to enable this:

  filesystems: {
    "tank<": true,
  }

On the receiving side the file systems would be setup identically.
e.g. both sides would look like this:
tank/foo
tank/bar/baz

[mdlayher]

I ran into the same thought when setting up zrepl today. I am backing up locally from one zpool to a hot backup pool and a cold backup pool that is a single disk.

I had assumed if I have primary/media, I'd get secondary/media, but the current hierachy is secondary/local/primary/media. Not really a huge deal I suppose (and I was able to rename a large existing dataset to match), but in my case I wouldn't ever anticipate receiving pool streams from multiple senders since I'm just doing a local copy.

[problame]

@papamoose I understand you want the ability to map arbitrarily between sender and receiver filesystem names.
Whereas for @mdlayher, it would probably be sufficient to support root_fs="".

For both of you, the issue that's worth subscribing to is probably #253.

A few follow-up questions to @mdlayher :

• So, you are pushing a set of filesystems from primary to secondary/local. And then, from secondary/local, to yet another cold backup pool?
• Regardless of the previous question: would you want primary's content to be replicated to secondary as well, then? Or are you not interested in replicating primary, but only its children?
  (Because: primary is not just the pool name, it's also a regular dataset, that could contain user data).

[mdlayher]

Thanks for following up! Here is my current generated configuration for reference: https://gist.github.com/mdlayher/36c201634be0e2b9593f266e0eb777ec

So, you are pushing a set of filesystems from primary to secondary/local. And then, from secondary/local, to yet another cold backup pool?

I would like to replicate primary to secondary, but since I only have a single client (the local host) I just set client_identity = local, so yep the computed path ends up being primary/media to secondary/local/primary/media.

I have two single disk pools called backup0 and backup1 and replicate directly from primary to those when the pools are imported and online. These are cold storage disks that I will plug in, replicate, and then probably take offsite or store in a safe.

Regardless of the previous question: would you want primary's content to be replicated to secondary as well, then? Or are you not interested in replicating primary, but only its children?
(Because: primary is not just the pool name, it's also a regular dataset, that could contain user data).

I'm only interested in replicating the children, yes! I've generally been treating non-leaf datasets as just containers for other datasets.

After thinking on this for a couple of days, it is a bit nice to have the secondary/local prefix because I can only set ZFS encryption on a child dataset rather than the entire zpool. This means that I can guarantee everything written in under that child is encrypted, and if I understand correctly, should also simplify unlocking the backup{0,1} pools using ZFS native encryption.

---

So really there are no necessary changes here since no software I am using demands a particular layout in my backup pools. I just wanted to share my initial thoughts.

[problame]

Yes, encryption is another problem child wrt root_fs="". I think you can set it on the root dataset of a pool, but you can't recv -F then.

Regarding encryption on the encrypt-on-recv: make sure to use zrepl 0.5. see changelog for details.

[mdlayher]

NixOS 21.11 installed 0.4 by default so I learned that one the hard way unfortunately, but I can confirm that 0.5 worked great! Thanks.

[mdlayher]

As a follow up to #547 (reply in thread) and as requested by @problame, I would like a way to be able to trigger replication on a schedule without necessarily creating a snapshot first.

I have a workaround that works, but frankly, I get the feeling I should just embrace the zrepl periodic snapshotting mechanism rather than trying to make zfs-auto-snapshot (as configured by NixOS) play nicely.

But perhaps it'd be nice to just have some way to say "run this replication job every hour" or similar, since you could pretty easily set up some Go tickers to send a signal at regular intervals.

[mdlayher]

I switched over to zrepl snapshotting today and have disabled zfs-auto-snapshot, so I think my request is now obsolete!

[eoli3n]

#577 : introduce '>' value for sub datasets only in filesystems key
#579 : store bucket value for snapshots in a property

[sabirovrinat85]

Is there anything like that zfs-check ? If no, then maybe it's worth considering as an additional feature