generate-zbm from git master & zfs-2.2.99-r200+ with all feature flags enabled = ?

[comrade-meowski]

Hi devs me again testing the bleeding edge and running into - entirely expected - problems.

When the zfs-2.2.0-rc versions appeared before the released zbm could boot their pools when all the new feature flags were enabled I could fix it easily enough myself. Procedure was to install the latest zfs-2.2.0-rc version on an arch system and use generate-zbm from git master to make a portable .EFI from it. Simple.

I've failed to repeat the same trick with the latest zfs-2.2.99 from git - as I'm sure you're aware zfs-master as of version r200+ (ish) introduced RAIDZ expansion and some new feature flags, namely:

•   redaction_list_spill
  

•   raidz_expansion
  

So on my same arch system I used kernel 6.6.1 + zfs-2.2.99-209 + generate-zbm (from git master) to make a new.zbm.efi as before. However it can't boot any of my test VMs with zfs-2.2.99-200+ pools with all feature flags enabled. The warning about incompatible feature flags has disappeared and the zbm menu is accessible but booting any kernel results in failure.

Am I right in thinking you just haven't - yet - merged support in zbm yet and I'm jumping the gun?

[classabbyamp]

If you generated the zfsbootmenu image locally, it should use the system zfs modules, so there's no reason it shouldn't support whatever feature flags the system zfs modules support.

• I would not recommend using ZFSBootMenu's git master, as it is not guaranteed to work at all times (the v2.2.x branch is considered stable)
• Are you sure the ZBM EFI was regenerated after the zfs modules were built? what's zfs version in ZBM?

[comrade-meowski]

If you generated the zfsbootmenu image locally, it should use the system zfs modules, so there's no reason it shouldn't support whatever feature flags the system zfs modules support.

That's my understanding as well and up until now I've not had a problem locally generating ZBM EFIs either.

Checking in the rescue shell of my new ZBM EFI the zfs version is correct: 2.2.99-205_blah for utils and kmod. From the rescue shell I can check the rpool is detected and imported (it is) and whether I can detect and import other pools (I can).

Booting any of the rpool kernels drops me at the initramfs prompt with error: "Failed to load ZFS modules". If I modprobe zfs it silently fails and any attempt to modprobe or import pools fail. The most helpful error message I can coax out of the initramfs prompt is:

Begin: Importing ZFS root pool 'rpool' ... Begin importing 'rpool' using defaults ... Failure: 1
Begin: Importing pool 'rpool' using cachefile ... Failure: 1

Now I appreciate the obvious thing to do is blame everything but ZBM for this but there's a problem: the test system was working perfectly using exactly the same packages and kernels/initramfs immediately prior. To eliminate any possible confusion - and because I was anticipating issues anyway - the only thing I changed for testing was to update the rpool to enable the two new feature flags and then drop the new ZBM EFI into place. With only those two changes the exact same system with the same kernel(s) immediately failed. Therefore by definition the only thing that can be the cause is the new ZBM EFI.

That leaves two options: either the ZBM EFI is at "fault" (not in the pejorative sense, it's just the cause) OR the generation of the new ZBM EFI went wrong, which is what I will test next. The ZBM EFI was generated on a separate system that I use to generate all of my ZBM EFIs and as it happens I did use the git master version of generate-zbm - thanks to your tip I will go back and use the stable ZBM 2.2.2 generate-zbm script instead. Whilst I'm at it I will also check to see if I can boot a cloned test system with exactly the same characteristics as my fails-to-boot one with the same ZBM EFI but with the pools not upgraded to the latest feature flags.

I shall report back - thanks for your help.

[classabbyamp]

that's probably an initramfs issue not a ZBM issue, if zbm can start the boot process. sounds like your initramfs is missing zfs

[zdykstra]

When you're in the emergency shell of your initramfs, can you check if spl.ko is loaded / loads successfully? As @classabbyamp said though, once your BEs kernel and initramfs are booted it's out of ZBMs hands.

[comrade-meowski]

Ok I've found the issue even if I can't explain it - predictably it wasn't ZBM at all so I can close this now. Turns out that the initial generate-zbm process worked perfectly the first time and as soon as I moved the new ZBM EFI to a couple of different VMs to test the same scenario out everything worked as expected.

So be clear kernel 6.6.1 + zfs-2.2.99-200 (or higher) + generate-zbm (from git master or ZBM 2.2.2) makes an EFI that will boot a zfs boot pool even with all the new feature flags enabled. Apologies for the noise!

Ironically it seems my carefully prepared matched testing pair of a physical system and it's virtualized clone had somehow borked their kernels and been rendered incapable of generating a working initramfs. Both passed subsequent zpool scrubs ok and had been working and booting immediately prior so I really can't explain that - reinstalling the 6.6 kernels in question and generating fresh new initramfs for them resolved the issue on the problem systems.

I see a lot of weird stuff in my busy sysadmin life testing new software but I'm really at a loss to explain how those two test systems ended up in that state or how they were even booting in the first place. How did they manage to generate their initial initramfs successfully but then not subsequently? I guess my mistake was only double checking my work instead of triple checking it... ¯_(ツ)_/¯

Thanks as usual.