v2.2.1 initcpio generated image silently fails

[gardar]

Hello,

I've encountered an issue when using generate-zbm on zbm v2.2.1. The images I generate just don't boot — there's no error message; they just don't launch.

I've been using the same configuration that worked flawlessly in the past. Here are the configurations for reference:

config.yaml

Global:
  ManageImages: true
  BootMountPoint: /boot/esp
  DracutConfDir: /etc/zfsbootmenu/dracut.conf.d
  PreHooksDir: /etc/zfsbootmenu/generate-zbm.pre.d
  PostHooksDir: /etc/zfsbootmenu/generate-zbm.post.d
  InitCPIO: true
  InitCPIOConfig: /etc/zfsbootmenu/mkinitcpio.conf
Components:
  ImageDir: /boot/esp/EFI/zbm
  Versions: 3
  Enabled: false
  syslinux:
    Config: /boot/syslinux/syslinux.cfg
    Enabled: false
EFI:
  ImageDir: /boot/esp/EFI/zbm
  Versions: false
  Enabled: true
Kernel:
  CommandLine: ro quiet loglevel=0

mkinitcpio.conf

MODULES=()
BINARIES=()
FILES=()
HOOKS=(base udev keyboard autodetect keymap modconf block filesystems zfsbootmenu)

For some context, I'm on ArchLinux, and here are the package details:

zfs-dkms: 2.2.0-1
zfs-utils: 2.2.0-1
linux: 6.5.7.arch1-1
linux-headers: 6.5.7.arch1-1

Would appreciate any insights or assistance on this matter. Thanks in advance!

[zdykstra]

When you say they don't boot or launch, what does that entail? How are they launched, and what happens when they are (presumably) launched?

2.2.1 is a maintenance release that's really just timed to the release of OpenZFS 2.2.0, so that we can publish new binaries. Which is to say this is likely a local issue that could be difficult to track down.

[gardar]

The image is installed with efibootmgr and basically nothing happens, it just returns to the efi boot menu.
If I attempt to execute the image from the uefi shell I get this error: Command Error Status: Load Error.

The prebuilt binary works fine and if I execute it from the uefi shell it launches immediately.
Can I enable some verbose/debug output on the image?

Out of curiosity I tried extracting a old working zfs 2.1 image, the new troublesome zfs 2.2 image, and the 2.2 prebuilt image not sure if this gives any clues:

zfs image 2.1:

total 30M
drwxr-xr-x 2 g g   15 Oct 19 03:20 .
drwxr-xr-x 7 g g    9 Oct 19 03:20 ..
-rw-r--r-- 1 g g  512 Nov 23  2022 .cmdline
-rw-r--r-- 1 g g 7.2K Nov 23  2022 COFF_SYMBOLS
-rw-r--r-- 1 g g  13K Nov 23  2022 .data
-rw-r--r-- 1 g g  512 Nov 23  2022 .dynamic
-rw-r--r-- 1 g g  512 Nov 23  2022 .dynsym
-rw-r--r-- 1 g g  20M Nov 23  2022 .initrd
-rw-r--r-- 1 g g  11M Nov 23  2022 .linux
-rw-r--r-- 1 g g  512 Nov 23  2022 .osrel
-rw-r--r-- 1 g g 4.0K Nov 23  2022 .rela
-rw-r--r-- 1 g g  512 Nov 23  2022 .reloc
-rw-r--r-- 1 g g  512 Nov 23  2022 .sbat
-rw-r--r-- 1 g g  512 Nov 23  2022 .sdmagic
-rw-r--r-- 1 g g  36K Nov 23  2022 .text

zfs 2.2:

total 33M
drwxr-xr-x 2 g g   12 Oct 19 03:20 .
drwxr-xr-x 7 g g    9 Oct 19 03:20 ..
-rw-r--r-- 1 g g  512 Oct 19 01:01 .cmdline
-rw-r--r-- 1 g g 1.0K Oct 19 01:01 .data
-rw-r--r-- 1 g g  20M Oct 19 01:01 .initrd
-rw-r--r-- 1 g g  13M Oct 19 01:01 .linux
-rw-r--r-- 1 g g  512 Oct 19 01:01 .osrel
-rw-r--r-- 1 g g  512 Oct 19 01:01 .reloc
-rw-r--r-- 1 g g 9.0K Oct 19 01:01 .rodata
-rw-r--r-- 1 g g  512 Oct 19 01:01 .sbat
-rw-r--r-- 1 g g  512 Oct 19 01:01 .sdmagic
-rw-r--r-- 1 g g  48K Oct 19 01:01 .text

zfs 2.2 prebuilt image:

total 47M
drwxr-xr-x 2 g g   13 Oct 19 03:22 .
drwxr-xr-x 8 g g   11 Oct 19 03:22 ..
-rw-r--r-- 1 g g  512 Oct 17 01:06 .cmdline
-rw-r--r-- 1 g g 9.2K Oct 17 01:06 COFF_SYMBOLS
-rw-r--r-- 1 g g 8.0K Oct 17 01:06 .data
-rw-r--r-- 1 g g  512 Oct 17 01:06 .dynamic
-rw-r--r-- 1 g g 1.0K Oct 17 01:06 .dynsym
-rw-r--r-- 1 g g  37M Oct 17 01:06 .initrd
-rw-r--r-- 1 g g  11M Oct 17 01:06 .linux
-rw-r--r-- 1 g g  512 Oct 17 01:06 .osrel
-rw-r--r-- 1 g g 4.0K Oct 17 01:06 .rela
-rw-r--r-- 1 g g  512 Oct 17 01:06 .reloc
-rw-r--r-- 1 g g  28K Oct 17 01:06 .text

[zdykstra]

I've been able to reproduce this issue locally. The problem seems to be that the section offsets that are used by generate-zbm no longer work with the EFI stub provided by Arch. However, I have been able to successfully use mkinitcpio to produce a working ZFSBootMenu EFI with the Arch EFI stub. Until we can rework how EFI bundles are made with generate-zbm, the best option is to stick with the stub that Void ships, or use our binary release.

[gardar]

Interestingly the sections on the void stub one and the arch stub one look the same:

arch:

 512 Oct 19 01:01 .cmdline
1.0K Oct 19 01:01 .data
 20M Oct 19 01:01 .initrd
 13M Oct 19 01:01 .linux
 512 Oct 19 01:01 .osrel
 512 Oct 19 01:01 .reloc
9.0K Oct 19 01:01 .rodata
 512 Oct 19 01:01 .sbat
 512 Oct 19 01:01 .sdmagic
 48K Oct 19 01:01 .text

void:

 512 Oct 19 23:44 .cmdline
1.0K Oct 19 23:44 .data
20M Oct 19 23:44 .initrd
13M Oct 19 23:44 .linux
512 Oct 19 23:44 .osrel
512 Oct 19 23:44 .reloc
9.0K Oct 19 23:44 .rodata
512 Oct 19 23:44 .sbat
512 Oct 19 23:44 .sdmagic
48K Oct 19 23:44 .text

The debug log from generate-zbm also looks the same for both the void stub and arch stub
generate-zbm-void.log
generate-zbm-generate.log
generate-zbm-uki.log

[gardar]

I've been able to reproduce this issue locally. The problem seems to be that the section offsets that are used by generate-zbm no longer work with the EFI stub provided by Arch. However, I have been able to successfully use mkinitcpio to produce a working ZFSBootMenu EFI with the Arch EFI stub. Until we can rework how EFI bundles are made with generate-zbm, the best option is to stick with the stub that Void ships, or use our binary release.

Fantastic!
How were you able to produce a working one with mkinitcpio? Is it simply a matter of some configuration or is it more involved than that?

[zdykstra]

You can bypass generate-zbm entirely if you want and call mkinitcpio directly. In my test environment, I ran:

mkinitcpio --config /local/environments/test.everything/mkinitcpio.conf -v --hookdir /home/zdykstra/source/zfsbootmenu/initcpio --hookdir /usr/lib/initcpio -A zfsbootmenu --uki /local/environments/test.everything/arch-stub.EFI --uefistub source/zfsbootmenu/testing/stubs/local.arch.efi.stub

You should be able to just call mkinitcpio --config /etc/zfsbootmenu/mkinitcpio.conf -v -A zfsbootmenu --uki /tmp/zfsbootmenu.EFI and have it produce a functional image.

[gardar]

Great, that works! I also managed to tweak my config.yml so that it works for generating/installing the uki image.

Global:
  ManageImages: true
  BootMountPoint: /boot/esp
  DracutConfDir: /etc/zfsbootmenu/dracut.conf.d
  PreHooksDir: /etc/zfsbootmenu/generate-zbm.pre.d
  PostHooksDir: /etc/zfsbootmenu/generate-zbm.post.d
  InitCPIO: true
  InitCPIOConfig: /etc/zfsbootmenu/mkinitcpio.conf
  InitCPIOFlags: ['--uki /boot/esp/EFI/zbm/vmlinuz-linux.EFI']
EFI:
  ImageDir: /boot/esp/EFI/zbm
  Versions: false
  Enabled: true
Kernel:
  CommandLine: ro quiet loglevel=0
  Prefix: dummy

With that being said, now when I boot I get the message

Unsupported features detected
Upgrade ZFS modules in ZFSBootMenu with generate-zbm
Even though everything seems to work.