Use zfsbootmenu for multi disk pools

[SimonBard]

I am using two HDD as a raid and a SSD as cache. How do I install zfsbootmenu here correctly?
Do I still need to define one partition of a specific HDD as the boot partition?

I am referring to this document here:
https://docs.zfsbootmenu.org/en/v2.3.x/guides/ubuntu/uefi.html#disk-preparation

I want to create the zpool via:
zpool create system1 mirror sda sdb cache sdc

lsblk give me so far:

sda      8:0    0   3,6T  0 disk
└─sda1   8:1    0   3,6T  0 part
sdb      8:16   0   3,6T  0 disk
├─sdb1   8:17   0   3,6T  0 part
└─sdb9   8:25   0     8M  0 part
sdc      8:32   0 931,5G  0 disk
├─sdc1   8:33   0     1M  0 part
├─sdc2   8:34   0   513M  0 part
├─sdc3   8:35   0     2G  0 part [SWAP]
├─sdc4   8:36   0     2G  0 part
└─sdc5   8:37   0   927G  0 part
sdd      8:48   1  29,3G  0 disk
├─sdd1   8:49   1   4,7G  0 part /cdrom
├─sdd2   8:50   1   4,9M  0 part
├─sdd3   8:51   1   300K  0 part
└─sdd4   8:52   1  24,6G  0 part /var/crash
                                 /var/log

[SimonBard]

Why are they creating a Create EFI boot partition in the docu instead of a Create EFI boot pool?

[ahesford]

A couple of us use mdraid to create a software array compatible that an EFI firmware can still read. If you prefer to stay away from mdraid, you can either

• Pick one of the EFI system partitions on one of your disks and just use that; or
• Adapt the contrib script that syncs EFI system partitions to your needs.

[SimonBard]

Many thanks! So that means:

1. I format the discs
2. Make an EFI System Partition on each disk:

ESP_DISKS="/dev/sda /dev/sdb /dev/sdc"
for disk in $ESP_DISKS; do
    sgdisk -n "1:1m:+512m" -t "1:ef00" "$disk"
done

3. Create the mdraid array:

(install mdadm via "apt install mdadm" if not installed)

mdadm --create --verbose --level 1 --metadata 1.0 \
    --homehost any --raid-devices 3 /dev/md/esp \
    /dev/sda1 /dev/sdb1 /dev/sdc1
mdadm --assemble --scan
mdadm --detail --scan >> /etc/mdadm.conf

5. Format the array as vfat, create an fstab entry, and mount:

mkfs.vfat -F32 /dev/md/esp

cat << EOF >> /etc/fstab
/dev/md/esp /boot/efi vfat defaults 0 0
EOF

mkdir -p /boot/efi
mount /boot/efi

6. Set BOOT_DISK to /boot/efi

export BOOT_DISK="/boot/efi"
export BOOT_PART="1"
export BOOT_DEVICE="${BOOT_DISK}${BOOT_PART}"

7. Create 2nd partition on each disc:

ESP_DISKS="/dev/sda /dev/sdb /dev/sdc"
for disk in $ESP_DISKS; do
    sgdisk -n "2:0:-10m" -t "1:bf00" "$disk"
done

8. Create zpool

zpool create rpool mirror sda sdb cache sdc

9. Copy my backup of rpool from USB-drive to the rpool created in step 7.

[SimonBard]

I already have an error at step 3:

 mdadm --create --verbose --level 1 --metadata 1.0 \
>     --homehost any --raid-devices 3 /dev/md/esp \
>     /dev/sda1 /dev/sdb1 /dev/sdc1
mdadm: /dev/sda1 appears to contain an ext2fs file system
       size=3907016704K  mtime=Thu Jun  6 06:14:09 2024
mdadm: /dev/sdb1 appears to contain an ext2fs file system
       size=3907008512K  mtime=Thu Jun  6 06:15:42 2024
mdadm: size set to 524224K
Continue creating array? y
Continue creating array? (y/n) y
mdadm: array /dev/md/esp started.
root@ubuntu:~# mdadm --assemble --scan

mdadm: No arrays found in config file or automatically

[SimonBard]

Output of fdisk is:

Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EZAZ-00S
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1C21AC50-B283-6947-9F3C-E8C98417095E

Device     Start     End Sectors  Size Type
/dev/sda1   2048 1050623 1048576  512M EFI System


Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68W
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9C2649AB-F4CB-0F4D-94D0-A09D73893AD8

Device     Start     End Sectors  Size Type
/dev/sdb1   2048 1050623 1048576  512M EFI System


Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 860
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: CA08197F-FC09-4406-8464-1FA55B2CCF2A

Device     Start     End Sectors  Size Type
/dev/sdc1   2048 1050623 1048576  512M EFI System


Disk /dev/sdd: 29.3 GiB, 31457280000 bytes, 61440000 sectors
Disk model: USB3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F45E2FA1-C5A6-4D79-876F-C8245AF921E0

Device       Start      End  Sectors  Size Type
/dev/sdd1       64  9828451  9828388  4.7G Microsoft basic data
/dev/sdd2  9828452  9838519    10068  4.9M EFI System
/dev/sdd3  9838520  9839119      600  300K Microsoft basic data
/dev/sdd4  9842688 61439936 51597249 24.6G Linux filesystem


Disk /dev/loop8: 53.26 MiB, 55844864 bytes, 109072 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop9: 91.69 MiB, 96141312 bytes, 187776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop10: 452 KiB, 462848 bytes, 904 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@ubuntu:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0     3G  1 loop /rofs
loop1    7:1    0     4K  1 loop /snap/bare/5
loop2    7:2    0  73.9M  1 loop /snap/core22/858
loop3    7:3    0 237.2M  1 loop /snap/firefox/2987
loop4    7:4    0  63.4M  1 loop /snap/core20/1974
loop5    7:5    0  12.3M  1 loop /snap/snap-store/959
loop6    7:6    0 349.7M  1 loop /snap/gnome-3-38-2004/143
loop7    7:7    0 485.5M  1 loop /snap/gnome-42-2204/120
loop8    7:8    0  53.3M  1 loop /snap/snapd/19457
loop9    7:9    0  91.7M  1 loop /snap/gtk-common-themes/1535
loop10   7:10   0   452K  1 loop /snap/snapd-desktop-integration/83
sda      8:0    0   3.6T  0 disk
└─sda1   8:1    0   512M  0 part
sdb      8:16   0   3.6T  0 disk
└─sdb1   8:17   0   512M  0 part
sdc      8:32   0 931.5G  0 disk
└─sdc1   8:33   0   512M  0 part
sdd      8:48   1  29.3G  0 disk
├─sdd1   8:49   1   4.7G  0 part /cdrom
├─sdd2   8:50   1   4.9M  0 part
├─sdd3   8:51   1   300K  0 part
└─sdd4   8:52   1  24.6G  0 part /var/crash
                                 /var/log

[SimonBard]

And in step 7:

root@ubuntu:~# for disk in $ESP_DISKS; do
>     sgdisk -n ":0:-10m" -t "1::bf00" "$disk"
> done
Information: Moved requested sector from 7814016654 to 7814014976 in
order to align on 2048-sector boundaries.
Could not change partition 1's type code to !
Error encountered; not saving changes.
Information: Moved requested sector from 7814016654 to 7814014976 in
order to align on 2048-sector boundaries.
Could not change partition 1's type code to !
Error encountered; not saving changes.
Information: Moved requested sector from 1953504654 to 1953503232 in
order to align on 2048-sector boundaries.
Could not change partition 1's type code to !
Error encountered; not saving changes.

[no-usernames-left]

If you already created the pool, destroy it.

Wipe both sda and sdb (sgdisk --zap-all should be sufficient).

On both sda and sdb create two partitions, the first being 512 MB (or whatever) of type ef00 and the second being the rest of the disk of type bf00.

Use mdadm to mirror sda1 and sdb1 with the metadata argument that puts the metadata at the end of the partition.

Create the ZFS pool specifying sda2 and sdb2 as the mirror devices with sdc as a cache device.

[no-usernames-left]

It looks like you have an ESP on your cache drive, which is probably a waste of space.

Also you have given sda and sdb device names for ZFS, which can cause problems as which device gets which identifier at boot time depends on several factors; I recommend using the UUIDs.

[SimonBard]

Try this instead:
However, a simpler idea is just to install ZBM into sda1 and sdb1 separately, adding each one as a separate boot entry using efibootvars. Then if the first disk fails, the EFI BIOS will simply go down the list and boot from the other one.

I am trying this and followed your instructions. How do I now install ZBM?

I tried:

~# ls -lA /dev/disk/by-partuuid/ | grep -E sd[ab]1
lrwxrwxrwx 1 root root 10 Jun  8 16:54 5942e75e-5842-43cc-9ed1-bdacd7d2fdac -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun  8 16:54 8b39c805-4b4e-4e23-851b-759fa00791e3 -> ../../sda1

root@ubuntu:~# mkfs.vfat -F32 /dev/disk/by-partuuid/8b39c805-4b4e-4e23-851b-759fa00791e3
mkfs.fat 4.2 (2021-01-31)

root@ubuntu:~# cat << EOF >> /etc/fstab
> $( blkid | grep /dev/disk/by-partuuid/8b39c805-4b4e-4e23-851b-759fa00791e3 | cut -d ' ' -f 2 ) /boot/efi vfat defaults 0 0
> EOF

root@ubuntu:~# mkdir -p /boot/efi
root@ubuntu:~# mount /boot/efi
mount: vfat: der Einhängepunkt ist nicht vorhanden.

As you see, it fails. I do not see my mistake. Please help me.

[no-usernames-left]

German is a language mainly used in only four countries (one of which many people haven't even heard of), so try to get your output in English when seeking help so that the people trying to help you in their spare time don't have to go to extra effort to do so.

You have pasted the steps taken to create /etc/fstab, but not its actual contents. Can you please provide the exact contents of it?

You are using complex expressions which seem to be unaligned with your skill level. I recommend hand-creating the small number of files you need in nano etc; as one example, the layout of the fstab file is well documented (man fstab or here).

It says the mount point isn't available. It sounds like you should investigate why that is the case.

If you are going to be using mdraid for your ESP, you should be operating on the md device and never the underlying devices. Did you change your mind?

[ahesford]

@no-usernames-left watch yourself. If you see a message in a foreign language that you don't understand, you can either ignore it, or you can feed it into a translator.

[no-usernames-left]

I was simply suggesting the user switch the language to English for that session; the more effort volunteers have to put into helping, the fewer problems they will have time to solve, and perhaps the less willing they will be to engage in the first place.

I will cease replying in this discussion.

[SimonBard]

I am now stuck at the point:
https://docs.zfsbootmenu.org/en/v2.3.x/guides/ubuntu/uefi.html#create-a-vfat-filesystem
It gives me:
mkfs.vfat: unable to open /dev/sdc1: Device or resource busy

[SimonBard]

Its not mounted, by the way. @ahesford

 df -h
Filesystem         Size  Used Avail Use% Mounted on
rpool/ROOT/ubuntu  3.6T  406M  3.6T   1% /
rpool/home         3.6T  128K  3.6T   1% /home

[ahesford]

Why are you pinging me? I have no idea what's going wrong with your procedure.

[SimonBard]

Why are you pinging me? I have no idea what's going wrong with your procedure.

Sorry. I am following the docu and got the error. Thought there might be something missing.
Like unmounting or exiting from the chroot to liveusb system or so.

[SimonBard]

Probably there is a chance to create a vfat also via gparted or so.

[SimonBard]

Why are you pinging me? I have no idea what's going wrong with your procedure.

And it probably has to do with mdadm and you suggested this. So I thought you might give some hints here.