Trying to install grub-btrfs on Fedora 35

[SysGuides]

Hello,

I've installed snapper on Fedora 35 and tweaked it such that I can successfully roll back from a previous snapper snapshot. When I installed grub-btrfs, I noticed that I couldn't boot into a snapshot from grub-btrfs menu.

I'm getting the error
error: ../../grub-core/fs/btrfs.c:2041:file '/@/.snapshots/1/snapshot/boot/vmlinuz-5.14.xxx' not found

The reason I found this was because the grub-btrfs script appends '/@' to the beginning of the kernel/initrd path. I could boot into a snapshot if I could only delete '/@' at the beginning from the kernel/initrd path.

It would be really helpful if you could tell me what I need to do to get rid of the '/@' in the kernel path.

My Setup:

# btrfs subvolume list /
ID 256 gen 191 top level 5 path @
ID 257 gen 190 top level 5 path @home
ID 258 gen 195 top level 5 path @var
ID 263 gen 185 top level 258 path @var/lib/machines
ID 265 gen 187 top level 256 path .snapshots
ID 266 gen 177 top level 265 path .snapshots/1/snapshot
ID 267 gen 180 top level 265 path .snapshots/2/snapshot

# btrfs subvolume get-default /
ID 256 gen 196 top level 5 path @

# cat /etc/fstab
UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a /                       btrfs   subvol=@,compress=zstd:1 0 0
UUID=35B6-0728                            /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a /home                   btrfs   subvol=@home,compress=zstd:1 0 0
UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a /var                    btrfs   subvol=@var,compress=zstd:1 0 0
UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a /.snapshots             btrfs   subvol=@/.snapshots,compress=zstd:1 0 0

# grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-5.15.6-200.fc35.x86_64"
args="ro rhgb quiet {extra_cmdline}"
root="UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a"
initrd="/boot/initramfs-5.15.6-200.fc35.x86_64.img"
title="Fedora Linux (5.15.6-200.fc35.x86_64) 35 (Workstation Edition)"
id="693311b03223462c82428147bb88cfd9-5.15.6-200.fc35.x86_64"

cat /etc/default/grub-btrfs/config 
GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@")
GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")
GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"
GRUB_BTRFS_MKCONFIG=/usr/sbin/grub2-mkconfig
GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check

# tail -n18 /boot/grub2/grub-btrfs.cfg 
    menuentry '  vmlinuz-5.14.10-300.fc35.x86_64 & initramfs-5.14.10-300.fc35.x86_64.img' --class snapshots --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-snapshots-955aa367-ec05-4d42-8f98-f52ee2d64f9a' {
        if [ x$feature_all_video_module = xy ]; then
        insmod all_video
        fi
        set gfxpayload=keep
        insmod btrfs
        if [ x$feature_platform_search_hint = xy ]; then
            search --no-floppy --fs-uuid  --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  955aa367-ec05-4d42-8f98-f52ee2d64f9a
        else
            search --no-floppy --fs-uuid  --set=root 955aa367-ec05-4d42-8f98-f52ee2d64f9a
        fi
        echo 'Loading Snapshot: 2021-12-03 14:53:20 @/.snapshots/1/snapshot'
        echo 'Loading Kernel: vmlinuz-5.14.10-300.fc35.x86_64 ...'
        linux "/@/.snapshots/1/snapshot/boot/vmlinuz-5.14.10-300.fc35.x86_64" root=UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a rhgb quiet  rootflags=compress=zstd:1,subvol="@/.snapshots/1/snapshot"
        echo 'Loading Initramfs: initramfs-5.14.10-300.fc35.x86_64.img ...'
        initrd "/@/.snapshots/1/snapshot/boot/initramfs-5.14.10-300.fc35.x86_64.img"
    }
}

Thanks

[Antynea]

Hello,

grub-btrfs uses the following two commands to know the path of a snapshot.
Can you provide the return of :
btrfs sub list -sa /
and
grub2-mkrelpath /.snapshots/1/snapshot/

According to all the information you have provided (thank, this is helpful), /@ shouldn't be present.

[SysGuides]

Thanks for the reply. Following are my results:

# btrfs sub list -sa /
ID 266 gen 177 cgen 177 top level 265 otime 2021-12-03 14:53:20 path <FS_TREE>/@/.snapshots/1/snapshot
ID 267 gen 180 cgen 180 top level 265 otime 2021-12-03 14:54:23 path <FS_TREE>/@/.snapshots/2/snapshot

# grub2-mkrelpath /.snapshots/1/snapshot/
/@/.snapshots/1/snapshot

Please note, earlier, the path to kernel/initrd was like this:

# grubby --info=ALL
index=0
kernel="/@/boot/vmlinuz-5.15.6-200.fc35.x86_64"
args="ro rhgb quiet {extra_cmdline}"
root="UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a"
initrd="/@/boot/initramfs-5.15.6-200.fc35.x86_64.img"
title="Fedora Linux (5.15.6-200.fc35.x86_64) 35 (Workstation Edition)"
id="693311b03223462c82428147bb88cfd9-5.15.6-200.fc35.x86_64"
index=1
kernel="/@/boot/vmlinuz-5.14.10-300.fc35.x86_64"
args="ro rhgb quiet ${extra_cmdline}"
root="UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a"
initrd="/@/boot/initramfs-5.14.10-300.fc35.x86_64.img"
title="Fedora Linux (5.14.10-300.fc35.x86_64) 35 (Workstation Edition)"
id="693311b03223462c82428147bb88cfd9-5.14.10-300.fc35.x86_64"
index=2
kernel="/@/boot/vmlinuz-0-rescue-693311b03223462c82428147bb88cfd9"
args="ro rhgb quiet ${extra_cmdline}"
root="UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a"
initrd="/@/boot/initramfs-0-rescue-693311b03223462c82428147bb88cfd9.img"
title="Fedora Linux (0-rescue-693311b03223462c82428147bb88cfd9) 35 (Workstation Edition)"
id="693311b03223462c82428147bb88cfd9-0-rescue"

Because I was not able to rollback kernel updates, (1) I added this to /etc/default/grub file
SUSE_BTRFS_SNAPSHOT_BOOTING="true"

and (2) removed '/@' from the kernel/initrd path

# cat /boot/loader/entries/693311b03223462c82428147bb88cfd9-5.15.6-200.fc35.x86_64.conf 
title Fedora Linux (5.15.6-200.fc35.x86_64) 35 (Workstation Edition)
version 5.15.6-200.fc35.x86_64
linux /boot/vmlinuz-5.15.6-200.fc35.x86_64
initrd /boot/initramfs-5.15.6-200.fc35.x86_64.img
options root=UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a ro rhgb quiet {extra_cmdline}
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora

And finally (3) updated the file

# cat /boot/efi/EFI/fedora/grub.cfg 
set btrfs_relative_path="yes"
search --no-floppy --fs-uuid --set=dev 955aa367-ec05-4d42-8f98-f52ee2d64f9a
set prefix=($dev)/boot/grub2

Now, I can successfully rollback to the previously kernel with the command

# snapper ls
 # | Type   | Pre # | Date                                | User | Cleanup | Description                   | Userdata
---+--------+-------+-------------------------------------+------+---------+-------------------------------+---------
0  | single |       |                                     | root |         | current                       |         
1  | pre    |       | Friday 03 December 2021 02:53:20 PM | root | number  | /usr/bin/dnf -y update kernel |         
2  | post   |     1 | Friday 03 December 2021 02:54:23 PM | root | number  | /usr/bin/dnf -y update kernel |     

# snapper --ambit classic rollback 1

But the sad part is I cannot boot into snapshots from grub-btrfs menu

[Antynea]

Thanks.

1. ID 265 gen 187 top level 256 path .snapshots is a nested subvolume, its parent is ID 256 gen 191 top level 5 path @

2. # btrfs sub list -sa / and grub2-mkrelpath /.snapshots/1/snapshot/ confirms that .snapshots is a nested subvolume.

3. SUSE_BTRFS_SNAPSHOT_BOOTING="true" I didn't know, reading the sources of the Grub package available on OpenSUSE,
   I discovered what this parameter does:

# grub-mkconfig_lib file on fedora

make_system_path_relative_to_its_root ()
{
  if [ "x${SUSE_BTRFS_SNAPSHOT_BOOTING}" = "xtrue" ] ; then
  "${grub_mkrelpath}" -r "$1"
  else
  "${grub_mkrelpath}" "$1"
  fi
}

So, if the parameter is set to "true" grub2-mkrelpath is executed with the -r option.

4. Bingo:

# grub2-mkrelpath -r /.snapshots/1/snapshot/
/.snapshots/1/snapshot

But

# grub2-mkrelpath -r /@/.snapshots/1/snapshot
/@/.snapshots/1/snapshot

/@/.snapshots/1/snapshot is the path used by grub-btrfs:

# btrfs sub list -sa /
ID 266 gen 177 cgen 177 top level 265 otime 2021-12-03 14:53:20 path <FS_TREE>/@/.snapshots/1/snapshot <------ # @ is here
ID 267 gen 180 cgen 180 top level 265 otime 2021-12-03 14:54:23 path <FS_TREE>/@/.snapshots/2/snapshot

Supporting nested subvolumes is not easy.

What snapper users usually do is :
Delete the nested subvolume, and create a subvolume at the root of the fs.
fedora / Arch Linux

[SysGuides]

@Antynea, Thanks for reply

As far as I can recall, I've tried the step in the link you provided, and it doesn't work in Fedora 35. When you mount .snapshots subvolume as ID=5, it WILL rollback successfully, but when you try to boot from the snapshot from the script you provided, I get the error 'kernel not found'. I've noticed, when the .snapshot subvolume is mounted as ID=5, the /@/.snapshots/1/snapshot will not be mounted. But when you mount as a nested subvolume (snapper default), the /@/.snapshots/1/snapshot will be mounted. The only thing is I have to remove '/@' from the kernel path.

But, I'll try again. Perhaps I made some errors.

Also, in Fedora 35, the SUSE_BTRFS_SNAPSHOT_BOOTING="true" definition is like this

# cat /etc/grub.d/10_linux
 67 case x"$GRUB_FS" in
 68     xbtrfs)
 69         if [ "x${SUSE_BTRFS_SNAPSHOT_BOOTING}" = "xtrue" ]; then
 70         GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} \${extra_cmdline}"
 71         else
 72         rootsubvol="`make_system_path_relative_to_its_root /`"
 73         rootsubvol="${rootsubvol#/}"
 74         if [ "x${rootsubvol}" != x ]; then
 75             GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
 76         fi
 77         fi;;
 78     xzfs)
 79         rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
 80         bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
 81         LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs%/}"
 82         ;;
 83 esac

I'll give it another shot and see what happens.

Thanks

[Antynea]

Step by step:

1. delete snapper snapshots
   snapper -c root delete x-y # where "x" is first snapshot and "y" latest (1-3)

2. Delete .snapshots subvolume
   btrfs subvolume delete /.snapshots

3. Create the directory /.snapshots
mkdir /.snapshots

4. mount your root fs in /mnt:
   mount -o subvolid=5 /dev/disk/by-uuid/955aa367-ec05-4d42-8f98-f52ee2d64f9a /mnt

5. Create a @snapshots subvolume # You can use another name for the subvolume, but this one in general is recommended.
   btrfs sub create /mnt/@snapshots

6. Unmount /mnt
umount /mnt

7. Mount @snapshots in /.snapshots
mount -o subvol=@snapshots /dev/disk/by-uuid/955aa367-ec05-4d42-8f98-f52ee2d64f9a /.snpashots

8. Adds this line to your fstab:
   UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a /.snapshots btrfs subvol=@snapshots,compress=zstd:1 0 0
   This line is no longer useful, comment or delete it.
   UUID=955aa367-ec05-4d42-8f98-f52ee2d64f9a /.snapshots btrfs subvol=@/.snapshots,compress=zstd:1 0 0

Now when snapper creates a snapshot, it will be stored in an independent subvolume, and grub-btrfs should work..