Two system pools on 1 disk – 1 or 2 /boot directories and WTB grub?
I have a legacy root-on-zfs install that was once made by installing to an ext4 / with /boot on a separate ext4 partition, and then moving the contents of / to a ZFS pool.
I had various *BSD installations on the same disk which never caused any trouble but now I’m considering to do a Devuan install on an unused partition. To be more precise: copy an already set-up install from another machine (that uses XFS) to a fresh pool on the partition in question.
I’ll want to stick to the approach with a /boot on its own partition and think I’d better not use a single /boot for both distros. So, two /boot partitions.
I’ve never done this before, not even without ZFS in the equation. How will this work with grub, esp. given they will be 2 different versions and each distro will at some point want to generate a new grub config? Will it figure out which boot partition goes with what pool based on matching the kernels (on the boot) with the kernel modules (on the pool), or will I have to verify the concordance manually (in grub.cfg) after each grub update? I see that grub.cfg just references the UUID of the partition holding /boot, plus the name of the pool as a kernel argument.
FWIW, the machine this would be on does have UEFI support but doesn’t seem to give me an option to boot off the EFI partition (into rEFInd for instance), possibly because the grub bootloader is also installed.
I wouldn’t have an issue with having to step through a few hoops to boot the secondary distro.
It turns out this is possible but takes a bit of work.
Grub itself will not detect the second Linux install if it doesn’t share the same /boot directory (or partition). I can remember that was an issue too for detecting the BSD installs I used to have next to a Linux install.
The good news is that this means grub can be run on each install without risk of messing up the other, as long as it doesn’t get installed to the MBR bootsector of course.
The tricky part is getting rEFInd to detect the different kernels each distro may have installed, and also to launch the selected initrd with the correct parameters so it will load the correct ZFS pool. The easiest way I know to achieve this for 1 distro is to use grub, and ubuntu/grubx64.efi. As it turns out, this loader has the path to grub.cfg hardcoded; it has to be /EFI/ubuntu/grub.cfg so copying it to an additional directory won’t solve anything the way I hoped.
Fortunately Debian (and thus also Devuan) has its own version of the loader that expects grub.cfg in /EFI/debian/grub.cfg, so that solved the problem for me. (NB: the Devuan version created a directory /EFI/$HOSTNAME so I had to rename it.)
My problem isn’t entirely solved. I can currently boot into my 2nd install via a VM with rawdisk access to the EFI partition and that install’s /boot and ZFS pool partitions. My laptop doesn’t give me an option to bring up a boot manager at startup; it always boots straight into grub which I have installed on the MBR bootsector. I added a grub stanza to find and start rEFInd off the EFI partition but that doesn’t work yet – is it even possible to do that?