davidnewall.com
bespoke business solutions

Booting FreeBSD from an LVM volume

David Newall
July 12, 2014

Why?

I consider Linux to be poorly engineered in many areas, particularly the horrendous design of it's memory management. Whatever were they thinking when they decided that the right way to deal with out of memory was to kill processes unrelated to the one that was seeking more memory than could be provided? This is actually a major problem. The SSH daemon on one of my important production servers, located in a data centre 1500KM away, was recently killed from this, preventing any acces until the machine could be restarted. I'm looking to switch to something more robust and less brittle.

I probably would prefer OpenBSD, because of it's record of excellent security, but all of my files are currently in various LVM2 volumes, and OpenBSD does not support that. FreeBSD does support LVM2, and as my friend, Greg Lehey, is something of a master in it, that is what I have chosen to try, at least for now. I'm sure this will provide much needed excuses to chat with him or even visit.

Installation

The first hurdle was how to install FreeBSD. My computer has a very satisfactory SSD with plenty of available space, but fully allocated to LVM2 (plus a recovery partition and another for some nasty software made in Redmond, USA, about which I deny knowing!) Obviously I want to put FreeBSD into a new LVM2-based volume, and that is not one of the standard recipes baked into the installer.

It was actually very easy to install. First, I created a logical volume for FreeBSD. I allocated 10GB. Then I downloaded the memstick image of FreeBSD 10.0-RELEASE, copied it to a USB memory stick, and booted the computer. I went through the normal installation process, up to the point of partitioning, and there chose the Shell option, which gave me a root shell. The installer explained that I had to manually partition the drives, create the file system(s), populate /tmp/bsdinstall_etc/fstab, and mount the file systems under /mnt. I loaded the geom_linux_lvm kernel module, formatted the logical volume that I had previously created, mounted it and created a minimal fstab.

The commands were:

# kldload geom_linux_lvm
# newfs /dev/linux_lvm/vg-FreeBSD
# mount /dev/linux_lvm/vg-FreeBSD /mnt
# echo "/dev/linux_lvm/thales-FreeBSD  /  ufs  rw,noatime  1 1" > /tmp/bsdinstall_etc/fstab
# exit

The balance of installation completed as normal, albeit I created a single file-system for everything rather than separate slices for root, swap, /boot, /home, /usr, /sys, /tmp and /var. I'm satisfied with a single filesystem, at least for now; I can always revise this at a later stage if I feel the need. Likewise, I'm currently not using a swap because I think I won't need one having 4GB of RAM. Again, I can revise that decision later, and if I do, I will use the existing swap volume that I created for Linux.

Configuring Grub2

The only difficult issue was working out how to make the machine boot. Popular wisdom strongly suggested that I couldn't; that I would need to boot from a FreeBSD partition (c.f. MS-DOS partition table.) This is not necessary. You can boot from an LVM2 logical volume, using grub2, which was the boot loader that I already was using.

Working out how took me many hours, searching the web for clues, then trial and error. The answer was actually very simple. Grub already knows how to load a FreeBSD kernel (likewise NetBSD and OpenBSD), including loading kernel modules. The final grub.cfg stanza was:

menuentry "FreeBSD" {
	insmod lvm
	insmod ufs2
	kfreebsd /boot/kernel/kernel
	kfreebsd_module_elf /boot/kernel/geom_linux_lvm.ko
	set kFreeBSD.vfs.root.mountfrom=ufs:/dev/linux_lvm/vg-FreeBSD
	set kFreeBSD.vfs.root.mountfrom.options=rw
}

ZFS instead of LVM2

I suppose life would have been simpler had I used ZFS when I setup the machine. It has very similar features to LVM2, works under Linux as well as FreeBSD, and booting FreeBSD from ZFS is better documented, not to mention (now) officially supported. It would have given me access to my Linux files from FreeBSD, and vice versa, for that matter. As it was, most of my Linux files were in ext4 filesystems, which is poorly supported on FreeBSD (e.g. read-only), so I created a new logical volume for use as a ZFS storage pool, which sounds like the wrong to do but does work, created new ZFS filesystems and copied the files from corresponding ext4 filesystems. This allows switching between FreeBSD and Linux as needed (but I hope I won't need to.)

Even had I setup ZFS from the first, I think I still would have needed LVM2 to contain virtual machines.

—end—