libvirt/libxl on FreeBSD

Few months ago FreeBSD Xen dom0 support was announced. There's even a guide available how to run it: http://wiki.xen.org/wiki/FreeBSD_Dom0.

I will not duplicate stuff described in that document, just suggest that if you're going to try it, it'd probably be better to use the port emulators/xen instead of compiling stuff manually from the git repo.. I'll just share some bits that probably could save some of your time.

X11 and Xen dom0

I wasn't able to make X11 work under dom0. When I startx with the x11/nvidia-driver enabled in xorg.conf, kernel panics. I tried to use an integrated Intel Haswell video, but it's not supported by x11-drivers/xf86-video-intel. It works with x11-driver/xf86-video-vesa, however, the vesa driver causes system lock up on shutdown that triggers fsck every time on the next boot and it's very annoying. Apparently, this behavior is the same even when not under Xen. I decided to stop wasting my time on trying to fix it and just started using it in a headless mode.

IOMMU

You should really not ignore the IOMMU requirement and check if your CPU supports that. If you boot Xen kernel and you don't have IOMMU support, it will fail to boot and you'll have to perform some boot loader tricks to disable Xen to boot your system (i.e. do unload xen and unset xen_kernel). Just google up your CPU name, e.g. 'i5-4690' and follow the link to ark.intel.com. Make sure that it lists VT-d as supported under the 'Advanced Technologies' section. Also, make sure it's enabled in BIOS as well.

UEFI

At the time of writing (May / June 2015), Xen doesn't work with the UEFI loader.

xl cannot allocate memory

You most likely will have to modify your /etc/login.conf to set memorylocked=unlimited for your login class, otherwise the xl tool will fail with some 'cannot allocate memory' error.

libvirt

It's very good that Xen provides the libxl toolkit. It should have been installed when you installed the emulators/xen port as a dependency. The actual port that installs it is sysutils/xen-tools. As the libvirt Xen driver supports libxl, there's not so much work required to make it work on FreeBSD. I made only a minor change to disable some Linux specific /proc checks inside libvirt to make it work on FreeBSD and pushed that to the 'master' branch of libvirt today.

If you want to test it, you'd need to checkout libvirt source code using git:

git clone git://libvirt.org/libvirt.git

and then run ./bootstrap. It will inform if it needs something that's not installed.

For my libxl test setup I configure libvirt this way:

./configure --without-polkit --with-libxl --without-xen --without-vmware --without-esx --without-bhyve CC=gcc48 CFLAGS=-I/usr/local/include LIBS=-L/usr/local/lib

The only really important part here is the '--with-libxl', other flags are more or less specific to my setup. After configure just run gmake and it should build fine. Now you can install everything and run the libvirtd daemon.

If everything went fine, you should be able to connect to it using:

virsh -c "xen://"

Now we can define some domains. Let's check these two examples:

The first one is for a simple pre-configured FreeBSD guest image. The second one defines CDROM device and hard disk devices. It's set to boot from CDROM to be able to install Linux. Both domains are configured to attach to the default libvirt network on the virbr0 bridge. Additionally, both domains support VNC.

You could get domain VNC display number using the vncdisplay command in virsh and then connect to a VM with your favorite VNC client.

I've been using this setup for a couple of days and it works fine. However, more testers are welcome, if you're using it and have some issues please drop me an email to novel@`uname -s`.org or poke me on twitter.