An upcoming release of libvirt, 1.2.8 that should be released early September, will include an initial support of managing ZFS volumes.
That means that it's possible to boot VMs and use ZFS volumes as disks. Additionally, it allows to control volumes using the libvirt API. Currently, supported operations are:
- list volumes in a pool
- create and delete volumes
- upload and download volumes
It's not possible to create and delete pools yet, hope to implement that in the next release.
Defining a pool
Assume we have some pools and want to use one of them in libvirt:
# zpool list NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT filepool 1,98G 56,5K 1,98G 0% - 0% 1.00x ONLINE - test 186G 7,81G 178G 0% - 4% 1.00x ONLINE -
Let's take filepool and define it with libvirt. This could be done using this virsh command:
virsh # pool-define-as --name zfsfilepool --source-name filepool --type zfs Pool zfsfilepool defined virsh # pool-start zfsfilepool Pool zfsfilepool started virsh # pool-info zfsfilepool Name: zfsfilepool UUID: 5d1a33a9-d8b5-43d8-bebe-c585e9450176 State: running Persistent: yes Autostart: no Capacity: 1,98 GiB Allocation: 56,50 KiB Available: 1,98 GiB virsh #
As you can see, we specify a type of the pool, its source name, such as seen in zpool list output and a name for it in libvirt. We also need to start it using the pool-start command.
Managing volumes
Let's create a couple of volumes in our new pool.
virsh # vol-create-as --pool zfsfilepool --name vol1 --capacity 1G Vol vol1 created virsh # vol-create-as --pool zfsfilepool --name vol2 --capacity 700M Vol vol2 created virsh # vol-list zfsfilepool Name Path ------------------------------------------------------------------------------ vol1 /dev/zvol/filepool/vol1 vol2 /dev/zvol/filepool/vol2 virsh #
Dropping a volume is also easy:
virsh # vol-delete --pool zfsfilepool vol2 Vol vol2 deleted
Uploading and downloading data
Let's upload an image to our new volume:
virsh # vol-upload --pool zfsfilepool --vol vol1 --file /home/novel/FreeBSD-10.0-RELEASE-amd64-memstick.img
... and download
virsh # vol-download --pool zfsfilepool --vol vol1 --file /home/novel/zfsfilepool_vol1.img
Note: if you would check e.g. md5 sum of the downloaded files, the result would be different as downloaded file will be of the same size as a volume. However, if you trim zeros, it'll be the same.
$ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c MD5 (zfsfilepool_vol1.img) = a77c3b434b01a57ec091826f81ebbb97 $ truncate -r FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img $ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c MD5 (zfsfilepool_vol1.img) = e8e7cbd41b80457957bd7981452ecf5c $
Booting a VM from volume
Finally got to the most important part. In use a volume as disk device for VM 'devices' section of the domain XML should be updated with something like this:
<disk type='volume' device='disk'> <source pool='zfsfilepool' volume='vol1'/> <target dev='vdb' bus='virtio'/> </disk>
Few notes
Note #1: this code is just a few weeks old, so quite likely there are some rough edges. Feel free to report problems to novel%freebsd.org if you spot any problems.
Note #2: this code is FreeBSD-only for now. However, it should not be hard to make it work on Linux with zfsonlinux.org. Its developers were kind enough to add some useful missing flags in some of the CLI tools. However, these changes are not available in any released version so far. There are some more minor differences between zfs on Linux and FreeBSD, but that should not be hard to address. I was planning to get to it as soon as a new version of zfs on linux with the necessary flags is available. However, if you are interested in that and ready to help with testing -- feel free to poke me so it could be done sooner.
Thanks for adding zfs support for libvirt! I'm waiting for the Linux support, since all my VM hosts use that.
ReplyDeleteAlso in my configuration I have the zvols for the vms inside a filesystem hierarchy:
/rpool/DATA/vm/vm1
/rpool/DATA/vm/vm2
...
Any chance of adding support for such a configuration?
Regards,
Markus
Markus, could you please email me a bit more details about your setup to bogorodskiy-at-gmail-dot-com?
DeleteIs this already implemented?
DeleteI can't find documentation about it and would like to have my zvols in an existing zpool (zpool/virtualisation/zvols).
Regards,
Max
Okay. Just gave it a try and it worked. For anybody else landing on this page:
Delete# virsh pool-define-as --name zpool --source-name zpool/virtualisation/zvols --type zfs
# virsh pool-start zpool
# virsh vol-create-as --pool zpool --name test1 --capacity 1G
# virsh vol-list zpool
Name Path
------------------------------------------------------------------------------
test1 /dev/zvol/zpool/virtualisation/zvols/test1
Hi, do you know the whether zfsonlinux is ready yet for making this feature work on Linux hosts?
ReplyDeleteActually, it should be fine as of 0.6.4 or newever.
Deletelibvirt needs a minor patch to make things work on linux: http://people.freebsd.org/~novel/misc/libvirt_linux_zfs_002.diff
I'm planning to add a run-time detection for that, so the code works both on FreeBSD and Linux without extra patches.
When I try to define the zfs pool I get: missing backend for pool type 11 (zfs)
ReplyDeleteMy libvirt version is 1.3.4, OS is Arch
If you run "virsh -V", can you see "ZFS" in supported store list? If no, what were the args passed to "./configure"?
DeleteI'm using the default arch package.
ReplyDeleteAs seen here on line 105 in https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/libvirt for 1.3.4
there are no flags for zfs.
I'll rebuild it with the ArchBuildSystem, which flags i have to set?
Thanks for the help.
PS: Decoupled my Google+ Acc
Flag needed for configure is "--with-storage-zfs". Also, it's required to have zfs tools installed (i.e. zfs and zpool) so configure script could find them.
ReplyDeleteThis is working perfectly for me on Ubuntu 16.04 LTS (xenial)
ReplyDeleteHi, thank you for your writeup! I have a problem that I posted to libvirt-users (https://www.redhat.com/archives/libvirt-users/2017-April/msg00016.html) and got no reply. Maybe there's somebody listening here to help me?
ReplyDelete---
Hi,
I’m new here so apologies if this has been answered before.
I have a box that uses ZFS for everything (ubuntu 17.04) and I want to
create a libvirt pool on that. My ZFS pool is named „big"
So i do:
> zfs create big/zpool
> virsh pool-define-as --name zpool --source-name big/zpool --type zfs
> virsh pool-start zpool
> virsh pool-autostart zpool
> virsh pool-list
> virsh vol-create-as --pool zpool --name test1 --capacity 1G
> virsh vol-list zpool
Everything seems to work (no error message, vol-list shows the created
volume, I can see the volume
via zfs list -t all). -BUT- I cannot use that volume via virt-manager
and after a short while it’s
no longer listed via virsh vol-list zpool. The very same thing works
as expected if I create a new zfs
pool which I hand into libvirt. So instead of creating a pool from
"big/zpool“ I create a pool names
„somepool“ on a free device and -voila- everything works.
Hope I did make myself clear?
Best regards,
thies
Hi, I've replied to the message on libvirt-users.
Delete