Tuesday, August 26, 2014

ZFS support in libvirt

An upcoming release of libvirt, 1.2.8 that should be released early September, will include an initial support of managing ZFS volumes.

That means that it's possible to boot VMs and use ZFS volumes as disks. Additionally, it allows to control volumes using the libvirt API. Currently, supported operations are:

  • list volumes in a pool
  • create and delete volumes
  • upload and download volumes

It's not possible to create and delete pools yet, hope to implement that in the next release.

Defining a pool

Assume we have some pools and want to use one of them in libvirt:

# zpool list
NAME       SIZE  ALLOC   FREE   FRAG  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
filepool  1,98G  56,5K  1,98G     0%         -     0%  1.00x  ONLINE  -
test       186G  7,81G   178G     0%         -     4%  1.00x  ONLINE  -

Let's take filepool and define it with libvirt. This could be done using this virsh command:

virsh # pool-define-as --name zfsfilepool --source-name filepool --type zfs
Pool zfsfilepool defined

virsh #  pool-start zfsfilepool
Pool zfsfilepool started

virsh # pool-info zfsfilepool
Name:           zfsfilepool
UUID:           5d1a33a9-d8b5-43d8-bebe-c585e9450176
State:          running
Persistent:     yes
Autostart:      no
Capacity:       1,98 GiB
Allocation:     56,50 KiB
Available:      1,98 GiB

virsh # 

As you can see, we specify a type of the pool, its source name, such as seen in zpool list output and a name for it in libvirt. We also need to start it using the pool-start command.

Managing volumes

Let's create a couple of volumes in our new pool.

virsh # vol-create-as --pool zfsfilepool --name vol1 --capacity 1G
Vol vol1 created

virsh # vol-create-as --pool zfsfilepool --name vol2 --capacity 700M
Vol vol2 created

virsh # vol-list zfsfilepool
 Name                 Path                                    
------------------------------------------------------------------------------
 vol1                 /dev/zvol/filepool/vol1                 
 vol2                 /dev/zvol/filepool/vol2                 

virsh #

Dropping a volume is also easy:

virsh # vol-delete --pool zfsfilepool vol2
Vol vol2 deleted

Uploading and downloading data

Let's upload an image to our new volume:

virsh # vol-upload --pool zfsfilepool --vol vol1 --file /home/novel/FreeBSD-10.0-RELEASE-amd64-memstick.img 

... and download

virsh # vol-download --pool zfsfilepool --vol vol1 --file /home/novel/zfsfilepool_vol1.img

Note: if you would check e.g. md5 sum of the downloaded files, the result would be different as downloaded file will be of the same size as a volume. However, if you trim zeros, it'll be the same.

$ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img 
MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c
MD5 (zfsfilepool_vol1.img) = a77c3b434b01a57ec091826f81ebbb97
$ truncate -r FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img
$  md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img             
MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c
MD5 (zfsfilepool_vol1.img) = e8e7cbd41b80457957bd7981452ecf5c
$

Booting a VM from volume

Finally got to the most important part. In use a volume as disk device for VM 'devices' section of the domain XML should be updated with something like this:

    <disk type='volume' device='disk'>
      <source pool='zfsfilepool' volume='vol1'/>
      <target dev='vdb' bus='virtio'/>
    </disk>

Few notes

Note #1: this code is just a few weeks old, so quite likely there are some rough edges. Feel free to report problems to novel%freebsd.org if you spot any problems.

Note #2: this code is FreeBSD-only for now. However, it should not be hard to make it work on Linux with zfsonlinux.org. Its developers were kind enough to add some useful missing flags in some of the CLI tools. However, these changes are not available in any released version so far. There are some more minor differences between zfs on Linux and FreeBSD, but that should not be hard to address. I was planning to get to it as soon as a new version of zfs on linux with the necessary flags is available. However, if you are interested in that and ready to help with testing -- feel free to poke me so it could be done sooner.


13 comments:

  1. Thanks for adding zfs support for libvirt! I'm waiting for the Linux support, since all my VM hosts use that.

    Also in my configuration I have the zvols for the vms inside a filesystem hierarchy:
    /rpool/DATA/vm/vm1
    /rpool/DATA/vm/vm2
    ...

    Any chance of adding support for such a configuration?

    Regards,
    Markus

    ReplyDelete
    Replies
    1. Markus, could you please email me a bit more details about your setup to bogorodskiy-at-gmail-dot-com?

      Delete
    2. Is this already implemented?
      I can't find documentation about it and would like to have my zvols in an existing zpool (zpool/virtualisation/zvols).

      Regards,
      Max

      Delete
    3. Okay. Just gave it a try and it worked. For anybody else landing on this page:

      # virsh pool-define-as --name zpool --source-name zpool/virtualisation/zvols --type zfs
      # virsh pool-start zpool
      # virsh vol-create-as --pool zpool --name test1 --capacity 1G
      # virsh vol-list zpool
      Name Path
      ------------------------------------------------------------------------------
      test1 /dev/zvol/zpool/virtualisation/zvols/test1

      Delete
  2. Hi, do you know the whether zfsonlinux is ready yet for making this feature work on Linux hosts?

    ReplyDelete
    Replies
    1. Actually, it should be fine as of 0.6.4 or newever.

      libvirt needs a minor patch to make things work on linux: http://people.freebsd.org/~novel/misc/libvirt_linux_zfs_002.diff

      I'm planning to add a run-time detection for that, so the code works both on FreeBSD and Linux without extra patches.

      Delete
  3. When I try to define the zfs pool I get: missing backend for pool type 11 (zfs)
    My libvirt version is 1.3.4, OS is Arch

    ReplyDelete
    Replies
    1. If you run "virsh -V", can you see "ZFS" in supported store list? If no, what were the args passed to "./configure"?

      Delete
  4. I'm using the default arch package.
    As seen here on line 105 in https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/libvirt for 1.3.4
    there are no flags for zfs.

    I'll rebuild it with the ArchBuildSystem, which flags i have to set?

    Thanks for the help.

    PS: Decoupled my Google+ Acc

    ReplyDelete
  5. Flag needed for configure is "--with-storage-zfs". Also, it's required to have zfs tools installed (i.e. zfs and zpool) so configure script could find them.

    ReplyDelete
  6. This is working perfectly for me on Ubuntu 16.04 LTS (xenial)

    ReplyDelete
  7. Hi, thank you for your writeup! I have a problem that I posted to libvirt-users (https://www.redhat.com/archives/libvirt-users/2017-April/msg00016.html) and got no reply. Maybe there's somebody listening here to help me?

    ---

    Hi,

    I’m new here so apologies if this has been answered before.

    I have a box that uses ZFS for everything (ubuntu 17.04) and I want to
    create a libvirt pool on that. My ZFS pool is named „big"

    So i do:

    > zfs create big/zpool
    > virsh pool-define-as --name zpool --source-name big/zpool --type zfs
    > virsh pool-start zpool
    > virsh pool-autostart zpool
    > virsh pool-list
    > virsh vol-create-as --pool zpool --name test1 --capacity 1G
    > virsh vol-list zpool

    Everything seems to work (no error message, vol-list shows the created
    volume, I can see the volume
    via zfs list -t all). -BUT- I cannot use that volume via virt-manager
    and after a short while it’s
    no longer listed via virsh vol-list zpool. The very same thing works
    as expected if I create a new zfs
    pool which I hand into libvirt. So instead of creating a pool from
    "big/zpool“ I create a pool names
    „somepool“ on a free device and -voila- everything works.

    Hope I did make myself clear?

    Best regards,
    thies

    ReplyDelete
    Replies
    1. Hi, I've replied to the message on libvirt-users.

      Delete