bhyve vs VirtualBox benchmarking, part2

"There are in order of increasing severity: lies, damn lies, statistics, and computer benchmarks",
man 8 diskinfo

I got some feedback to my previous post about benchmarking of bhyve and VirtualBox:

@rbogorodskiy Try the e1000 with vbox for networking, and a raw device for the i/o test. For bhyve, virtio-blk is best.

— bhyve dev (@bhyve_dev) October 17, 2016

So I decided to do some more tests and include e1000 for networking tests and try different drivers and image types for I/O tests.

Networking test

This is a little more extensive test than the one in my previous blog post, now it includes e1000 and virtio-net for both VirtualBox and bhyve. Setting for the test is still the same: iperf and bridged network mode. Commands remain the same, on VM side I run:

iperf -s

On the host, I run:

iperf -c $vmip

And calculate the average of 8 runs. Results:

And actual values (in Gbits/sec) are:

	VirtualBox	bhyve
e1000	1.42875	1.57375
virtio-net	0.43275	2.4

The most shocking part here is that in VirtualBox e1000 is more than 3x times faster than virtio-net. This seems a little strange, esp. considering that e1000 performance in bhyve is almost the same, but virtio-net in bhyve is approx. 1.5x times faster than e1000 (that's probably a very huge difference too, but at least it's expected virtio to be faster).

I/O testing

I decided to check things suggested in the tweet above and started with disk configuration. I've converted my image to the "fixed size" type image like this:

VBoxManage clonehd uefi_fbsd_20gigs.vdi uefi_fbsd_20gigs_fixed.vdi --variant Fixed

Then I conducted tests for the following configurations:

• VirtualBox + fixed size image
• VirtualBox + dynamic size image
• bhyve + virtio-blk
• bhyve + ahci-hd

The same image was used for all tests. Fixed size image was produced using the command specified above, raw image for bhyve was created from the VirtualBox image using the qemu-img(1) tool.

I started with bonnie++ test at results surprised me, to put it softly:

Here, we can see that bhyve with virtio-blk shows the best write speed (that is expected), but then shows the worst rewrite speed (which is a little surprising, but the gap is minimal) and shows the worst read speed (more than 2 times slower than VirtualBox; extremely surprising). After that I decided to take a few days break and then to try some different ways of benchmarking.

So, for read performance I used the diskinfo(8) tool this way:

diskinfo -tv ada0 | grep middle

and use the average of 16 runs. For writing, I used dd(1):

dd bs=1M count=2048 if=/dev/zero of=test conv=sync; sync; rm test

and also use the average of 16 runs. I got the following results:

And the numbers (all values are kbytes/sec):

	vbox (fixed size img)	vbox (dynamic img)	bhyve (ahci-hd)	bhyve (virtio-blk)
diskinfo	1232397	1528779	1296055	2647685
dd	113737	135088	113889	115924

Frankly, this didn't help to figure out state of things, because these results are kind of opposite to what bonnie++ showed: bhyve with virtio-blk shows very high read speed as demonstrated by diskinfo, 1.7x faster than VirtualBox. On the other hand, the numbers that diskinfo is showing are crazy: 2647 mbytes/sec. This feels more like RAM transfer rates, so it looks like some sort of caching is involved here.

As for the dd(1) test part, bhyve with virtio-blk is 16.5% slower than VirtualBox using dynmanic size image.

Conclusion

• For VirtualBox, if choosing between e1000 and virtio-net, e1000 definitely provides better performance over virtio-net, at least on FreeBSD hosts with FreeBSD guests
• virtio-net in bhyve is approx. 1.5x faster than VirtualBox with e1000
• I'll refrain from comments on I/O tests.

Further Reading

• Raw output of commands used for testing
• Previous round of benchmarks