Bhyve Networking Options

Once in a while people on the #bhyve IRC channel on freenode ask questions about bhyve networking configuration, i.e. how to configure things to let a VM have network access.

There are at least 3 ways to do that (that I'm aware of, maybe there are more):

• Bridged networking
• NAT
• NIC Passthrough

I'll try to go over each of those and describe how things work in each scheme.

Common configuration

Common things for all the setups: I'm running FreeBSD 12-CURRENT amd64, I'm having two NICs (re0 and re1), both connected to a home router.

Bridged Networking

Bridged networking, just like name suggests, means bridging together VM interfaces and the uplink interface, putting those in the same L2 segment.

Configuration is relatively straight-forward. Let's start from a completely fresh configuration, where we don't even have re1 (uplink) configured:

kloomba# ifconfig re1
re1: flags=8802 metric 0 mtu 1500
        options=8209b
        ether 18:a6:f7:01:66:52
        nd6 options=29
        media: Ethernet autoselect (100baseTX )
        status: active
kloomba#

Now let's create a bridge named brextand add re1 to it. Also, we'll run dhclient on it to obtain an IP address (in my setup it comes from a DHCP server running on my home router):

kloomba# ifconfig bridge create name brext                                                                                                                                                
kloomba# ifconfig brext addm re1
kloomba# ifconfig brext up
kloomba# ifconfig re1 up
kloomba# dhclient brext

As a result we have an IP address assigned to the brext bridge:

brext: flags=8843 metric 0 mtu 1500
        ether 02:29:bb:66:56:01
        inet 192.168.87.46 netmask 0xffffff00 broadcast 192.168.87.255 
        nd6 options=1
        groups: bridge 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re1 flags=143
                ifmaxaddr 0 port 2 priority 128 path cost 200000

Now we need to create a tap (that will be tap1 in my case) interface for VM and boot it up:

kloomba# ifconfig tap create up
kloomba# ifconfig brext addm tap1

And boot a VM like in a way you like, for example:

bhyve -c 2 -m 4G -w -H \
        -s 0,hostbridge \
        -s 3,ahci-cd,/home/novel/FreeBSD-11.0-CURRENT-amd64-20160217-r295683-disc1.iso \
        -s 5,virtio-net,tap1 \
        -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
        -s 30,xhci,tablet \
        -s 31,lpc -l com1,stdio \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        vm0

Now we can open up a VNC client and connect to this VM (I use vncviewer :0) and do the following:

vm# dhclient vtnet0

If things go well, you'll get an IP address from the same subnet as IP address on host's re1 and, obviously, it's served by the same DHCP server that serves the host's re1.

### host ###
kloomba# ifconfig brext
brext: flags=8843 metric 0 mtu 1500
        ether 02:29:bb:66:56:01
        inet 192.168.87.46 netmask 0xffffff00 broadcast 192.168.87.255
        nd6 options=1
        groups: bridge 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap1 flags=143
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: re1 flags=143
                ifmaxaddr 0 port 2 priority 128 path cost 200000
kloomba# 

### vm ###
root@vm0:~ # ifconfig vtnet0
vtnet0: flags=8943 metric 0 mtu 1500
        options=80028
        ether 00:a0:98:1b:c8:07
        inet 192.168.87.47 netmask 0xffffff00 broadcast 192.168.87.255
        nd6 options=29
        media: Ethernet 10Gbase-T 
        status: active
root@vm0:~ # 

To understand a little better what's going on here, let's run ping on a VM:

vm# ping 8.8.8.8

... and check how it looks like on the host:

kloomba# tcpdump -qni brext -c2 -e host 8.8.8.8 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on brext, link-type EN10MB (Ethernet), capture size 65535 bytes
16:09:44.755083 00:a0:98:1b:c8:07 > 40:4a:03:76:de:1d, IPv4, length 98: 192.168.87.47 > 8.8.8.8: ICMP echo request, id 25603, seq 4, length 64
16:09:44.783499 40:4a:03:76:de:1d > 00:a0:98:1b:c8:07, IPv4, length 98: 8.8.8.8 > 192.168.87.47: ICMP echo reply, id 25603, seq 4, length 64
2 packets captured
6 packets received by filter
0 packets dropped by kernel
kloomba# 

What we can see here? Packets from our VM are leaving the host with the IP address it has on vtnet0, MAC address is also vtnet0's MAC. BTW, 40:4a:03:76:de:1d is MAC of my router.

That is it, it works, does not need stuff like firewalls or routing configuration to work, so this approach is relatively easy. There are downsides of that, however. It's pretty common that the router your PC is connected to is configured to only pass only single MAC per networking port, or maybe even only a whitelisted MAC (that's quite common in office environments) or if your home router does not support that. In that case you'll have to go with the NAT approach that I'll describe next.

NAT Networking

Let's assume we're starting from scratch and don't have that brext bridge we created in the previous section. And we're again starting with creation of the new bridge:

kloomba# ifconfig bridge create name brnat up
kloomba# ifconfig tap create up
kloomba# ifconfig brnat addm tap1
brnat: flags=8843 metric 0 mtu 1500
        ether 02:29:bb:66:56:01
        nd6 options=1
        groups: bridge 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap1 flags=143
                ifmaxaddr 0 port 10 priority 128 path cost 55
kloomba# 

As we can see, we no longer have our uplink interface re1 in the bridge. Now let's start a VM, command will be exactly the same as in the previous section, so I won't repeat it here.

Now, if we go to the VM and try to do dhclient vtnet0 nothing happens, because there are no DHCP server reachable from this VM. It's a good time to decide what IP range we'll use for our VM(s). Let's go with something like 10.0.0.0/24. Let's configure pf to do NATing for us. Basic /etc/pf.conf for this purpose might look like this:

ext_if="re1"

virt_net="10.0.0.0/24"

scrub all

nat on $ext_if from $virt_net to any -> ($ext_if)

pass log all

What we're doing here? For packets coming from $virt_net (our VMs range) we're translating its source address from 10.0.0.0/24 internal net to address of our external interface (re1). Now we can load the rules, enable pf and check if that works.

kloomba# pfctl -f /etc/pf.conf
kloomba# pfctl -e
pfctl: pf already enabled
kloomba# 

We also need to assign a proper IP address to our bridge:

kloomba# ifconfig brnat inet 10.0.0.1/24

Also, it's a good time to ensure that IP forwarding is enabled on the host: sysctl net.inet.ip.forwarding=1.

Now VM is expected to get connectivity if we manually assign an IP address to it:

vm0# ifconfig vtnet0 inet 10.0.0.2/24 up
vm0# route add default 10.0.0.1

Now things should work and we can actually see what's going on with the packets. Let's start ping again in our VM: ping 8.8.8.8 and tcpdump on interfaces on the host. Let's start with brnat:

kloomba# tcpdump -ni brnat -c 2 -e host 8.8.8.8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on brnat, link-type EN10MB (Ethernet), capture size 65535 bytes
17:28:36.101514 00:a0:98:1b:c8:07 > 02:29:bb:66:56:01, ethertype IPv4 (0x0800), length 98: 10.0.0.2 > 8.8.8.8: ICMP echo request, id 21507, seq 62, length 64
17:28:36.129840 02:29:bb:66:56:01 > 00:a0:98:1b:c8:07, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.0.2: ICMP echo reply, id 21507, seq 62, length 64
2 packets captured
2 packets received by filter
0 packets dropped by kernel
kloomba# 

And on our uplink interface re1:

kloomba# tcpdump -ni re1 -c 2 -e host 8.8.8.8                                                                                                                                                                       
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:31:37.898499 18:a6:f7:01:66:52 > 40:4a:03:76:de:1d, ethertype IPv4 (0x0800), length 98: 192.168.87.44 > 8.8.8.8: ICMP echo request, id 19102, seq 236, length 64
17:31:37.926781 40:4a:03:76:de:1d > 18:a6:f7:01:66:52, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 192.168.87.44: ICMP echo reply, id 19102, seq 236, length 64
2 packets captured
3 packets received by filter
0 packets dropped by kernel
kloomba# 

We can see that at this point no information about VM is exposed here (i.e. no VM subnet 10.0.0.0/24, no vtnet0 MACs etc); NAT works as expected.

As you can see, NAT networking is a little more complex configuration-wise. Though it's probably the most general solution, you don't have to rely on external routers configuration, bridging support in the hardware/drivers and so forth.

This configuration can be simplified though, a good step to it would be configuring DHCP server on brnat to serve IP addresses from our VM range. This could be done for example using the dns/dnsmasq tiny DHCP server.

NIC Passthrough

This is a somewhat fun way to setup networking because a) you'll need 1 (one) physical NIC per VM b) you'll need one more physical NIC for host if you want it to keep connected. This might be much better with SR-IOV though I've never tried SR-IOV cards on FreeBSD. Anyway, back to the point.

I'm going to passthrough re1, I'm running pciconf -l -v to find its PCI address:

re1@pci0:3:0:0:        class=0x020000 card=0x85051043 chip=0x816810ec rev=0x09 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

So I add pptdevs="3/0/0" to /boot/loader.conf and reboot. After reboot it looks this way:

ppt0@pci0:3:0:0:        class=0x020000 card=0x85051043 chip=0x816810ec rev=0x09 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

Now starting a VM like this:

bhyve -c 2 -m 1G -w -H -S \
        -s 0,hostbridge \
        -s 4,ahci-hd,/home/novel/img/uefi_fbsd.raw \
        -s 6,passthru,3/0/0 \
        -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
        -s 30,xhci,tablet \
        -s 31,lpc -l com1,stdio \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        vm0

If things go well (i.e.: host supports IOMMU, device supports passthrough, ...), we'll see this device in a VM exactly like it would appear in host:

re0@pci0:0:6:0: class=0x020000 card=0x85051043 chip=0x816810ec rev=0x09 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

At this point it can be used just like it was not a VM but another host connected to a network with its own NIC. Once can run dhclient re0 etc.

Further reading

• https://www.openbsd.org/faq/pf/nat.html
• https://wiki.freebsd.org/bhyve/pci_passthru
• https://www.freebsd.org/doc/handbook/config-network-setup.html
• https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

Update Oct, 17th, 2016: added pics.

Re-using arguments and environment variables for autotools' configure script

A thing that I'm going to describe is nowhere difficult, however it's not very obvious and actually I was pretty surprised that I only managed to think about it after so many years working with autotools.

From time to time I find myself needing to re-run the configure script because I either need to add or remove some option or just run it with sh -x for debugging purposes or with scan-build for example.

Usually I have a number of arguments and environment variables defined and I don't keep that in my head. I just hit ctrl-r and find the last execution of configure with all the stuff included and continue with that. But if I don't touch it for some time then it goes away from the HISTFILE. In that case I have to go the config.log file and copy/paste stuff from there. However, copy/pasting sucks and, moreover, it doesn't have proper quotes.

Apparently, there's a better solution for that: the config.status can provide everything that's needed.

$ ./config.status --config
'--with-hal' '--without-uml' '--without-polkit' 'CFLAGS=-g -I/usr/local/include' 'LDFLAGS=-L/usr/local/lib'
$

For example, to re-run all the same using scan-build just do:

$ eval scan-build ./configure `./config.status --config`

So it's pretty neat.

More info on the config.status scripts is available in its official documentation.

Changing partition type on FreeBSD

I had an ufs partition that I've used as a swap device. I was pretty satisfied with this setup until I decided to use it as a kernel dump device. I've ran dumpon on it and figured out it's not possible to use it as dump device. So I decided to change partition type to swap.

I executed bsdlabel and got such error: bsdlabel: Class not found. I've been googling and reading quite a lot of time, lots of such questions are not answered, but finally was able to find a solution. The solution is to use gpart tool. It appeared to be a handy tool and to change type of my ad0s2b partition I executed these commands:



Works great.

Using mutt with Gmail IMAP: complete guide

There are some guides around on the net, but I wasn't able to find a complete guide to make mutt as usable as gmail web interface, so I will try to put stuff together in one single post. I will also try to explain why each certain option is needed. Also, I don't pretend it to be the most optimal or correct way of configuration, but it works in practice. :-)

Basic IMAP configuration

So, the basic IMAP options follow:


set realname = "Your Name"
set imap_user = 'yourname@gmail.com'
set spoolfile = imaps://imap.gmail.com:993/INBOX
set smtp_url = "smtp://yourname@smtp.gmail.com:587/"
set folder = "imaps://imap.gmail.com:993"
set record="+[Gmail]/Sent Mail"
set postponed="+[Gmail]/Drafts"
set header_cache="~/.mutt/cache/headers"
set message_cachedir="~/.mutt/cache/bodies"
set certificate_file=~/.mutt/certificates


These lines are quite obvious and could be seen in a number of guides.

That's probably a minimal set of setting you need to be able to check your gmail mails using mutt. You might want to put your password to the config file in order to prevent mutt asking it all the time.

Ah, yeah, and I forgot one obvious thing that you probably know about anyway, the config file is ~/.muttrc.

sorting

Gmail was the first app I've used that sorted threads by the date of the last message in the thread. This is quite nice, because in the times when I used mutt with POP, I sorted thread by the first message, so if somebody sends a message to some ancient thread few screens of scrolling back I could easily miss it.

So, I've configured similar sorting in mutt, with the only difference that I prefer newer threads to be in the bottom, not on top.

So I've added the following lines to my ~/.muttrc:


set sort=threads
set sort_aux=last-date-received


The first line says that messages should be grouped in thread, the second one determines the order of the threads, so in our case we sort by newest message in the thread. You might want to consult muttrc(5) manpage for other sorting options.

Address book

When I've performed configuration steps described above, I've realized I really really really miss the built in address book of Gmail! The thing is that our company uses gmail to handle its mail and all the people who have an account automatically appear in the address book, which is quite handy and I've got used to this feature very very quickly.

So, in order to use the address book from mutt, query_command feature can be used. It allows to use an external application as address book.

There's a nice script which allows to be used as query_command in mutt, it's called goobook.

In order to install it, create some directory, e.g. ~/opt, cd into it and execute the following command:

svn checkout http://goobook.googlecode.com/svn/trunk/ goobook-read-only

Note: you will have to have subversion client installed.

Then cd to "~/opt/goobook-read-only" (or whatever dir you used), execute:

cp settings_example.py settings.py, then edit settings.py and fill in your gmail credentials. Now while you're in the working dir of goobook, check if it's configured correctly by executing something like:

./contactos.py e

Instead of 'e' you can type any letters which you sure will match somebody's name or address in your address book. If the script output something sensible then it works.

Now, teach the mutt to use this script by adding these lines to ~/.muttrc:


set query_command = '/home/user/opt/goobook-read-only/contactos.py "%s"'


Now, in mutt you can type 'Q' and type some query to make sure stuff is working correctly. If it works nice, then it means you can use address book when composing new messages. For example, if you hit 'm' to create a new message, in 'To:' prompt you can type some stuff and click 'ctrl-t' and mutt will complete stuff from the address book.

Yeah, one more thing about goobook. I had to insert empty 'print' right after arguments check in the contactos.py script as it seems that mutt use first line of the output to show in the status bar. So the scripts looks this way for me:


...

74 if __name__ == '__main__':
75 if len(sys.argv) < 2:
76 sys.exit(1)
77 print
78
79 try:

...


I haven't had time to look at mutt sources to investigate this problem, so I'm not sure if it's an expected behavior.

New messages info

What else... the other very important option is 'imap_check_subscribed'. To make mutt automatically add all the folders to mailboxes (so you can see a number of new/unread messages in each folder, new mail confirmation in different mailboxes etc), the following setting should be made:

set imap_check_subscribed=yes

If you're moving from Gmail or maybe some GUI MUA, you might want to see a sidebar with list of folders and mail counts. It's possible to implement in mutt as well, however it requires applying third-party patches, sidebar patch specifically. I don't provide instructions how to apply it as in FreeBSD it's as simple as building mail/mutt-devel port with "-DWITH_MUTT_SIDEBAR_PATCH" flag. Configuration is as simple as adding these lines to ~/.muttrc:


set sidebar_visible=yes
set sidebar_width=30


Summary

So, here's my complete ~/.muttrc file, only 50 lines long:


set realname = "yourname"
set imap_user = 'yourname@gmail.com'
set spoolfile = imaps://imap.gmail.com:993/INBOX
set smtp_url = "smtp://yourname@smtp.gmail.com:587/"
set folder = "imaps://imap.gmail.com:993"
set record="+[Gmail]/Sent Mail"
set postponed="+[Gmail]/Drafts"
set header_cache="~/.mutt/cache/headers"
set message_cachedir="~/.mutt/cache/bodies"
set certificate_file=~/.mutt/certificates
set imap_check_subscribed=yes
set move = no
set delete = yes
set edit_hdrs
set include
set reply_to
set abort_nosubject=no
set sig_dashes = no # don't append -- as I use only my name as a sig
set attribution = " %n wrote:\n" # I don't like lengthy attributions as well
alternates '(username|altusername)([-+].*)?@.*'
set hdr_format="%4C %Z %{%m/%d} %-15.15F (%4c) %s" # format of the index

ignore *
unignore from subject to cc date x-mailer x-url user-agent
hdr_order from to cc date subject x-mailer user-agent

set query_command = '~/opt/goobook-read-only/contactos.py "%s"'

set sort=threads
set sort_aux=last-date-received

# see beyond for hooks explanations
folder-hook . 'set from = "yourname@gmail.com"; \
set smtp_url= "smtp://yourname@smtp.gmail.com:587/"'

folder-hook my 'set from = "i@mywork.com"; \
set smtp_url = "smtp://i@mywork.com@smtp.gmail.com:587/";'

folder-hook foobar 'set from = "yo@example.org";'

# colors stuff outside
source ~/.mutt/themes/mytheme

# gpg
source ~/.mutt/gpg.rc
set crypt_autosign = yes

# sidebar
set sidebar_visible=yes
set sidebar_width=30


Few words about the hooks. I have two mail accounts, work and personal, both hosted at gmail. As I don't want to check both accounts, I've configured my work account to forward all the mail to my personal account and configured personal account to place all such mail into a separate folder.

So, the first default '.' hook just says that we generally use personal account. The second hook for 'mywork' says that when in 'mywork' folder, we change 'from' address to work address and use smtp from work account for sending (could be configured on gmail side as well to add working address for account, but I don't like this way for various reasons).

The third hook is for an alias at example.org which I use for some maillists, so I change 'from' address to it so maillist software allows me to write there. In this case I've added this address to gmail account, so no special 'smtp_url' magic is required.

So, it doesn't seem to be complex to configure mutt this way and I'm totally happy with such a configuration.

Update @ 24th, May 2010: I've wrote a replacement for goobook and switched to it instead: http://empt1e.blogspot.com/2010/05/introducing-goocaa-or-on-way-to-my.html