KVM Revisited Part I - Multiple VLANs

Posted on Tue 15 September 2015 in misc

I've decided to rebuild my KVM lab on OpenSUSE 13.2 and to try and finish documenting the process of a fully functional KVM lab on multiple VLANs.

Install KVM with yast2

I used the "Install Hypervisor and Tools" in yast2 to install KVM.  I also allowed it to create the bridge (br0)

Access with virt-manager

This is the same as the steps I outlined in my previous post  http://pyn00b.blogspot.co.uk/2014/01/opensuse-kvm-getting-started-notes.html - just make sure to add yourself to the libvirt group and to restart the libvirtd.service

Multiple VLANs

The objective here is to have a single KVM host with a single Ethernet NIC to have multiple and routable segregated subnets. This is achieved by Bridging and VLAN tagging. We do this so that we can use the existing network infrastructure rather than using the KVM host to do any NATing or using KVM virtual networks.

Our solution is to have a single NIC, to which we create a VLAN interface, a Bridge is created from the VLAN interface and KVM attaches virtual NICs to the Bridge. The bridge is specified when creating a host and is therefore attached to the VLAN. This can be repeated multiple times for any number of VLANs. For example, a hierarchy of network devices might look like this:

Physical Nic (enp3s0) -> VLAN interface (enp3s0.69) -> Bridge (br69) -> vnet0

Physical Nic (enp3s0) -> VLAN interface (enp3s0.69) -> Bridge (br69) -> vnet1

Physical Nic (enp3s0) -> VLAN interface (enp3s0.101) -> Bridge (br101) -> vnet2

So in summary, for each new VLAN, a new VLAN interface is created with a VLAN tag, that interface is then Bridged and any number of KVM hosts attach their virtual NIC to the bridge and their traffic is VLAN tagged transparently.

Setting up VLAN interfaces Let's start by creating an virtual interface that will tag traffic for VLAN 69.  We do so by creating the file /etc/sysconfig/network/ifcfg-enp3s0.69

BOOTPROTO='none'
STARTMODE='auto'
ETHERDEVICE='enp3s0'

Notice that by using the format ifcfg-enp3s0.VLANID sets the VLAN tag to 69 for us.  You can use VLAN_ID= as an option instead if you want to.  (see man ifcfg-vlan)

Also worth noting is that I have set BOOTPROTO='none' rather than 'static' and using IPADDR=''.  This is somewhat in contradiction to man ifcfg which states "Do  not  use  (none) to  just  skip  the IP setup".  However I noticed that when yast2 creates the bridge during the KVM install it sets BOOTPROTO='none'.  In my testing either option had the same result.  I used 'none' for consistency.  The yast convention is also to create ifcfg-vlanX which I did not care for.

Setting up Bridges

We now need to created a bridge from our VLAN interface.  We do so by creating /etc/sysconfig/network/ifcfg-br69

BOOTPROTO='none'
BRIDGE='yes'
BRIDGE_FORWARDDELAY='0'
BRIDGE_PORTS='enp3s0.69'
BRIDGE_STP='off'
STARTMODE='auto'

Changes to sysctl

Disable any traffic over the bridges getting sent to iptables ( http://wiki.libvirt.org/page/Net.bridge.bridge-nf-call_and_sysctl.conf ) I created  /etc/sysctl.d/kvm.conf and reloaded the config with sudo sysctl --system

net.ipv4.ip_forward = 0
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

Checking config

I initially had issues getting wicked to work properly with the VLANs (see https://bugzilla.suse.com/show_bug.cgi?id=939580) I upgraded wicked from OBS and it was better after that although using ifup was still unreliable.  systemctl restart network.service works much better for now.

It turns out wicked network manager has some nice features too.  I particularly like this one as an alternative to ifconfig:

$ sudo wicked ifstatus all
...
 br69            up
      link:     \#8, state up, mtu 1500
      type:     bridge
      config:   compat:suse:/etc/sysconfig/network/ifcfg-br69
...
 enp3s0.69       enslaved
      link:     \#11, state up, mtu 1500, master br69
      type:     vlan enp3s0\[69\], hwaddr 67:89:d5:65:1e:66
      config:   compat:suse:/etc/sysconfig/network/ifcfg-enp3s0.69

vnet1           device-unconfigured
      link:     \#12, state up, mtu 1500, master br69
      type:     tap, hwaddr fe:54:00:ac:0a:29

kvm vlan