Xen and vlans

During a big lab move at work a few months ago, we decided that our utility virt server needed VLAN support. The dhcp vm needed interfaces on three different networks and it seemed rather silly to add extra physical interface for the minimal traffic generated.

The first issue we encountered was the rather interesting bridging script installed by default. It does wonders for being able to bridge the primary interface and can be used to bridge multiple interface, but it fails entirely for VLAN interfaces. Best bet is just to disable any network scripts in /etc/xen/xend-config.sxp and let the os handle it. We’re using RHEL5, so we created the VLAN interface along with the bridge using the normal configs in /etc/sysconfig/network-scripts. Our naming scheme for the devices was ${DEVICE_TYPE}{$VLAN_NUMBER}.conf.

Example vlan123.conf:

DEVICE=vlan123
VLAN=yes
VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD
PHYSDEV=eth1
BOOTPROTO=static
ONBOOT=yes
BRIDGE=xenbr123

As you can see, eth1 is the physical interface connected to the switch port tagged with the vlans. We added the ‘VLAN_PLUS_VID_NO_PAD’ param to use the vlan${NUMBER} scheme. We aren’t bringing the interface up with an ip as it’s gonna be part of a bridge.

xenbr123.conf:

DEVICE=xenbr123
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes

This brings up our bridge without an ip address. The dom0 doesn’t need an ip on this VLAN, so no point in enabling it. To use the bridge in your domu’s, just specify the interface in the config file (or at creation time).

Example domu def file with multiple interfaces:

name = "example"
uuid = "62e4f71f-a46c-25f7-e947-f161aaad6f00"
maxmem = 512
memory = 512
vcpus = 1
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ ]
disk = [ "phy:/dev/vm/example,xvda,w" ]
vif = [ "mac=00:16:3e:4b:a5:46,bridge=xenbr123", "mac=00:16:3e:4b:a5:4a,bridge=xenbr456", mac=00:16:3e:4b:a5:47,bridge=xenbr789"]

The above was pretty straightforward, but after putting it in place we ran into a very odd issue. The vm’s couldn’t actually communicate via the vlan’d interfaces. After a bit of tcpdumping we discovered the default firewall was allowing outbound traffic on the bridge, but incoming was getting rejected. Easy fix was to add the following lines to /etc/sysconfig/iptables:


-A RH-Firewall-1-INPUT -i xenbr+ -j ACCEPT
-A RH-Firewall-1-INPUT -o xenbr+ -j ACCEPT

Note that this allows all traffic to pass on all xenbr devices. Since the dom0 doesn’t have an ip bound it’s not an issue in our configuration since the only traffic on the bridges are for the domu’s. If you do use the devices in your dom0, you’ll need to adjust the firewall accordingly or you’ll end up with a gaping hole in your security scheme.

3 thoughts on “Xen and vlans

  1. I have been trying to follow your comments on configuring the network and am a bit confused. You said you used the normal configs in /etc/sysconfig/network-scripts. but you called your configuration files vlanxx.conf. I must be missing something because I thought the standard files were all ifcfg-vlanxx. Where did you put the .conf files and how did you use them?

  2. // Andrew
    I think he meant /etc/sysconfig/network-scripts/ifcfg-vlan123 and /etc/sysconfig/network-scripts/ifcfg-xenbr123. That’s a common mistake for experienced users. The problem is that the one who knows, rarely verifies that on a live system. So that’s just what went out of his mind…

    Or here is another idea: you can adopt source_config function in /etc/sysconfig/network-scripts/network-functions to use any naming scheme you like. But I wouldn’t do that 😉

  3. If anyone knows how to dump the interface traffic *including* the vlan header – this would be terrific info to help with debugging things when “stuff doesn’t work”…

Leave a Reply