If you, like me, are using Xen with a HVM guest machine (i.e. with an hardware assisted virtualization) you may have experienced horrible network performances for the domU.
I tested the performance with the netperf tool. First result where embarrassing, about 2.5 Mbps. The problem is very common, and it could be someway related to the host NIC. I have run the tests over an HP DL 380 G5, with two Broadcom 5708 on board (it’s the standard configuration for an HP DL server. People have reported that with an Intel chipset based NIC everything works fine).
The problem was that there were a lot of dropped packets and errors, that force TCP entering the congestion avoidance phase, so performances are almost zero.
I’ve unsuccessfully tried to do the following:
updating the Broadcom firmware to the latest release (1.9.6);
disabling the Broadcom managing interface (that, as a bonus, requires booting a DOS disk, thanks Broadcom for firmly being in the 80’s);
PCI blacklisting the eth1 device and giving full access to it to the two domU;
increasing the TCP window size on dom0, domU and increasing the transmission queue length of the physical and virtual network interface;
All these, combined, give me something, about 50 Mbps, but there are still dropped packets and transmission errors.
On the RHEL 5 mailing list, someone reported that are now available the para-virtualized drivers for RHEL 5. I’ve tried them, following the Red Hat guide, and performances jumped to an incredible 345 MBps of sustained throughput (for two domU on the same dom0: i.e. they are memory-to-memory transfers).
(funny enough, now I’ve found an italian guide describing the same problem, although it seems to me they’re wrong in identifying the root cause: it’s not the virtual NIC speed the problem, the errors are).