Improving network throughput between Hyper-V R2 virtual machines

My colleague Norbert Westland asked me if I could explain the noticeable difference in network throughput between two Windows Server 2008 virtual machines on the same physical server in a Hyper-V R2 cluster compared to an identical file copy between the same VM and a physical host or from the physical host to the VM.

Our configuration:

• HP c7000 enclosure with a few BL460c G6 blades
• HP Virtual Connect Flex-10
• HP Brocade Blade SAN Switches
• HP EVA4400

Storage configuration
Five 500GB Cluster Shared Volumes, spread across 10 Fibre Channel disks configured with VRAID5; the EVA virtual disks are spread across both array controllers)

Network configuration
Thanks to Flex-10 we could split up the dual 10Gb ports on the HP blade server into four flexNICs with appropriate network speeds.

Management Team:  2Gb/s
Live Migration Team: 2 Gb/s
VM Network Team: 14Gb/s
VM Network DMZ Team: 2 Gb/s

VM to VM tests
When copying a number of files between the two VM’s we saw a speed of 30MB/sec. In this case the datacopy only traversed the virtual network adapter of VM1, across the VMBus directly to the virtual network adapter of VM2. No physical network was touched at all.

A similar test was performed between VM1 and VM2 on different nodes in the cluster. In this example the path was extended from the VMBus in VM1 to the physical VM Network Team of the one cluster node to the physical VM Network Team of the second cluster node and back to the VMBus and the virtual network adapter of VM2. The result was almost identical. So far so good.

VM to Physical (and vice versa)
When copying the same amount of data from either VM1 or VM2 to the physical host, the speed consistently increased to 100MB/sec. This sounded like bad news. What could explain this enormous difference in throughput.

Things examined:

• Did the VM have an emulated or a synthetic network adapter installed? In our case the Hyper-V integrations were installed and the much faster hypervisor aware network adapter was configured.
• Were there any hidden network adapters still left behind from? Using set devmgr_show_nonpresent_devices=1 did not reveal any hidden network adapters in device manager.
• Were network optimizations turned on on the virtual network adapter? Optimizations were turned on.
• Are supported network drivers and teaming software installed?  All network software was up-to-date.

After some search we found that several other people had experienced disappointing performance in network speed. On many occasions there were references to disabling TCP Offload. The HP blade server contains an embedded NC532i Flex-10 10GbE Multifunction Network adapter which supports which supports TCP Offload and Large Send Offload.

Results with TCP Offload disabled

The VM to Physical and Physical to VM copy test remained at 100MB/sec so that switch did not do much for this test.

However, the VM to VM test jumped to 100MB/sec and higher after the change. The two VM’s could be on one host or on separate hosts in the cluster. The speed would remain the same.

In the end the difference in network throughput disappeared and physical and virtual were fully on par again.

Because I was aware that virtualization MVP Aidan Finn (@joe_elway on Twitter) was also running almost the same kind of hardware, I asked Aidan if he could post his results as well.

The next day I saw this great post:
http://www.aidanfinn.com/index.php/2009/11/w2008-r2-hyper-v-network-speed-comparisons/

Microsoft has published a whitepaper on network optimization in Windows Server 2008 R2 which discusses all new networking features:

Search for: http://www.google.nl/search?hl=nl&source=hp&q=8EDE21BC-0E3B-4E14-AAEA-9E2B03917A09%2FHSN_Deployment_Guide.doc&meta=&aq=f&oq=

Whitepaper in HTML:
http://74.125.77.132/search?q=cache:17UM0bosaocJ:download.microsoft.com/download/8/E/D/8EDE21BC-0E3B-4E14-AAEA-9E2B03917A09/HSN_Deployment_Guide.doc+8EDE21BC-0E3B-4E14-AAEA-&cd=1&hl=nl&ct=clnk&gl=nl

You can follow Hans Vredevoort on Twitter