I encountered a pesky issue recently. Before I get into the details, first a quick overview of the setup. A Windows Server 2012 cluster consisting of two cluster nodes. The cluster nodes are brand new HP DL 360 G8p servers with 256Gb Memory and two six-core processors. Networking is based on 10Gb Emulex NICs for converged fabric connected to a HP Procurve 5406zl. The storage for the cluster has two members, an Equallogic 4100E and an Equallogic 4100XV. The ISCSI traffic is on a dedicated network with separate 1Gb NICs in the cluster nodes.
When I connected to a cluster node the response in the RDP session sometimes had a little delay. Typing in PowerShell for example felt like watching a movie with the audio out of sync from time to time. The first time I thought the lack of sleep was taking its toll. But after experiencing a couple of delays I concluded that I had some troubleshooting to do.
After bypassing the Remote Desktop Gateway that I connected through, I singled out one cluster node having the issue. I looked at the event log, but came up empty handed. My next thought made me look at the networking infrastructure. I checked that both servers had the correct and identical NIC firmware and drivers. I also verified that the switch had the latest firmware applied. I compared the complete converged fabric configuration on both servers. All parts checked out fine. I looked at the task manager and the processor utilization was close to idle.
The next thing to rule out was the NIC hardware. Since only one of the two servers was subject to the issue I decided to swap the 10Gb NICs between the servers. After this swap the issue seemed to have disappeared. I did not experience the issue on the other server.
I am unable to let go of an issue without a proper technical explanation and since the NIC hardware swap seemed to make the issue disappear I run a diagnostic test on both servers. All green checkmarks. Suddenly the delay appeared again on the same server where I experienced the issue before. We can now rule out the NIC hardware.