How are you testing the NFS (just vmotions?)? Does the time to "degrade" appear to be the same after restarting the appliance? Can you track the amount of data read (to try to correlate it to a memory/disk limit) before it degrades. Have you changed the NFS version in Nexenta (I don't use NFS, but I've read that ESXi uses version 3 and might be getting confused on the v4 default of Nexenta)? Are you using jumbo frames (might try without if you are)?

SSH into the esxi host and use esxtop to look at the v, d, and u screens to see if you can spot any high latency issues that would point back to a network config.

Jeff Gibson wrote:

How are you testing the NFS (just vmotions?)? Does the time to "degrade" appear to be the same after restarting the appliance? Can you track the amount of data read (to try to correlate it to a memory/disk limit) before it degrades. Have you changed the NFS version in Nexenta (I don't use NFS, but I've read that ESXi uses version 3 and might be getting confused on the v4 default of Nexenta)? Are you using jumbo frames (might try without if you are)?

SSH into the esxi host and use esxtop to look at the v, d, and u screens to see if you can spot any high latency issues that would point back to a network config.

Hi Jeff

I'm testing using both the vmware hypervisor directly and cloning also through the vms themselves. The time to degrade is the same. As soon as we push high IO the appliance dies after about 5 mins. Yes we've tracked the data read and its somewhere around 20gb well within the ARC cache. I've tried both V3 and V4 on the Nexenta without any noticable improvements also tried with and without jumbo frames. I'm speculating that it's something with the NFSD process that i'm just unaware of.

I'll try esxtop i don't think it's an network issue but just to exclude that option. We're running on an 10G nexus switch right now with no other load.

You might double check using iperf to test the network BW between nexenta and ESXi. I'd run the test twice as long as it takes to cause the degraded status just to be sure.

When you're doing you're test watching esxtop, change to the "u" screen, then expand the device with the "e" command and type in the device name (I believe it'll start with {NFS}) and then watch the DAVG/cmd column to see what the average command latency is (mine was spiking to >300ms when I was having MTU issues)

I just ran the iperf tests.

It appears the network is fine(and stable) no matter how many iterations I throw at it. No noticable spikes on DAVG/cmd either. I can't provoke the error with iperf - only using NFS.

I've also tested CIFS and it appears unaffected by this. Even when NFS repports reads of 3-7mb/s the CIFS service can deliver the expected read performance.

I'm not sure where to look but I believe this "problem" is isolated to the NFS Server service on the Nexenta however i'm not sure how to debug it on the appliance.

And thanks for your help and tips in trying to solve this issue Jeff.

You're welcome, but I think you're about to reach the end of what I can offer since I don't use NFS. One last bit of clarification, when you ran the iperf test, did you do bidirectional or trade? Or did you only test one direction?

It would be one-direction i think. I only ran it against the nexenta appliance with an iteration of 100.

Try running it in dual mode (sending traffic in both directions at once) or switch which is the server for another test. I found I had a problem in only one direction but affected performance when I tried to use the system.