Forums » Help »

Very strange NFS behaviour (performance)

Added by Koen Calliauw over 2 years ago

Hi all,

I'm testing out NexentaStor (the community version for now, RC5) for it's performance with NFS, as we'd like to start using this for our VM storage backend. So after everything is connected and working (nfs mount is ok), I did a simple test with 'dd' on the linux client. What happens is very weird:

  • nfs client does absolutely nothing for the first part of the copy (testing with 3,2GB file from /dev/zero). I can tell this running a 'dstat' in a separate console. I can see no network traffic being generated. I can also tell this by the fact I see no network traffic or IOPS in the analytics section of NexentaStor.

  • then, after about 2 minutes it starts copying after a while, saturating my gigabit link and I start seeing network traffic and disk i/o in the analytics.

  • disk i/o keep going at about 1800-2100 iops, but network falls back to 0
  • network starts again, disk i/o still going strong
  • network stops again
  • disk i/o keep going for for about 15 seconds
  • after both disk i/o and network have fallen back to 0, it takes another 20 seconds for the client to complete the copy.

So, I can get 2000 IOPS, I can saturate my gigabit link but my file copy amounts to 18MB/sec. Somehow something is seriously messed up but I feel there must be some configuration options that I messed up.

Can anybody help me troubleshoot this? Thanks

Kind regards, K!


Replies

RE: Very strange NFS behaviour (performance) - Added by Koen Calliauw over 2 years ago

setting 'sync' to 'always' instead of 'standard' made the speed double. Network transfer starts immediately after dd starts, IOPS follow (why is that?).

RE: Very strange NFS behaviour (performance) - Added by Koen Calliauw over 2 years ago

Koen Calliauw wrote:

setting 'sync' to 'always' instead of 'standard' made the speed double. Network transfer starts immediately after dd starts, IOPS follow (why is that?).

I meant 'disabled'

RE: Very strange NFS behaviour (performance) - Added by Roman Strashkin over 2 years ago

this is normal.

NFS is sync-operation process. 'sync=disabled' = FORCE disable all sync-operations (including NFS).

RE: Very strange NFS behaviour (performance) - Added by Koen Calliauw over 2 years ago

Hi,

So I did some more testing and the more testing I did the more frustrating it became. I ended up doing dd tests on the storage appliance itself to take networking out of the equation and got the exact same results:

  • small dd copies work OK, not superfast, but OK

  • large dd copies are not working at all. Trying to create an 8GB file of zeroes just locks up everything. I end up having to CTRL+C out of it (and then wait for another minute or 2). Over the network it's the same, except I see that the NFS server cannot be reached in /var/log/messages It's almost like writing beyond a certain point in the storage pool causes the issue. I checked the DAS and no disks are reported faulty.

All settings are pretty default. The only thing I tried (to no avail) was setting Syszfsnocacheflush to Yes (is this effectively disabling the ZIL?)

Thanks, Koen

RE: Very strange NFS behaviour (performance) - Added by Angel Vidal over 2 years ago

What are the specs on your Nexentastor box? Im concerned with RAM, Controller and Disks. Also, do you have any dedup or compression enabled? What about the layout of your pool?

RE: Very strange NFS behaviour (performance) - Added by Koen Calliauw over 2 years ago

Hello,

The Appliance has the following specs: HP ProLiant DL360 G5 2x Intel Xeon E5420 Quad-Core 32GB Memory Controller: HP SC80Ge (LSI) HBA DAS: HP MSA2012sa 12x 450GB SAS2 15K rpm

Dedup and compression are both off. I have created 12 vdisks and 12 volumes on the MSA so it's basically JBOD. I created a pool with 4 x 3 disks in RAIDZ.

Thanks, Koen