Yes, esxi absolutely forces sync mode. To test this, set sync=disabled on the folder shared via NFS and re-test. If confirmed, get a low-latency SSD and add as log (zil) device.

Hi,

sorry for not coming back earlier. I have test the different sync option and finally found that in ANY case ESXi will force synced writes on NFS. The only option to increase spped/lower latency is to use SSD. In my case I use two mirrored OCZ Vertex 3 SSD as log device and the latency dropped below 20 ms in worst case.

Does anybody know, if zil will use the whole device for logging or are there some limits?

Greetings

Not sure I understand you. Are you saying you disabled sync mode and that didn't speed things up? If so, no offense, but I don't believe that.

Hi,

yes, enabling async or disabling sync does not change a bit in ESXi behavior. This is said to be related to the use of the o_sync option when ESXi writes data causing the write request to unexceptional being in sync mode. The only way to work around this is to place a solid state caching device in the write queue, which will lead to faster commits of the writes. In my case, I decided to invest into not to expensive but fast SSD for ZIL.

Maybe there are some hacked NFS implementations that ignore the o_sync option but Nexenta seems to respect it.

Greeting,

Chris

Like I said, I don't believe this. I and numerous others have contradictory experience. sync=disabled causes the server to act as if the request were asynchronous. e.g. the client is acknowledged then and there. There is nothing 'hacked' about sync=disabled doing what I said. If nexenta is 'respecting' o_sync in this case, there is a (new) bug. This has been discussed more than once on these forums, and it's always worked the way I am describing.

Hi Dan,

hmm... I have definitively not seen any real performance benefits in ESXi NFS Datapools under workload when switching between sync and async at zfs folder level and have read a lot of articles regarding the o_sync behavior with different results, especially in ZFS environments. But I would be interested in figures that show what impact can be expected from the specific measures. I would be great if someone would have done this. If I got time and equipment, I will do some additional testing on my own and post the results here.

It isn't complicated. Create and share a folder via NFS. Mount it as a datastore on ESXi. Create a VM on it, and have that VM do something like "time dd if=/dev/zero of=testfile bs=1M count=8K" to write 8GB data. It will be very slow. Then, 'zfs set sync=disabled FOLDER' and repeat the test. It should be several times faster. I just re-ran this with a ubuntu guest and 1gb network.

Sync disabled:

8589934592 bytes (8.6 GB) copied, 91.5043 s, 93.9 MB/s

Sync enabled:

8589934592 bytes (8.6 GB) copied, 254.069 s, 33.8 MB/s

Keep in mind, this is only going to be noticeable with write-heavy workloads. I don't know what your specific tests have been. But the above result is absolutely in keeping with my and other folks experience (and logic - if the NFS client has to wait for the write to be 100% committed to disk, this makes sense and also explains why you adding a low-latency SSD helps a lot.)