I'm betting this is due to ESXi sync access on writes. To test this, set sync=disabled on the pool and retest.

Wow, thank you!

The tests now give me: 4K 3475 IOPS / 13,5 MBps 4K QD4 12536 IOPS / 49 MBps 4K QD16 26341 IOPS / 103 MBPS

This is way better, and a lot more in the range I was looking at. I will also enable L2ARC as well now.

This is not recommended in general for production - it was to prove/disprove the theory. Even though zfs won't get corruption from a crash, an esxi guest's FS might have corruption depending on what was being written and when. Check the forums here for ZIL/SSD and etc...

What is the problems with sync set to disabled? A powerloss that can make the writeoperation not to be written because they are in RAM?

I am fairly well protected with UPS and shutdowns of the servers.

I would really like to have a ZIL, but unfortunately it is not going to happen this year. Good SLC based SSD are still way to expensive.

Not just that. Consider that ESXi presents a virtual disk to a guest OS. The guest might have a journaling based filesystem (like ntfs or ext3 or whatever) that does certain writes in a certain order for consistency. Those writes go to the virtual disk, which ESXi then writes to the file on the NFS datastore. The ZFS appliance then lies about the writes being complete so if you crash in that window, the guest OS can in theory suffer from filesystem corruption (not just writes to a file being lost.) If you are on a UPS, you can take the chance. I do, but then my ESXi datastore is snapshotted every night and my ZFS box is on a UPS...

Thank you for the good explanation. The Nexenta is wellbacked with two UPS-s. And I do take snapshots every day on the ZFS-appliance. I also do backup evry hour in the guests that do writes locally. So I am staying at sync to disabled.

Do you have some recommendations for well priced SSD for ZIL-usage?

Not really. Like you, it is not in my budget (this is basically a home server).

To improve performance you may also want to change that 128KB block size. You will want to make it match your VM OS block size, if windows the default is 4KB. This will reduce the number of I/Os needed to write (if the block isnt in arc/l2arc) as you would need to read the 128KB block, modify the 4KB and then write it to its new location, instead of just writing the block to its new location.

Is it possible to change the block size in a running system without being afraid of what is going to happen? Or do I have to regenerate the folder?

I thought that was only applicable to iSCSI? For NFS, the host (nexenta) is doing the writes...

Its the same, you have a structured file stored on a file system, that structure is dependent upon the file system of the VM file system, 4KB, 8KB, etc blocks. These blocks are read and written from the file stored on zfs in those block sizes, so if your VM writes to its 4KB block, it writes that to the file on ZFS, which is holding that, say 200GB file, in 128KB blocks. When you write that 4KB block in the VM, you will then have to update that 128KB block on ZFS. With iSCSI you have an additional alignment to take into account, the VMFS for vmware. See nexentas best practice guide for nfs and esxi: http://info.nexenta.com/rs/nexenta/images/doc3.15000-nxs-v0.0-000004-AUsingNexentaStorNFSwithESXi_5.pdf

http://info.nexenta.com/rs/nexenta/images/doc_3.1_5000-nxs-v0.0-000004-A_Using_NexentaStorNFS_with_ESXi_5.pdf

Sorry this was the best practice, but that one also provides the required info and benchmarks.

Page 14:

http://info.nexenta.com/rs/nexenta/images/5000-nxs-v0.0-000002-A_nxstor_vmware_best_practices.pdf

I would also try upping the queue depth to apx 30 maybe more with the 15k drives. You know you have gone too high if you start seeing cpu spikes. The cpu will have to manage the queue if it is too high

Linda, I presume you are talking about Syszfsvdevmaxpending? I can try to set that to 30?

And what about changing the block size? can that be done on a running system? and access the files that are stored with the 128K block size?

David, this is interesting. Reading that doc, it does in fact recommend 8K-16K for NFS folder record size. I wonder why the default is 128K? I guess the smaller recordsize is specific to vmware usage?

128kb will be good for general file access, it will us smaller blocks for small files, big for big ones, improving io and throughput. But when it comes to structured files, databases, VMs, its not, as they are large files, making their block sizes 128kb. With structured files you need to align the file system blocks with the internal structure to get the best io performance.

I asked the writers of the paper how they determined the 8-16k block sizes. They found the 8k was the sweet spot for latency, but 16k was close and also had the best throughput for vmotion.

The larger 128k block would work well for vsphere writes as i believe the default block size for that is 1mb(didn't double check this).

The best thing to do for block size is run your workload and figure out your average io size as set accordingly.

There are 2 block sizes for vmware, the vmfs uses 1,2,4 or 8MB for the unified blocks, with sub blocks broken up into 64KB prior to esxi5. ESXi5 uses 8KB blocks, ESXi also supports 1KB blocks, by storing the data in the metadata block. But that is if you are using VMFS, ie block storage. NFS has only the vmdk files, which are broken up into the file system blocks of the vm os (from what i understand).

this would make sense then why the 8k blocks are the sweet spot.

This is all getting very confusing. I am using NFS for my ESXi datastore. Should I be using the default 128KB, 8/16KB or does it not matter?

so for nfs use 8k, for iscsi use 128k

This is really confusing now. I seem to recall the defaults with nexentastor are the other way around? Or am I misremembering?

My understanding is, you would use 8KB for both, the VMFS file block allocation size doesnt matter. Reads and writes to the VMDK, whether it is to NFS or block storage via VMFS are what the OS requests. VMware can coallesce reads and writes, so the block size read or written can be bigger than actually requested by the OS.

See: http://myvirtualcloud.net/?p=988 also read the comments by Chad Sakac (EMC), giving examples of the read/write stack from OS through vmware to the storage.

Of course this may be inaccurate, in my attempt to improve performance since we implemented virtualisation, it has taken alot of searching and reading to get information, VMware doesnt really provide concrete info on how the VMFS blocks affect read / writes, all I have read on the site is a post saying that performance isn't affected.

I also just posted a using nfs with vmware guide in the links forum. maybe that will be helpful?

Yes, thanks! I am in the process of storage vmotioning all the VMs off the datastore and then back to it after changing the recordsize to 8K.