You don't say what the size of network interfaces? 1gb? 10gb?

One of the nice things about zfs is it's ability to stage traffic.

The one thing that will help me say how this will perform is knowledge of your working set size. You maybe able to get your entire workload into arc and then it will be very fast. But is you have a working set size that is greater than 500GB you may have to be looking outside your l2 on a regular basis.

Is you workload heavy read or write?

btw, it is kind of general consensus that nfs maybe faster.

It will be 1GB NIC.

This setup is for the lab use. So read and write might be 50/50

but i don't understand this: "The one thing that will help me say how this will perform is knowledge of your working set size. You maybe able to get your entire workload into arc and then it will be very fast. But is you have a working set size that is greater than 500GB you may have to be looking outside your l2 on a regular basis."

Is it possible to mix HDD in the same ZFS pool?

Yes, it is very common. We use the ssd's tactically in that we use them for read and write cache's. So you could potentially create one big pool with all the hdd and use the ssd to stage writes into the pool and use ssd to create a large read cache for the pool. It is called a hybrid pool and it is very effectively for getting much better speed out of slower hdd.

Using write cache is especially effectively when running nfs. The write cache's all sync writes, which is all that nfs uses..

how to config in the nexenta for the maximum performance using this setup? Is 60Gb ssd for LOG enough since there would be 63GB of ARC.