A few things:

• if your goal is a cheap start, why go for the 16gig RAM sticks? Too expensive IMHO. And netlist is nice (we use it for our ESXi servers) but also too expensive for this IMHO. Go to the MOBO page on the SMC site and just look up the tested memory and go with whatever you can get the best price on that is ECC reg.

• That mobo is really new, make sure the SAS chip C606 will work with Nexenta/ZFS and make sure it supports or you can at least flash it into some sort of IT/HBA mode if your not going to add any internal HBA's.

• If your going with SAS expander backplanes to save on SAS interconnects (and this is for biz not home), personally, i'd stick with SAS disks over SATA.

Thank you for your comments.

As it turns out right now, it appears non-netlist 16GB dual rank sticks are price equivalent to multiple 8GB sticks from one of our suppliers currently, so it makes little sense to blow ram slots with 8GB (I am assuming those sticks are not actually Netlist HCDIMM's based on the description, but they very well could be Netlist). Going higher than that is a huge price jump though. You are correct that Netlist Hypercloud HCDIMM's are both high performance and expensive, currently 2x-3x the price of conventional 16GB dual rank or quad rank RDIMM's, though that may change as the 32GB Hypercloud DIMM's become more common. I could go with 4GB UDIMM to further reduce costs, but that puts an upper limit on what I can do with the motherboard, unless I commit to throwing away those sticks later.

As I understand it, Nexenta itself is actively developing the new Intel SAS chipset SAS drivers for the new C600 series chipsets that go with the E5 Xeons. Naturally that means I must wait until the driver is released. 3.13 is the expected release to support basic booting of E5, so the SAS driver might be available then, otherwise wait until summer for 4.0.

I figured the expander is the achilles heel of this build, if I go SATA in the hotswap bays. An interesting side effect/thought is using the downstream ports of the expanders to connect to 2 SAS devices on the 2.5inch HDD mounts rather than the motherboard onboard SATA ports, so perhaps something like an OCZ Talos. Though that brings up whether those 2 should be ZIL or L2ARC.

As for the vdev layout, the 6x6 and the 11x3 both yield 72GB. 11x3 gets me more space earlier, but the 6x6 doubles the vdev count, potentially doubling performance. Considering this is intended for ESXi 4.1 iSCSI mounts, would the performance side be more important here?

This post is old, but let me add an update for future people that as of the most current 3.1.3.5 Community Edition, the Intel C600 storage controller is NOT supported. I am in the process of installing Nexenta on a new build (learning process for me), and it does not support the Intel C600 storage controller, or the thermal management controller (though this seems to be making little different on the system's own control of its fans).

I have the X9DR7-LN4F+, so I actually have an onboard LSI 2308 controller for SAS. I have it flashed to IT mode, and Nexenta is seeing that just fine. Current issues seems to be the i350 Gigabit NICs. Nexenta sees them but I don't seem to be able to get an IP address through DHCP. This is very early in the process however and there could be something wrong with my current configuration.

To summarize as of 3.1.3.5:

C600 is NOT working LSI 2308 in IT mode IS working.

EDIT: i350 is working for at least 10 months now. This is probably an issue with my configuration :)

I am actually getting closer to ordering the original setup now actually, but shifting some parts closer to Mr. Weaver's setup.

• Supermicro CSE-847E26-R1K28LPB revision M (36x3.5inch hotswap bays plus space for 4x2.5 disks internally)
• Supermicro MCP-220-84701-0N double 2.5 inch mounts (2x actually to provide space for 4 2.5 inch SSD's eventually deep inside)
• Supermicro CBL-0281L-01 SAS cables (SFF 8087 links to the backplanes, 4x)
• Supermicro MBD-X97DR7-LN4F-0
• Intel CM8062107185605 Xeon E5-2643 3.3GHz LGA2011 Sandy Bridge-EP (1x to start)
• Supermicro SNK-P0048AP4 active 2U chassis type fanned heatsink
• Kingston KVR16R11D4/16HM 16GB 1600MHz DDR3 ECC RDIMM (Hynix)
• Supermicro MEM-IDSAVS1-064G 64GB SATADOM (2x for syspool, these seem to actually be Innodisk Inno-Lite MLC SATADOM's)
• Seagate ST33000650SS Constellation ES 3.5 inch 7200RPM 3TB 64MB SAS2

The 2 starter SSD's for ZIL/L2ARC is a a sticking point. Since I had intended to keep the hotswap bays for spinning HDD's, I am forced to keep the SSD's on fixed internal mounts. The easiest way of dealing with them is ZIL on SATA3 and maybe SATA3 for the L2ARC, as the motherboard has 2xSATA3 ports and 8xSATA2 ports, though if I move on to an eventual 4xSSD configuration, I have to make the the choice of keeping 2xZIL on SATA3 and shift L2ARC exclusively to SATA2. I gave it some thought, but since newer ZFS versions support dead ZIL's, and since mirroring ZIL is largely for safety (since the ZIL write is single threaded so there is no performance increase), I had entertained the thought of 1 ZIL and 3 L2ARC.

As for which SSD's to buy I was originally thinking the OCZ Deneva 2R 200GB SATA3 type 2.5 inch SSD which is available now. Or go for the slower Intel 320 300GB SATA2 2.5 inch SSD available now. Or wait a little for the new Intel DC S3700 200GB SATA3 2.5 inch SSD which will be out soon.

I had thought the SATA driver part for C60x chips was already working, but I guess not. I wasn't really intending to go through with this build until Nexenta 4.0 comes out anyways, which should have all the necessary drivers in place since Nexenta and Supermicro spend time with each other (fingers crossed).

N A wrote:

I am actually getting closer to ordering the original setup now actually, but shifting some parts closer to Mr. Weaver's setup.

• Supermicro CSE-847E26-R1K28LPB revision M (36x3.5inch hotswap bays plus space for 4x2.5 disks internally)
• Supermicro MCP-220-84701-0N double 2.5 inch mounts (2x actually to provide space for 4 2.5 inch SSD's eventually deep inside)
• Supermicro CBL-0281L-01 SAS cables (SFF 8087 links to the backplanes, 4x)
• Supermicro MBD-X97DR7-LN4F-0
• Intel CM8062107185605 Xeon E5-2643 3.3GHz LGA2011 Sandy Bridge-EP (1x to start)
• Supermicro SNK-P0048AP4 active 2U chassis type fanned heatsink
• Kingston KVR16R11D4/16HM 16GB 1600MHz DDR3 ECC RDIMM (Hynix)
• Supermicro MEM-IDSAVS1-064G 64GB SATADOM (2x for syspool, these seem to actually be Innodisk Inno-Lite MLC SATADOM's)
• Seagate ST33000650SS Constellation ES 3.5 inch 7200RPM 3TB 64MB SAS2

The 2 starter SSD's for ZIL/L2ARC is a a sticking point. Since I had intended to keep the hotswap bays for spinning HDD's, I am forced to keep the SSD's on fixed internal mounts. The easiest way of dealing with them is ZIL on SATA3 and maybe SATA3 for the L2ARC, as the motherboard has 2xSATA3 ports and 8xSATA2 ports, though if I move on to an eventual 4xSSD configuration, I have to make the the choice of keeping 2xZIL on SATA3 and shift L2ARC exclusively to SATA2. I gave it some thought, but since newer ZFS versions support dead ZIL's, and since mirroring ZIL is largely for safety (since the ZIL write is single threaded so there is no performance increase), I had entertained the thought of 1 ZIL and 3 L2ARC.

As for which SSD's to buy I was originally thinking the OCZ Deneva 2R 200GB SATA3 type 2.5 inch SSD which is available now. Or go for the slower Intel 320 300GB SATA2 2.5 inch SSD available now. Or wait a little for the new Intel DC S3700 200GB SATA3 2.5 inch SSD which will be out soon.

I had thought the SATA driver part for C60x chips was already working, but I guess not. I wasn't really intending to go through with this build until Nexenta 4.0 comes out anyways, which should have all the necessary drivers in place since Nexenta and Supermicro spend time with each other (fingers crossed).

If you are going to go at it with a single log device then make sure you are happy with the performance of your pool under load BEFORE you add it. Sure, you might not lose any data to a failure, but you'd be surprised how slow your system will become once you get used to the log ssd...

Personally, I'd wait for the S3700. The most recent box I built used (2) Kingston E100 100GB drives (purchased after I sent back the OWC drives I was using because I had one fail after a few months and another dead out of the box) for LOGs and (2) Intel 520 240GB drives (running as 192GB for better performance and lifespan) for CACHE. I'm happy with the drives I got, but as soon as any fail, I'm swapping to the S3700 for the consistent performance. Based on reviews, I really thing Intel hit it out of the park on those.

Jason, aren't you afraid of using Intel 520 SSD's with SandForce controllers? The SF controllers in SSD's of several manufacturers aren't really famous of their reliability. It's one of those things holding us back for using SF based SSD's. The Samsung 840 Pro's came out a couple of weeks ago and are not only more performant, but Samsung also uses their own controllers. Just like Intel DID, which were (as far as I know) pretty reliable.

eMiz0r . wrote:

Jason, aren't you afraid of using Intel 520 SSD's with SandForce controllers? The SF controllers in SSD's of several manufacturers aren't really famous of their reliability. It's one of those things holding us back for using SF based SSD's. The Samsung 840 Pro's came out a couple of weeks ago and are not only more performant, but Samsung also uses their own controllers. Just like Intel DID, which were (as far as I know) pretty reliable.

No, there were several manufacturers who had reliability issues that happened to be using SF controllers. I don't believe that there is anything wrong with the SF chipsets. I've got somewhere around 40 Intel 330/520 SSDs in service (mostly 520s), some in client machines and some in servers, none of which have failed. I've also got some SF-15xx drives from Smart in my IBM SAN which have been rock-solid. My experience with OCZ and OWC, on the other hand, has been absolutely miserable, regardless of controller or intended usage.

The concerns I expressed above about the 520 were purely from a TBW standpoint because They're not rated for a very long lifespan. I still use them because they're cheap, perform really well (especially when over-provisioned), and are easily replaced locally.

I remember reading somewhere recently that even with Sandforce controllers, the firmware makes a ton of difference. Supposedly Intel sends their Sandforce firmware through at least 6 months of QA, but due to some sort of exclusivity agreement with Sandforce, Sandforce isn't allowed to backport bugfixes in the Intel firmware back into the generic base firmware available to all their partners for a set period of time. That premium you pay is for the enhanced QA/firmware, all other things being equal.

The real issue with using SandForce based drives is that they rely on compression for most of their performance. This is not possible when a lot of the data is encrypted. The drives also depend on a lot of spare area to perform well, which is not possible, or not recommended for ZFS because ZFS wants whole disks, not slices. Therefore, you cannot reserve spare area manually (unless your SSD supports it).

This is where Enterprise SSD's like the new Intel 710 series and the Fusion I/O show their benefits. One, they have extremely consistent performance under high workloads, and two, you can manually set the spare area in the firmware to ensure that only a single disk (or multiple disks in the case of Fusion I/O) are presented to the operating system while spare area is maintained.

So keep in mind above that you should not try to over-provision your ZFS through partitions because you'll only have slices to work with that way. Instead, you should look at Consumer SSDs that have good resiliency.

I kept this in mind, and chose to go with the OCZ Vertex 4s. Their new Indilinx controller does not rely on compression for performance like Sandforce, and is much more resilient to high-duration writes. Their newer OCZ Petrol SSD's are also a good choice.

Good SSD's for Cache = high read speeds. Good SSD's for ZIL/Log = high write speeds.

OCZ Vertex 4 OCZ Petrol Samsung 830 Crucial M4

Bad SSD's

3Gbit/s Intel SSD's. These are seriously antiquated in performance compared to modern SSDs. Sandforce-based SSD's. Intel 520, Vertex 1, 2, and 3, Agility, etc. etc. Asynchronous Nand-based SSD's (Agility, Crucial V4, pretty much all second-tier SSD's). TLC (tri-level cell) based SSD's (Samsung 840 is the only one in production. Great speeds but its longevity is in serious question due to the fragile nature of TLC during sustained writes and reads)

To the rest of your build, I'm using the 2650 and so far, its impossible to bog down. You and me are using two different philosophies as you're using 4 high-powered cores (3.3Ghz) while I'm using 8 low-speed cores (2.0Ghz). Still, with a SMB share pushing 80MB/s and two VMWare instances loaded off the SAN, I barely break 20% load. This is with de-dup and compression on. If you are not doing either of those options, save your money and get a simple dual core processor. In fact, right now I seriously question putting in the money for a dual processor box. Chances are I will never saturate this processor unless they implement live encryption. If you imitate my setup, on your processor, assuming things scale fairly linearly (and you still have a much more aggressive clock speed), you would probably barely be topping 40% at my current load.

With memory, unless you are planning to stick to just one processor and really need density, I would stick to 8GB modules. I went with 32GB of memory to start with, and even with de-dup, it's running fine. With 8GB sticks, you could do 64GB / processor on that board. Unless you are really leveraging dedup over many TB's of data, 64GB / processor should be more than enough for the build. You can also run at full 1333 speed (I believe 16GB and up modules are limited to 1066?) Also keep in mind your ranking, as that's important as well. Stick to dual-rank sticks (I believe you can get dual rank in everything except 32GB DIMMS). Dual-rank sticks will let you maintain memory speeds and put less stress on the controller than quad-rank modules.

As a last point, knowing your load is very important. My SSD Cache was a great investment, and really keeps my array fast. On the flipside, ZIL is only important for highly synchronous writes, like used by Databases. If you aren't doing those sorts of writes (most file transfers are not synchronous), then the ZIL log is pretty much useless. I know my ZIL log is barely being tapped and I could have saved some money by removing it all-together (or moving it to 15K spindles instead of SSD's).

You are correct that the end goals may be dfferent for different people. My specific orientation is a NFS/iSCSI box for one or more ESXi VM servers, with a potentially larger buildout basing on this rig alone. Dedupe and compression are focus points, but since I am doing a gradual buildup of components I get shorthanded early on, so am forced to pick higher end parts compared to immediate needs to avoid having to throw away stuff, in particular I will probably get only one CPU to start, and probably 2 SSD, 6 SAS HDD, and 16GBx4 RAM. This allows easier acquisition of HDD's only to grow out. Since some ZFS stuff is single threaded, I felt going for higher clock speed might be helpful, and it's not like I can't use Intel HT to boost logical cores. Eventually, as 10G gets cheaper, I wanted to put in a 10G ethernet card, which is where any performance issues will rear their ugly heads, though at that point I may also be forced to get the second CPU as I may be in need a of DAS JBOD, which would probably warrant a new separate external link SAS controller, which consumes PCI slots along with the 10G card(s), which may push me over what the single CPU can do on a dual CPU motherboard that is operating singleheaded.

Jason Litka wrote:

eMiz0r . wrote:

Jason, aren't you afraid of using Intel 520 SSD's with SandForce controllers? The SF controllers in SSD's of several manufacturers aren't really famous of their reliability. It's one of those things holding us back for using SF based SSD's. The Samsung 840 Pro's came out a couple of weeks ago and are not only more performant, but Samsung also uses their own controllers. Just like Intel DID, which were (as far as I know) pretty reliable.

No, there were several manufacturers who had reliability issues that happened to be using SF controllers. I don't believe that there is anything wrong with the SF chipsets. I've got somewhere around 40 Intel 330/520 SSDs in service (mostly 520s), some in client machines and some in servers, none of which have failed. I've also got some SF-15xx drives from Smart in my IBM SAN which have been rock-solid. My experience with OCZ and OWC, on the other hand, has been absolutely miserable, regardless of controller or intended usage.

The concerns I expressed above about the 520 were purely from a TBW standpoint because They're not rated for a very long lifespan. I still use them because they're cheap, perform really well (especially when over-provisioned), and are easily replaced locally.

I can HIGHLY recommend the Supersspeed SLC's : [[http://www.supersspeed.com/productAll.asp?sid=50]] The are true SLC ssd's, but they use the Sandforce SF2281 controller. Sure this controller has had it's share of nastiness in the past, but i believe a the firmware fixed that beautifully.

here's the link that got me wondering [[http://www.tweaktown.com/reviews/4917/supersspeeds302hypergoldmlc240gbssd_review/index.html]] I have 4 x 60 GB for ZIL ( 2 x mirrored) and 2 x 240 GB ( 2 x mirrored) and arc.. just 1 word.. WOW!

sure it's from a unknown manufacturer (or is it, i believe the make the ssd's compellent uses in their systems), but you can order them through ebay , and they are the cheapest SLC's you'll ever see ( 140 USD for the 60 GB, 250 for the 120GB... plus they outperform everything out there ( including , at least in my setup, 1 zeusmaxiops, had 1 for testing)

was wondering if anyone else has had a go at these ?

kinda wondering if anyone else has had a go at these

You really want to avoid desktop SSDs for your ZIL. Your ZIL will see a lot of write activity, and that'll just kill your desktop SSDs. Look for something more enterprise-class for the ZIL, like an SLC-based SSD, or something like the new Intel DC S3700 series. You'll pay more, but you won't be swapping them out every couple of months, either.

The Intel DC S3700 is an MLC-based SSD, but it manages the writes better using a capacitor and standard RAM to cache writes before committing them to flash. This allows the drive to reduce the number of individual writes it performs to flash drastically, and the capacitor allows the drive to ensure cached writes make it to flash, even if the power is lost (it acts like a short-term battery). This design allows the DC S3700 handle several orders of magnitude greater writes than a standard desktop-class SSD will.

FYI, I am phasing out my 240GB 520s in favor of the 400GB DC S3700 drives. My working data set has increased...

Paul Reid wrote:

You really want to avoid desktop SSDs for your ZIL. Your ZIL will see a lot of write activity, and that'll just kill your desktop SSDs. Look for something more enterprise-class for the ZIL, like an SLC-based SSD, or something like the new Intel DC S3700 series. You'll pay more, but you won't be swapping them out every couple of months, either.

The Intel DC S3700 is an MLC-based SSD, but it manages the writes better using a capacitor and standard RAM to cache writes before committing them to flash. This allows the drive to reduce the number of individual writes it performs to flash drastically, and the capacitor allows the drive to ensure cached writes make it to flash, even if the power is lost (it acts like a short-term battery). This design allows the DC S3700 handle several orders of magnitude greater writes than a standard desktop-class SSD will.

Well, the superspeeds may be sold as consumer ssds, they are infact SLC ssd's which make use of intel SLC and a LSI Controller.. Just make very sure that if you use them in combination with a sas interposer when used in a JBOD