We have a similar configuration, although I haven't really looked at performance, so I don't have any clue if it's optimal or not.

We have two SSD ZIL drives (mirrored) and two SSDs for cache (not mirrored). Instead of 12 we have 13 regular drives. One is a spare and the other 12 are in three four drive raidz1 groups.

- Matt

-----Original Message----- From: NexentaStor.org [mailto:tracker@nexentastor.org] Sent: Wednesday, July 13, 2011 3:09 PM Subject: [NexentaStor Project - General Discussion - msg2760] Best zRAID / configuration for 9 or 10 HDDs?

http://www.nexentastor.org/boards/1/topics/2760 Sean Leyne

I am looking for some help in figuring out the best configuration for my Nexenta SAN configuration. I am new to the whole ZFS/zRAID world and need some pointers.

I am planning to use 2 SSDs for ZIL and Cache along with 12 HDDs (thinking of 2 HDD for hot spares, so 10 available for use), and I want a configuration that provides the best iSCSI performance for random IO operations. The SAN system will have a dual port 10Gbe network adapter, along with at least 1GB of RAM per 1TB of zPool space, the HDD will be 7200 rpm 6Gps SAS.

What zRAID / configuration would be best for my purpose?

Thanks in advance

-- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://www.nexentastor.org/my/account

I think the best configuration for random I/O would be mirrors. The more the better. Since you would have a ZIL and since random writes are turned into sequential writes by the system, you will definitely feel a difference when flushing your cache to disk if you have more vdevs. You have to be careful with the size of your L2ARC since there is some overhead in physical memory pointers to your SSDs. Im not clear on the specifics on how much RAM your L2ARC will use, but Im sure google knows.

Angel Vidal wrote:

I think the best configuration for random I/O would be mirrors. The more the better. Since you would have a ZIL and since random writes are turned into sequential writes by the system, you will definitely feel a difference when flushing your cache to disk if you have more vdevs.

So, you would recommend 3 RAIDZ1 volumes with 3 HDDs each? Or, 5 mirrored volumes with 2 HDDs?

Matthew Goheen wrote:

We have a similar configuration, although I haven't really looked at performance, so I don't have any clue if it's optimal or not.

We have two SSD ZIL drives (mirrored) and two SSDs for cache (not mirrored). Instead of 12 we have 13 regular drives. One is a spare and the other 12 are in three four drive raidz1 groups.

Matt,

Why do you have your ZIL and cache devices mirrored? For performance or for redundancy?

Also, I have read that SSD for cache is also strongly recommended but you only have SSD for cache, any particular reason?

Not sure exactly how you got that out of what I said...

We have two mirrored SSDs for the ZIL (32Gb Intel X25-E). We mirror these because if you lose one you lose all the writes that were queued up.

We have two NON-mirrored SSDs for the cache (80Gb Intel X25-M, I think) because if one fails you lose basically nothing (you lose some stuff in the read cache).

I also don't really understand your last sentence...do you?

"I read that airplanes are strongly recommended for flying but you only have airplanes for flying, any particular reason?" Huh? Sounds good to me... :-)

- Matt

-----Original Message----- From: NexentaStor.org [mailto:tracker@nexentastor.org] Sent: Wednesday, July 13, 2011 8:06 PM Subject: [NexentaStor Project - General Discussion - msg2760] RE: Best zRAID / configuration for 9 or 10 HDDs?

Matt,

Why do you have your ZIL and cache devices mirrored? For performance or for redundancy?

Also, I have read that SSD for cache is also strongly recommended but you only have SSD for cache, any particular reason?

Sean, I would recommend 5 mirrored volumes with 2 drives each.

Matt, If Im not mistaken, you wont lose your writes in the event of a ZIL failure unless the system also crashes before the writes in your ARC have been flushed. Its my understanding that the ZIL will not be read unless there is a need to replay those writes such as when the system is shut down uncleanly. I believe synchronous writes go into the ARC like all the other writes, but also get written to the ZIL. Once the writes in the ARC are committed, then the copies stored in the ZIL are removed.

I could be wrong, but Im pretty sure thats how it works.

It appears you may be correct...I've read various somewhat conflicting articles, but the consensus appears to be that a ZIL failure can be tolerated by the OS. However, that still leaves the ZIL as a critical resource if the system happens to crash -- the data in the ZIL then becomes the only copy, so one could argue that it should therefore be mirrored. I'm not necessarily buying into that, as the window for that failure becomes pretty damn small (i.e. the system crashes AND your ZIL disk fails).

I found this article useful: http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flash-memory-ssds-and-zfs

- Matt

-----Original Message----- From: NexentaStor.org [mailto:tracker@nexentastor.org] Sent: Thursday, July 14, 2011 4:12 AM Subject: [NexentaStor Project - General Discussion - msg2760] RE: Best zRAID / configuration for 9 or 10 HDDs?

http://www.nexentastor.org/boards/1/topics/2760 Angel Vidal

Matt, If Im not mistaken, you wont lose your writes in the event of a ZIL failure unless the system also crashes before the writes in your ARC have been flushed. Its my understanding that the ZIL will not be read unless there is a need to replay those writes such as when the system is shut down uncleanly. I believe synchronous writes go into the ARC like all the other writes, but also get written to the ZIL. Once the writes in the ARC are committed, then the copies stored in the ZIL are removed.

I could be wrong, but Im pretty sure thats how it works.

-- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://www.nexentastor.org/my/account

Matthew Goheen wrote:

Not sure exactly how you got that out of what I said...

I don't know what happened, but I read your original post as saying that your had mirrored HDDs for ZIL... my bad.

I also don't really understand your last sentence...do you?

I was in the context of: Was there a reason why you are not using SSDs for ZIL... but that can be ignored, given my mis-read.

In my environment, the larger problem with the ZIL failing is the horrible NFS performance that we would experience. We opted to go for a DDRdrive instead of an SSD since we never really see more than 3GB of sync writes "in play" plus the lack of wear in a RAM device vs SSD.

Angel Vidal wrote:

Sean, I would recommend 5 mirrored volumes with 2 drives each.

What about the setup of redundancy groups?

Redundancy groups? What do you mean?

Hey Angel, i don't comment on the redundant groups. My suggestion as mentioned before by another guy: To get the best performance use mirrored vdevs. And use a zil as the iSCSI is everytime sync.

And: Don't ever use the X25E as ZIL device. Either go with ddrdrive or Stecs.

Angel Vidal wrote:

Redundancy groups? What do you mean?

I got a little confused when reading a Nexenta doc, they were discussing Redundancy Groups (vdevs) when the doc turned to discussing stripping across redundancy groups... Sorry for the question

Florian X wrote:

And use a zil as the iSCSI is everytime sync.

Use ZIL as iSCSI???

And: Don't ever use the X25E as ZIL device. Either go with ddrdrive or Stecs.

What's wrong with X25E?

Sean Leyne wrote:

Florian X wrote:

And use a zil as the iSCSI is everytime sync.

Use ZIL as iSCSI??? No. I mean as iSCSI and NFS are producing synchronous IO you should use a ZIL device (slog)

And: Don't ever use the X25E as ZIL device. Either go with ddrdrive or Stecs.

What's wrong with X25E? It does not really fulfill the flush command correctly when zfs wants it to do. Therefore zfs might think it is in a consistent state yet it isn't. And it also doesn't writes data correctly. In tests it came out that you can prove with the ability of zfs that is written has not a correct checksum. I have another statement which i have to look up.

Florian X wrote:

What's wrong with X25E?

It does not really fulfill the flush command correctly when zfs wants it to do. Therefore zfs might think it is in a consistent state yet it isn't. And it also doesn't writes data correctly. In tests it came out that you can prove with the ability of zfs that is written has not a correct checksum.

According to the Nexenta HCL, the X25-E is supported but when connected to a SAS expander. Perhaps when 'direct connected' the X25-E doesn't show this problem.