Our DB is about 200 Gb and growing. Is it acceptable to use Ethernet or should I look at other links like fibre or whatever..?

So i am about to give you alot more than what you asked for..

I used to work at sun as an se, where we sized storage for database all the time based just on size of the database. All of the components(disk, and bus) were kind of fixed so we could figure out how many disks and buses we needed to hit capacity. If i have a 200GB database but only write to it a couple times an hour.. then i barely need any throughput.. just capacity.

How many io's and how fast are the metrics we use now. You should be able to find out from a dba what their averages iops and their size. Read and write ratio's are also helpful. We also need to know latency, how fast do you need them? are they sync or async? Also there are lots of options on how you are feed iops.. different disks and network, so there is alot more math.

Everything after that is math. Everything i am writing here is gross, GROSS, approximation but..

I am going to assume an 8k io size because it is a typical block size for user of an oracle database.

Let say your workload requires 1k iops.

Disks on average drive about 100-200 iops @8k. So with the best disk, you would need 5 disks. There are different buses including SAS which is running at about 6gb/sec so we can see it driving apx 60-80k iops, so you should be able to drive the ios you need from that. If you then have to package that up and send it across the network, 1gb nic can run about 12k iops, so you are still just limited by disk.

Probably the best invention in the last decade has been the use of ssd. The average ssd can drive between 20-80k 8k iops at extremely low latency. But they vary greatly by price, capacity and reliability. The faster architecture would be 10gbe and all ssd, but that is cost prohibitive.

The zfs in nexentastor has the ability to greatly improve performance at a low cost by adding tactical ssd's. The single best thing you can do is add ssd for zil(write cache) and arc(read cache). You may even want to get enough ssd to put your working set of your database into read cache. For the write cache we recommend slc ssd.. it is a little more expensive, but it doesn't wear as fast for heavy write workloads.

After all of that the network becomes easier to determine, if i can get the disk fast enough, i will be able trunk network to the right size for workload.

So let's say you need 80k iops? well.. to do this without ssd would require 100's of disk. In this case you probably would want a few ssd's, a couple for zil and a couple for arc and then the disk behind it becomes a place to flush to.

Oh yea, and did i happen to mention RAM? If i have enough ram(and 200GB is not uncommon these days) i maybe able to keep the entire database cached in ram..Then we have to do the math all over again..

Is a ok /common practice to expose NFS shares to the VM's? I might give a dedicated nic to the VM with PCI pass-through. Or better to create a disk image in Dom0, and expose to VM as disk?

Again, the answer is it depends on the workload inside the vm. It is extremely common for vm's to be put on the network. On either nfs or iscsi. I personally like the idea of a single disk image per vm, mostly just for ease of management.

How do I regularly and most efficiently backup my nexenta data in case of disaster? Can I sync in almost real time to another nexenta node on separate hardware?

This isn't a question we can answer... most companies have there own business requirements for backup and dr. We have several tools inside nexentastor to help. We have auto-tier, which allows you to run send files to another system at a regular interval, auto-sync which does replication is closer to real time, and also send all the zfs metadata..and auto-cdp(extra $) which gives closer to sync replication. We also have a couple of cluster addon products(extra $) for automatic failover.

Is it worth looking at OpenFiler or some other open solution, - reason for asking is that we are essentially a Centos operation, and also future of OpenSolaris? (In my mind I am leaning towards Nexenta and ZFS, need ammunition to convince the rest)

The one thing we have that most of the other open products don't have is we have a very solid existing user base and a good sized support organization. We also have partnerships with apx 200 hardware vendors and a fairly large channel. We also started and are core contributors to illumos. When oracle closed the doors on opensolaris, we were instrumental in creating illumos which is a fork of opensolaris. All of the changes we make to the core kernel go upstream into illumos and are shared with the community and are put into our products. We have been growing this community steadily and there are now a number of companies dependent on illumos including us, joyent, delphix and everycity. This community is attracting more developers every week because it is what people hoped opensolaris would be, open, innovative and collaborative.

Small co., budget conscious, solution must be rock solid for live environments -

Rock solid can built. Look for single points of failure. Having more than one of everything is best. 2 heads, 2 jbods, mirrored drives, multiple buses, nics, and power.. Look at your architecture and say.. what if that fails?

We just sell the software. We rely on our hardware partners to work out the hardware details. We are built on the idea of enterprise features with commodity components.

Also, We are at version 3.1 of our product. It has gone through many growth pains, but it is seen to be very stable at this point.

Thanks for the 'proper' reply. I will digest and keep this forum updated on exciting developments re our Nexenta project!