Nested ESX5i instances and virtual Nexenta SANs - possible test strategy for Nexenta

Added by Ashley Watson over 2 years ago


We have been looking at various ways of increasing the amount of flexibility for our development projects that require simulations of full vSphere 5 managed clusters.

We have storage presented to 6xUCS blades (12 cores, 96GB ram each) via an HP MSA 2312G2 (15k RPM, 48 spindles, 4Gb/s FC interface - no T10 capabilities). We have another IBM DS san with 48 spindles (4Gb/s FC - no T10 capability). Each blade is running ESX5i and is part of a vCentre managed instance.

We didn't want to tie up specific blades to specific projects so we created nested ESX5i instances that are capable of running 64 bit guests - this is a great feature os ESX5i;

We have 2xnested ESX5i instances.

We created a VM on our main ESX5i farm for a virtualised NexentaStor instance (6GB ram, 2xvCPU, 15GB VMDK for NexentaOS, 6x100GB VMDKs - one from each raid set on our FC SAN for performance). We have 3xe1000 NICs - 2 for iSCSI traffic, 1 for management. The syspool is attached to a virtual LSI Logic parallel controller, the 6x100GB VMDKs are configured within the NexentaStor instance to be 1xRaidZ volume on a second virtual LSI Logic parallel controller. We go through the process of presenting zvols (150GB+150GB+100) to our 2xvESX5i instance via iSCSI.

We use storage DRS rules on our main cluster to keep the VMs for ESX5i-a and ESX5i-b and vNexenta on the same blade to prevent cross blade traffic and we configure the failover on our FC so that we keep the traffic on a single leg of the FC infrastructure.

With this set up we can use NexentaStor as a virtualised iSCSI SAN to present storage to our nested ESX5i hosts within our FC architecture. We can immediately reproduce issues like hardware accelerated locking etc without additional physical hardware or putting any other VMs at risk.

If Nexenta could invest some time setting up this type of setup (like we have - it took us about 2 days), then they'd maybe able to get to the bottom of some of the nasty VAAI issues and also understand better some of the issues people seem to be having with Nexenta in virtualised environments.

Initial testing shows our virtual Nexenta and Nested ESX5i cluster performing not that much slower than our native FC SAN to physical ESX5i hosts.