Testing the Dell EMC PowerEdge FX2s VSAN All-Flash Ready-Node
I was privileged enough to get my hands on a Dell EMC PowerEdge FX2s VSAN All-Flash Ready-Node for testing.
- The Dell PowerEdge FX2s VSAN All-Flash Ready-Node consists of the following hardware:
- 4 x Dell PowerEdge FC430 servers
- 2 x All-Flash disk sleds
- 2 x Dell PowerEdge FN410s IO Modules
- Each Dell PowerEdge FC430 has the following specifications:
- CPU – 2 x Intel Xeon E5-2680 v4 (14 cores at 2.4GHz)
- Memory – 256 Gb
- The Dell PowerEdge FX2s OOBM is via the Chassis Management Controller (CMC).
- The are two FN410s IO modules have OOBM IP addresses which are accessed via the CMC OOBM link.
- All Dell PowerEdge FC430 servers iDRAC have OOBM IP addresses which are accessed via the CMC OOBM link.
- Chassis, modules and servers will require 7 IP addresses for OOBM access (1 x CMC, 2 x IOM, 4 x iDRAC).
- The Dell PowerEdge FX2s chassis supports multiple configurations of hardware including combinations of compute and storage sleds. At maximum it can support up to 8 quarter-width compute sleds or 4 quarter-width compute sleds and 2 storage sleds.
- The VSAN All-Flash Ready-Node consists of 4 quarter-width compute sleds and 2 storage sleds. As such, PCIe slot mappings to compute sleds are shown below. However, it is not recommended to use PCIe cards as it inhibits the use of stateless computing. For example, if compute1a sled has a HBA card in slot PCIe 6, the WWPN of the HBA is locked to that slot. Moving the compute sled to a different chassis slot would cause the WWPN to change unless the PCIe card is moved as well.
- Physical KVM can be accessed from the front of the FX2s chassis. As shown in the picture below:
- Item 4 – KVM select button enables mapping of the KVM to the sleds
- Item 6 – Video connect to connect a monitor
- Item 7 – USB connector for keyboard or mouse
Dell Chassis Management Controller
- Connecting the power cables to the Dell PowerEdge FX2s chassis powers on the CMC. However, this does not power on the chassis.
- By default, the CMC is configured with static IP address 192.168.0.120 with username/password of root/calvin. Use a laptop with IP address on the same subnet to connect to the CMC and change the IP address.
- The CMC has 2 network interfaces which operates in stacking (default) mode or redundant.
IMPORTANT: Connecting both CMC interfaces to the same switch will cause a network loop if the CMC is NOT configured in redundant mode.
- Stacking mode with single physical link to switch is recommended as it reduces the number of cables when there are multiple chassis and also removes any risk of a network layer-2 loop if both CMC interfaces are connected to the same switch in redundant mode but the CMC is accidentally reverted to stacking mode.
- In stacking mode, the 2nd CMC network interface is used to daisy-chain additional Dell PowerEdge FX2s chassis to be accessed by the single IP address.
- Licensed features for the CMC are shown in the table below. Note that features such as multi-chassis management and one-to-many firmware updates require Enterprise License to be installed.
Feature Express Enterprise CMC Network Yes Yes CMC Serial Port Yes Yes RACADM (SSH, Local, and Remote) Yes Yes WS-MAN Yes Yes SNMP Yes Yes Telnet Yes Yes SSH Yes Yes Web-based Interface Yes Yes Email Alerts Yes Yes CMC Settings Backup No Yes CMC Settings Restore Yes Yes Remote Syslog No Yes Directory Services No Yes Single Sign-On Support No Yes Two-Factor Authentication No Yes PK Authentication No Yes Remote File Share No Yes Enclosure level power capping No Yes Multi-chassis management No Yes FlexAddress Enablement No Yes One-to-many Server Firmware Update No Yes One-to-many configuration for iDRAC No Yes
- CMC supports Windows Active Directory for authentication. An Active Directory service account is NOT required. Users are authenticated to Active Directory during logon.
Dell PowerEdge FN I/O Module
- The Dell PowerEdge FN I/O Module supports 5 configuration modes and is highly configurable to as a Layer-2 or Layer-3 switch supporting ACLs, CoPP, STP, BGP, OSPF, RIP, VRRP, LLDP, PIM, etc.
- The default mode of operation is Standalone Mode which is sufficient for most use-cases. In this mode, it acts as a simple Layer-2 switch with all VLANs enabled. This mode provides the easiest form of administration as VLANs are configured on the servers instead of the I/O modules. Other modes of operation require configuration to be done using the CLI and not supported with the CMC.
- Each I/O Module has 12 interfaces, 8 of which are internal to the servers and 4 which are external. All internal and external interfaces are automatically configured as a LAG with ID 128.
- Details regarding the other configuration modes is available on Dell’s website at http://www.dell.com/support/manuals/au/en/audhs1/poweredge-fx2/fn-iom-126.96.36.199-config-pub/operational-modes?guid=guid-4130e522-8102-44a9-8765-73ece70c7337&lang=en-us
- The diagram below shows the internal interface mappings of the compute sleds to the I/O modules.
NOTE: In the case of the VSAN Ready-Node, the sleds labelled Compute3 and Compute4 are actually storage sleds. As such, the corresponding ports 5-8 on the I/O modules are not active.
QuickDeploy and Server Profiles – Limitation in CMC
Note: Do NOT use Server Profiles in the CMC for full stateless computing.
Dell OpenManage Essentials is required to create Server Profiles and virtual MAC / WWN addresses.
- FlexAddress allows CMC to assign WWN/MAC IDs to a particular slot and override the factory IDs. Hence, if the server module is replaced, the slot based WWN/MAC IDs remain the same. Every server module is assigned unique WWN and/or MAC IDs as part of the manufacturing process. Without FlexAddress, if a server had to be replaced with another server module, the WWN/MAC IDs changes.
- If the server is inserted in a new slot or chassis, the server-assigned WWN/MAC is used unless that chassis has the FlexAddress feature enabled for the new slot.
- QuickDeploy can be used to automatically assign a server profile and/or iDRAC settings to a compute sled when the server is inserted. However, the minimum number of iDRAC IP addresses is 8 which wastes 4 addresses if there are only 4 compute sleds.The MAC Address pool can only be used in a Boot Identity Profile as the intention is to use the virtual MAC address for Boot From SAN or Boot From iSCSI. The virtual MAC addresses can only be assigned to one NIC (either NIC 1-1-1 or 2-1-1) at a time but NOT both NICs. This is a limitation of the CMC as it does not support full stateless computing.
- The MAC Address pool can only be used in a Boot Identity Profile as the intention is to use the virtual MAC address for Boot From SAN or Boot From iSCSI. The virtual MAC addresses can only be assigned to one NIC (either NIC 1-1-1 or 2-1-1) at a time but NOT both NICs. This is a limitation of the CMC as it does not support full stateless computing.
- For true stateless computing, Dell OpenManage Essentials needs to be used.
Dell OpenManage Essentials
- Dell OpenManage Essentials is a free download from Dell and is installed on a Windows server. An SQL database is required although SQL Express edition can be use for small deployments.
- Dell PowerEdge FX2s needs to have the CMC Enterprise license and Dell servers require the iDRAC Enterprise license to be managed by Dell OpenManage Essentials.
- OpenManage Essentials can be used for inventory, monitoring, patch management, configuration management and deployment of Dell servers, chassis and/or IO Aggregator modules. In addition, it is also used to manage the virtual identify of a server.
- After a server has been allocated an iDRAC IP address and root password, it can then be discovered and inventoried by OpenManage Essentials.
- Upon successful discovery and inventory, the server can be added for deployment in the “Repurpose and Bare Metal Devices” section.
- Server templates are created by importing the attributes of an existing server and then can be modified in OpenManage Essentials as required. Settings such as iDRAC and BIOS can be modified and applied to other servers.
- Virtual IO Pool is used to define a pool of MAC addresses, WWPN and WWNN if required. The “Create Virtual IO Pool” task can be used to create a new Virtual IO Pool.
- Devices can then be added into a compute pool for which a server template is applied. Use the “Create Compute Pool” task to create a new compute pool specifying the devices added in the “Repurpose and bare metal devices” section.
- Use the “Deploy Template” task to assign a template to devices in a compute pool specifying the server template and Virtual IO pool.
VSAN Performance Testing
- 4 x FC430 compute sleds with 2 x storage sleds.
- Each storage sled is shared with 2 x compute sleds where disks 0-7 are allocated to compute sled1 and disks 8-15 are allocated to compute sled2.
- Each compute sled is allocated 2 x 380Gb SSD and 6 x 1.5TB SSD.
- VSAN 6.6.1 is installed on each host and configures 2 disk groups on each server with each disk group comprising of 1 x 380Gb cache SSD and 3 x 1.5TB capacity SSDs.
Test Scenario 1 – VSAN RAID5 Erasure Coding / 4 VMs 1TB disk usage
- VSAN disk policy configured as RAID5 Erasure Coding with 1 failure to tolerate. Deduplication and compression is enabled.
- 4 x VMs (2 x vCPU / 4Gb / 250Gb disk) running Windows 2016 Standard with IOMeter.
- VM placement on the 4-node ESXi cluster is as follows:
- iometer03 and iometer04 – Host 02
- iomoter02 – Host 03
- iometer01 – Host 04
- IOMeter setup on each VM with 5 worker threads:
- Testing is based on 8K block size with 33% Write / 67% Read ratio and 50% Sequential/Random distribution.
- Test results shows a total of 85K IOPs max for the 1TB workload run.
- vSAN Backend IOPs results shows a much higher number (approx. total 170K max IOPs) as this is the raw disk IOPs ignoring RAID penalties.
- Performance stats per VM shown below:
Test Scenario 2 – VSAN RAID1 Mirroring / 4 VMs 1TB disk usage
- The VSAN disk policy is reconfigured with RAID1 instead.
- Same test is re-run again after the VMs have re-synced with RAID1.
- Test results shows a total of 100K IOPs max for the 1TB workload run.
- VSAN backend IOPs shows slightly higher (approx. 132K max total IOPs) number not accounting for RAID1 penalties. As expected with RAID1, write and read IOPs are the same.
Test Scenario 3 – VSAN RAID1 Mirroring / 8 VMs 2TB disk usage
- Test Scenario 2 is repeated with 8 identical VMs instead of 4 VMs to demonstrate if the observed IOPs are doubled.
- Test results show approx. 113K max total IOPs for 2TB workload run which is similar to the 1TB workload run in Test Scenario 2.
- VSAN Backend IOPs show approx. 148K max total IOPs.
- Looking at one of the VMs which were used during the 4 VM / 1TB test, the observed IOPs reduced significantly during the 8VM / 2TB test with latency increasing to approx. 67ms.
- This test shows that the system has reached its maximum IOPs limit which tops out at approx. 110K IOPs. Adding additional VMs will just reduce the performance of existing VMs.
Test Scenario 4 – VSAN RAID1 Mirroring with SIOC Enabled / 8 VMs 2TB disk usage
- Disk policies are modified to include VSAN Storage I/O Control.
- 3 SIOC tiers are configured and applied to different VMs.
- High IOP tier is created with 2000 IOP shares with no limit. This is applied to high performance VMs.
- Normal IOP tier is created with 10K IOP limit and 1000 IOP shares. This is applied to majority of the VMs in the cluster and is the default storage policy.
- Low IOP tier is created with 1K IOP limit and 500 IOP shares.
- The same IOMeter tests are run to see if the VMs affected by “noisy neighbour” after the new SIOC storage policies are applied.
- The normal storage policy with 10K IOP limit is applied to VM iometer01. The storage policy is then changed to “low” which limits the IOPs to 1K.
As can be seen from the graph, the IOPs has decreased to 1000 but resulting in high latencies which is expected.
The VM is still functioning fine but performing very slowly due to the increased latencies.
- Performance of other VMs with the 10K IOP limit are also quite stable with higher latency which demonstrates that the limit is being enforced successfully.
This particular configuration of the Dell PowerEdge FX2s VSAN All-Flash Ready-Node (4 x Dell FC430 compute sleds + 2 All-Flash storage sleds) is able to push approximately 110K IOPs for the 35TB usable capacity after RAID1 overhead.
VMware best practice is to leave 20% available for VSAN which leaves approximately 28TB usable capacity in the cluster. Deduplication ratio of approximately 2x is observed during the test.
Switching on VSAN Storage I/O Control would be mandatory to prevent noisy-neighbour scenarios as it was demonstrated that a high IO workload had the potential of crippling other VMs in the cluster without the use of Storage I/O Control. Storage I/O control is a vSphere Enterprise Plus feature.