Selecting SuperMicro components for a Hadoop & Spark Cluster


When creating a Hadoop & Spark Cluster, two types of server configurations (Management and Data nodes) are desired. Unlike many server vendors, SuperMicro not only sells traditional server models with a limited number of option but a wide range of server components (cases, power supplies, motherboards, and adapters) that can be used to create unique server models.

To create the best SuperMicro Management and Data node for Hadoop & Spark required not only a review of available server models but a review of newly released SuperMicro server components that support newer processors and memory.

Presented are two server models created using SuperMicro components. Each server model was selected based on the feature set it offers. Additional adapter cards maybe required in servers to fully realize cluster requirements.

Each server model presented has been validated in cluster deployments and found to meet design expectations for cost, performance and reliability.

 Server Types Service Placement Examples Server Design Focus
Management Node Name node, Job Tracker, HBase Master, Zookeeper, Hive, Oozie, etc… Fast CPU, Memory and Network
Data Node HDFS, GPFS, Task Tracker, HBase Region Fast Dense Storage and Network

Management nodes – SuperMicro SYS-6018R-WTRT

Case – CSE-815TQ-R706WB

The SuperMicro case selected for our Management node is 1U in size to reduce rack space requirements when storage is not critical to the stated purpose of the server. The case supports up to 4 LFF 3.5″ drives allowing the server to support RAID1 or RAID10 storage requirements for a Management node.

Motherboard options available in 1U cases are more limited due to space restrictions imposed by card slot requirements, redundant power supply and drive form factor requirements.

Power Supplies – 750W Redundant

Motherboards – X10DRW-iT

Our selected motherboard supports up to 1TB of memory, dual Intel X540 network ports matching our Data node network selection. One IPMI with KVM over LAN, One 6Gb SATA drive controller with RAID. Two full PCIe slots and one Mezz slot.

CPU and Memory

CPU and memory resources are adjusted based on the workload planned for the Hadoop Cluster. The selected motherboard will support the recommended CPU & Memory.

Cluster Purpose, Management Node(s) CPU Memory (min)
IA – Internet Analytics E5-2640 v3 64GB (1866 MHz)
LZ / DL – Landing Zone / Data Lake E5-2660 v3 128GB (2133 MHz)
NS / CA – NoSQL / Complex Data Analytics E5-2680 v3 128GB (2133 MHz)

Data nodes – SuperMicro SSG-6028R-E1CR12T

Case – CSE-826BE1C-R920LPB

The SuperMicro 826 server case supports up to 12 LFF 3.5″ drives with an optional rear drive cage to support two additional (02) SFF 2.5″ drives. This case support a large selection of motherboards with various memory and PCIe slot combinations.

Power Supplies – 920W Redundant, high-efficiency

Motherboard – X10DRH-CT

Our selected motherboard supports up to 1TB of memory, dual Intel X540 network ports matching our Management node network chip set selection. One IPMI with KVM over LAN, Dual Drive controllers supporting 12Gb SAS & 6Gb SATA both with RAID. Seven PCIe slots are available allowing for feature expansion.

CPU and Memory

Cluster Purpose, Data Nodes CPU Memory (min)
IA – Internet Analytics E5-2620 v3 64GB (1866 MHz)
LZ / DL – Landing Zone / Data Lake E5-2640 v3 64GB (1866 MHz)
NS / CA – NoSQL / Complex Data Analytics E5-2680 v3 128GB (2133 MHz)

CPU and memory resources are adjusted based on the workload planned for the Hadoop Cluster. The selected motherboard will support the recommended CPU & Memory.