Return to site

New Toys.. Hardware Specs

And why create a portable home lab.

· VMware,Home Lab Hacking

Lets talk about hardware..

by Simon Richardson - VMware

Apologies for the delay, I finally found some time in between endless zoom meetings to write this guest piece on the Lab Hardware we are working on.

The first question we get asked is: Why are we doing this? Is this a repeat of previous work on Edge use cases?

In my role at VMware, I have been spending a portion of my time working on Edge use cases, industry-specific solutions and what that hardware/software solution for the Edge might look like.

We are increasingly seeing customers and partners with a desire to run workloads in a secure, performant manner that is good enough to enable workloads at Edge locations. These workloads could be anything from simple Virtual Machines, to complex k8s deployments, the common theme we are seeing is the desire for lower cost, smaller footprint, lower heat, lower power requirements, whilst retaining high availability and the ability to manage in a consistent manner from a central location.

Our first work was on Project nanoEDGE (Blog Link), hit the link for all the details of the work we performed to validate the solution. The blog talks about the answer to a simple question “what would a small format SDDC look like if not constrained by traditional rack/rails that were good enough to run 25 workloads”. I’m confident that our work on Project nanoEDGE provided a validated solution and BOM so that a customer and partner could simply replicate the work we had done to provide an SDDC at the Edge.

Ok, what's the difference?

We have a fully validated solution from our colleagues at Supermicro, after the Project nanoEDGE work we reached out to Supermicro and asked if there was a selection of servers we could use that would enable us to start looking at what a ‘Kubernetes at the Edge’ solution may look like.

To do this we wanted to make sure that we had a validated solution that was truly representative of what a customer could buy today, that was fit for purpose and performant. The solution also needed to provide all the benefits of the VMware stack, security, performance, consistency and operational management.

Our requirements were a number of ‘small’ single socket, multi-core, 64-128Gb RAM, 10GBe NICs, vSphere 7 compatible servers, configurable with multiple flash storage devices, essentially a validated small vSAN Server. The Supermicro E300-9D-8CN8TP was the perfect fit for our Edge requirements.

We have worked previously with Supermicro Servers using the E200-8D for a Proof of Concept on what a very small vSphere Server might be, to my knowledge, this is still one of if not the smallest vSphere Servers on the VMware Compatibility Guide

A big benefit for us is that we now have the servers in the UK making it much easier to work on, fortunately, we managed to arrange shipping just before the COVID-19 hit the UK shores.

Four servers were provided with a configuration considered as a realistic vSphere/vSAN configuration, the servers are the Supermicro E300-9D-8CN8TP -

Our plan is to work through a number of configurations over time and get those written up, its key to note that the Hardware Bill of Materials provides validated vSAN components, which equally means that the solution should be able to provide the nodes for a VMware Cloud Foundation approach.

The Supermicro servers are validated for vSphere 6.7 U3 and vSphere 7.0 which is crucial depending on the configuration we choose to host our Kubernetes based workloads.

VMware Compatibility Guide - Link

OK, get to the good stuff, what is the configuration?

Item

Description

Qty

E300-9D-8CN8TP

Skylake D, X11SDV-8C-TP8F, E300 + 150W power adapter

4

MEM-DR432L-SL03-ER26

32GB DDR4-2666 2Rx4 ECC REG DIMM

16

HDS-MMT-MTFDDAV1T0TDL-12

Micron 1300 1TB SATA M.2 22X80mm TLC SED <1DWPD

4

HDS-IMN0-SSDPELKX010T8

Intel DC P4511 1T NVMe PCIe3.1x4 M.2 22x110mm 1DWPD

4

HDS-IMN0-SSDPEL1K375GA

Intel3D XPointDC P4801X 375G PCIe3.0x4 60DWPD M.2 22x110

4

AOC-SLG3-2M2

LP, PCIe3 x8, Dual port NVMe M.2 carrier

4

RSC-RR1U-E8

PCIe x8 Riser Card

4

Form Factor

The Server itself is small, not quite E200-D small, but, small when compared to a traditional rack and rails server. The E300-9D is small and light enough to fit in a rucksack and taken as carry on on a plane, 1.56kg for the server itself. It is surprising how small a Server can be that is performant and capable of hosting multiple Virtual Machines and Containers.

Dimensions

Width 10" (254mm)

Height 1.7" (43mm)

Depth 8.9" (226mm)

Weight Net Weight: 3.45 lbs (1.56 kg)

CPU

We have the Supermicro E300-9D-8CN8TP, using the Skylake D 8 Core processor with 16 threads, although a single socket, this provides a good bang for buck on performance and physical size, power requirements and heat output.

There is always a trade-off between cost/performance/power/size/heat, in my mind the Skylake-D and previous generation Xeon-D provide an excellent processor for these smaller form factor servers, the ideal processor for Edge workloads.

Paul Branen at Tinkertry does an excellent write up on Skylake-D -

From the picture above it's clear to see how tightly packaged the whole server is, air is drawn through the front to the back.

Interestingly there are also 2x U2 Connectors available for connection to compatible nvme devices.

The Server itself has a single 150W power supply, some would question the wisdom of an Enterprise Server with a single point of failure, I’d argue that a single power supply allows us to have a smaller form factor, lower cost and generates less heat, all requirements typical for Edge type use cases. We can mitigate some of the missing hardware features in software, utilising High Availability and Shared Storage in software we can greatly simplify the requirements of the Hardware.

Memory

Each Server can hold 512GB DDR4 LRDIMM RAM, our configuration is a generous 128GB, prices of RAM remain stubbornly high at the time of writing, as such 128GB is a good cost/capacity balance. Sticking with RDIMM reduces the capacity to 256GB RAM. The ability to move up to 512GB RAM offers some significant headroom for future scaling.

Storage

There are a number of storage devices within the servers, we intend to use the 1TB SATA SSD as the Boot Device, this leaves us with two remaining M2 storage devices, in our configuration the two vSAN storage devices are nvme the remaining device is SATA based.

A supported configuration is to use a SATADOM for the boot device, we chose to use an M2 device instead, the supplied Micron M2 is a 1TB SATA Device which is large in size, a 256GB would be ideal - SATA DOM Solutions

Clearly, for a vSAN flash based configuration we would want to take advantage of the nvme devices, one huge advantage of using nvme devices is the controller is on the device, there is no requirement for a separate storage controller, this gives greater flexibility in the packaging of small form factor servers.

A write-up of why nvme is such an important step for flash devices - VSAN now supporting NVMe devices - Virtual Blocks

The two nvme devices are attached to an add-on card, the AOC-SLG3-2M2, which is in turn plugged into a right angle PCIe Adapter, part code RSC-RR1U-E8.

The two Storage devices used for vSAN Storage are a 1TB M.2 Intel NVMe TLC SSD for capacity and a 375GB M.2 Intel Optane for cache.

Details below :

Intel DC P4511 1T NVMe PCIe3.1x4 M.2 22x110mm 1DWPD

Intel3D XPointDC P4801X 375G PCIe3.0x4 60DWPD M.2 22x110

Network

Networking is well catered for, importantly for vSAN we have

  • 2x 10GBase-T ports,

  • 2x 10G SFP+ ports and

  • 4x 1Gbe

  • 1x IPMI

An excellent writeup from William Lam on the networking and drivers available - Supermicro E300-9D (SYS-E300-9D-8CN8TP) is a nice ESXi & vSAN kit

  • 4 x 1GbE is based on Intel I350 and uses the igbn driver

  • Both 2 x 10GbE (RJ45) and 2 x 10GbE (SFP+) is based on Intel X722 and uses the i40en driver

In our case we are running a 1Gbe switch, although it is recommended to use 10Gbe where possible to prevent any throughput issues on vSAN.

Recap

We have a supported 4-node vSphere 7 vSAN solution, totalling 5128GB RAM, 32 Cores with 64 threads, 16x 10Gbe connections with 8 available with RJ45 and 8 available via SFP+, cache and capacity tiers on nvme devices.

All of which weights less than 7kgs and would still fit in a carry on bag for most airlines, we’ve built a powerful Edge solution able to demonstrate a number of valid solutions.

With the hardware up and running we have a number of solution areas that we intend ot work through ..

  • Initial install and configuration of k8s utilising Tanzu Kubernetes Grid

  • Sizing of k8s and what workloads can be run on Edge systems

  • Bare Metal k8s vs VM based k8S

  • Deployment of ‘interesting’ workloads.

With Thanks ..

We certainly aren’t the first to look at the use of E200 or E300 Supermicro servers for Edge use cases, there are a number of excellent Blogs that have a significant amount of research and information on these form factors.

With thanks to both William Lam, Paul Branen and David Cheung who inspired our pursuit of Supermicro small form factor servers.

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OK