Comparing fabric efficiencies of Fixed vs Chassis based designs, non-blocking
Recently I made the observation that Fixed switches continue to outpace Chassis switches in both power and space efficiency. Simply put, with Fixed switches you can cram more ports into fewer RUs, and each port will draw less power when compared to Chassis switches. Those are the indisputable facts.
A common objection when these facts are presented is that ultimately when you go to build a fabric of Fixed switches, that fabric will consume more total power, more total RU, and leave you with a lot more switches and cables to manage when compared to a single Chassis switch. For example, one 384 port line rate chassis switch (Arista 7508) consumes less power and RU when compared to the (10) Dell Force10 Z9000 fixed switches you would need to build a 384 port fabric. While that is true, this is purely academic with no relevance to the real world. Who in their right mind runs their 384 port non-blocking fabric on one switch? Nobody. To carry this flawed logic out a bit further, the largest fabric you could have would be equal to the largest chassis you can find. Nobody wants that kind of scalability limit.
In the real world we build scalable fabrics with more than one switch. So with that in mind lets look at some various non-blocking fabric sizes, staring at 384 ports and on up to 8192 ports. In each fabric size lets compare the total power and RU of the network switches. I’ll make the observation that when you actually construct a real world non-blocking fabric, designs with all fixed switches consume less power and less space than comparable designs with chassis switches. Another interesting observation is the non-blocking fabrics constructed with fixed switches available today result in fewer switches and cables to manage - compared to designs a chassis vendor might propose.
The chassis based design uses a typical 1RU fixed switch as the Leaf (Arista 7050S-64), connecting to a Chassis switch Spine layer (Arista 7508 or 7504), something Arista would likely propose.
The design of all fixed switches uses a 2RU switch at both the Leaf and Spine layer, the Dell Force10 Z9000. The intention here is not to pick on Arista - quite the contrary - I’m using Arista as an example because of the current crop of monstrous power sucking chassis switches, Arista’s are the more efficient (sucking less). Kudos to them for that.
To get straight to the point, lets first look at the overall summary charts. The individual designs and data will follow for those interested in nit-picking.
Fabric Power Efficiency
The chart above shows that fully constructed non-blocking fabrics of all fixed switches are more power efficient than the typical design likely proposed by a Chassis vendor. As the fabric grows the efficiency gap widens. Given we already know that fixed switches are more power efficient than chassis switches, this data should make sense.
Fabric Space Efficiency
Again, the chart above shows a very similar patter with space efficiency. A fully constructed non-blocking fabric of all fixed switches consumes less data center space than the typical design of Chassis switches aggregating fixed switches.
Designs & Data
As you look at the designs below, notice that non-blocking fabrics with fixed switches actually have fewer switches and cables to manage than non-blocking fabrics with Chassis switches – contrary to the conventional wisdom.
Above: (6) Leaf fixed switches, (4) Spine fixed switches interconnected with 40G and providing 384 line rate 10G access ports at the Leaf layer, and 96 inter-switch links. (10) switches total, each with a max rated power consumption of 800W.
Above: (12) Leaf fixed switches, (2) Spine chassis switches interconnected with 10G. Each Leaf switch at 220W max power has 32 x 10G uplink, and 32 x 10G downlink for 384 line rate access ports, and 384 inter-switch links (ISL). The (2) chassis switches are 192 x 10G port Arista 7504 each rated at 7RU and 2500W max power.
Above: (32) Leaf fixed switches, (16) Spine fixed switches interconnected with 40G providing 2048 line rate 10G access ports at the Leaf layer, and 512 inter-switch links. (48) switches total, each with a max rated power consumption of 800W.
Above: (64) Leaf fixed switches each at 220W max power and 1RU, with 32 x 10G inter-switch links, and 32 x 10G non-blocking fabric access ports. (8) Arista 7508 Spine chassis each with (6) 48-port 10G linecards for uniform ECMP. Because each 11RU chassis switch is populated with 6 linecards of 8 possible, I’ve factored down the power from the documented max of 6600W, down to 5000W max. (72) total switches.
Above: (64) Leaf fixed switches, (32) Spine fixed switches interconnected with 10G providing 4096 line rate 10G access ports at the Leaf layer, and 4096 inter-switch links. (96) switches total, each with a max rated power consumption of 800W.
Above: (128) Leaf fixed switches each at 220W max power and 1RU, with 32 x 10G inter-switch links, and 32 x 10G non-blocking fabric access ports. (16) Arista 7508 Spine chassis each with (6) 48-port 10G linecards for uniform ECMP. Because each 11RU chassis switch is populated with 6 linecards of 8 possible, I’ve factored down the power from the documented max of 6600W, down to 5000W max. (144) total switches.
Above: (64) Leaf fixed switches, (128) Spine fixed switches interconnected with 10G providing 8192 line rate 10G access ports at the Leaf layer, and 8192 inter-switch links. (192) switches total, each with a max rated power consumption of 800W.
Above: (256) Leaf fixed switches each at 220W max power and 1RU, with 32 x 10G inter-switch links, and 32 x 10G non-blocking fabric access ports. (32) Arista 7508 Spine chassis each with (6) 48-port 10G linecards for uniform ECMP. Because each 11RU chassis switch is populated with 6 linecards of 8 possible, I’ve factored down the power from the documented max of 6600W, down to 5000W max. (288) total switches.
Conclusion
When building a non-blocking fabric, a design of all fixed switches scales with better power and space efficiency, and with fewer switches and cables (if not the same), when compared to designs with chassis switches.
Follow-up post: Architecting Data Center Networks in the era of Big Data and Cloud
Cheers, Brad