On “Why TRILL wont work for the data center”

Today I came across “Why TRILL won’t work for data center network architecture” by Anjan Venkatramani of Juniper. Anjan’s article makes a few myopic and flawed arguments in slamming TRILL, setting up a sale for QFabric.  The stated problems with TRILL include FCoE, L3 multi-pathing, VLAN scale, and large failure domains. The one and only Ivan Pepelnjak has already tackled the flawed FCoE argument, be sure to read that, so I’ll opine here on the L3, VLAN scale, and failure domain arguments.

Anjan writes this about L3 gateways in a TRILL network:

While TRILL solves multi-pathing for Layer 2, it breaks multi-pathing for Layer 3. There is only one active default router with Virtual Router Redundancy Protocol (VRRP), which means that there is no multi-pathing capability at Layer 3.

This is a bit shortsighted and assumes that we are simply stuck with existing L3 gateway protocols of today like VRRP, and therefore you just have to use those in a TRILL network. Why? As the L2 technology evolves it makes perfect sense to look at how L3 protocols should evolve with it.  For example, it’s entirely possible that a simple Anycast method could be used for the L3 gateway in TRILL. In short, each L3 TRILL switch would have the same IP address and same MAC address for the L3 default gateway. The server resovles the L3 gateway to this MAC address which is available on ALL links, because each TRILL spine switch is originating it as if it were its own. The L2 edge switch now makes a simple ECMP hash calculation to decide which L3 switch and link receives the flow.  Simple, right?  The same Anycast concept can also be used for services, such as load balancers and firewalls.

Anjan’s L3 gateway argument is a setup for stating why fewer VLANs should be used with TRILL, thus requiring more hosts on each VLAN (to reduce the L3 switching bottlenecks) thereby adding to scalability problems. Therefore, any subsequent argument related to having too many hosts on one VLAN can be dismissed as FUD based on a shortsighted premise. There’s no reason to change the number of VLANs you deploy with TRILL, or the number of hosts per VLAN.

Anjan continues about TRILL failure domains:

Security and failure isolation are a real concern in the TRILL architecture. Both issues stem from being artificially forced into large broadcast domains. Flapping interfaces, misbehaving or malicious applications and configuration errors can cause widespread damage and in a worst case scenario result in a data center meltdown.

Again, the “large broadcast domains” can be dismissed as myopic FUD. There would be no reason to have larger than normal broadcast domains in a TRILL deployment.

Now, lets talk about “configuration error” and the resulting “wide spread damage”. Coming from Juniper, lets acknowledge the somewhat obvious ulterior motive of selling their (still in slideware) QFabric architecture. Given that, the assumption Juniper would like you to believe is that QFabric would be much less vulnerable to wide spread damage from configuration error than a TRILL network.  But how can that be possible?  On what basis?

The QFabric architecture resembles that of one big proprietary 128-slot switch. One configuration change affects the entire architecture, for better or worse. How is Juniper proposing their architecture is any less vulnerable to disastrous configuration mistakes? If anything, a single network-wide configuration input such as you get with QFabric only increases this risk.  No?  Why not?  Furthermore, why would “security and failure isolation” be less of a concern with Juniper QFabric, compared to any other standards based architecture such as TRILL?

Would any of the Juniper folks like to state their case, and enlighten me? :-)


Disclaimer: The author is an employee of Cisco Systems, Inc. However, the views and opinions expressed by the author do not necessarily represent those of Cisco Systems, Inc. The author is not an official media spokesperson for Cisco Systems, Inc.

“Jawbreaker”, merchant silicon, QFabric, and flat networks

Brad, can you elaborate on Cisco’s Jawbreaker project? What exactly is it? Is it a response to Juniper’s Q-Fabric? Is it an attempt to rectify the inconsistencies in the differing purpose-built approaches of the N7K and N5K?

Why create a new architecture?

It seems like Cisco is really in trouble – creating a new architecture, abandoning its own silicon for merchant silicon…they seem to have missed the boat with regard to flat networks.

Here is an article worth commenting on:



For reasons that should be fairly obvious I can’t discuss in detail rumored Cisco R&D projects such as “Jawbreaker” in a public forum.

I did read the Network World article and found it suspiciously interesting how many presumptions were made about a Cisco R&D project, such as ship dates, architecture, motivations, etc. As far as I’m concerned, some of these assertions were just flat-out wrong.

Cisco, like any other tech company, is going to have R&D projects with cool sounding names. Some will survive and turn into real products (e.g. Cisco UCS), others will never see the light of day. It’s how any good tech company flushes out the good ideas from the bad.

You can also bet that Cisco is looking at ways to evolve the Nexus platform with dense and highly scalable 40G, 100G, and beyond. I think that’s a fairly obvious assumption to make. No? Just because a project may have a name like “Jawbreaker” doesn’t mean it’s not Nexus.

Merchant Silicon

Depending on the application, sometimes merchant silicon does the job well. Take for example the Nexus 3000. If your application just needs the lowest possible latency for say, high frequency trading, the merchant silicon available today does that well, really well. Therefore it makes sense for both Cisco and it’s customers to have timely access to these products at competitive prices, running industry proven and feature rich switch software (NX-OS), and backed by Cisco’s global partner network and world-class 24×7 Cisco TAC support.

While merchant silicon does speeds and feeds well, it’s not the place to find innovation. Custom software, custom hardware (ASIC), or a clever combination of the two is where new and innovative technologies will be introduced for immediate customer benefit. I don’t see that changing one bit. And you can bet Cisco will continue to lead in this arena.

There is a significant trend underway in software driven innovations, such as software defined networking (SDN). At some point you can only drive so many features into hardware so fast, while the innovation potential and velocity of software development is almost limitless, as far as I can tell. Some will say SDN means the end of custom hardware, just use merchant silicon everywhere and innovate only in software. Sorry, I don’t buy it. I tend to believe the best possible outcomes will result from an intentional mix of purpose-built software (SDN) and purpose-built hardware (ASIC).

Those that are only capable of innovating in one of the two areas (hardware or software) will do OK.  However those that can engineer both software and hardware innovations in a single system are best positioned for the next wave of innovation, IMHO.

Juniper QFabric

I have to give credit to Juniper for reaching into new territory with QFabric. It’s an interesting and bold concept. There, I said it.

Bold in the sense that Juniper is asking the customer to invest in one giant proprietary 128 “slot” switch. Most people are comfortable with an 18-slot switch, it’s not too much capacity to commit to one vendor, and not too big a failure domain. But a 128 slot switch? That’s unprecedented territory. I’ll be curious to see to how well that message is received once QFabric becomes a reality (still slideware as of today).

Interesting in the sense that each edge QF-Node has the posture much like a distributed forwarding linecard in a chassis switch. One of the challenges with this architecture is hardware consistency across all of the “linecards”. Those of you familiar with distributed forwarding chassis switch architecture know that if you have a chassis full of linecards, the entire switch has to dumb down to that of the least capable card in the system, to maintain system wide consistency. I’m curious to see how that will be managed in a 128 slot QFabric as customers try to simply add or migrate in-flight to newer QF-Node edge technology.

Flat networks

“[Cisco] seemed to have missed the boat with regard to flat networks”

Really? I don’t get that. Help me understand because Cisco is the only vendor shipping a 16-way “Spine” today with FabricPath, based on TRILL. Consider that each Spine can be an 18-slot modular switch with 512 10G ports. That’s a tremendous amount of capacity and bandwidth to build very a large “flat network”, available today. There are lots of vendors talking about “flat networks”, but which ones are actually shipping?

Of course today’s offering isn’t perfect and there will be improvements. In the near future you will have Nexus 5500 supporting FabricPath, perfect for the ToR “Leaf”. You will also have newer generations of Nexus 7000 linecards with higher 10G density and L2/L3 line rate switching, supporting FabricPath as well in the same 16-way Spine topology. All of this allowing for an even larger and higher performance “flat network” than is already available today.

Cisco “missed the boat” with regard to “flat networks”? I beg to differ. The boat is actually carrying Cisco spine/leaf “flat network” gear to customer door steps today, while the others are still showing slides. :-)


Disclaimer: The author is an employee of Cisco Systems, Inc.  However, the views and opinions expressed by the author do not necessarily represent those of Cisco Systems, Inc. The author is not an official media spokesperson for Cisco Systems, Inc.