Seven reasons VMware NSX, Cisco UCS and Nexus are orders of magnitude more awesome together

Note: This article was originally written for and published at the VMware Network Virtualization Blog. The following is a verbatim re-post of the original content.

“VMware NSX, Cisco UCS and Cisco Nexus, TOGETHER solve many of the most pressing issues at the intersection of networking and virtualization.”

Executive Summary

VMware NSX brings industry-leading network virtualization capabilities to Cisco UCS and Cisco Nexus infrastructures, on any hypervisor, for any application, with any cloud management platform.  Adding state of the art virtual networking (VMware NSX) to best-in-class physical networking (Cisco UCS & Nexus) produces significant optimizations in these key areas:

  • Provision services-rich virtual networks in seconds
  • Orders of magnitude more scalability for virtualization
  • The most efficient application traffic forwarding possible
  • Orders of magnitude more firewall performance
  • Sophisticated application-centric security policy
  • More intelligent automation for network services
  • Best-of-breed synergies for multi data center
  • More simplified network configurations

Cisco UCS and Nexus 7000 infrastructure awesomeness

A well-engineered physical network always has been and will continue to be a very important part of the infrastructure. The Cisco Unified Computing System (UCS) is an innovative architecture that simplifies and automates the deployment of stateless servers on a converged 10GE network. Cisco UCS Manager simultaneously deploys both the server and its connection to the network through service profiles and templates; changing what was once many manual touch points across disparate platforms into one automated provisioning system. That’s why it works so well. I’m not just saying this; I’m speaking from experience.

Cisco UCS is commonly integrated with the Cisco Nexus 7000 series; a high-performance modular data center switch platform with many features highly relevant to virtualization, such as converged networking (FCoE), data center interconnect (OTV), Layer 2 fabrics (FabricPath, vPC), and location independent routing with LISP. This typically represents best-in-class data center physical networking.

With Cisco UCS and Nexus 7000 platforms laying the foundation for convergence and automation in the physical infrastructure, the focus now turns to the virtual infrastructure. VMware NSX, when deployed with Cisco UCS and Cisco Nexus, elegantly solves many of the most pressing issues at the intersection of networking and virtualization. VMware NSX represents the state of the art for virtual networking.

1) Virtualization-centric operational model for networking

VMware NSX adds network virtualization capabilities to existing Cisco UCS and Cisco Nexus 7000-based infrastructures, through the abstraction of the virtual network, complete with services such as logical switching, routing, load balancing, security, and more. Virtual networks are deployed programmatically with a similar speed and operational model as the virtual machine — create, start, stop, template, clone, snapshot, introspect, delete, etc. in seconds.

The virtual network allows the application architecture (including the virtual network and virtual compute) to be deployed together from policy-based templates, consolidating what was once many manual touch points across disparate platforms into one automated provisioning system. In a nutshell, VMware NSX is to virtual servers and the virtual network what Cisco UCS is to physical servers and the physical network.

2) More headroom for virtualization, by orders of magnitude (P*V)

VMware NSX provides the capability to dynamically provision logical Layer 2 networks for application virtual machines across multiple hypervisor hosts, without any requisite VLAN or IP Multicast configuration in the Cisco UCS and Cisco Nexus 7000 infrastructure. For example, thousands of VXLAN logical Layer 2 networks can be added or removed programmatically through the NSX API, with only a few static infrastructure VLANs; compared to what was once thousands of manually provisioned VLANs across hundreds of switches and interfaces.

Figure: NSX dynamic logical Layer 2 networks

Two of the most common breaking points when scaling a network for virtualization are:

  1. Limited number of STP logical port instances the switch control plane CPUs can support, placing a ceiling on VLAN density.
  2. Limited MAC & IP forwarding table resources available in switch hardware, placing a ceiling on virtual machine density.

VLANs and virtual machines; two things you don’t want a visible ceiling on. Fortunately, VMware NSX provides significant headroom for both, by orders of magnitude, for the simple reason that VLAN and STP instances are dramatically reduced; and hardware forwarding tables are utilized much more efficiently.

Consider (P1 * V1) = T. Switch ports * number of active VLANs = STP logical ports.

One thousand fewer infrastructure VLANs with VMware NSX translates into one thousand times fewer STP logical port instances loading the Cisco UCS and Nexus 7000 control plane CPUs. This can only help ongoing operational stability, along with the obvious scaling headroom.

Consider (P2 * V2) = D. Physical hosts * VMs per host equals virtual machine density.

Normally, the size of the MAC & IP forwarding tables in a switch roughly determines the ceiling of total virtual machines you can scale to (D), as each virtual machine requires one or more entries. With VMware NSX, however, virtual machines attached to logical Layer 2 networks do not consume MAC & IP forwarding table entries in the Cisco UCS and Nexus 7000 switch hardware. Only the physical hosts require entries. In other words, with VMware NSX, the ceiling is placed on the multiplier (P2), not the total (D).

Reduced VLAN sprawl and logical Layer 2 networks compound to both simplify the Cisco UCS and Nexus configurations and significantly extend the virtualization scalability and virtual life of these platforms.

3) Most efficient application traffic forwarding possible

Have you ever noticed the paradox that good virtualization is bad networking? For example, the network design that works best for virtualization (Layer 2 fabric) isn’t the best design for Layer 3 traffic forwarding, and vice versa. That is, until now.

VMware NSX provides distributed logical Layer 3 routing capabilities for the virtual network subnets at the hypervisor kernel. Each hypervisor provides the Layer 3 default gateway, ARP resolver, and first routing hop for its hosted virtual machines.  The result is the most efficient forwarding possible for east-west application traffic on any existing Layer 2 fabric design, most notably Cisco UCS.

Figure: NSX Distributed Layer 3 routing — intra host

In the diagram above, VMware NSX distributed logical routing provides east-west Layer 3 forwarding directly between virtual machines on the same Cisco UCS host, without any hairpin hops to the Cisco Nexus 7000 — the most efficient path possible.

VMware NSX spans multiple Cisco UCS hosts acting as one distributed logical router at the edge. Each hypervisor provides high performance routing only for its hosted virtual machines in the kernel I/O path, without impact on system CPU. Layer 3 traffic between virtual machines travels directly from source to destination hosts inside the non-blocking Cisco UCS fabric — the most efficient path possible.

Figure: NSX Distributed Layer 3 routing — inter host

This efficient Layer 3 forwarding works with the existing Cisco UCS Layer 2 fabric, keeping more east-west application traffic within the non-blocking server ports, minimizing traffic on the fewer uplink ports facing the Cisco Nexus 7000 switches.

With Layer 3 forwarding for the virtual network handled by the hypervisors on Cisco UCS, the Cisco Nexus 7000 switch configurations are simpler; because VMware NSX distributed routing obviates the need for numerous configurations of virtual machine adjacent Layer 3 VLAN interfaces (SVIs) and their associated HSRP settings.

Note: HSRP is no longer necessary with the VMware NSX distributed router, for the simple reason that virtual machines are directly attached to one logical router that hasn’t failed until the last remaining hypervisor has failed.

The Cisco Nexus 7000 switches are also made more scalable and robust as the supervisor engine CPUs are no longer burdened with ARP and HSRP state management for numerous VLAN interfaces and virtual machines.  Instead, VMware NSX decouples and distributes this function across the plethora of x86 CPUs at the edge.

4) More awesome firewall, by orders of magnitude (H*B)

Similar to the aforementioned distributed logical routing, VMware NSX for vSphere also includes a powerful distributed stateful firewall in the hypervisor kernel, which is ideal for securing east-west application traffic directly at the virtual machine network interface (inspecting every packet) with scale-out data plane performance. Each hypervisor provides transparent stateful firewall inspection for its hosted virtual machines, in the kernel, as a service – and yet all under centralized control.

The theoretical throughput of the VMware NSX distributed firewall is some calculation of (H * B). The number of Hypervisors * network Bandwidth per hypervisor. For example, 500 hypervisors each with two 10G NICs would approximate to a 20 Terabit east-west firewall.

Figure: NSX Distributed Firewall — intra host

As we see in the diagram above, the distributed firewall provides stateful east-west application security directly between virtual machines on the same Cisco UCS host, without any hairpin traffic steering through a traditional firewall choke point. Zero hops. The most efficient path possible.

The VMware NSX distributed firewall spans multiple Cisco UCS hosts, like one massive firewall connected directly to every virtual machines. Each hypervisor kernel provides the stateful traffic inspection for its hosted virtual machines. In other words, traffic leaving a Cisco UCS host and hitting the fabric has already been permitted by a stateful firewall, and is therefore free to travel directly to its destination (where it’s inspected again).

Figure: NSX Distributed Firewall — inter host

Given the VMware NSX distributed firewall is directly adjacent to the virtual machines, sophisticated security policies can be created that leverage enormous amount of application-centric metadata present in the virtual compute layer (things such as user identity, application groupings, logical objects, workload characteristics, etc.); far beyond basic IP packet header inspection.

As a simple example, a security policy might say that protocol X is permitted from the logical network “Web” to “App”  – no matter the IP address.  Consider a scenario where this application is moved to a different data center, with different IP address assignments for “Web” and “App” networks; and having no affect on the application’s security policy.  No need to change or update firewall rules.

Finally, we can see again that more east-west application traffic stays within the low latency non-blocking Cisco UCS domain — right where we want it.  This can only help application performance while freeing more ports on the Cisco Nexus 7000 previously needed for bandwidth to a physical firewall.

5) More awesome network services

One of the more pressing challenges in a virtualized data center surrounds efficient network service provisioning (firewall, load balancing) in a multi-tenant environment. Of particular importance are the services establishing the perimeter edge — the demarcation point establishing the application’s point of presence (NAT, VIP, VPN, IP routing). Typical frustrations often include:

  • Limited multi-tenancy contexts on hardware appliances
  • Static service placement
  • Manually provisioned static routing
  • Limited deployment automation
  • Service resiliency

To address this, VMware NSX includes performance optimized multi-service virtual machines (NSX Edge Services), auto deployed with the NSX API into a vSphere HA & DRS edge cluster. Multi-tenancy contexts are virtually unlimited by shifting perimeter services from hardware appliances to NSX Edge virtual machines on Cisco UCS.

Figure: Sample VMware NSX logical topology on Cisco UCS

Dynamic IP routing protocols on the NSX Edge (BGP, OSPF, IS-IS) allow the Cisco Nexus 7000 switches to learn about new (or moved) virtual network IP prefixes automatically — doing away with stale and error prone static routes.

VMware NSX Edge instances leverage HA & DRS clustering technology to provide dynamic service placement and perpetual N+1 redundancy (auto re-birth of failed instances); while Cisco UCS stateless computing provides the simplified and expedient restoration of service capacity (re-birth of failed hosts).

Figure: Application traffic flow. Before & After

With VMware NSX, traffic enters the Cisco UCS domain where all required network services for both north-south and east-west flows are applied using high performance servers within the non-blocking converged fabric, resulting in the most efficient application flows possible.

Note: VMware NSX is also capable of bridging virtual networks to physical through the NSX Edge, where specific VXLAN segments can be mapped to physical VLANs connecting physical workloads, or extended to other sites.

6) Divide and Conquer multi data center

Solving the multi data center challenge involves tackling a few very different problem areas related to networking. Rarely does one platform have all the tools to solve all of the different problems in the most elegant way. It’s usually best to divide and conquer each problem area with the best tool for the job. In moving an application from one data center to another, the networking challenges generally boil down to three problem areas:

  1. Recreate the application’s network topology and services
  2. Optimize Egress routing
  3. Optimize Ingress routing

In abstracting the virtual network, complete with Logical Layer 2 segments, distributed logical routing, distributed firewall, perimeter firewall, and load balancing, all entirely provisioned by API and software, VMware NSX is the ideal tool for quickly and faithfully recreating the applications network topology and services in another data center. At this point the NSX Edge provides the application a consolidated point of presence for optimized routing solutions to solve against.

Figure: Multi data center with VMware NSX, Cisco OTV and LISP

The next problem area — optimized egress routing — is ideal for a tool like OTV on the Cisco Nexus 7000 series, where the virtual network’s NSX Edge is given a consistent egress gateway network at either data center, with localized egress forwarding. Cisco OTV services are focused on the DMZ VLAN and the NSX Edge, and not burdened with handling every individual network segment, every virtual machine, and every default gateway within the application. With this simplicity the OTV solution becomes more scalable to handle larger sets of applications, and easier to configure and deploy.

With the Cisco Nexus 7000 and OTV keying on the NSX Edge (via VIPs and IP routing) for the application’s point of presence, this serves as in ideal layering point for the next problem area of optimized ingress routing. This challenge is ideal for tools such as BGP routing, or LISP on the Cisco Nexus 7000 switches and LISP capable routers; delivering inbound client traffic immediately and directly to the data center hosting the application.

7) A superior track record of integration and operational tools

It’s hard to think of two technology leaders with a better track record of doing more operationally focused engineering work together than Cisco and VMware. Examples are both recent and plenty; such as the Cisco Nexus 1000V, Cisco UCS VM-FEX, Cisco UCS Plugin for VMware vCenter, the Cisco UCS Plugin for VMware vCenter Orchestrator, and so on.

Operational visibility is all about providing good data and making it easily accessible. A comprehensive API is the basis on which two industry leaders can engineer tools together exchanging data to provide superior operational visibility. Cisco UCS and VMware NSX are two platforms with a rich API engineered at its core (not a bolted on afterthought). When looking at both the track record and capabilities of VMware and Cisco, working together to serve their mutual customer better, we’re excited about what lies ahead.

In closing

VMware NSX represents best-in-class virtual networking, for any hypervisor, any application, any cloud platform, and any physical network.  A well-engineered physical network is, and always will be, an important part of the infrastructure. Network virtualization makes it even better by simplifying the configuration, making it more scalable, enabling rapid deployment of networking services, and providing centralized operational visibility and monitoring into the state of the virtual and physical network.

The point of this post is not so much to help you decide what your data center infrastructure should be, but to show you how adding VMware NSX to Cisco UCS & Nexus will allow you to get much more out of those best-in-class platforms.

Brad Hedlund
Engineering Architect
VMware NSBU

About the Author ()

Brad Hedlund is an Engineering Architect with the CTO office of VMware’s Networking and Security Business Unit (NSBU), focused on network & security virtualization (NSX) and the software-defined data center. Brad’s background in data center networking begins in the mid-1990s with a variety of experience in roles such as IT customer, systems integrator, architecture and technical strategy roles at Cisco and Dell, and speaker at industry conferences. CCIE Emeritus #5530.

Comments (27)

Trackback URL | Comments RSS Feed

  1. DWP says:

    Interesting how this runs contrary to the market. As VMware continues to extend its solutions, hardware assets trend toward commoditization. How would the solutions above be impacted if Arista, HP, Huawei networking and any x86 compute assets were supplemented? Will vSAN diminished the lethargic uptake of FCoE?

    I see the power of NSX in enabling a flat datacenter network of highly available commodity assets. Why pay extra for an IP bus?

  2. Cawnpore Charlie says:

    What if vSAN becomes very popular – could you please describe an alternate physical and logical network architecture with support for vSAN?

  3. Josh says:

    Interesting blog post. Nsx is a really great set of technologies that really complements a virtualized private cloud infrastructure well.

    However I’m a little unclear about the benefit cisco brings to the table here. At the end of the day, thanks to nsx, all the benefits and whiz bang features of ucs and n7k seem to be much less important, and with virtualization ratios increasing rapidly, the benefits of blades are of less importance as well.

    It seems to me that you can get equally optimized traffic flows and an equally great private cloud using nexus 6000/5000, brocade dcx, Arista, force 10, or any of the other data center type switches and an hp 1u rackmount server for a lot less money….

    Hate to be Debbie downer on the Cisco offering. I love the n7k (I operate many of them in my networks), but I think nsx is far more disruptive to cisco than vmware and Cisco are letting on…

    • Luke D says:

      I’m with Josh here… what benefits does UCS and Nexus bring to this discussion? I can see how NSX add enormous capability but you haven’t explained how the Cisco components add/combine value to whole solution. Just that some of the constraints of the UCS stack (eg. East-West traffic) are somewhat mitigated in an NSX environment.

      What can you do in UCS/Nexus/NSX that you couldn’t on a comparable Dell/Arista/NSX design (to pick two arbitrary products) ?

  4. wd says:

    - How can you be sure in Fig inter host L3, that both Interfaces are on the same fabric, A in your case; if one is in A, the other in B, traffic has to go via N7K, assuming UCS EHM ?
    - Any Interoperability Matrix OSPF,…. between NSX and any Cisco Router ?
    - EIGRP Support ?
    - NSX … for any hypervisor …… does it mean Support for Hyper-V, KVM,….

    • wd says:

      After some digging, I found most of the answers; see eg. http://www.vmware.com/products/nsx/resources.html

      Cisco is not on the list of supported “Layer 2 and Layer 3 Gateway Services to connect physical and virtual Environments”

      Multihypervisor Support = Xenserver / KVM

    • Brad Hedlund says:

      Hi Walter,

      In the Figure “NSX dynamic Logic Layer 2 networks” I am showing a Transport VLAN attached to vnic0 on each UCS blade. The vmknics created for VXLAN would be assigned to this VLAN. As the UCS admin you could configure the system such that all your vnic0 interfaces are on the same fabric A, failing over to fabric B.

      NSX is designed to support any hypervisor. Today that means KVM, Xen, and ESX. This can easily be extended to other hypervisors such as Hyper-V. It’s software, so it’s just a matter of porting the NSX vSwitch over to those systems.

      As for routing protocols, we implement the RFC specs for standard protocols OSPF, BGP, and ISIS, and expect that the link partner is also implementing the well known RFCs.

      Cheers,
      Brad

      • Hywel Burris says:

        Hi,

        With VM mobility across multiple sites (for both SRM & vmotion) what routing protocol would you suggest for adjacent layer 2 subnets. The “normal” cisco way would be to implement LISP but is there another way in NSX?

        Cheers

        • Brad Hedlund says:

          Hi,
          With NSX, you wouldn’t need to bother with stretching Layer 2 subnets between sites, because you can recreate the complete L2-L7 logical network (with a high degree of granularity), as quickly as you can start a VM.
          Stretching a subnet between data centers is what you do when you know two things are true:
          1) it takes a long time to manually configure networking (VLANs, L3 gateways, etc.)
          2) you never know when action #1 is going to be needed.
          So you place the same VLAN in two places at once, in advance, and accept all of the L3 traffic trombone gotchas and configuration complexities.
          At some point you realize that stretching VLANs was only half the battle and you still need to figure out how to stretch or quickly move the rest of the L3-L7 stack, trying to solve each layer ala cart with a different bag of tricks.
          Wouldn’t it be better to just template the entire L2-L7 network, move that template along with the application VMs, and recreate the network as quickly as you can recreate compute? ;-)

          Cheers,
          Brad

          • wd says:

            VMware still requires L2 for VMotion ! VMotion over L3 is discussed since many years, VMware created this caveat, why should they fix it ?

          • Brad Hedlund says:

            Not really true anymore. While not specifically documented yet, VMware customers can request that vMotion over L3 be supported for their environment.

          • Hywel Burris says:

            Thanks for the reply, sounds great but have a couple more questions:-

            1) Does SRM (or a vCO workflow) have the capability today to initiate the L2-7 migration of the NSX (vCNS) VM’s and networks?
            2) How is routing northbound of the NSX environment announced so that other devices in different sites know that the gateway address to this network has moved sites? Is BGP suggested for this?
            3) What are the current suggested options for interfacing with third party physical load balancers (F5) for advanced LB functionality. Realise that virtual F5′s may be a better fit, but sometimes hard to justify additional spend.

            Apologies for, what may seem dumb questions, but VMware guy dipping toe into new network territory!

            Hywel

  5. Eric says:

    Hi all,

    one simple question , as we can find lot of information about NSX, any resources we can evaluate and test NSX in the lab?
    Thanks!

    -Eric

  6. Anthony Samuel says:

    This is a nice site. Keep up the outstanding work. Nothing compares to a virtual network now does it. It’s bloody good.

  7. Alex says:

    I’m especially amazed with example #3, because it illustrates clearly that in most cases, the VM traffic won’t even hit the nexus 7k.

    Hypothetically, would it be possible to run NSX on 10-15 C-series UCS servers, and connect their VIC 1225 or Intel X520 10GbE cards straight to stacked whitebox 10GbE switches (cumulus linux-supported, with hardware VXLAN VTEP termination support)?

    Then these switches could just have a perimeter firewall as their gateway? Or is there some other component I’m missing?

    Thanks a lot for these posts, and the previous UCS series, very informative!

    • Josh says:

      I feel exactly the same way.

      Thanks to NSX, the N7K’s in the core (in these examples) do almost nothing as long as everything is functional.

      Now if a link fails here and there you’ll get some traffic traversing the core, but not much.

      IMO: Thanks to NSX, there’s no longer a need to invest in big core switches. any number of 10GbE switches will do just as nicely. And even the benefits of UCS suddenly don’t matter quite as much (the VIC in particular loses it’s value as a CNA, since the underlay doesn’t need nearly as much intelligence)

      • Alex says:

        Thanks Josh – I realize many big companies might have already invested in the nexus and it might make sense for them — but for a startup like ours the nexus costs can be overwhelming. I had been looking at these whitebox switches in particular which seem to have the chipset with VTAP hardware support: whiteboxswitch.com/collections/10-gigabit-ethernet-switches . Then everything else should be able to run on the NSX software if I understand correctly, plus a hardware perimeter firewall

        • Josh says:

          I’m with you. I am a consultant who does mostly government, and in my sector as well, nexus can be a huge nut. Though I admit white boxing kind of scares me as well.

          The good news is if you go 100% virtual you don’t even need a hardware vtep. If you do need a hardware vtep, there are some great low cost options out there. I have a friend who is an account manager for brocade who insists brocade vdx stuff is very cost effective. Arista also has one, as well as avaya, and even staying with Cisco, simply getting out of the n7k and into a l2 only n5k can save a boatload of money.

  8. Bill Smith says:

    So playing devil’s advocate – who runs an entire VM cluster on a single blade or even on a single chassis? How much traffic actually stays local to a physical server? Anyone building a resilient environment will have at least 2x blade chassis with VMs existing on multiple chassis. That means at some point you have to leave the virtual environment and go across the physical underlay. How does NSX know if a link is congested before it sends the tunnel that way? This overlay thing is nothing new, MPLS has been doing exactly the same thing for decades and that is a spaghetti mess to manage with all those p2p pseudowires running everywhere. It also required the underlay to communicate with the overlay to avoid physical network issues and increase adoption. That physical network awareness does not exist with NSX unless I am missing something, it is operating in a vacuum. I see NSX as interesting in a single blade chassis or maybe a very small environment or lab potentially but not very useful across a large network for that reason.

  9. Dan Barber says:

    Brad,

    So my question, which I have had some mixed answers on—lots of “I think so”s—is this: East/West Layer 2 Inter-Host (blade 1 to blade 2) within the same UCS chassis, VMs on the same portgroup… will that traffic exit the chassis because ALL traffic on UCS B-series has to reach the Fabric Interconnect, or will the vDS be smart enough to do L2 host-to-host on the backplane?

    • Brad Hedlund says:

      Hi Dan,
      The FEX modules in the UCS chassis do not do any switching. Any traffic from one blade to another within the same chassis will at least traverse the Fabric Interconnect. There would be no way for a virtual switch to bypass that.

    • Josh says:

      That traffic will leave the chassis, as from my understanding that’s the was ucs works

  10. Ali says:

    Can Fabric Path help with the east west traffic in this scenario ?

Leave a Reply

Your email address will not be published. Required fields are marked *