New design guide: VMware NSX with Cisco UCS and Nexus 7000

Back in September 2013 I wrote a piece on why you would deploy VMware NSX with your Cisco UCS and Nexus gear.  The gist being that NSX adds business agility, a rich set of virtual network services, and orders of magnitude better performance and scale to these existing platforms.  The response to this piece was phenomenal with many people asking for more details on the how.

The choice is clear.  To obtain a more agile IT infrastructure you can either:

  1. Rip out every Cisco UCS fabric interconnect and Nexus switch hardware you’ve purchased and installed, then proceed to repurchase and re-install it all over again (ASIC Tax).
  2. Add virtualization software that works on your existing Cisco UCS fabric interconnects and Nexus switches, or any other infrastructure.

To help you execute on choice #2, we decided to write a design guide that provides more technical details on how you would deploy VMware NSX for vSphere with Cisco UCS and Nexus 7000.  In this guide we provide some basic hardware and software requirements and a design starting point.  Then we walk you through how to prepare your infrastructure for NSX, how to design your host networking and bandwidth, how traffic flows, and the recommended settings on both Cisco UCS and VMware NSX.  As a bonus there is 48 x 36 poster that includes most of the diagrams from the guide and some extra illustrations.

Download the 48 x 36 poster here (PDF):


Download the full design guide here (PDF):






  1. David Klebanov says

    Hi Brad,

    Nice drawings and interesting write-up. Unfortunately, I was late to post my comments on VMware’s blog site, so here I am.

    Server Virtualization really became relevant to the wider market with introduction of specific hardware support in processors (Intel/AMD). Wikipedia summarizes it rather well here I wonder if we should have called it “ASIC tax” back then and dismissed it as being too constraining and too tied to the hardware…

    Network virtualization is the first step, which allows decoupling workload location in the topology from its network identifiers, such as IP and MAC addresses. This functionality already existed on the market for a few years now with encapsulation technologies like VXLAN and LISP.

    We need to elevate the discussion beyond that to solve real customer issues in the form of agile application deployment and not agile VM connectivity provisioning.

    BTW, can I do vMotion over VXLAN already? If not, it would make an interesting argument for validity of overlays on top of vPC topology, as described in the guide.

    My 2 cents.

    • says

      Hi David,
      The way I see it, the enhancements in commercial CPU silicon (Intel/AMD) for server virtualization offloads are analogous to what we’re already seeing today with VXLAN aware NICs, and VXLAN capable Top of Rack switches in commercial networking silicon (Broadcom/Intel). You can start today with or without the hardware offloads.

      There’s a big difference however with network virtualization in that, when implemented in edge host software, we inherently leverage the power of a distributed system. As a simple example, a single edge host CPU provides the processing for a virtual server. However, with network virtualization, many, many, edge host CPUs (running in parallel) provide the processing for a virtual network (such as east-west routing & firewall). Whereas in a non-virtual network, a single physical CPU (ASIC) is the processing chokepoint for eg. east-west routing and firewall. As such, network virtualization software alone (even before hardware offloads) provides better performance compared to a non-virtual network.


      p.s. Yes. You can use an L3 network for vMotion traffic if that’s what you’re asking.

      • Drew says

        Hi Brad,

        I am curious about the East to West routing, etc.

        Have there been studies to show how much faster, if at all, running your traffic “East to West” via edge host CPUs rather than via 10Gb up to an ASIC on a Fabric Interconnect, or L3 up to a Nexus 5k / 7k?

        If I currently have 50 to 100 switches running in my enviroment today, is there a sizing guide that says, “I need x amount of edge host CPUs” in order to achieve the same performance as a dedicated network ASIC built for 10GB networks?

  2. Stimit Oak says

    Hi Brad…I work on the switching silicon side of things and am interested to know where you see the future? If all the control is in the hypervisor software where does that leave the ASIC? I don’t think vSwitches are practical. So what do you need from the silicon of tomorrow?

  3. Bhargav says

    Hi Brad,

    Thanks for getting this out, Was thinking about the best practices guide with NSX. Question time :-)

    1) Pre-NSX can understand the requirement for Infra-VLANs, do we still need to have Infra-VLANs Post-NSX ?. Why cannot one virtualize Infra traffic like Management, VMotion traffic etc. What i mean by virutalize is to create a separate VxLAN tunnel for infra-traffic ?. In that way can remove even the one time configurartion for Infra VLANs ?

    2) Does transport VLAN refer to VLAN added after VxLAN encap ?

    3) In the examples, routing is done by the N7K. How does it work when routing is done by the vSphere using DLR. Is N7K required then ?

    4) Design doc referes to seamless migration from UCS based infra to any network infra. Is it really seamless ?. If i move my DB from UCS fabric to an env having DB virtualized, will it work ?. Is it not the port-group configuration of source and destination different ?


    • says

      Hi Bhargav,

      1) No. There really isn’t much benefit to placing the infrastructure Management and vMotion traffic on a VXLAN Logical Switch. Also consider that Management traffic is what configures NSX on a host. By placing host Management traffic on a VXLAN built by NSX, you’ve created a circular dependency.

      2) The transport VLAN is simply the VLAN on which VXLAN encapsulated traffic will traverse. It’s the VLAN on which each host has a vmknic (or multiple vmknics) soley dedicated to VXLAN.

      3) In the doc, I actually show east-west routing being handled by the DLR. At some point your north-south traffic will need to exit the virtual network and enter the physical. This happens on the NSX Edge Services Router, and the N7K is simply the next router hop to reach the rest of the data center or outside world.

      4) One of the beautiful things about NSX is that it doesn’t care what make/model you have for the physical infrastructure, and you can have different make/models in the same NSX deployment. Consider a scenario where you have NSX running just fine on your Cisco UCS and N7Ks. You then stand up a new pod of (pick your favorite make/model switch) (e.g. I depicted Nexus 9000 in the guide) right next to your UCS. You can then take your new end hosts connected to your new Nexus 9000s and add them to the existing NSX domain. You now have VXLAN Logical Switches, and Logical Routers, spanning both infrastructures. Your VMs and NSX Edges can live migrate (vMotion) from one infrastructure to the next without any changes or downtime.


  4. Bhargav says

    Hi Brad

    Thanks for the clarification.

    1) Agree with circular dependency created by placing management traffic on VxLAN.

    2) This Edge VLAN seems tricky. What i understand is that, there a VLAN called Edge VLAN that span across all the DLR’s, Control VM, ESR VM and Nexus 7K. All applications has one interface in Edge VLAN and therefore are in same subnet. This VLAN is also a Infra VLAN and plumbed during fabric installation. There is BGP session between N7K to all DLR’s, Control VM and ESR VM. The client facing ip address of the application stack (Web-IP) is advertised to N7K with next-hop set to connected interface. Basically, the virutal IP address is advertised to physical world using BGP running on N7K. Is this fair summary of the use of Edge VLAN?. I still don’t get the usecase of Edge VLAN. Why should they be in same VLAN (Same subnet), why cannot their be multihop-BGP between ESR’s/DLR/control-VM to N7k. Is there something i am missing ?


  5. Bhargav says

    Adding to my earlier questions.

    Agree with circular dependency created by placing management traffic on VxLAN. I am still not clear about VMotion & Storage (say iSCSI). Assume an inter-DC or a fabric having lot of hops. Will the current approach, one may have to plumb VLAN across the fabric. If one were to place VMotion on a VxLAN what is dis-advantage be like ?. DCB-PFC ?. The advantage is to remove VLAN plumbing for VMotion across the fabric.


  6. Alex says

    Great guide!

    Would it be possible to follow this design guide, just with Cisco UCS rack servers, and a pair of white-box switches running Cumulus networks linux? for example the ($6,000 for 48 10GbE ports). If I understand correctly they support VXLAN VTEP termination. Then the two whitebox switches would connect to our PA3020 cluster which handles our layer 3.

    In my situation I have 8 ESXi hosts (C220 M3 with 2x 10GbE each) in the tenant cluster and might add another 8 hosts in the next year. Using FAS3240 for IP storage (NFS). I can not justify investing on the Nexus 7k or even 5k, especially because we have fairly basic needs for L3 (only one ISP, and two datacenter locations)

    If I understand correctly, since most of the magic happens inside the NSX, the Nexus 7000 or 5000 would be really underutilized in our setup. we just have a small 100Mbps colo connection…

    Thanks for any advice you can provide.

  7. Tommy says

    Great guide, really giving us some great places to start from. I’m curious why you maintain both vPC and standard port-channels. I understand why you’re required to have to standard port-channels since the vPC peer link won’t support routing protocol communication, but why maintain this disjoint configuration?

    • says

      I’m assuming that an existing setup (pre-NSX) has vPC uplinks to the Nexus 7000. So the point here is that you can leave that as-is, just don’t run the Edge VLANs on them if you plan on dynamac IP Routing.

      Thanks for the comment.


  8. Alessandro Lorusso says

    Hi Brad,
    great guide!

    I have only one note: in the scenario with different uplinks for edge vlans and other vlans, is it necessary to use the disjoint layer2 configuration on UCS? If yes, it will be necessary to use separated vNICs for edge VLANs and other VLANs…


    • says

      Hi Alessandro,

      Different uplinks for Edge and “Other” VLANs is only necessary if you’re using vPC uplinks for the “Other” VLANs. In other words, the Edge VLANs should not be placed on vPC uplinks to a Nexus 7000.
      By the way, Cisco UCSM 2.0 supports the “disjoint layer2” configuration, so multiple vNICs are not necessary. Here’s a video I found on YouTube showing how to configure it:


  9. ron hehemann says

    Brad, would NSX help us to overcome the reserved vlan range blocked by the UCS, but in use at a customer.
    FI can we install on a C-serie server ESXi, install NSX and “rewire” the clans and then connect to the UCS Flexpod?
    As you are very knowledgeable in all components.


Leave a Reply

Your email address will not be published. Required fields are marked *