Cisco UCS Networking Best Practices (in HD)

UPDATE 3/8/2011: This video series has been obsoleted by a new and updated series posted here: Cisco UCS Networking videos (in HD), Updated & Improved!

This is a presentation I developed covering networking best practices for Cisco UCS, and now have recorded in High Definition for your viewing pleasure! Sweet! :-)

This presentation assumes familiarity with basic networking and server VNIC concepts in UCS, and familiarity with virtual port channels.

This version of the presentation (v2.5) focuses primarily on the Ethernet uplinks. SAN uplinks and VMware networking scenarios are briefly discussed but not covered extensively. Those topics and others such as QoS, the Cisco VIC, and vNIC fabric failover may be included in future versions of this presentation.

Stay tuned for updates! RSS feed:

UPDATE 3/8/2011: This video series has been obsoleted by a new and updated series posted here: Cisco UCS Networking videos (in HD), Updated & Improved!

Part 1 – Cisco UCS Networking Overview

In Part 1 we take start with a baseline overview of Cisco UCS Networking. At the heart of the system is the Fabric Interconnect (6100) “the Brains of UCS” which provides 10GE & FC networking for all the compute nodes in its domain as well as being the central configuration, management, and policy engine for all automated server and network provisioning.

Part 2 – Switch Mode vs. End Host Mode

Part 2 is an examination of the two different switching modes supported by the Fabric Interconnect, “Switch Mode” and “End Host Mode”. With “Switch Mode”, the Fabric Interconnect behaves like a normal Layer 2 switch on all server ports and uplinks, and therefore attaches to the upstream data center network as a spanning tree enabled “Switch”.

“End Host Mode”, on the other hand, while still providing local Layer 2 switching on the server ports, does not behave like a normal Layer 2 switch on its uplinks.  Instead, server NICs are “pinned” to a specific uplink, and no local switching happens from uplink to uplink.  This allows “End Host Mode” to attach to the network like a “Host” without spanning tree, and all uplinks forwarding on all VLANs.

End Host Mode is the preferred mode, and it’s enabled by default.

Part 3 – End Host Mode – Individual Uplinks

In Part 3 we take a look how the individual uplinks behave in End Host Mode, and how the system reacts to uplink failures. When an uplink fails, the Fabric Interconnect will move the server NICs to a new uplink in under a second without causing any disruption to the server NIC.  This uplink failover process is called dynamic re-pinning.

After the dynamic re-pinning process, the Fabric Interconnect will send Gratuitous ARP messages for all of the MAC address that were previously using the failed uplink. This GARP process aids the upstream network in quickly learning the new location of the affected MAC address now using the new uplink.

Part 4 – Port Channel Uplinks

Here we take a look at the benefits of using Port Channel uplinks with Cisco UCS. The key advantages to port channel uplinks is the minimal impact of a physical link failure and the potential for better overall uplink load balancing. During individual physical link failures fewer moving parts required to provide a fast recovery.  For example, Gratuitous ARP messages and dynamic re-pinning are not required when an individual physical member link fails in a port channel uplink.  Port Channel uplinks are definitely recommended whenever possible.

Part 5 – Virtual Port Channel Uplinks (vPC)

Part 5 covers the advantages of using virtual port channel (vPC) uplinks with Cisco UCS. With vPC uplinks, there is minimal impact of both physical link failures and upstream switch failures. With more physical member links in one larger logical uplink, there is the potential for even better overall uplink load balancing and better high availability than with a standard Port Channel uplink discussed in Part 4. Using a virtual port channel uplink is highly recommended if you have vPC capabilities present in your upstream network switches.

Part 6 – Connecting Cisco UCS to separate networks

In Part 6 we discuss the scenario of connecting a single Cisco UCS system in End Host Mode to separate Layer 2 networks. When the system is in End Host Mode, it expects and assumes that all uplinks are connected to the same common Layer 2 domain. If some uplinks are connected to physically separate networks you will have connectivity problems.  The Fabric Interconnect will randomly pick one of its uplinks to process broadcast messages for all VLANs.  As a result, only servers associated with the chosen network will be able to see and process broadcasts messages on their network.  The solution is create a common Layer 2 network for the Fabric Interconnect in End Host Mode and each of the separate networks to attach to, or, use Switch Mode.  If creating a common Layer 2 network or using Switch Mode is not an option, you can always deploy a unique Cisco UCS system per separate network to preserve the existing “silos”.

Part 7 – Inter Fabric Traffic Examples

This is a brief look at some the common types of traffic flows that may flow between Fabric-A and Fabric-B within a single Cisco UCS system. With this understanding, the subsequent material will make more sense.

Part 8 – Don’t: Connect Cisco UCS to vPC domains without vPC uplinks

This is a fairly extensive look at the scenario of attaching UCS to upstream switches configured for vPC, without using vPC uplinks. Here we will show that this scenario doesn’t make much sense and in fact can cause some unwanted traffic black holes under some failure scenarios. This is a prelude to Part 9 where we illustrate that if your upstream network is configured for virtual port channel capability (vPC), you should always attach UCS with vPC uplinks.

Part 9 – Do: Connect Cisco UCS to vPC domains with vPC uplinks

This section shows that if you have virtual port channel capabilities in your upstream switches, you have everything to gain and nothing to loose by connecting Cisco UCS with vPC uplinks. You will gain the benefit of the upstream switch locally switching all Fabric-A to Fabric-B traffic, and acheiving more bandwidth scalability for inter-fabric traffic because all inter-fabric traffic will travel on the vPC uplinks, rather than on less abundant inter-switch links. Additionally, you will avoid potential black hole failure scenarios discussed in Part 8, if vPC is already present in the upsteam network switches.

Part 10 – Connecting Cisco UCS without vPC

While there are certainly advantages to uplinking Cisco UCS with virtual port channels, vPC is certainly not required. Cisco UCS easily and efficiently connects to any data center network environment with or without vPC. This section discusses best practices connecting UCS to networks without vPC.  The key best practice here is to always dual attach each Fabric Interconnect to two upstream network switches, whether its with vPC uplinks, or multiple individual uplinks.  Another suggested practice is to avoid attaching Cisco UCS to a second tier Layer 2 switch with spanning tree blocking links.  A better approach is to either have vPC capabilites at the second tier Layer 2 switch, or connect Cisco UCS directly to the tier 1 switch, avoiding a traffic bottlenecks induced by spanning tree.

Follow up post and whitepaper: Cisco Nexus 7000 connectivity solutions for Cisco UCS


  1. says


    I really appreciate this post. It has been extremely valuable in our configuration and evaluation of the UCS platform.

    One question I would like clarified about UCS best practices has to do with 1000v. I was wondering if you might be able to shed some light. In the “Best Practices in Deploying Cisco Nexus 1000V Series Switches on Cisco UCS B Series Blade Servers” gude, found here: there is a statement regarding the placement and connectivity of VSMs in a 1000v implementation on UCS. It gives several options “listed in order of recommended customer use” but says little to justify the order of preference. I am looking for any information that would help me understand the reasoning behind the recommendation. Also I am wondering why VSM Inside the Cisco Unified Computing System on the Cisco Nexus 1000V Series VEM is not recommended or at least not mentioned at all.


  2. Andy says

    I have configuration question. when connecting a Fabric Interconnect running in End Host Mode to a nexus 5k vPC, what spanning-tree port type should the 5k port channel be? i.e. spanning-tree port type edge or spanning-tree port type network?


    • says

      The Fabric Interconnect in End Host Mode connects to the upstream switch like a “Host”, not as a “Switch”. Therefore you should configure the upstream switch port (or port channel in your case) just as you would if it were a server attaching to it. Therefore the best configuration on the 5K, in your case, would be: ‘spanning-tree port type edge trunk


  3. Joe Ignatius says

    Hi Brad,

    These videos are really informative and clear. It is great to have a place to review topics.

    I would love to see something similar where the subject is “How Nexus platform works with UCS, in the data center”.


  4. Karan Bhagat says

    awsome post…i was just having a tech session with your Cisco peer Steve Fishman… good stuff… I’d like to see the same for BP on FCoE to FC from the 6100 to MDS91xx… and how to scale FC to large SAN env from UCS.

  5. Tejo Prayaga says

    Awesome videos. It greatly helped me to understand the connectivity and traffic flow path in the UCS system. Thank you

  6. JM says


    Awesome vids.

    I give this a A+ for the presentation of very useful information.

    I found this very helpful but the delivery and tone…. knocked me out. I could hardly stay awake.

    This is an A++ as a sedative. Dude you should probably consider moonlighting as a hypnotist.



    Great job though. Keep up the good work.

  7. robert r says

    In watching video#2 and looking at the difference between switch mode and end host mode I don’t see where / how I would team nics to a host. In your example vlan 10 is only on 6100A and vlan 20 is only on 6100B which suggests that if we lost 6100A we would lose vlan 10 completely.
    Wouldn’t a better approach be to put vlans 10 & 20 on both 6100s and trunk and team them to the servers via LACP so that a failure would allow the traffic to still have a path to server ?

    • says

      Yes, in most cases the same VLANs will exist on both 6100s. You would not use LACP for NIC teaming because that would require the 6100s to be clustered together in a vPC domain, and that is not supported. For NIC teaming you could use standard active/standby teaming, the unique vNIC “fabric failover” capability in UCS, or in the case of a VMware host you can also use active/active NIC load balancing based on vPort-ID or Mac Pinning.


  8. Matthew says

    Hi Brad,

    I enjoyed these great videos, thanks!

    To your statement “For NIC teaming you could use standard active/standby teaming,”
    assuming active/standby alternating (server1 active to FEX1, server 2 active to FEX2), how is the traffic flow between server 1 and 2 ?


    • says

      In this scenario:
      (server1 active to FEX1, server 2 active to FEX2), how is the traffic flow between server 1 and 2 ?
      Traffic between Server 1 and Server 2 would travel through the upstream network. This is another example of “Inter-Fabric Traffic” discussed in Part 7.


  9. Paul C says

    Excellent information, brief and straight to the point info on very relevant concepts for SEs pushing this solution to clients and needing to know all the facts (not just the cool marketing stuff).

    Many thanks,

  10. Ambi says

    Great post Brad

    However i have a question

    how would the traffic flow be if the 2 VMs are on different network … would it have to go to an upstream L3 switch to get switched??


  11. Mohan Muthu says

    Excellent Brad. I am very new to UCS and have one question on the videos.

    What does vEth1, vEth2, vEth3 etc. shown in your video inside the 6100? I was unable to find any documentation about this.

    Does this mean these are virtual ethernet ports corresponding to every vNIC on the blade servers? Can these ports be configured like any other switch port?

    Any information on these would be really helpful.

    thank you for the excellent video.


    • says


      Does this mean these are virtual ethernet ports corresponding to every vNIC on the blade servers?


      Can these ports be configured like any other switch port?

      In theory, Yes. In practice, No. Cisco UCS Manager does all the configuration of the vEth ports for you, based on the settings of the servers vNICs in the Service Profile for that server.


  12. Mohan Muthu says

    Thank you Brad! Appreciate your quick response.

    Your page is at the top of my “favourites” bookmarks!!!


  13. Eric T says

    Hi Brad,
    I just watched the videos and really enjoyed them. First, Thank You for taking the time to do this. It was very clear and clearly you took the time to ensure your audience was following you. Second, I found myself wishing you included the failure scenarios from a VM perspective. The box vs the bowtie design I am happy with from a network design perspective. When I review all your failure scenarios in total you have covered all my bases; however I see the coming convergence of SysAdmin and NetworkAdmin responsiblities thus covering the failures from a virtual machine perspective would be very helpful, then I can send the SysAdmins to this page and tell them all their networking questions are answered here. :)
    For example I can see them wanting to load-balance across the Fabric A and Fabric B, nic teaming in VM. Is this good or bad from a network perspective? Personally I don’t like it since the network has taken care of the failure and load-balancing but I need to be able to articulate why to SysAdmins. Heck I don’t even know if VM supports this but I would like to know.
    My statment isn’t a complain by the way, only an observation about who your audience is and who they will be in a couple of years.
    Thanks Again!

  14. Raghu says

    Hi Brad,

    Thanks for sharing excellent video’s about UCS. I would like to know one infromation regarding UCS.

    Is the UCS cluster ID is unique across the network ? If possible can you share any information about cluster configuration ?

    Thanks again :-)

  15. ChalieM says

    Hello Brad..

    Great presentation. I’m a newbie to the Cisco UCS solution and your presentation spells it all out. I do have a question though. I’ve been recently approved to upgrade our VMware infrastructure and network core. We are currently a Dell shop, but I’m seriously considering replacing it with UCS. Can you tell me in a sentence or two the advantages/disadvantage going with Cisco solution over the DELL blade chassis? Out side of the management and switch configuration on the DELL platform, what’s different?

    • says

      I have some recent experience with a large financial customer evaluating Cisco UCS vs. DELL blade solutions. In every conversation or visit with this customer I would espouse many unique virtues of Cisco UCS; the simplicity of implementation, 10GE wire once infrastructure, stateless computing, the unique innovations with VMware, etc, etc. After carefully evaluating both platforms this customer told me, in a nutshell, paraphrasing here: “It’s UCS Manager, stupid!” (Coining the popular “It’s the Economy, stupid!”).
      For this customer, the out of the box capabilities in UCS Manager was the killer differentiator that radically simplified the management of all the servers and network settings.

      For example, template driven provisioning and updates. Not only can you deploy server configurations from templates within minutes, you can upgrade the firmware on a template and the hundreds of servers tied to that template are automatically upgraded by the system. Need to add a new adapter to hundreds of servers? No problem, add the new adapter to the template and you’re done. This is just scratching the surface of the UCS Manager capabilities built in to the system, not a bolt on software package with extra license fees.


  16. aiwa says

    hi Brad, gr8 post.

    One question i cudnt find an answer –

    Why is NEXUS needed for UCS? Feature advantages? Dependencies?
    Can a UCS system be connectd to a conventional Cisco 4500/6500 based data network instead of using Nexus?

    • says

      Yes – you can connect UCS to any standard 1GE or 10GE switch.
      Nexus is a good choice for the 10GE density, performance, high availability, and vPC – as discussed in one of the videos.

  17. Brad Morgan says


    What would the “best practice” recommendation be for a vmware deployment on UCS with redundant fiber interconnects connected to Nexus switches when the “SAN” connectivity is via iSCSI rather than FC. Would it be better to group the iSCSI vlan in with the other wan/lan traffic vlans so that it could share in the benefits present with vPC, or would it be best to set up the 6100s in switch mode to isolate the iSCSI vlan to protect it from broadcast storms or loops in the other vlans that might choke its bandwidth off. Is there a way to guarantee bandwidth to the iSCSI vlan to make sure that it won’t be adversely affected if an issue arises in one of the other vlans?

    Your posts are exceptinally helpful.

    Thank you

    • says

      To partition BW for iSCSI, you can keep UCS in End Host mode, use a LAN Pin Group and call it “iSCSI” or something. Pick the uplink you want dedicated for iSCSI traffic and put it in your iSCSI Pin Group. Take the vNICs used for iSCSI and assign them to the “iSCSI” LAN Pin Group.


  18. VIJAY SHEKHAR says

    Great work Brad!! really!

    I have a question w.r.t Fabric Interconnect EHV Mode.
    EHV mode does not learn MAC addresses on BI, It only learns MAC on SI.
    now when server is sending a packet out of the POD to ethernet cloud, it will ARP and learn the MAC address of the nexy layer3 Hop, All good till here.
    Now when the FI gets that Frame from server destined to L3 hop for that network it will be a Unlearnt MAC.
    I also read that all Unlearnt MAC recieved on SI are Flooded to All other SI and Pinned BI.
    If tha bove statement is true does FI flood traffic leaving the POD to ALL SI interfaces EVERYTIME?
    It sounds too silly, but asking based on what I ave been reading.
    Vijay Shekhar.

  19. Leo says


    one question regarding connecting a UCS to two or more L2 domains: To avoid switching the fabric interconnects to switch mode, would it be feasible / recommended to build several vPC Channels and pin the vNICS manually to these channels?


    • says

      No – static pinning will not work. The FI will still only pick one uplink as the broadcast listener. That’s the case for UCSM version 1.4 or older.

      In the next release of UCSM, End Host Mode will be enhanced to support separate upstream L2 domains.


      • Gab says

        Hi Brad,
        one more time i find answers to our questions on your site!! Thanks! i’d like to know about this post i’m replying to. “In the next release of UCSM, End Host Mode will be enhanced to support separate upstream L2 domains.”. I was also thinking about the solution to have different VPC’s per L2 domains and static pinning, but a s you says it does not work for UCS 1.4 or older. Is this already solved on the new versions?. We are under scenario where we are sharing a chassis and FI’s to connect to 2 different L” domains and it causing some trouble. Is there any alternative to not use switching mode?


  20. Praveen says

    Hi ,

    Awesome information!!!

    I am designing a Network where we need to connect the 6120Xp Fabric extender to Catalyst 6509 . I found that we can connect the fabric Interconnect (FI) to 1G port of an uplink switch using The first 8 fixed ports can be configured to operate in 1 Gigabit Ethernet mode with the transceiver options specified for use with SFP-compatible ports using 1000BASE-T SFP GLC-

    Since I have 2Nos 6509 Switches in the Datacentre I can connect only 4 Ports from 6120XP to each of 650. Can I configure channel from 6120Xp to 6509 between these 4 links?

    Thanks and Regards

  21. William Fleitz says

    Brad, Excellent presentation. I have a quick question perhaps you or someone else can answer. My server team wants to connect a VMotion switch to the UCS so that we can Vmotion legacy VM servers to the UCS ESX farm. I’m wondering what is the best practice to accomplish this? It would need to be accessible across both 6100s. Thanks!

  22. Marwan says

    Hi Brad,
    i know its a late comments or question but just wondering about best practice UCS/N1K networking
    where cisco Docs recomend to use LACP port channel with clustered upstream switches (vPC,VSS)
    and vpc_HM with non-clustered switches with mac-pining
    the question here
    if we present one vNIC to the ESX host and N1K and in UCS this vNIC has failover enabled from FIA to FIB
    and in the back end this vNIC has the UCS port-channel to the physical switch
    why we need to worry about port channelling if its already will be done by UCS and the esxi host and N1K will see one uplink/vNIC???
    ( understood with palo we can vitalize these vNICs without enabling fail over to send traffic selectively over certain ports if required ! )


  23. Eddie Wieder says

    Our UCS is version 2.0w and is configured for switched mode. We have a 3750x Stack with 10GB uplink ports configured in Port Channel with two 6100’s. We now need to connect our Two DMZ’s to our UCS but keep the traffic separated, if possible. We are currently running VMWare for all of our 4 server hosts. My question is based on Video 6. I am unsure on how to do this correctly as it seems to keep bridging/broadcasting each sides networks. Any help would be greatly appreciated.

    • says

      In switch mode there should be no issue. Just connect your DMZ switches to the 6100s, pick the VLANs within UCS that will be the DMZ VLANs, and make sure only those VLANs are forwarding on the links connecting to the DMZ.

  24. Shashidhar Patil says

    Hi Brad,
    I do not see that failure cases of Fabric explained. If Fabric A fails
    all vNIC0 generated traffic is disrupted ? Can a VM use one port each from vNIC0 and vNIC1 for
    redundancy and team those ports ?

  25. Joe L says


    Thanks so much for these. They are loaded with great information.

    Question: I see that you are using slide decks with Cisco on them. Are these slide decks available online? It would be very helpful if I could pull these down to my local machine to flip through.

    Any information is appreciated.

    Thank you again!

  26. Chad says

    Is it possible to connect C-Series to 6248 Fabric Interconnects directly without requiring the Fabric Extenders if there is no intent to centrally manage the C-Series? I simply want to use 10GB to access the same SAN as our blade chassis.


  27. says

    Hi Brad!

    Congratulations for yours presentations!!
    I´m planning this change in the UCS enviroment:
    I have 2 port-channels (1 Fabric A + 1 Fabric B) connecting to a upstream network (3750 Stack). The actual portchannel is a 1G and I need to migrate to a 10G portchannel.
    How can I do that without disruption to the createds LAN Pinning Groups and vNICs configurations?
    I had tried this scenario with the Cisco UCSPE, but I´m not sure about the good way because at the Virtual enviroment the uplinks ports doesn´t came up.

    Thank you very much!!

    Best Regards!


  28. Ray says

    FYI: I’m new to this UCS product….

    On the FI, can you do traffic shaping? For example, I have a FI with two modules and if I want to use my module two for certain type of traffic (NFS, SAN, Database, etc.) The goal here is to streamline and configure my FI to isolate types of traffic to get most performance out of it.


  29. Wael says

    Hi Brad.

    i have a great confusion regarding the VIC card, how it is seen from the hypervisor side from example VMware vSphere, i mean the physical adapters from the hypervisor perspective. second i have a great wonder if it is right that the every VM can granted a VEM and every VEM have one path through a VNIC or HBA out of the server to one FEX and a back up path the same way way through the second FEX.

    im so confused with that subject, i hope u can dedicate a video for that Brad. and if u can refer me to any good URL for me to grasp the idea u would be so thankfull


  30. jason says

    Good stuff,

    Any recommendations on trunking multiple pairs of UCS fabrics together? I am somewhat surprised the ucs 6248 and 6296 do not support this currently.

  31. says

    hey brad,

    in part 6 you make reference to your design not having overlapping vlans. if you did have overlapping vlans, what is the best practice?



  32. Danny says

    thanks for the vids.
    I have one thing about inter fabric traffic: As I know, Fabric Interconnect handles layer2 traffic inside itself, so if you connect all vnics within the same vlan with the same fabric, the traffic will not go out to the l2 switch.

    If you design your UCS as follow;
    – all vnics 0 (VLAN10) pinned to Fabric A,
    – all vnics 1 (VLAN 20) pinned to Fabric B,
    Can you connect UCS directly to layer3 switch, can’t you?

  33. GaryS says

    Hi Brad, just starting with UCS, enjoyed the Video’s quick question on UCS Up-links using Port Channels. We are looking at VC1225 VIC’s and I’d like to create 2 PC’s and split the port channel across to Switches (1 Port from each PC in each switch), the switches are in the same L2 Domain are there any issues or reasons why I can’t (the Upstream Datacentre Switches are HP Procurve units?



  34. GaryS says

    Hi Brad

    Thanks for the reply, I read a little more (and watched another of your videos) after posting and realised that without vPC it wouldn’t work but as you suggest as a stack it would. That’s not the config so it looks like it would have to be individual links to the 2 switches in the L2 Domain (so redundancy is in place) and as they share the same broadcast domain it would work.

    Enjoyed watching the Video explains of UCS has helped me get my head around how it works!



    • says

      That’s the beauty of the End Host Mode in UCS. You can connect multiple uplinks to multiple switches in the same L2 domain, and not worry about spanning tree or blocking links. All paths will be utilized. Glad you liked the videos!


  35. gourav says

    hi brad , i need your help in one of our setup , there is only 1 n/w switch and 1 SAN switch, both FI will connect to same SAN switch and same LAN switch.please suggest it will work in end host mode for san and lan or i have to change it from end host mode to switch mode.

  36. Abdulrahiman says


    I have some clarifications regarding Cisco C-series servers, We are using VIC 1225.

    1) We don’t have Nexus switches in our lab , shall we connect C-series servers to a catalyst switch for the data connectivity with VIC 1225?
    2) How the management port of the servers connected, can I connect to a catalyst switch for OOB and Inband purpose?
    3) How many LoM ports by default in a C-series servers and how many of them I can use for data connection and management connection?
    4)Is there any 10G ports available as LoM?

  37. Rou says

    Hello brad,

    We are about to install UCS with 2 FI , 2 N5K and MDS SAN switchs. The question is:
    is it better to connect FI to MDS SAN directly or the MDS to the 2 N5K Nexus switchs ? any difference in terms of configurations cabapibilities ( may be QoS for FC flows ,..) ? drawbacks ?


Leave a Reply

Your email address will not be published. Required fields are marked *