Cisco UCS Networking Best Practices (in HD)

Filed in Cisco UCS, Featured, Video by on June 22, 2010 56 Comments

UPDATE 3/8/2011: This video series has been obsoleted by a new and updated series posted here: Cisco UCS Networking videos (in HD), Updated & Improved!

This is a presentation I developed covering networking best practices for Cisco UCS, and now have recorded in High Definition for your viewing pleasure! Sweet! :-)

This presentation assumes familiarity with basic networking and server VNIC concepts in UCS, and familiarity with virtual port channels.

This version of the presentation (v2.5) focuses primarily on the Ethernet uplinks. SAN uplinks and VMware networking scenarios are briefly discussed but not covered extensively. Those topics and others such as QoS, the Cisco VIC, and vNIC fabric failover may be included in future versions of this presentation.

Stay tuned for updates! RSS feed: http://www.bradhedlund.com/feed/

UPDATE 3/8/2011: This video series has been obsoleted by a new and updated series posted here: Cisco UCS Networking videos (in HD), Updated & Improved!


Part 1 – Cisco UCS Networking Overview

In Part 1 we take start with a baseline overview of Cisco UCS Networking. At the heart of the system is the Fabric Interconnect (6100) “the Brains of UCS” which provides 10GE & FC networking for all the compute nodes in its domain as well as being the central configuration, management, and policy engine for all automated server and network provisioning.


Part 2 – Switch Mode vs. End Host Mode

Part 2 is an examination of the two different switching modes supported by the Fabric Interconnect, “Switch Mode” and “End Host Mode”. With “Switch Mode”, the Fabric Interconnect behaves like a normal Layer 2 switch on all server ports and uplinks, and therefore attaches to the upstream data center network as a spanning tree enabled “Switch”.

“End Host Mode”, on the other hand, while still providing local Layer 2 switching on the server ports, does not behave like a normal Layer 2 switch on its uplinks.  Instead, server NICs are “pinned” to a specific uplink, and no local switching happens from uplink to uplink.  This allows “End Host Mode” to attach to the network like a “Host” without spanning tree, and all uplinks forwarding on all VLANs.

End Host Mode is the preferred mode, and it’s enabled by default.


Part 3 – End Host Mode – Individual Uplinks

In Part 3 we take a look how the individual uplinks behave in End Host Mode, and how the system reacts to uplink failures. When an uplink fails, the Fabric Interconnect will move the server NICs to a new uplink in under a second without causing any disruption to the server NIC.  This uplink failover process is called dynamic re-pinning.

After the dynamic re-pinning process, the Fabric Interconnect will send Gratuitous ARP messages for all of the MAC address that were previously using the failed uplink. This GARP process aids the upstream network in quickly learning the new location of the affected MAC address now using the new uplink.


Part 4 – Port Channel Uplinks

Here we take a look at the benefits of using Port Channel uplinks with Cisco UCS. The key advantages to port channel uplinks is the minimal impact of a physical link failure and the potential for better overall uplink load balancing. During individual physical link failures fewer moving parts required to provide a fast recovery.  For example, Gratuitous ARP messages and dynamic re-pinning are not required when an individual physical member link fails in a port channel uplink.  Port Channel uplinks are definitely recommended whenever possible.


Part 5 – Virtual Port Channel Uplinks (vPC)

Part 5 covers the advantages of using virtual port channel (vPC) uplinks with Cisco UCS. With vPC uplinks, there is minimal impact of both physical link failures and upstream switch failures. With more physical member links in one larger logical uplink, there is the potential for even better overall uplink load balancing and better high availability than with a standard Port Channel uplink discussed in Part 4. Using a virtual port channel uplink is highly recommended if you have vPC capabilities present in your upstream network switches.


Part 6 – Connecting Cisco UCS to separate networks

In Part 6 we discuss the scenario of connecting a single Cisco UCS system in End Host Mode to separate Layer 2 networks. When the system is in End Host Mode, it expects and assumes that all uplinks are connected to the same common Layer 2 domain. If some uplinks are connected to physically separate networks you will have connectivity problems.  The Fabric Interconnect will randomly pick one of its uplinks to process broadcast messages for all VLANs.  As a result, only servers associated with the chosen network will be able to see and process broadcasts messages on their network.  The solution is create a common Layer 2 network for the Fabric Interconnect in End Host Mode and each of the separate networks to attach to, or, use Switch Mode.  If creating a common Layer 2 network or using Switch Mode is not an option, you can always deploy a unique Cisco UCS system per separate network to preserve the existing “silos”.


Part 7 – Inter Fabric Traffic Examples

This is a brief look at some the common types of traffic flows that may flow between Fabric-A and Fabric-B within a single Cisco UCS system. With this understanding, the subsequent material will make more sense.


Part 8 – Don’t: Connect Cisco UCS to vPC domains without vPC uplinks

This is a fairly extensive look at the scenario of attaching UCS to upstream switches configured for vPC, without using vPC uplinks. Here we will show that this scenario doesn’t make much sense and in fact can cause some unwanted traffic black holes under some failure scenarios. This is a prelude to Part 9 where we illustrate that if your upstream network is configured for virtual port channel capability (vPC), you should always attach UCS with vPC uplinks.


Part 9 – Do: Connect Cisco UCS to vPC domains with vPC uplinks

This section shows that if you have virtual port channel capabilities in your upstream switches, you have everything to gain and nothing to loose by connecting Cisco UCS with vPC uplinks. You will gain the benefit of the upstream switch locally switching all Fabric-A to Fabric-B traffic, and acheiving more bandwidth scalability for inter-fabric traffic because all inter-fabric traffic will travel on the vPC uplinks, rather than on less abundant inter-switch links. Additionally, you will avoid potential black hole failure scenarios discussed in Part 8, if vPC is already present in the upsteam network switches.


Part 10 – Connecting Cisco UCS without vPC

While there are certainly advantages to uplinking Cisco UCS with virtual port channels, vPC is certainly not required. Cisco UCS easily and efficiently connects to any data center network environment with or without vPC. This section discusses best practices connecting UCS to networks without vPC.  The key best practice here is to always dual attach each Fabric Interconnect to two upstream network switches, whether its with vPC uplinks, or multiple individual uplinks.  Another suggested practice is to avoid attaching Cisco UCS to a second tier Layer 2 switch with spanning tree blocking links.  A better approach is to either have vPC capabilites at the second tier Layer 2 switch, or connect Cisco UCS directly to the tier 1 switch, avoiding a traffic bottlenecks induced by spanning tree.



Follow up post and whitepaper: Cisco Nexus 7000 connectivity solutions for Cisco UCS

About the Author ()

Brad Hedlund is an Engineering Architect with the CTO office of VMware’s Networking and Security Business Unit (NSBU), focused on network & security virtualization (NSX) and the software-defined data center. Brad’s background in data center networking begins in the mid-1990s with a variety of experience in roles such as IT customer, systems integrator, architecture and technical strategy roles at Cisco and Dell, and speaker at industry conferences. CCIE Emeritus #5530.

Comments (56)

Trackback URL | Comments RSS Feed

  1. Mark Sennott says:

    Great stuff here Brad. Thanks educating us on this stuff, and keep up the good work!

  2. Brad,
    Wow these are great videos! Thanks for taking the time to make all of them and put them on your site.

  3. Shahid says:

    Brad….amazing videos and great blog…thanks!

  4. Jason says:

    Brad,

    I really appreciate this post. It has been extremely valuable in our configuration and evaluation of the UCS platform.

    One question I would like clarified about UCS best practices has to do with 1000v. I was wondering if you might be able to shed some light. In the “Best Practices in Deploying Cisco Nexus 1000V Series Switches on Cisco UCS B Series Blade Servers” gude, found here: http://cisco.biz/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-558242.html there is a statement regarding the placement and connectivity of VSMs in a 1000v implementation on UCS. It gives several options “listed in order of recommended customer use” but says little to justify the order of preference. I am looking for any information that would help me understand the reasoning behind the recommendation. Also I am wondering why VSM Inside the Cisco Unified Computing System on the Cisco Nexus 1000V Series VEM is not recommended or at least not mentioned at all.

    Jason

  5. Andy says:

    Brad,
    I have configuration question. when connecting a Fabric Interconnect running in End Host Mode to a nexus 5k vPC, what spanning-tree port type should the 5k port channel be? i.e. spanning-tree port type edge or spanning-tree port type network?

    thanks!

    • Brad Hedlund says:

      Andy,
      The Fabric Interconnect in End Host Mode connects to the upstream switch like a “Host”, not as a “Switch”. Therefore you should configure the upstream switch port (or port channel in your case) just as you would if it were a server attaching to it. Therefore the best configuration on the 5K, in your case, would be: ‘spanning-tree port type edge trunk

      Cheers,
      Brad

  6. Joe Ignatius says:

    Hi Brad,

    These videos are really informative and clear. It is great to have a place to review topics.

    I would love to see something similar where the subject is “How Nexus platform works with UCS, in the data center”.

    Joe

  7. Karan Bhagat says:

    Brad,
    awsome post…i was just having a tech session with your Cisco peer Steve Fishman… good stuff… I’d like to see the same for BP on FCoE to FC from the 6100 to MDS91xx… and how to scale FC to large SAN env from UCS.

  8. Tejo Prayaga says:

    Awesome videos. It greatly helped me to understand the connectivity and traffic flow path in the UCS system. Thank you

  9. JM says:

    Brad:

    Awesome vids.

    I give this a A+ for the presentation of very useful information.

    I found this very helpful but the delivery and tone…. knocked me out. I could hardly stay awake.

    This is an A++ as a sedative. Dude you should probably consider moonlighting as a hypnotist.

    :-)

    JM

    Great job though. Keep up the good work.

  10. robert r says:

    Hello,
    In watching video#2 and looking at the difference between switch mode and end host mode I don’t see where / how I would team nics to a host. In your example vlan 10 is only on 6100A and vlan 20 is only on 6100B which suggests that if we lost 6100A we would lose vlan 10 completely.
    Wouldn’t a better approach be to put vlans 10 & 20 on both 6100s and trunk and team them to the servers via LACP so that a failure would allow the traffic to still have a path to server ?

    • Brad Hedlund says:

      Robert,
      Yes, in most cases the same VLANs will exist on both 6100s. You would not use LACP for NIC teaming because that would require the 6100s to be clustered together in a vPC domain, and that is not supported. For NIC teaming you could use standard active/standby teaming, the unique vNIC “fabric failover” capability in UCS, or in the case of a VMware host you can also use active/active NIC load balancing based on vPort-ID or Mac Pinning.

      Brad

  11. Matthew says:

    Hi Brad,

    I enjoyed these great videos, thanks!

    To your statement “For NIC teaming you could use standard active/standby teaming,”
    assuming active/standby alternating (server1 active to FEX1, server 2 active to FEX2), how is the traffic flow between server 1 and 2 ?

    Thanks,
    Matthew

    • Brad Hedlund says:

      Matthew,
      In this scenario:
      (server1 active to FEX1, server 2 active to FEX2), how is the traffic flow between server 1 and 2 ?
      Traffic between Server 1 and Server 2 would travel through the upstream network. This is another example of “Inter-Fabric Traffic” discussed in Part 7.

      Cheers,
      Brad

  12. Paul C says:

    Excellent information, brief and straight to the point info on very relevant concepts for SEs pushing this solution to clients and needing to know all the facts (not just the cool marketing stuff).

    Many thanks,
    Paul

  13. Ambi says:

    Great post Brad

    However i have a question

    how would the traffic flow be if the 2 VMs are on different network … would it have to go to an upstream L3 switch to get switched??

    Ambi

  14. Joe says:

    dude, you rock! thank you for taking the time to create this knowledge.

  15. Mohan Muthu says:

    Excellent Brad. I am very new to UCS and have one question on the videos.

    What does vEth1, vEth2, vEth3 etc. shown in your video inside the 6100? I was unable to find any documentation about this.

    Does this mean these are virtual ethernet ports corresponding to every vNIC on the blade servers? Can these ports be configured like any other switch port?

    Any information on these would be really helpful.

    thank you for the excellent video.

    Mohan

    • Brad Hedlund says:

      Mohan,

      Does this mean these are virtual ethernet ports corresponding to every vNIC on the blade servers?

      YES!

      Can these ports be configured like any other switch port?

      In theory, Yes. In practice, No. Cisco UCS Manager does all the configuration of the vEth ports for you, based on the settings of the servers vNICs in the Service Profile for that server.

      Cheers,
      Brad

  16. Mohan Muthu says:

    Thank you Brad! Appreciate your quick response.

    Your page is at the top of my “favourites” bookmarks!!!

    Regards,
    Mohan

  17. Eric T says:

    Hi Brad,
    I just watched the videos and really enjoyed them. First, Thank You for taking the time to do this. It was very clear and clearly you took the time to ensure your audience was following you. Second, I found myself wishing you included the failure scenarios from a VM perspective. The box vs the bowtie design I am happy with from a network design perspective. When I review all your failure scenarios in total you have covered all my bases; however I see the coming convergence of SysAdmin and NetworkAdmin responsiblities thus covering the failures from a virtual machine perspective would be very helpful, then I can send the SysAdmins to this page and tell them all their networking questions are answered here. :)
    For example I can see them wanting to load-balance across the Fabric A and Fabric B, nic teaming in VM. Is this good or bad from a network perspective? Personally I don’t like it since the network has taken care of the failure and load-balancing but I need to be able to articulate why to SysAdmins. Heck I don’t even know if VM supports this but I would like to know.
    My statment isn’t a complain by the way, only an observation about who your audience is and who they will be in a couple of years.
    Thanks Again!

  18. Raghu says:

    Hi Brad,

    Thanks for sharing excellent video’s about UCS. I would like to know one infromation regarding UCS.

    Is the UCS cluster ID is unique across the network ? If possible can you share any information about cluster configuration ?

    Thanks again :-)

  19. ChalieM says:

    Hello Brad..

    Great presentation. I’m a newbie to the Cisco UCS solution and your presentation spells it all out. I do have a question though. I’ve been recently approved to upgrade our VMware infrastructure and network core. We are currently a Dell shop, but I’m seriously considering replacing it with UCS. Can you tell me in a sentence or two the advantages/disadvantage going with Cisco solution over the DELL blade chassis? Out side of the management and switch configuration on the DELL platform, what’s different?

    • Brad Hedlund says:

      Charles,
      I have some recent experience with a large financial customer evaluating Cisco UCS vs. DELL blade solutions. In every conversation or visit with this customer I would espouse many unique virtues of Cisco UCS; the simplicity of implementation, 10GE wire once infrastructure, stateless computing, the unique innovations with VMware, etc, etc. After carefully evaluating both platforms this customer told me, in a nutshell, paraphrasing here: “It’s UCS Manager, stupid!” (Coining the popular “It’s the Economy, stupid!”).
      For this customer, the out of the box capabilities in UCS Manager was the killer differentiator that radically simplified the management of all the servers and network settings.

      For example, template driven provisioning and updates. Not only can you deploy server configurations from templates within minutes, you can upgrade the firmware on a template and the hundreds of servers tied to that template are automatically upgraded by the system. Need to add a new adapter to hundreds of servers? No problem, add the new adapter to the template and you’re done. This is just scratching the surface of the UCS Manager capabilities built in to the system, not a bolt on software package with extra license fees.

      Cheers,
      Brad

  20. aiwa says:

    hi Brad, gr8 post.

    One question i cudnt find an answer –

    Why is NEXUS needed for UCS? Feature advantages? Dependencies?
    Can a UCS system be connectd to a conventional Cisco 4500/6500 based data network instead of using Nexus?

    • Brad Hedlund says:

      aiwa,
      Yes – you can connect UCS to any standard 1GE or 10GE switch.
      Nexus is a good choice for the 10GE density, performance, high availability, and vPC – as discussed in one of the videos.

  21. Brad Morgan says:

    Brad,

    What would the “best practice” recommendation be for a vmware deployment on UCS with redundant fiber interconnects connected to Nexus switches when the “SAN” connectivity is via iSCSI rather than FC. Would it be better to group the iSCSI vlan in with the other wan/lan traffic vlans so that it could share in the benefits present with vPC, or would it be best to set up the 6100s in switch mode to isolate the iSCSI vlan to protect it from broadcast storms or loops in the other vlans that might choke its bandwidth off. Is there a way to guarantee bandwidth to the iSCSI vlan to make sure that it won’t be adversely affected if an issue arises in one of the other vlans?

    Your posts are exceptinally helpful.

    Thank you

    • Brad Hedlund says:

      Brad,
      To partition BW for iSCSI, you can keep UCS in End Host mode, use a LAN Pin Group and call it “iSCSI” or something. Pick the uplink you want dedicated for iSCSI traffic and put it in your iSCSI Pin Group. Take the vNICs used for iSCSI and assign them to the “iSCSI” LAN Pin Group.

      -Brad

  22. VIJAY SHEKHAR says:

    Great work Brad!! really!

    I have a question w.r.t Fabric Interconnect EHV Mode.
    EHV mode does not learn MAC addresses on BI, It only learns MAC on SI.
    now when server is sending a packet out of the POD to ethernet cloud, it will ARP and learn the MAC address of the nexy layer3 Hop, All good till here.
    Now when the FI gets that Frame from server destined to L3 hop for that network it will be a Unlearnt MAC.
    I also read that all Unlearnt MAC recieved on SI are Flooded to All other SI and Pinned BI.
    If tha bove statement is true does FI flood traffic leaving the POD to ALL SI interfaces EVERYTIME?
    It sounds too silly, but asking based on what I ave been reading.
    Thanks!
    Vijay Shekhar.

  23. Leo says:

    Hello,

    one question regarding connecting a UCS to two or more L2 domains: To avoid switching the fabric interconnects to switch mode, would it be feasible / recommended to build several vPC Channels and pin the vNICS manually to these channels?

    __Leo

    • Brad Hedlund says:

      Leo,
      No – static pinning will not work. The FI will still only pick one uplink as the broadcast listener. That’s the case for UCSM version 1.4 or older.

      In the next release of UCSM, End Host Mode will be enhanced to support separate upstream L2 domains.

      Cheers,
      Brad

      • Gab says:

        Hi Brad,
        one more time i find answers to our questions on your site!! Thanks! i’d like to know about this post i’m replying to. “In the next release of UCSM, End Host Mode will be enhanced to support separate upstream L2 domains.”. I was also thinking about the solution to have different VPC’s per L2 domains and static pinning, but a s you says it does not work for UCS 1.4 or older. Is this already solved on the new versions?. We are under scenario where we are sharing a chassis and FI’s to connect to 2 different L” domains and it causing some trouble. Is there any alternative to not use switching mode?

        Thanks!

  24. Alex Wilkinson says:

    Pure Gold! Thank you for providing this for the community!

    -Alex

  25. Praveen says:

    Hi ,

    Awesome information!!!

    I am designing a Network where we need to connect the 6120Xp Fabric extender to Catalyst 6509 . I found that we can connect the fabric Interconnect (FI) to 1G port of an uplink switch using The first 8 fixed ports can be configured to operate in 1 Gigabit Ethernet mode with the transceiver options specified for use with SFP-compatible ports using 1000BASE-T SFP GLC-

    Since I have 2Nos 6509 Switches in the Datacentre I can connect only 4 Ports from 6120XP to each of 650. Can I configure channel from 6120Xp to 6509 between these 4 links?

    Thanks and Regards
    Praveen

  26. William Fleitz says:

    Brad, Excellent presentation. I have a quick question perhaps you or someone else can answer. My server team wants to connect a VMotion switch to the UCS so that we can Vmotion legacy VM servers to the UCS ESX farm. I’m wondering what is the best practice to accomplish this? It would need to be accessible across both 6100s. Thanks!

  27. Marwan says:

    Hi Brad,
    i know its a late comments or question but just wondering about best practice UCS/N1K networking
    where cisco Docs recomend to use LACP port channel with clustered upstream switches (vPC,VSS)
    and vpc_HM with non-clustered switches with mac-pining
    the question here
    if we present one vNIC to the ESX host and N1K and in UCS this vNIC has failover enabled from FIA to FIB
    and in the back end this vNIC has the UCS port-channel to the physical switch
    why we need to worry about port channelling if its already will be done by UCS and the esxi host and N1K will see one uplink/vNIC???
    ( understood with palo we can vitalize these vNICs without enabling fail over to send traffic selectively over certain ports if required ! )

    Thanks

  28. Eddie Wieder says:

    Our UCS is version 2.0w and is configured for switched mode. We have a 3750x Stack with 10GB uplink ports configured in Port Channel with two 6100’s. We now need to connect our Two DMZ’s to our UCS but keep the traffic separated, if possible. We are currently running VMWare for all of our 4 server hosts. My question is based on Video 6. I am unsure on how to do this correctly as it seems to keep bridging/broadcasting each sides networks. Any help would be greatly appreciated.

    • Brad Hedlund says:

      Eddie,
      In switch mode there should be no issue. Just connect your DMZ switches to the 6100s, pick the VLANs within UCS that will be the DMZ VLANs, and make sure only those VLANs are forwarding on the links connecting to the DMZ.

  29. Shashidhar Patil says:

    Hi Brad,
    I do not see that failure cases of Fabric explained. If Fabric A fails
    all vNIC0 generated traffic is disrupted ? Can a VM use one port each from vNIC0 and vNIC1 for
    redundancy and team those ports ?

  30. Joe L says:

    Brad,

    Thanks so much for these. They are loaded with great information.

    Question: I see that you are using slide decks with Cisco on them. Are these slide decks available online? It would be very helpful if I could pull these down to my local machine to flip through.

    Any information is appreciated.

    Thank you again!
    Joe

  31. Chad says:

    Is it possible to connect C-Series to 6248 Fabric Interconnects directly without requiring the Fabric Extenders if there is no intent to centrally manage the C-Series? I simply want to use 10GB to access the same SAN as our blade chassis.

    Thanks

  32. Leonardo says:

    Hi Brad!

    Congratulations for yours presentations!!
    I´m planning this change in the UCS enviroment:
    I have 2 port-channels (1 Fabric A + 1 Fabric B) connecting to a upstream network (3750 Stack). The actual portchannel is a 1G and I need to migrate to a 10G portchannel.
    How can I do that without disruption to the createds LAN Pinning Groups and vNICs configurations?
    I had tried this scenario with the Cisco UCSPE, but I´m not sure about the good way because at the Virtual enviroment the uplinks ports doesn´t came up.

    Thank you very much!!

    Best Regards!

    Leonardo

  33. Ray says:

    FYI: I’m new to this UCS product….

    On the FI, can you do traffic shaping? For example, I have a FI with two modules and if I want to use my module two for certain type of traffic (NFS, SAN, Database, etc.) The goal here is to streamline and configure my FI to isolate types of traffic to get most performance out of it.

    Thanks,
    -Ray

  34. Wael says:

    Hi Brad.

    i have a great confusion regarding the VIC card, how it is seen from the hypervisor side from example VMware vSphere, i mean the physical adapters from the hypervisor perspective. second i have a great wonder if it is right that the every VM can granted a VEM and every VEM have one path through a VNIC or HBA out of the server to one FEX and a back up path the same way way through the second FEX.

    im so confused with that subject, i hope u can dedicate a video for that Brad. and if u can refer me to any good URL for me to grasp the idea u would be so thankfull

    thanks

  35. jason says:

    Good stuff,

    Any recommendations on trunking multiple pairs of UCS fabrics together? I am somewhat surprised the ucs 6248 and 6296 do not support this currently.

  36. Dave Owen says:

    hey brad,

    in part 6 you make reference to your design not having overlapping vlans. if you did have overlapping vlans, what is the best practice?

    thanks,

    dave

  37. Danny says:

    Hey,
    thanks for the vids.
    I have one thing about inter fabric traffic: As I know, Fabric Interconnect handles layer2 traffic inside itself, so if you connect all vnics within the same vlan with the same fabric, the traffic will not go out to the l2 switch.

    If you design your UCS as follow;
    – all vnics 0 (VLAN10) pinned to Fabric A,
    – all vnics 1 (VLAN 20) pinned to Fabric B,
    Can you connect UCS directly to layer3 switch, can’t you?

Leave a Reply

Your email address will not be published. Required fields are marked *