UPDATE 3/8/2011: This video series has been obsoleted by a new and updated series posted here: Cisco UCS Networking videos (in HD), Updated & Improved!
This is a presentation I developed covering networking best practices for Cisco UCS, and now have recorded in High Definition for your viewing pleasure! Sweet! 
This presentation assumes familiarity with basic networking and server VNIC concepts in UCS, and familiarity with virtual port channels.
This version of the presentation (v2.5) focuses primarily on the Ethernet uplinks. SAN uplinks and VMware networking scenarios are briefly discussed but not covered extensively. Those topics and others such as QoS, the Cisco VIC, and vNIC fabric failover may be included in future versions of this presentation.
Stay tuned for updates! RSS feed: http://bradhedlund.com/feed/
UPDATE 3/8/2011: This video series has been obsoleted by a new and updated series posted here: Cisco UCS Networking videos (in HD), Updated & Improved!
Part 1 – Cisco UCS Networking Overview
In Part 1 we take start with a baseline overview of Cisco UCS Networking. At the heart of the system is the Fabric Interconnect (6100) “the Brains of UCS” which provides 10GE & FC networking for all the compute nodes in its domain as well as being the central configuration, management, and policy engine for all automated server and network provisioning.
Part 2 – Switch Mode vs. End Host Mode
Part 2 is an examination of the two different switching modes supported by the Fabric Interconnect, “Switch Mode” and “End Host Mode”. With “Switch Mode”, the Fabric Interconnect behaves like a normal Layer 2 switch on all server ports and uplinks, and therefore attaches to the upstream data center network as a spanning tree enabled “Switch”.
“End Host Mode”, on the other hand, while still providing local Layer 2 switching on the server ports, does not behave like a normal Layer 2 switch on its uplinks. Instead, server NICs are “pinned” to a specific uplink, and no local switching happens from uplink to uplink. This allows “End Host Mode” to attach to the network like a “Host” without spanning tree, and all uplinks forwarding on all VLANs.
End Host Mode is the preferred mode, and it’s enabled by default.
Part 3 – End Host Mode – Individual Uplinks
In Part 3 we take a look how the individual uplinks behave in End Host Mode, and how the system reacts to uplink failures. When an uplink fails, the Fabric Interconnect will move the server NICs to a new uplink in under a second without causing any disruption to the server NIC. This uplink failover process is called dynamic re-pinning.
After the dynamic re-pinning process, the Fabric Interconnect will send Gratuitous ARP messages for all of the MAC address that were previously using the failed uplink. This GARP process aids the upstream network in quickly learning the new location of the affected MAC address now using the new uplink.
Part 4 – Port Channel Uplinks
Here we take a look at the benefits of using Port Channel uplinks with Cisco UCS. The key advantages to port channel uplinks is the minimal impact of a physical link failure and the potential for better overall uplink load balancing. During individual physical link failures fewer moving parts required to provide a fast recovery. For example, Gratuitous ARP messages and dynamic re-pinning are not required when an individual physical member link fails in a port channel uplink. Port Channel uplinks are definitely recommended whenever possible.
Part 5 – Virtual Port Channel Uplinks (vPC)
Part 5 covers the advantages of using virtual port channel (vPC) uplinks with Cisco UCS. With vPC uplinks, there is minimal impact of both physical link failures and upstream switch failures. With more physical member links in one larger logical uplink, there is the potential for even better overall uplink load balancing and better high availability than with a standard Port Channel uplink discussed in Part 4. Using a virtual port channel uplink is highly recommended if you have vPC capabilities present in your upstream network switches.
Part 6 – Connecting Cisco UCS to separate networks
In Part 6 we discuss the scenario of connecting a single Cisco UCS system in End Host Mode to separate Layer 2 networks. When the system is in End Host Mode, it expects and assumes that all uplinks are connected to the same common Layer 2 domain. If some uplinks are connected to physically separate networks you will have connectivity problems. The Fabric Interconnect will randomly pick one of its uplinks to process broadcast messages for all VLANs. As a result, only servers associated with the chosen network will be able to see and process broadcasts messages on their network. The solution is create a common Layer 2 network for the Fabric Interconnect in End Host Mode and each of the separate networks to attach to, or, use Switch Mode. If creating a common Layer 2 network or using Switch Mode is not an option, you can always deploy a unique Cisco UCS system per separate network to preserve the existing “silos”.
Part 7 – Inter Fabric Traffic Examples
This is a brief look at some the common types of traffic flows that may flow between Fabric-A and Fabric-B within a single Cisco UCS system. With this understanding, the subsequent material will make more sense.
Part 8 – Don’t: Connect Cisco UCS to vPC domains without vPC uplinks
This is a fairly extensive look at the scenario of attaching UCS to upstream switches configured for vPC, without using vPC uplinks. Here we will show that this scenario doesn’t make much sense and in fact can cause some unwanted traffic black holes under some failure scenarios. This is a prelude to Part 9 where we illustrate that if your upstream network is configured for virtual port channel capability (vPC), you should always attach UCS with vPC uplinks.
Part 9 – Do: Connect Cisco UCS to vPC domains with vPC uplinks
This section shows that if you have virtual port channel capabilities in your upstream switches, you have everything to gain and nothing to loose by connecting Cisco UCS with vPC uplinks. You will gain the benefit of the upstream switch locally switching all Fabric-A to Fabric-B traffic, and acheiving more bandwidth scalability for inter-fabric traffic because all inter-fabric traffic will travel on the vPC uplinks, rather than on less abundant inter-switch links. Additionally, you will avoid potential black hole failure scenarios discussed in Part 8, if vPC is already present in the upsteam network switches.
Part 10 – Connecting Cisco UCS without vPC
While there are certainly advantages to uplinking Cisco UCS with virtual port channels, vPC is certainly not required. Cisco UCS easily and efficiently connects to any data center network environment with or without vPC. This section discusses best practices connecting UCS to networks without vPC. The key best practice here is to always dual attach each Fabric Interconnect to two upstream network switches, whether its with vPC uplinks, or multiple individual uplinks. Another suggested practice is to avoid attaching Cisco UCS to a second tier Layer 2 switch with spanning tree blocking links. A better approach is to either have vPC capabilites at the second tier Layer 2 switch, or connect Cisco UCS directly to the tier 1 switch, avoiding a traffic bottlenecks induced by spanning tree.
Follow up post and whitepaper: Cisco Nexus 7000 connectivity solutions for Cisco UCS
Great stuff here Brad. Thanks educating us on this stuff, and keep up the good work!
Brad,
Wow these are great videos! Thanks for taking the time to make all of them and put them on your site.
Brad….amazing videos and great blog…thanks!
Brad,
I really appreciate this post. It has been extremely valuable in our configuration and evaluation of the UCS platform.
One question I would like clarified about UCS best practices has to do with 1000v. I was wondering if you might be able to shed some light. In the “Best Practices in Deploying Cisco Nexus 1000V Series Switches on Cisco UCS B Series Blade Servers” gude, found here: http://cisco.biz/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-558242.html there is a statement regarding the placement and connectivity of VSMs in a 1000v implementation on UCS. It gives several options “listed in order of recommended customer use” but says little to justify the order of preference. I am looking for any information that would help me understand the reasoning behind the recommendation. Also I am wondering why VSM Inside the Cisco Unified Computing System on the Cisco Nexus 1000V Series VEM is not recommended or at least not mentioned at all.
Jason
Brad,
I have configuration question. when connecting a Fabric Interconnect running in End Host Mode to a nexus 5k vPC, what spanning-tree port type should the 5k port channel be? i.e. spanning-tree port type edge or spanning-tree port type network?
thanks!
Andy,
The Fabric Interconnect in End Host Mode connects to the upstream switch like a “Host”, not as a “Switch”. Therefore you should configure the upstream switch port (or port channel in your case) just as you would if it were a server attaching to it. Therefore the best configuration on the 5K, in your case, would be: ‘spanning-tree port type edge trunk‘
Cheers,
Brad
Hi Brad,
These videos are really informative and clear. It is great to have a place to review topics.
I would love to see something similar where the subject is “How Nexus platform works with UCS, in the data center”.
Joe
Brad,
awsome post…i was just having a tech session with your Cisco peer Steve Fishman… good stuff… I’d like to see the same for BP on FCoE to FC from the 6100 to MDS91xx… and how to scale FC to large SAN env from UCS.
Awesome videos. It greatly helped me to understand the connectivity and traffic flow path in the UCS system. Thank you
Brad:
Awesome vids.
I give this a A+ for the presentation of very useful information.
I found this very helpful but the delivery and tone…. knocked me out. I could hardly stay awake.
This is an A++ as a sedative. Dude you should probably consider moonlighting as a hypnotist.
JM
Great job though. Keep up the good work.
Hello,
In watching video#2 and looking at the difference between switch mode and end host mode I don’t see where / how I would team nics to a host. In your example vlan 10 is only on 6100A and vlan 20 is only on 6100B which suggests that if we lost 6100A we would lose vlan 10 completely.
Wouldn’t a better approach be to put vlans 10 & 20 on both 6100s and trunk and team them to the servers via LACP so that a failure would allow the traffic to still have a path to server ?
Robert,
Yes, in most cases the same VLANs will exist on both 6100s. You would not use LACP for NIC teaming because that would require the 6100s to be clustered together in a vPC domain, and that is not supported. For NIC teaming you could use standard active/standby teaming, the unique vNIC “fabric failover” capability in UCS, or in the case of a VMware host you can also use active/active NIC load balancing based on vPort-ID or Mac Pinning.
Brad
Hi Brad,
I enjoyed these great videos, thanks!
To your statement “For NIC teaming you could use standard active/standby teaming,”
assuming active/standby alternating (server1 active to FEX1, server 2 active to FEX2), how is the traffic flow between server 1 and 2 ?
Thanks,
Matthew
Matthew,
In this scenario:
(server1 active to FEX1, server 2 active to FEX2), how is the traffic flow between server 1 and 2 ?
Traffic between Server 1 and Server 2 would travel through the upstream network. This is another example of “Inter-Fabric Traffic” discussed in Part 7.
Cheers,
Brad
Excellent information, brief and straight to the point info on very relevant concepts for SEs pushing this solution to clients and needing to know all the facts (not just the cool marketing stuff).
Many thanks,
Paul
Great post Brad
However i have a question
how would the traffic flow be if the 2 VMs are on different network … would it have to go to an upstream L3 switch to get switched??
Ambi
Ambi,
Yes, traffic flow between two VM’s on different VLANs would travel through an upstream L3 switch, such as a Nexus 7000 or Catalyst 6500.
dude, you rock! thank you for taking the time to create this knowledge.
Excellent Brad. I am very new to UCS and have one question on the videos.
What does vEth1, vEth2, vEth3 etc. shown in your video inside the 6100? I was unable to find any documentation about this.
Does this mean these are virtual ethernet ports corresponding to every vNIC on the blade servers? Can these ports be configured like any other switch port?
Any information on these would be really helpful.
thank you for the excellent video.
Mohan
Mohan,
YES!
In theory, Yes. In practice, No. Cisco UCS Manager does all the configuration of the vEth ports for you, based on the settings of the servers vNICs in the Service Profile for that server.
Cheers,
Brad
Thank you Brad! Appreciate your quick response.
Your page is at the top of my “favourites” bookmarks!!!
Regards,
Mohan
Hi Brad,
I just watched the videos and really enjoyed them. First, Thank You for taking the time to do this. It was very clear and clearly you took the time to ensure your audience was following you. Second, I found myself wishing you included the failure scenarios from a VM perspective. The box vs the bowtie design I am happy with from a network design perspective. When I review all your failure scenarios in total you have covered all my bases; however I see the coming convergence of SysAdmin and NetworkAdmin responsiblities thus covering the failures from a virtual machine perspective would be very helpful, then I can send the SysAdmins to this page and tell them all their networking questions are answered here.
For example I can see them wanting to load-balance across the Fabric A and Fabric B, nic teaming in VM. Is this good or bad from a network perspective? Personally I don’t like it since the network has taken care of the failure and load-balancing but I need to be able to articulate why to SysAdmins. Heck I don’t even know if VM supports this but I would like to know.
My statment isn’t a complain by the way, only an observation about who your audience is and who they will be in a couple of years.
Thanks Again!
Hi Brad,
Thanks for sharing excellent video’s about UCS. I would like to know one infromation regarding UCS.
Is the UCS cluster ID is unique across the network ? If possible can you share any information about cluster configuration ?
Thanks again
Hello Brad..
Great presentation. I’m a newbie to the Cisco UCS solution and your presentation spells it all out. I do have a question though. I’ve been recently approved to upgrade our VMware infrastructure and network core. We are currently a Dell shop, but I’m seriously considering replacing it with UCS. Can you tell me in a sentence or two the advantages/disadvantage going with Cisco solution over the DELL blade chassis? Out side of the management and switch configuration on the DELL platform, what’s different?
Charles,
I have some recent experience with a large financial customer evaluating Cisco UCS vs. DELL blade solutions. In every conversation or visit with this customer I would espouse many unique virtues of Cisco UCS; the simplicity of implementation, 10GE wire once infrastructure, stateless computing, the unique innovations with VMware, etc, etc. After carefully evaluating both platforms this customer told me, in a nutshell, paraphrasing here: “It’s UCS Manager, stupid!” (Coining the popular “It’s the Economy, stupid!”).
For this customer, the out of the box capabilities in UCS Manager was the killer differentiator that radically simplified the management of all the servers and network settings.
For example, template driven provisioning and updates. Not only can you deploy server configurations from templates within minutes, you can upgrade the firmware on a template and the hundreds of servers tied to that template are automatically upgraded by the system. Need to add a new adapter to hundreds of servers? No problem, add the new adapter to the template and you’re done. This is just scratching the surface of the UCS Manager capabilities built in to the system, not a bolt on software package with extra license fees.
Cheers,
Brad
hi Brad, gr8 post.
One question i cudnt find an answer -
Why is NEXUS needed for UCS? Feature advantages? Dependencies?
Can a UCS system be connectd to a conventional Cisco 4500/6500 based data network instead of using Nexus?
aiwa,
Yes – you can connect UCS to any standard 1GE or 10GE switch.
Nexus is a good choice for the 10GE density, performance, high availability, and vPC – as discussed in one of the videos.
Brad,
What would the “best practice” recommendation be for a vmware deployment on UCS with redundant fiber interconnects connected to Nexus switches when the “SAN” connectivity is via iSCSI rather than FC. Would it be better to group the iSCSI vlan in with the other wan/lan traffic vlans so that it could share in the benefits present with vPC, or would it be best to set up the 6100s in switch mode to isolate the iSCSI vlan to protect it from broadcast storms or loops in the other vlans that might choke its bandwidth off. Is there a way to guarantee bandwidth to the iSCSI vlan to make sure that it won’t be adversely affected if an issue arises in one of the other vlans?
Your posts are exceptinally helpful.
Thank you
Brad,
To partition BW for iSCSI, you can keep UCS in End Host mode, use a LAN Pin Group and call it “iSCSI” or something. Pick the uplink you want dedicated for iSCSI traffic and put it in your iSCSI Pin Group. Take the vNICs used for iSCSI and assign them to the “iSCSI” LAN Pin Group.
-Brad
Great work Brad!! really!
I have a question w.r.t Fabric Interconnect EHV Mode.
EHV mode does not learn MAC addresses on BI, It only learns MAC on SI.
now when server is sending a packet out of the POD to ethernet cloud, it will ARP and learn the MAC address of the nexy layer3 Hop, All good till here.
Now when the FI gets that Frame from server destined to L3 hop for that network it will be a Unlearnt MAC.
I also read that all Unlearnt MAC recieved on SI are Flooded to All other SI and Pinned BI.
If tha bove statement is true does FI flood traffic leaving the POD to ALL SI interfaces EVERYTIME?
It sounds too silly, but asking based on what I ave been reading.
Thanks!
Vijay Shekhar.
Vijay,
The unknown dest MAC received from the server will be forwarded on the pinned uplink only. No flooding.
-Brad
Hello,
one question regarding connecting a UCS to two or more L2 domains: To avoid switching the fabric interconnects to switch mode, would it be feasible / recommended to build several vPC Channels and pin the vNICS manually to these channels?
__Leo
Leo,
No – static pinning will not work. The FI will still only pick one uplink as the broadcast listener. That’s the case for UCSM version 1.4 or older.
In the next release of UCSM, End Host Mode will be enhanced to support separate upstream L2 domains.
Cheers,
Brad
Pure Gold! Thank you for providing this for the community!
-Alex
Hi ,
Awesome information!!!
I am designing a Network where we need to connect the 6120Xp Fabric extender to Catalyst 6509 . I found that we can connect the fabric Interconnect (FI) to 1G port of an uplink switch using The first 8 fixed ports can be configured to operate in 1 Gigabit Ethernet mode with the transceiver options specified for use with SFP-compatible ports using 1000BASE-T SFP GLC-
Since I have 2Nos 6509 Switches in the Datacentre I can connect only 4 Ports from 6120XP to each of 650. Can I configure channel from 6120Xp to 6509 between these 4 links?
Thanks and Regards
Praveen
Brad, Excellent presentation. I have a quick question perhaps you or someone else can answer. My server team wants to connect a VMotion switch to the UCS so that we can Vmotion legacy VM servers to the UCS ESX farm. I’m wondering what is the best practice to accomplish this? It would need to be accessible across both 6100s. Thanks!
Hi Brad,
i know its a late comments or question but just wondering about best practice UCS/N1K networking
where cisco Docs recomend to use LACP port channel with clustered upstream switches (vPC,VSS)
and vpc_HM with non-clustered switches with mac-pining
the question here
if we present one vNIC to the ESX host and N1K and in UCS this vNIC has failover enabled from FIA to FIB
and in the back end this vNIC has the UCS port-channel to the physical switch
why we need to worry about port channelling if its already will be done by UCS and the esxi host and N1K will see one uplink/vNIC???
( understood with palo we can vitalize these vNICs without enabling fail over to send traffic selectively over certain ports if required ! )
Thanks
Our UCS is version 2.0w and is configured for switched mode. We have a 3750x Stack with 10GB uplink ports configured in Port Channel with two 6100′s. We now need to connect our Two DMZ’s to our UCS but keep the traffic separated, if possible. We are currently running VMWare for all of our 4 server hosts. My question is based on Video 6. I am unsure on how to do this correctly as it seems to keep bridging/broadcasting each sides networks. Any help would be greatly appreciated.
Eddie,
In switch mode there should be no issue. Just connect your DMZ switches to the 6100s, pick the VLANs within UCS that will be the DMZ VLANs, and make sure only those VLANs are forwarding on the links connecting to the DMZ.
Hi Brad,
I do not see that failure cases of Fabric explained. If Fabric A fails
all vNIC0 generated traffic is disrupted ? Can a VM use one port each from vNIC0 and vNIC1 for
redundancy and team those ports ?
Brad,
Thanks so much for these. They are loaded with great information.
Question: I see that you are using slide decks with Cisco on them. Are these slide decks available online? It would be very helpful if I could pull these down to my local machine to flip through.
Any information is appreciated.
Thank you again!
Joe
Is it possible to connect C-Series to 6248 Fabric Interconnects directly without requiring the Fabric Extenders if there is no intent to centrally manage the C-Series? I simply want to use 10GB to access the same SAN as our blade chassis.
Thanks