This question comes from a reader named Wei about how Cisco UCS behaves in End Host Mode with respect to MAC learning and flooding. Wei paints a scenario of two servers in the same VLAN, one inside Cisco UCS, the other outside of Cisco UCS. With the Fabric Interconnect in End Host Mode, what happens when the server inside UCS tries to communicate with the server outside UCS, starting with ARP and the subsequent unicast conversation.

Question:

I recently came across your website, it is really an awesome resource. Thanks for sharing your data center knowledge. I learned a lot about UCS networking from the HD videos you posted on http://bradhedlund.com/2010/06/22/cisco-ucs-networking-best-practices/. I do have a question regarding End Host mode and hoping you can help me with it.

Here is the scenario, the communication is with two devices on the same vlan but connected to different devices.

NIC 10 is on vlan 10 and is connected on a server port on FI 6100, NIC 20 is on vlan 10 and is connected to an upstream switch that connects to the uplink ports on the same FI 6100. Let&’s say NIC 10 ARPs for NIC 20, the frame is sent to the upstream switch via the uplink that handles the broadcast, and also other local vlan 10 server ports. Since there are no MAC learning on the uplink ports, so when the ARP reply comes back down the uplink, does this mean NIC 20&’s MAC will not be recorded in the MAC table on the FI 6100? what happens to subsequent unicast frames from NIC 10 to NIC 20? I know they will go up the uplink, but will they also be flooded to the local vlan 10 server ports also?

My Answer:

Here’s the setup: I’ll refer to the server inside Cisco UCS as Server A, and the server outside UCS as Server B. The Cisco UCS Fabric is in End Host Mode. Server A will initiate a conversation with Server B. Both servers are in VLAN 10. Server A’s primary NIC for VLAN 10 is connected to 6100-A.

When Server A issues an ARP message to learn Server B’s MAC address, this is a broadcast message that will be sent to all servers inside UCS on VLAN 10, connected to 6100-A. Additionally, the ARP message will also be sent out the UCS uplink Server A is pinned to. Note that 6100-A’s “Broadcast” link is an uplink it has chosen for receiving broadcasts, not sending broadcasts. When a server sends a broadcast message, that broadcast will always exit UCS on the same uplink designated for all other traffic from that server, the uplink automatically chosen via dynamic pinning or statically via LAN Pin Groups.

The upstream switch will receive the broadcast on the uplink from UCS pinned to Server A, and if the upstream switch has no prior knowledge of Server A it will use this broadcast message to learn Server A’s MAC address on this interface. The upstream switch will flood this broadcast ARP message on all other interfaces forwarding for VLAN 10 (including links connected to 6100-B). * Note that the upstream switch will also send this broadcast back to 6100-A on all the other interfaces it has facing 6100-A. However, only the link 6100-A has picked as its “Broadcast link” will actually process the broadcast. 6100-A will notice that the broadcast was originated from one of its own servers, and just drop it.

Server B will receive the broadcast ARP message and respond directly (unicast) back to Server A. The upstream switch receives Server B’s (unicast) ARP response to Server A’s MAC address on VLAN 10. Because the upstream switch has already learned which of its interfaces leads to Server A, it sends Server B’s response directly to this interface connected to 6100-A.

6100-A receives Server B’s unicast ARP response from the upstream switch on the uplink pinned to Server A. Because Cisco UCS is in End Host Mode, it will NOT learn the MAC address & location of Server B like the upstream switch did for Server A.

6100-A knows the server port and logical interface Server A is located on because of the authoritative knowledge provided by UCS Manager. Hence, 6100-A transmits Server B’s unicast response directly to Server A.

Server A receives the ARP response and sends a unicast message to Server B.

6100-A receives the unicast message destined to Server B. However, 6100-A does not have any knowledge of Server B’s location on the network. Server B is outside of UCS so there is no authoritative knowledge, no MAC table information for Server B. Because of this, 6100-A makes the assumption that Server B must be accessible in the upstream network and sends the unicast message for Server B out of the uplink pinned to Server A (because it was sent from Server A). The upstream switch receives the unicast message and simply transmits it directly to Server B.

In summary, Cisco UCS in End Host Mode uses this simple logic:

“If I receive traffic from one of my servers destined to something I dont know about, it must be out in the network somewhere so I’ll just send it out a pinned uplink.”*

“If I receive unicast traffic from an uplink destined to something I dont know about, I will just drop this traffic.”

“I will only pay attention to broadcasts received from my servers or received on my designated Broadcast uplink.”

“If I receive broadcast traffic from my designated Broadcast link, I will send the broadcast to my servers but not to my other uplinks.”

“If I receive broadcast traffic from my designated Broadcast link that originated from one of my own servers, I will just drop this traffic.”

Make sense?