I received some really good questions about FCoE, VN-Tag, FEX, and vPC from a reader named Lucas. Although I had 10 other things to do, I just couldn’t resist highlighting these questions, and my answers, in a new post that I thought my readers would enjoy!

Brad, You have amazing information about Nexus and UCS on your website. Please keep up the good work. I have a few queries would appreciate if you could please point me in the write direction.

1) FCOE with Vpc, How does this work. For Fcoe we must login to one fabric only, how will the load balancing offered by Vpc effect it?

From the perspective of the CNA installed in the server, you have to keep in mind that its really two different logical adapters hosted on one physical adapter, Ethernet & FC. The FC logical adapter on the CNA has no visibility or awareness of vPC - it still see’s two individual paths, one left, one right, and doesn’t behave any differently with or without vPC. More specifically, a dual-port CNA will actually have (2) logical FC adapters, each one with their own port, and each port typically connected to a separate fabric. The Ethernet logical adapters however are vPC aware and will treat the two paths as a single logical 20GE pipe for all the traffic it is handling (all the non-FCoE stuff).

UPDATE: Diagram below added for visual aid.

CNA with FCoE and vPC

2) VN-tag, is the tag applied by a vmware machine or only a fex(i/o in ucs or fex 2000) Can apply this tag?

VN-Tag is a modern day version of a virtual cable that connects a virtual NIC, or virtual HBA, hosted on an NIV capable adapter to an upstream virtual ethernet or virtual FC port on an NIV capable switch. In the case where the server does not have an NIV capable adapter, VN-Tag can also be used to connect a physical port on a fabric extender (FEX) to an upstream virtual ethernet port.

In a nutshell, an NIV capable adapter will apply the VN-Tag as traffic egresses from one of its virtual adapters. Any FEX in the path will just pass that traffic upstream to the switch terminating the other end of the virtual cable (VNTag). In this case you could think of the FEX as a virtual patch panel for virtual cables.

If you connect a plain non-NIV adapter to a fabric extender (FEX), it will be the FEX that applies the VN-Tag. In this sense, you still have a cable, but half of it is physical (server-to-FEX), and the other half is virtual (FEX-to-switch). In this case, you could think of the FEX as a physical to virtual media converter.

3) In FIP why do we need multiple Macs ( FPMA), I understand that FPMA will relieve the switch from creating a mapping between fcid and mac, but other than that why does the standard talk about multiple FC_LEPs on a single port. I am assuming each lep would need a separate mac, I am having a hard time visualizing it in real life.

Similar to Question #1 above, it helps to understand that the server CNA is really hosting two (or more) different logical adapters, Ethernet and FC. Each logical adapter will have its own link identity (MAC/WWN). Given that the actual physical medium is Ethernet, the logical FC adapter can’t use a WWN on the medium, so it uses a Ethernet MAC instead which will be the FC_LEP (link end point). As you point out, its most efficient when the FCoE switch can automatically provide this MAC to the Server, for administrative ease. Known as FPMA (fabric provided MAC address).

The same concepts hold true for the upstream switch. The FCoE switch is really two different switches hosted on one hardware platform, an Ethernet switch and a FC switch. The FC logical switch needs to look at the FC frames carried within the FCoE packets to process fabric logins and make a forwarding decision. In order to do that, it needs to receive the FCoE frames, decapsulate, make a decision, and re-encapsulate into FCoE again if necessary. The FC logical switch has an FC_LEP for this very reason, so that it can send and receive Ethernet frames carrying FC payload (FCoE).

If you could only read one book on FCoE to better understand these concepts, it would certainly be this one: http://www.ciscopress.com/bookstore/product.asp?isbn=158705888X

4) In a 2232PP FEX why is the straight through design preferred? will the fcoe break if we did active/active design? Please assist. Thanks, Lucas

As we discussed in Question #1, the logical FC adapters on the server CNA are oblivious to vPC. As a result, attaching a server CNA with vPC makes no difference to how FCoE is forwarded via two separate paths to two separate fabrics. However, this is not the case with a FEX or a switch which will forward the FCoE traffic on whatever Ethernet topology you place it on. If this topology includes a vPC that spans two different fabrics, then you will have FCoE traffic from one of your logical FC adapters landing on both fabrics. This could be confusing to determine where FCoE traffic is going, as well as breaking the traditional FC best practice of SAN A/B isolation. Although you certainly could do this, it’s just not a supported design right now.

As a result, as of right now Cisco does not recommend that you place FCoE traffic on an Ethernet topology spanning two fabrics (A/B, Left/Right, etc.). Therefore, if your Nexus 2232 FEX will be carrying FCoE traffic from CNA’s, you should NOT vPC attach the 2232 FEX to two different upstream Nexus 5000’s. Additionally, if your two upstream Nexus 5000’s are connected together for vPC, you should NOT forward FCoE VLANs on the vPC peer link. This will keep your FCoE forwarding deterministic and preserve two separate SAN fabrics. You haven’t lost any redundancy because your servers are all dual-attached to separate 2232 FEX’s which are each attached to separate Nexus 5000’s.

In the end you have something that looks like this:

Image from: Data Center Access Design Guide, Chapter 6

FCoE Design with Nexus 2232 and Nexus 5000

Make sense?

Thanks for the great questions! Keep’em coming :-)

Cheers,
Brad