Cisco UCS and VMWare vSwitch design with Cisco 10GE Virtual Adapter

Filed in Cisco UCS, Design Diagrams, FCoE, Featured, VMware by on July 5, 2009 34 Comments

This diagram is a sample design of Cisco UCS running vSphere 4.0 utilizing the VMWare vSwitch and Cisco’s virtualization mezzanine adapter.  The Cisco adapter is a dual port 10GE Converged Network Adapter supporting Fibre Channel over Ethernet (FCoE) and Network Interface Virtualization (NIV).  The Cisco adapter is “virtual” in the sense that this single physical adapter can be carved up into as many as 128 virtual adapters.  Each virtual adapter is then exposed to the operating system as a unique physical adapter.  The virtual adapters can be either Ethernet (vNIC) or Fibre Channel (vHBA).  This is just one of many possible VMWare + Cisco UCS designs I will be depicting in a series of future posts.

Many VMWare 3.5 installations today use as many as (4), (8), or even (12) 1GE adapters. Because the 10GE Cisco adapter can be presented to the ESX kernel as many adapters, this allows the designer to preserve existing multi-NIC VMWare designs using the traditional VMWare vSwitch for an easy migration to 10GE attached ESX hosts.

Some key observations and information related to this design:

  • Some familiarity with Cisco UCS and its automated provisioning of servers with Service Profiles are assumed.
  • The dual port 10GE Cisco virtual adapter on the UCS half height blade is exposing (4) Ethernet NIC’s and (2) Fibre Channel HBA’s to the ESX kernel.
  • No special drivers or intelligence is needed by the ESX kernel to see the virtual adapters.  When the operating system (ESX) scans the PCI bus it sees each virtual adapter as what it views to be a unique physical adapter plugged into its own slot on the PCI bus.
  • The virtual Ethernet adapters are called vNIC’s (not to be confused with a virtual machine’s virtual NIC also called a vNIC).  The virtual Fibre Channel HBA’s are called vHBA’s.
  • The number of vNIC’s and vHBA’s presented by the Cisco mezzanine adapter are defined in the Service Profile for this server within the UCS Manager located on the Fabric Interconnect.
  • Some vNIC’s can be given minimum guaranteed bandwidth or a higher QoS priority over other vNIC’s.  vNIC QoS settings might be a good fit for the Service Console or VMKernel vNIC’s (not shown here).  The vHBA’s supporting FCoE have QoS (minimum bandwidth guarantees, lossless ethernet) enabled by default.  The Cisco virtual adapter QoS capabilities will be covered in detail in a future post.
  • Each vNIC has a MAC address that was either manually defined or drawn from a pool of available MAC address when this Service Profile was created.  Similarly, the pWWN for the vHBA was either manually defined or drawn from a pool of available WWN’s.
  • The vNIC or vHBA is like a real adapter in the sense that it needs to be connected to its own dedicated upstream switch port with a dedicated cable.  However in this design each virtual adapter gets its own dedicated “virtual” switch port on the Fabric Interconnect, and the cable that connects the virtual adapter to its virtual switch port is a VNTag.
  • The Cisco virtual adapter applies a unique VNTag for all traffic egressing from a vNIC or vHBA.  The VNTag is then received by the Fabric Interconnect and associated to a virtual switch port — vEth for a virtual Ethernet switch port, or vFC for a virtual Fibre Channel switch port.
  • The Data Center IT team does not need to manually define, manage, or track VNTag’s.  The VNTag numbering and associations per link are entirely managed by the UCS Manager behind the scenes during the Service Profile provisioning.
  • Similarly, the Data Center IT team does not need to define or track the virtual switch ports on the Fabric Interconnect.  When a vNIC  or vHBA is defined in a Service Profile and then applied to a blade, the UCS Manager automatically provisions the necessary VNTags and virtual switch ports needed to complete the provisioning process.
  • Cisco UCS and the Cisco virtual adapter combined have a unique “Fault Tolerant” feature that can be defined with each vNIC.  The fault tolerant feature is shown in this design with the use of dashed lines.  vHBA’s do not support this fault tolerance feature. (Important: Please see the UPDATE below)
  • A vNIC uses one of the two Fabric Interconnects as its primary path.  Should any link or device failure occur along the primary path the vNIC will switch to the secondary path to the other Fabric Interconnect — this switchover occurs within micro seconds and is undetected by the operating system.  This fault tolerance eliminates the need for active/passive NIC teaming to be configured in the operating system.
  • When fault tolerance is defined for a vNIC the UCS Manager automatically provisions virtual switch ports (vEth) and VNTag’s on both Fabric Interconnects to assist in a speedy switchover process (one primary, the other standby). (Important: Please see the UPDATE below)
  • The Service Console and VMKernel port groups shown in this design do not have a NIC teaming configuration because the UCS fault tolerance is providing the high availability.
  • The vHBA’s do not support the UCS fault tolerant feature and therefore a standard HBA multi-pathing configuration is still required in the operating system (ESX kernel) for Fibre Channel high availability.
  • The VM port groups “vlan 10″ and “vlan 20″ do have a NIC teaming configuration for the purposes of vPort-ID based load sharing.  This will allow VM’s to utilize both 10GE paths from the adapter.
  • The Fabric Interconnect is a unified fabric switch and therefore plays the role of both an Ethernet access switch and a Fibre Channel access switch.
  • The Fibre Channel configuration of the Fabric Interconnect by default operates in NPV mode (N_Port Virtualization).  Therefore, the Fabric Interconnect does not need to be managed like an individual Fibre Channel switch, rather it connects to the SAN like an End Host.
  • The FLOGI of each vHBA is forwarded upstream to the SAN switch port configured as an F_Port with NPIV enabled (both Cisco and Brocade FC switches support NPIV).  The FC ports on the Fabric Interconnect connect to the SAN as an NP_port (N_port Proxy).  Therefore, each vHBA is visible and therefore zoned by the SAN administrator as if they were connected directly to the SAN switch.
  • The Ethernet configuration of the Fabric Interconnect can also attach to the Data Center LAN as an End Host, or as an Ethernet switch.  This design is showing the Fabric Interconnect connecting to the LAN as an Ethernet switch.
  • The Ethernet uplinks are standard 10GE and can connect to any standard 10GE LAN switch (Nexus 7000 or Catalyst 6500 are recommended).
  • If the LAN switch is Nexus 7000 or Catalyst 6500, the LAN administrator can use vPC (Nexus 7000) or VSS (Catalyst 6500) features to allow the Fabric Interconnect to uplink to a redundant core with a single logical Port Channel.  This provides full active/active uplink bandwidth from the Fabric Interconnect to the LAN with no Spanning Tree topology blocking links.

Related Posts:
Cisco UCS and Nexus 1000V design diagram with Palo adapter
Nexus 1000V with FCoE CNA and VMWare ESX 4.0 deployment diagram

IMPORTANT UPDATE: This design depicts the use of the Cisco UCS vNIC “Fabric Failover” feature, or referred to in this post as “Fault Tolerant”, used in conjunction with a hypervisor switch (vSwitch or N1K).  This design combination will be supported in Cisco UCS Manager version 1.4 and above.  If you are using Cisco UCS Manager version 1.3 or below, “Fabric Failover” used with a vNIC assigned to a hypervisor switch as depicted in this design is not supported. See this post for more information

NOTE:  The Cisco virtual adapter shown in this design is one of four possible adapter options in Cisco UCS.  The other three adapter options are the Intel Oplin, Emulex CNA, and Qlogic CNA.

CORRECTION:  Contrary to what was indicated when this post was originally published, drivers and hardware qualification for the Cisco virtualized adapter highlighted in this design will be available in vSphere 4.0 (update 1), however not for VI 3.5.  To run VI 3.5 on Cisco UCS you can use the Intel Oplin adapter, or the Emulex/Qlogic based adapters.

###

About the Author ()

Brad Hedlund is an Engineering Architect with the CTO office of VMware’s Networking and Security Business Unit (NSBU), focused on network & security virtualization (NSX) and the software-defined data center. Brad’s background in data center networking begins in the mid-1990s with a variety of experience in roles such as IT customer, systems integrator, architecture and technical strategy roles at Cisco and Dell, and speaker at industry conferences. CCIE Emeritus #5530.

Comments (34)

Trackback URL | Comments RSS Feed

  1. Duncan says:

    –> The Service Console and VMKernel port groups shown in this design do not have a NIC teaming configuration because the UCS fault tolerance is providing the high availability.

    This would however mean that you will need to suppress the warning that vCenter gives when there’s no Service Console redundancy. (I’m not familiar with UCS yet but I doubt that the host is aware of the UCS FT capabilities you are talking about.

    I also wonder what happens when the vNIC port fails where the SC is running on. Would this mean that the SC is isolated and HA would kick in?

    And something else to keep in mind, what happens when the driver/module fails? (That’s one of the reason why I like using multi vendor peripherals.)

    Thanks for the insights though! Really useful.

    Duncan Epping
    http://www.yellow-bricks.com
    ps: it’s VMware and not VMWare :-) (yeah I know…)

    • Brad Hedlund says:

      Duncan,
      It would be certainly possible to create (2) vNIC’s to handle just the Service Console and configure the typical redundancy between those two vNICs. The UCS Fault Tolerance capability at the vNIC level gives you the ability to hide link failures from the operating system and performing a seamless switchover. Whether or not you want to use UCS Fault Tolerance feature in your design is completely up to you, it’s entirely optional and not a required design element.

      what happens when the driver/module fails? (That’s one of the reason why I like using multi vendor peripherals.)

      In this design, since we are using the half height UCS B-200 blade that supports (1) dual-port mezzanine adapter, if the adapter completely failed it would be akin to the ESX host failing and VMware HA would need to respond.

      ps: it’s VMware and not VMWare

      Thanks for finally clearing that up! I had always wondered which way it was supposed to be, or if anybody really cared.

      Thanks for the comments Duncan. Come back anytime. :-)

      Cheers,
      Brad

      • Giuseppe Citerna says:

        Dear Brad, thanks for your amazing articles ! I am a newbie in this field and ask you some questions :

        - The virtual-ethernet interface on Fabric Interconnect was configured in trunking (.1q) ?
        - Is it correct to say that in this design we don’t have the full manageability of the vmnic/vNIC (of the
        VM) but only the full control/policy of the vminc (uplinks) of the Palo Adapter ?
        - How it is possible to associate/bind vNIC (of the VM) to the vmnic ? with NIC fail-over in esxi ?

        thanks
        giuseppe

  2. Justin says:

    How would the setup differ with the Cisco 1000v?

  3. Mike Hancock says:

    Agree with Duncan here, I’m pretty certain VMware HA is oblivious to UCS FT. I would use nic teaming for SC HA HB redundancy to avoid the warning across top of the main panel of vCenter.

    “… and the cable that connects the virtual adapter to its virtual switch port is a VNTag.” This analogy works very well!

  4. Duncan says:

    Well for this setup you could easily combine vNIC3 and vNIC4 for both the VMkernel and the Service Console. This way you would avoid having to use the advanced setting to suppress the errors. (vNIC3 Active for SC and vNIC4 standby for SC, vNIC4 active for VMkernel and vNIC3 standby for VMkernel)

    I do like the fact that you are keeping it simple by creating just one vSwitch!

    • Brad Hedlund says:

      Well for this setup you could easily combine vNIC3 and vNIC4 for both the VMkernel and the Service Console

      Yes that would work perfect! Exactly what I was thinking this morning eating breakfast.

      Thanks Duncan and Mike for the great feedback.

      Cheers,
      Brad

      Here is a quick update of how this would look:
      Service Console redundancy

  5. Duncan says:

    Right on…. good stuff Brad!

  6. chris stand says:

    I do not think you can currently team/etherchannel two CNA adapters from one ESX box that are going to use FOCE – https://www.myciscocommunity.com/message/4386
    and so you would need to connect the SAN A to 10GB_adap#1 and SAN B to 10GB_adap#2 But since there is only one 10 GB adapter this does seem problematic – I don’t want to have to failover because someone pulled/bumped THE cable.
    I would be curious if this is an issue with iSCSI if there were multiple adapters & they were teamed ?

    Are you cross connecting the fabric extenders to 2 5010s (or 7Ks) at the same time? If you lose a 5010 the FEX that is connected to it fails as well. It seems that your drawing has FEXs connected only to one “Fabric Interconnect” box with multiple(?) cables.
    Will cross-connections be available when the 50×0 NX-OS vPC feature set is release ?

    • Brad Hedlund says:

      Chris,
      While yes it is true that establishing an Etherchannel on two ports with FCoE enabled is not currently supported, understand this is NOT a necessary configuration, with or without Nexus 1000V.
      Yes, if Nexus 1000V has two physical adapters it will put those NICs into what it calls a “Host Port-Channel” (Etherchannel), however this does NOT require the two NICs to be connected to an Etherchannel on the network side. Nexus 1000V will learn via CDP (or manually defined) that those two NICs are not connected to the same switch and will create a Port-Channel “sub-group” for each NIC (or set of NICs) connected to the same physical switch. Each VM will stick to one “sub-group” or the other. This is very similar to VPORT-ID based Load Sharing on the traditional VMware vSwitch.

      Read more about forming a Nexus 1000V Port-Channel to two separate physical switches here:
      http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus1000/sw/4_0/interface/configuration/guide/if_5portchannel.html#wp1235649

      Hope this helps,
      Brad

  7. Gnijs444 says:

    Nice design. Actually, this is the standard 4-NIC ESX design i would say. However, is the Cisco Virtualisation NIC already available at FCS ? Do you have also designs for a half height server with standard Menlo CNA ? So with only 2 LAN NICs and 2 SAN HBAs ? How would you seperate VMotion traffic from production traffic in that case ? I have read somewhere that you could assign the vmotion traffic to a different FCoE lane using QOS settings etc.. any news on that ?

    regards,
    Geert

    • Brad Hedlund says:

      Geert,
      The Cisco Virtualization Adapter shown here is planned for release this Fall, along with the full height blade.
      Yes, I am in fact working on a similar design diagram (with what little spare time I have) using the standard Menlo based CNA which only presents (2) LAN NICs and (2) FC HBAs to the operating system. Your hunch is correct that using QoS settings to guarantee available bandwidth for VMotion and Service Console makes a lot of sense and is what I intend to represent.

      Cheers,
      Brad

  8. Sil says:

    Hi Brad,
    Excellent article!! I have two questions:

    1) Considering the blue connection linking the mezzanine card to each Fabric Extender and to each Fabric Interconnect and carrying traffic for the 5 VNtags: Is each blue ribbon a single physical connection? Thx.

    2) I assume you could use all four 10GigE uplink ports on the Fabric Extender to connect to four 10GigE incoming ports on the Fabric Interconnect using a single Etherchannel grouping the 4 physical 10GigE paths, right?

    2) Do you need the Fabric Interconnect FC expansion adapter to connect the FC uplink to the SAN FC switch, or can you just use one of the 10GigE ports (FCoE?) as uplink from the Fabric Interconnect to the SAN FC switch? Thx.

    Thanks much.

    • Brad Hedlund says:

      Sil,

      1) Yes, the thick blue line is representing a single physical 10G cable/connection.

      2) The Fabric Extenders (also called I/O modules) inside a UCS blade enclosure have (4) 10GE uplinks each that connect to a Fabric Interconnect (also called UCS Manager). Theses links between the Fabric Extender and the Fabric Interconnect cannot be configured as a single Etherchannel group. Rather, each downlink to a server mezzanine port is pinned to a specific uplink on the Fabric Extender.

      3) Yes, currently you need the FC expansion module to attach UCS to an FC SAN. Using FCoE “northbound” to uplink UCS to an upstream SAN or directly attached FCoE storage array is not yet supported in the current software, but no doubt will be a possibility in future software releases. Baby steps…

      Hope this helps,
      Brad

  9. Pierre-Louis says:

    Hi Brad,

    I’m working on a client file today and he asks me a good question : Is there a Cisco FCoE switch that could be inserted in HP BladeSystem c-Class (as you would fine with Brocade switch and Cisco CBS30x0 today) ???

    Today I dont see anything on CCO about integration of UCS 2100 in other systems that UCS 5100, could you confirm that ? Do you know if there is an evolution about this solution ?

    Thanks for your help.
    Regards,

    Pierre-Louis

    • Brad Hedlund says:

      Pierre-Louis,
      The UCS 2100 I/O module will never work in an HP c-class, just like HP Virtual Connect will never work in any other system than HP c-class. These components are specifically designed to work in a specific system – “tightly coupled”.
      You *might* see a Cisco Nexus 10G/FCoE blade switch for HP c-class in the near future (don’t ask me when). A Nexus blade switch for HP c-class would not be managed by UCS, rather it would act as a standalone switch much like a Nexus 5000 – “loosely coupled”.

      Cheers,
      Brad

  10. Craig says:

    It looks similar to the infiniband Xsigo to run on VMware

    • Brad Hedlund says:

      Craig,
      The vNIC and vHBA capability is similar to Xsigo with the exception that this is all *Ethernet*, not (gasp!) Infinband.

      Some things to be aware of with Xsigo and Infiniband:

      • Infiniband is a niche technology, not a lot of expertise exists in the market. You have to dig hard to find the word “Infiniband” on Xsigo’s site, why is that?
      • The Infiniband market is shrinking fast. The leading IB switch maker Voltaire has recognized it needs to sell Ethernet to remain viable.
      • Only one source for Infiniband adapters — Mellanox. Compare that to the wide availability and choices in Ethernet adapters.
      • Infiniband with Xsigo is a THIRD fabric in the Data Center, not a UNIFIED fabric.
      • How is the FC traffic transported over the Infiniband link? Can you explain it? Can your storage vendor explain it? Compare that to the broad industry acceptance and certification of FCoE.
      • FC over the Infiniband link is a specialized protocol and therefore requires the Xsigo Director be a STATEFUL gateway for FC traffic. Any software hiccup on the Xsigo Director may disrupt the state information and the server could loose all its LUNs. Compare that to the STATELESS transition from FCoE to FC.

      … I could go longer but you get the idea :-)

      Cheers,
      Brad

  11. Karan Bhagat says:

    Brad,

    Your blogs are invaluable. I’m in San Jose at the Cisco UCS Boot Camp and your blogs posts are helping me understand things much better. Thanks for all your time and effort in putting these posts together for everyone

    Karan

  12. Anirban Chakraborty says:

    Hi,

    Do you know if the Palo adapter’s 128 functions are already supported in vSphere 4.0 or will be supported in upcoming U1 update? Also, would the functions show up as 128 physical functions or as virtual functions? I believe vSphere does not have SR-IOV support yet.

    Thanks,
    Anirban.

    • Brad Hedlund says:

      Anirban,

      The Palo adapter itself will be supported in vSphere 4.0 update 1. The “functions” would show up as 128 physical adapters (if you configured that many). SR-IOV runs at the PCIe system level, and as such the OS running on the system requires no knowledge or awareness of SR-IOV.

  13. We had presentation from HP and Cisco last week for our Blade requirement. After their presentation I had question which one I should go with? Yeah I am new to Blade but quite old for VMware technologies.
    What I have understood is
    “Cisco UCS definitely has an edge compare to HP . It has the edge because they are so tightly integrated with VMware in terms of delivering solution for virtualization and example is Nexus 1000 vswitch . Only drawback which I see that they are new to the field and we need to have know their success story”

    “Where as HP has been in the market for a while and they had been delivering solution which is used by large scale customer. But we have to look at the our requirement. Our requirement is not that huge that we need HP solution. If I see the compare sheet for blade from Cisco VS HP I guess my requirement can be addressed by Cisco”

    Very confused customer I am but having said above I am not trying to prove that which one is better. After reading your blog I started having feeling that UCS will meet my requirement and I really don’t have to do heavy investment on HP solution. Other funniest thing which I came to know during those presentation is “Architect leading the efforts for Cisco was a key player at HP”.

    Nice One Brad keep it up.

  14. Nav says:

    How come the Vnic of the Palo card vNics have active uplinks that is non-dotted lines to both the I/O Module even no special pinning has been utilized for it.
    According to my understanding the dotted lines from the all the Palo vNics to one I/O Module .

  15. Michael says:

    This design is 4 years old. Would you do some things differently now?

Leave a Reply

Your email address will not be published. Required fields are marked *