Data Center Networking Q&A #1 – starring HP, Nexus 1000V, QoS

I thought it would be fun to pilot a series of posts where I pick out interesting search engine queries that were used to find my blog.  Often times these are good questions that deserve a good answer, or other interesting topics that can start a good discussion or fun debate.

This particular series will focus on Data Center Networking.  Another data center computing focused series may be started such as “Cisco UCS Q&A”. Stay tuned.

Each question or search query will be headlined in bold and the text beneath will be my answer, commentary, or general response.

Furthermore, if you have a question you think I might be able to answer please submit your question in the comments section to be considered for immediate answer or highlighted in a future Q&A post.

So here we go, this is the first Data Center Networking Q&A :) I hope you enjoy it!

hp qos marking configuration

I seem to be getting quite a number of these queries finding my site lately.  This person might be trying to find out how to configure their HP blade switch to classify and mark traffic for a QoS policy in their data center.  Well, there are two blade switches made by HP worth discussing: HP Virtual Connect Flex-10, and the HP Procurve 6120XG.

Lets start with HP Virtual Connect Flex-10.  I’ve got some real bad news for you on this one.  Flex-10 has no QoS capabilities what so ever.  If you follow my blog and tweets you have heard me point this out several times and I’ll do it here again.  Flex-10 is not capable of classifying traffic, not capable of marking traffic, and not capable of giving any special treatment or guaranteed bandwidth to important traffic.  If you search for “QoS” in the latest Virtual Connect User Guide it produces zero hits.  But don’t take my word for it, here is what HP says about Virtual Connect Flex-10 QoS:

VC does not currently support any user configurable Quality of Service features … these features are on the roadmap for future implementation.

Moving on to HP Procurve 6120XG; this blade switch made by HP actually has what  I will describe as “so-so” QoS capabilities, much better than Flex-10 anyway.  The Procurve 6120XG can give special treatment to traffic via (4) traffic queues, each with a differing degree of priority; Low, Normal, Medium, and High.  The High priority queue will always be serviced before the Medium queue, and so on.  This is a simple implementation of a QoS technique called Priority Queueing.  The downside to Priority Queueing is that the high priority queue (if busy) can starve bandwidth from all other queues, with no insurance or fairness that all traffic will get some portion of the bandwidth.  The 6120XG QoS implementation does not provide guaranteed bandwidth, it simply allows some packets to be transmitted before others, based on the queue they are serviced from.

The Procurve 6120XG can assign traffic to each of the (4) queues based on the incoming 802.1P COS value in the Ethernet header, or the IP DSCP or TOS value in the IP header.  One important thing to note here is that the HP Procure 6120XP expects the packet to already be marked when entering the switch.  The Procurve 6120XG is not capable of marking traffic based on MAC, IP, or TCP information.  If a packet enters the 6120XG with no marking, the packet cannot be classified and not provided any special treatment, no QoS.  The 6120XG can mark traffic based on incoming port, meaning all traffic from a certain port can be given a defined marking.  However, this rudimentary port-based classification has little use in a 10G server environment where different types of traffic will be converged on a single 10G server interface.

Given that the HP Procurve 6120XG’s QoS capabilities are only useful when the traffic has already been marked before entering the switch, it’s important to understand if your blade servers are capable of traffic classification and packet marking.  This becomes especially relevant with server virtualization hosts, such as with VMware.  The VMware vStandard Switch (VSS) or vNetwork Distributed Switch (VDS) does not have traffic classification or marking capabilities, however the optional Cisco Nexus 1000V has a comprehensive set of QoS classification and marking capabilities.  Hence, the Cisco Nexus 1000V can classify and mark important traffic leaving the VMware host (such as vMotion or management traffic) before entering the Procurve 6120XP where it can then be placed into one of the (4) QoS queues based on the marking it receives.

The downside to QoS on the HP Procurve 6120XG is that its basic Priority Queueing implementation does not provide any bandwidth guarantees to all traffic types, and the lack of traffic classification capabilities requires that your servers do the classification and marking, hence the need for Cisco Nexus 1000V on the vSphere Host.  This is why I give it a “so-so” rating.

You can find the complete QoS configuration details here in the HP Procurve 6120XP Traffic Management Guide (PDF)

For HP blade servers, my recommendation is to not use the HP Procurve 6120XG, and instead use the HP 10G Passthrough module.  This allows you to connect your HP blade servers directly to a Cisco Nexus switching environment, bypassing all of the other mediocre 10G blade switch options from HP.  The Cisco Nexus series has a rich set of QoS capabilities that do not just provide basic Priority Queuing, but rather offer an advanced Class Based Weighted Fair Queueing mechanism that can assign minimum guaranteed bandwidth to all traffic types.  This behaves in a manner very similar to how VMware provides reservations for CPU and memory to a virtual machine, but without limitations.  If more CPU and memory is available, the virtual machine can use it, but there is always a minimum guarantee provided by the reservation.  This is similar to how Cisco Nexus switches allocate bandwidth, minimum guarantees without limitations.

The Cisco Nexus switches are also capable of classifying and marking traffic based on MAC, IP, or TCP information.  So if a packet arrives unmarked you can mark it at the switch and provide granular QoS, with or without the Nexus 1000V.  You can later add the Nexus 1000V for all of the security, management, and network visibility reasons, but you can at least get started with rich QoS capabilities with or without the Nexus 1000V.

hp blades cisco nexus 1000v

Great news! You can absolutely run the Cisco Nexus 1000V on an HP blade server.  Furthermore, this can be done with any blade switch.  You can use Virtual Connect Flex-10, the Procurve 6120XG, the 10G Passthrough module, or any other standard Ethernet switch.

There is one particular note about using HP Virtual Connect Flex-10 with Nexus 1000V that I want to discuss.  The Flex-10 module can be setup in two different switching modes, mapped mode, or tunnel mode.  The most common mode is mapped mode because that is the default mode.  In mapped mode the Flex-10 administrator defines vNets, which are basically VLANs inside the Virtual Connect domain.  The Virtual Connect administrator can decide the VLAN ID used for these vNets independently of the network administrator who might be configuring the physical network switches and the Nexus 1000V.  The Flex-10 uplinks to the network with VLANs defined by the network administrator, however the VLAN IDs on the network uplink are mapped to a VLAN ID of a vNet, which might be a different VLAN ID, or the same.

If the Virtual Connect administrator is using mapped mode and mapping the VLAN ID on the network to a different VLAN ID inside the Virtual Connect domain, this can cause a problem with the Nexus 1000V.  If the Nexus 1000V VSM was configured with the VLAN ID’s known on the physical network, the Nexus 1000V VEM running on the server is expecting to see the same VLAN ID’s that were defined on the VSM.  However if the Flex-10 mapped mode configuration is changing (mapping) the VLAN ID’s to something different, you will have broken the linkage between Nexus 1000V VSM and VEM.

Tunnel mode, on the other hand, simply takes all VLAN ID’s defined on the network uplinks and sends them down to the server ports, without changing or mapping the VLAN ID’s to anything different.

The moral of the story here is that if you are going to use Nexus 1000V with Virtual Connect Flex-10, you can, just make sure you are not changing VLAN ID numbers.  If using mapped mode, do not map the network VLAN ID’s to a different VLAN ID at the server, keep them the same.  Or, you can also use tunnel mode which does not provide any option of changing VLAN ID numbers.

One you have insured VLAN ID consistency inside of Virtual Connect you are all set to have a very successful implementation of Nexus 1000V on HP blades, even with Virtual Connect Flex-10.

That’s all for now.

Hope you enjoyed the pilot episode of Data Center Networking Q&A #1

Please remember to submit questions or comments in the comment section below.  Some questions may be answered here or featured in a future installment.


  1. says

    Brad, VLAN mapping in Virtual Connect does not change the VLAN tags. In mapped mode Virtual Connect lets you define (map) which VLAN or VLANs are assigned to a specific NIC. In tunnel mode, the whole VLAN trunk is passed through to the NIC, and no VLAN processing happens in Virtual Connect.


    • says


      Either you or your documentation team need to spend some more time understanding your own product a little better. Here is a direct quote from Page 75 of the latest Virtual Connect 2.30 User Guide:

      If Virtual Connect is set to map VLAN tags, VC-Enet modules accept incoming frames with valid server VLAN tags and translate server-assigned VLANs into corresponding internal network VLANs, thus placing the server packet on the correct network. When these frames reach the uplinks, network VLANs are once again translated into external data center VLANs. Similarly, VLANs are translated back to server-assigned VLANs (or stripped and untagged) before sending frames out to servers. This function allows for the possibility of configurations where server VLANs might not match external VLANs used on uplinks.

      Am I reading that wrong?


  2. says

    Thanks for reading the documentation, and pointing out that section. It obviously needs some clarification. Just to be clear, Virtual Connect DOES NOT alter the VLAN ID. The server NICs will always see the same VLAN ID as it exists on the upstream switch. There is no mechanism within Virtual Connect to change the VLAN ID.

  3. says

    Brad, I just realized there is the option to change the VLAN tags on a mapped VLAN. I’ve never used it, nor have I ever seen it used. I’m not sure why anyone would use it for the reason you mentioned above.

    Ken Henault

  4. says

    Brad, there is something about the Nexus 1000v deployment that I dont understand.

    The industry roadmap with regard to virtual server networking is clearly to bypass the ESX server’s hypervisor and offload all the switching functions from the software switch to the adjacent physical bridge, as in VN-link in hardware and VEPA.

    If this is the case, I really don’t understand why an organization will invest tens or hundreds of thousands of dollars on the N1Kv?

    • says

      The Nexus 1000V provides management visibility and security policies that the alternative hypervisor switch (vSwitch) does not provide. While networking options that bypass the hypervisor are certainly interesting, it doesn’t necessarily mean that everybody will ultimately use those technologies. Most implementations today are based on hypervisor switches, and will continue to be for some time. That is why customers are investing in the Nexus 1000V.

      VMware themselves will be quick to disagree with you and make the case that the hypervisor software switch model provides the most flexibility, feature velocity, scalability, and more intelligent automated provisioning for the cloud. This was the case VMware made in some of the sessions at VMworld 2010. While VMware has a vested interest in the software switch model, some of that argument is pure marketing and turf defending, however they do make some valid points that I tend to agree with.

      The hypervisor software switch will always be able to bring features to market faster than hardware based solutions. Example: Netflow, DHCP Snooping, ERSPAN, etc, already present in Nexus 1000V.

      The hypervisor software switch does not have the same hard scalability limits (VM’s per host) as compared to hardware based solutions (VM-FEX, VEPA, etc.). Example with Cisco VM-FEX you can have around 50 VMs per host, versus 248 VM’s per host with Nexus 1000V.

      If the hypervisor has all the visibility into the information about VMs, including a VMs relationship to other VMs (vApp), can it perhaps make a more intelligent decision about network services and policies that should be provisioned? I think so. How much this really matters remains to be seen.


  5. says

    Brad, I agree that the 1000v offers value as opposed to the VMware vDS. No doubt.

    I’m not sure I understand your stance, though, with regard to distributed forwarding intelligence at the chassis level. When I made the comment before that I wasn’t too crazy about the fact that the UCS does not have any forwarding intelligence in the chassis, and that it has to forward traffic to the 6100s just to go from server blade to server blade in the same chassis, you opined that its a better approach to have predictable traffic flows.


    “By the way, that brings up another point I’m not too crazy about: the fact that the UCS chassis does not provide distributed intelligence, which forces intra-chassis traffic to get forwarded to the FEX, then up to the FI, and then right back down again. ”


    “Localizing intra-chassis traffic might be important if your data center is nothing more than (1) chassis, and there isn’t much bandwidth between your chassis and the network — but that’s simply not the case anymore. Once you add a second chassis you now have inter-chassis traffic that has a difference performance profile than the intra-chassis traffic.

    That’s not the case with UCS. Whether you have (1) chassis, (2) chassis, or (20) chassis, all traffic patterns are consistent and predictable. This makes it easier to keep a consistent SLA in a virtual data center with VM’s moving from one chassis to the next. With UCS, the location of the workload is of little significance.”

    So, I am trying to understand your philosophy (read; Cisco’s philosophy) when it comes to distributed forwarding intelligence. Is it better to forward all traffic to the network and have predictable flows and consistent SLAs, as you put it before? Or is it OK to have a hybrid model in which some flows will remain in the chassis and others will be forwarded to the LAN, as you are saying now?


    • says

      My comments above about Nexus 1000V had nothing to with “distributed forwarding intelligence” or “flows will remain in the chassis”.
      The fact that Nexus 1000V may keep some traffic local to a server is largely incidental. I never cited that as reason for Nexus 1000V or hypervisor software switches in general.
      As I said before, worrying about keeping traffic local to a chassis (or local to a server) is largely a pointless and futile exercise in a virtual data center. I think I’ve been pretty consistent about that “philosophy”.


Leave a Reply

Your email address will not be published. Required fields are marked *