HP Flex-10 versus Nexus 5000 & Nexus 1000V with 10GE passthrough

Filed in FCoE, Nexus, QoS by on February 9, 2010 40 Comments

I had an interesting discussion with a customer the other day where both Cisco (myself included) and HP account teams where on the same call to discuss Flex-10, Nexus 1000V, or other approaches that may work better. — Yeah, awkward.

Anyway, for most of the time we (the Cisco team) focused on Flex-10′s total lack of QoS capabilities. Flex-10 requires the customer to carve up bandwidth on the Flex-NICs exposed to the vSphere ESX kernel.

For example (1) FlexNIC set at 2Gbps for Service Console, another FlexNIC set at 2Gbps for vMotion, and a 3rd FlexNIC set at 6Gbps for VM data.  Even if the 10GE link is completely idle a FlexNIC doesn’t get anymore bandwidth than what’s been manually assigned to it.

The question then becomes … Why not just feed the server a 10G interface and let vMotion or VM’s grab as much bandwidth as is available? — with minimum guarantees provided by QoS.
The Nexus 1000V can apply the QoS markings and the Nexus 5000 can act upon it. However, with Flex-10 as a man-in-the-middle the QoS model breaks.
What value does the FlexNIC bandwidth assignments have with a 10G attached server, other than being a workaround for a lack of QoS intelligence in Flex-10?

Moving on to operations, with Nexus 1000V, the server team is handing over the VM networking configuration and responsibility to the network team. The Nexus 5000 is obviously managed by the network team as well …. so … who manages Flex-10? Server team or Network team?
If the Server team manages it, there is no continuity in troubleshooting from Nexus 5000 to Nexus 1000 because you have this other object in the path (Flex-10) managed by a separate team — everybody seems to agree that’s not an ideal situation.
If the Network team manages Flex-10, where is the CLI? where’s the QoS? how do you upgrade the code? do you need to reboot servers or interact with the server team for every config change? — nobody is quite sure.

The HP people on the call kept interjecting … “but, but, but … with Flex-10 you can move or replace the server without reconfiguring MAC/WWNs” That was all they could really say.
The customer didn’t seem to give this argument much weight because plenty of fail over capacity would be built into the VMware environment. If a server failed, VM’s will be immediately started on another host, and how fast you can reconfigure a physical server becomes less important.
It would take (2) or more servers failing at the exact same time (highly unlikely) for the speed of physical server provisioning becoming the bottle neck to restoring capacity to the vSphere environment.

It became quite clear towards the end of the meeting that 10GE pass-through to Nexus 5000 was a solid approach without much downside … and VMware vSphere + Nexus 1000V had a lot to do with the customer coming to that conclusion.
The nice migration path to FCoE with 10GE pass-through to Nexus 5000 is also a huge plus. The HP 10GE pass-through module does support FCoE today. So, if HP is “going to announce FCoE soon”, then why buy a Flex-10 switch today that doesnt support FCoE? Why not buy a pass-through device that supports FCoE now, and spend less money doing so?

More information on the HP 10GE passthrough module can be found here on HP’s website:
http://h18004.www1.hp.com/products/blades/components/ethernet/10gbe-p/

###

About the Author ()

Brad Hedlund (CCIE Emeritus #5530) is an Engineering Architect in the CTO office of VMware’s Networking and Security Business Unit (NSBU). Brad’s background in data center networking begins in the mid-1990s with a variety of experience in roles such as IT customer, value added reseller, and vendor, including Cisco and Dell. Brad also writes at the VMware corporate networking virtualization blog at blogs.vmware.com/networkvirtualization

Comments (40)

Trackback URL | Comments RSS Feed

Sites That Link to this Post

  1. uberVU - social comments | March 2, 2010
  1. Network Engineer says:

    Brad, few points of note here:

    (1) You are assuming that the network team deploys & manages Nexus-1000 soft switches. N1K switches are clearly not a requirement in ESX virtualized environment. They are more of a nice-to-have for those who for some reason need to do more than ESX “native” soft switches can offer.

    In our organization we decided to draw the line at the N5K aggregation layer, and let our Server team manage everything below that (which includes Flex-10 devices and all ESX software components). Flex-10 is no longer “in the middle”, because there is no Cisco piece at the bottom.

    (2) I can argue that FlexNIC bandwidth allocation method is superior to QoS. QoS is much more complex to provision and manage on large scale. And N5K doesn’t exactly offer good QoS capabilities in its current form (I can’t speak for N1K as we haven’t tried it). FlexNIC bandwidth carving is bullet-proof and it works with minimal provisioning complexity.

    (3) Your point about Flex-10 lack of network-friendly management capabilities is absolutely valid. I don’t think it was developed as a true “networking” device, but rather as a “server NIC on steroids”.

    • Brad Hedlund says:

      Hi “Network Engineer”,

      Thanks for the comments. Allow me to respond to a couple of things you said:

      N5K doesn’t exactly offer good QoS capabilities in its current form

      I couldn’t disagree with you more on this. The Nexus 5000 QoS capabilities stand out among all other 10GE L2 switches. Nexus 5000 can classify traffic based on COS, IP, UDP/TCP ports, IP Precedence, DSCP, and Protocol Type. From there you can classify traffic into 8 different service levels called “QoS Groups”. Each QoS group can then be given a % of guaranteed bandwidth, strict priority, and no-drop service if you want. Each port has dedicated 480KB buffers for queuing — compare that to Flex-10′s 2MB buffer shared across all ports.

      I can argue that FlexNIC bandwidth allocation method is superior to QoS … FlexNIC bandwidth carving is bullet-proof and it works with minimal provisioning complexity.

      So your argument is that carving bandwidth at the FlexNIC is superior because its just ‘easier’ — and because of that you are willing to limit vMotion bandwidth to 2Gbps (for example), versus having access to all 10GE? I’m sorry, but that sounds lazy! Especially when you find out how easy it is to enable QoS on the Nexus 1000V and Nexus 5000. Minimal provisioning complexity? I don’t know about you, but configuring just (2) 10GE ports per server sounds a lot easier than configuring (8) FlexNIC’s per server, maybe that’s just me though.

      Here’s how easy the configuration can be for QoS on Nexus 1000V and Nexus 5000. This is a simple example of guaranteeing 10% bandwidth to vMotion, with no rate limiting, all 10G is accessible.

      Easy Nexus 1000V QoS Configuration:

      policy-map type qos Mark-VMotion
      class class-default
      set cos 4

      port-profile VMotion
      service-policy type qos input Mark-VMotion

      With that config we have marked the vMotion traffic with CoS 4. That’s all you need to do on the Nexus 1000V. Now lets go the Nexus 5000 and have it act upon the classification made by the Nexus 1000V…

      Easy Nexus 5000 QoS Configuration:

      class-map type qos VMotion
      match cos 4

      policy-map type qos Classify-VMware-Traffic
      class VMotion
      set qos-group 2

      class-map type queuing VMotion
      match qos-group 2

      policy-map type queuing VMware-Bandwidth-Settings
      class type queuing VMotion
      bandwidth percent 10
      class type queuing class-default
      bandwidth percent 40

      Note: FCoE already has a 50% bandwidth setting on by default

      system qos
      service-policy type qos input Classify-VMware-Traffic
      service-policy type queuing input VMware-Bandwidth
      service-policy type queuing output VMware-Bandwidth

      Thats it! You’re done! And you only need to configure that once. Every ESX Host connected to the Nexus 1000v and Nexus 5000 will have the benefit of this intelligent QoS. With this simple configuration, your VMware vSphere ESX servers attached to Nexus 5000 can use as much of the 10GE bandwidth available for vMotion, with a guaranteed minimum of 10% (1Gbps). Now that’s efficiency.

      On the other hand, with Flex-10, since it does not have this intelligent QoS capability, you will need to configure a FlexNIC for vMotion and give it a very low bandwidth setting, such as 1 or 2Gbps, and it will never be able to go higher than that, even if the bandwidth is available. What a waste!

      Cheers,
      Brad

  2. Etherealmind says:

    The QoS debate is important to networking people, but in my experience, Server people have no concept of oversubscription, QoS, or ordered allocation. To whit, the debate on allowing VMware to oversubscribe memory even with resource controls. Most Server Admin cannot conceive of flexible resource allocation.

    Until that happens, I suspect that HP is selling “what the customer wants” not “what the customer needs”.

    The largest concern with HP Flex-10 is the operational risk to the network. The HP sales pitch that Flex-10 saves time on network provisioning is seductive but false. I wonder how many networks will suffer overloading outages, or security breaches, or VLAN exhaustion as the Server Admins treat the network as a static but infinite pool of resources. That is, “just keep configuring until it breaks” deployment plan.

    Adding servers without consideration for backbones, or dynamic load movement, or security boundaries is serious risk that is currently being ignored.

    Ah, well. Once again, Networks will pick up the pieces later while the Server folks just keep doing what they always did.

    Etherealmind

  3. quaich says:

    @Etherealmind: i totally agree with you in all points…….i often have the feeling that the network gets loaded and loaded with more intelligence but on the other side the upper layer intelligence and folks that are administering those upper layers stay on the same step or even worse……

    quaich

  4. net@work says:

    I totally agree with you in all points.
    We have N5k in place in our company, but we have no chance to use FCoE, because the server guys give us no chance. Even worse we swap all our FC switches this month without having a look to FCoE. Off cource FCoE is not the solution for all systems, but …
    The big problem is, that server administrators gather more and more network tasks, but they do not realy understand what they are doing. – In some years we will not see any netwokers within the datacenter. For me it seems that server people will win this game. Networkers have no chance to argument, because they use a language which is not spoken in the world of server guys. And security,availability and relaiability – who cares about this?

  5. serverguy says:

    You know what’s really depressing?

    I am a server guy (Unix & more recently VMware) and yet I find I can’t argue with the other comments regarding server guys.

    Obviously I exclude myself :)

    Too often though I work with server people with little clue of how to run on a contended/shared infrastructure let alone how to manage one themselves. The concepts of risk & impact analysis, capacity management, availability, failure domains, etc. seem meaningless and incomprehensible to the major of server people these days. Particularly those who’s experience has been centered on Linux or Windows.

    What depresses me most though, and it’s a cliche, is when network and server (and even storage) people are unable to work together or even just talk the same language.

  6. thehevy says:

    Great discussion and posts.

    We have been looking at network bandwidth and QoS on 10G Ethernet in our labs. While we can use benchmarking and load generation tools to get 9.5+G of throughput on a 10G port, the question that we keep asking ourselves is, will we see this in the real world?

    I am sitting in a VMware vSphere Fast Track course with 20+ Admins that still run their network connections on 1G. They are running less then 10 VMs per host with 8 1G ports of connectivity. With that few of VMs and since each 1G port is limited to 1G of traffic the likelihood of a host actually being able to max out a pair of 10G ports seems pretty low.

    So in most situation why would we want to spend the additional time and overhead of setting up QoS policies. I would say setup monitoring of the network and watch for bandwidth issues and only setup QoS if needed.

    I put more of these thoughts in a white paper…
    http://download.intel.com/support/network/sb/10gbe_vsphere_wp_final.pdf

    With all that said, I believe in two 10G ports that are wide open for all traffic types to share the bandwidth, UNLESS, you have a real need to manage your bandwidth of a specific traffic type. Having an Admin manually carve a 10G in to 4 segments that don’t share unused bandwidth doesn’t seem to match the new paradigm of dynamically changing virtualized data centers.

    I look forward to reading about the areas / use cases that QoS is really needed and is more efficient then just adding another 10G port to the host. The reality is 10G is not that expensive especially when compared to paying for the additional operational expenses to implement and manage QoS on every host in a data center.

  7. Sri says:

    Brad,
    I totally agree with you that VC and Flex-10 has no value proposition. I also agree with thehevy that 10Gig is more that what you need today, and no point in managing traffic when there is no traffic congestion. So, in fact no need for carving or QoS as well. In virtualized environments 8 1Gig NICs at a max are handle today’s needs, so having 2 10Gig pipes from one server means 2+ times more bandwidth will be available. Always you could add few more 10G ports.

    All this can be done with VMware and their native vSwitches. In addition, VMware supports traffic shaping with rate limiting feature, just in case you still want to limit bandwidth per traffic/port group..So I see no need for customer to spend money on Nexus 1000V soft switch. No value proposition, similar to Flex-10.

    Again, I agree that pass thru is the best option today, leaving path to FCoE support which comes with industry standards DCB based Converged Enhanced Ethernet, later part of 2010. So I would say customer should wait till on buying any convergence gear till the standard based products hit the market in by end of 2010. So that in fact means wait till Nexus 5K and UCS 6100 FI all these move to DCB standards based CEE..These products need to move on from Cisco proprietary DCE…and that will happen later in the years along with rest of the industry.

    There is no point in rushing in to all these products that are based on pre-standard technologies….

    Don’t you agree? If not, they may end up with expensive paper weights….
    Cheers
    Sri

  8. Juan Lage says:

    Leaving QoS aside, what about scaling the solution and network management? A solution with ESX and Flex-10 effectively has three network layers all of which are managed differently in interfaces, tools and even nomenclature. At the ESX layer, you have to manage the vSwitch, at the server chassis level you need to manage Flex-10, and then you need to manage your network layer. If you have workloads that need to exists beyond a single server chassis, this means for VM networking policy you need to touch at least those three components for (perhaps) every change.

    Say for instance you need to have a group of VMs on a PVLAN and they need to live in blades which expand multiple chassis (say more than 4, which is the limit on VC stacking today). Leave aside the fact that PVLAN may or may not be supported in all three mentioned layers, and if they are supported, will certainly work in different ways. Still, you need to configure the network on all three layers, think about it. If you are on vSphere Ent, no distributed switches, means also you need to touch every vSwitch on every host, plus every Flex-10 module, plus network. Let’s add vlan limitations in Flex-10 to make it more appealing … In large scale environments it can become a nightmare. In small scale, a burden.

    If you go for a solution like the one outlined above by Brad (N1K + Passthru + N5K) you can etherchannel every host to the N5K with a trunk. PVLAN required? It works the same everywhere (NX-OS) and you configure on a single point for up to 256 host: the N1Kv VSM. A lot simpler. Simple = reliable. Not to mention the savings in Ops costs.

    My 0.2 cents.

    Juan

  9. Sri says:

    Brad,
    My regards and compliments to you for all this great wonderful work you do with great technical details and creative blogging. You definitely are a good technology blogger’s role model and inspiration…I wish you the very best :).

    Keep up the good work!
    Cheers
    Sri

  10. Brad-

    Morning! Couldn’t you do something similar using 1 distributive virtual switch and using traffic shaping. I was under the impression that in utilization of Dvs that you can do both egress and ingress traffic shaping.

    I appreciate your comments.

    Disclaimer: Not a network guy

  11. thehevy says:

    The traffic shaping in vSphere works but it only provides shaping of traffic between the VM and a port in a port group. You can limit the amount of traffic a specific port can tx or rx but you have to keep in mind that the each port on the port group gets the same traffic shaping. That means that if you have shaping limited to 1G per port and 20 VMs end up on the same host that use that port group you can have up to 20Gs of traffic on that host.

    Traffic shaping based on a port has limited use. Traffic shaping based on a traffic type in conjunction with guaranteed minimum bandwidth allocations is where we need to get so all traffic types can take advantage of a large pipe. This will allow traffic to burst when no one else is using it while allowing higher priority traffic to get bandwidth when it needs it in a congested network.

    I can see using a port group based traffic shaping model for a backup connection in VMs or in a DMZ to match other connections are slower.

    One quick way to see your traffic is look at esxtop or resxtop’s networking screen during vMotion and under high traffic. I don’t think you will see any where near the 18Gb+ bandwidth that dual 10GbE ports are capable of…

    I still say, keep the network model as simple as possible, apply good monitoring and management tools and only implement traffic shaping and QoS if needed.

  12. Dan Robinson says:

    So I have a few questions about what seem to be holes in this theory that Nexus solves all problems.

    1) If you dont have N1K -OR- dont have N5K, does QoS still work? Your point about the HP Approach being lazy seems to me that its just meant to be more flexible and work in numerous environments, including those not under the Cisco thumb.

    2) The nexus series doesnt properly support FCoE today, because the CEE standard is not entirely ratified yet. So to say that FCoE works on the N5K when you will have to turn around the REPLACE some of the hardware down the line seems a little premature. You might argue that Cisco will push the standard their way, but its not a guarantee. And for those who say it will only be a software upgrade, ask your Cisco rep to put that in writing.

    3) Your argument about Pass Through 10GB confuses me.
    3a) The HP Proposition is that Flex-10 is natively supported by the LOM on the G6/G7 blades. So put pass through there (Interconnect 1 and 2) and you get 10GB Ethernet. how do you upgrade this to FCoE? Oh that’s right you have to buy a whole new server or put a CNA in a mezz slot and then move your 10GB Pass Through down to match. This might seem like a knock on HP, but see my point above that FCoE isnt ratified yet.
    3b) 10GB Ports on the N5K are free then? because your asking people to use 32 ports per chassis on your N5K. Using Flex-10 (or any 10GB Switch) would allow you to agregate your port usage a little. As mentioned before, you dont really need 10Gbx2 per server, so not having a 1:1 server:uplink ratio doesnt really matter. And for those who would argue this, does that mean your N5K has 30+ 10GB uplinks to your core? I didnt think so.

    Lastly, keep in mind as per #1 above that not all Cisco customers have Nexus across the board. Most still have CAT6K and cannot move to an entirely Nexus solution.
    Using Flex-10, at the last place I worked, we were able to have 4 chassis tied together with a 30Gbps private network for vMotion/HA/FT and then only using 4 x 10Gb uplinks to Cat6K Distribution layer for Management and VMs.

    • Brad Hedlund says:

      Hmm. Another HP employee posing as a customer?

      If you dont have N1K -OR- dont have N5K, does QoS still work?

      Sure it will, if you have the vSwitch connected to a physical switch that has QoS capabilities, which isn’t Flex-10 that’s for sure.

      The nexus series doesnt properly support FCoE today…

      Really? Tell that to EMC, NetApp, and oh by the way, HP, all who have certified, support, and resell the Nexus 5000 for FCoE deployments.

      …because the CEE standard is not entirely ratified yet

      CEE isn’t a standard. CEE (also known as DCB) is a collection of standards designed to enhance the data center network. Not all standards in CEE/DCB are required for FCoE.
      The standards that are required for FCoE to work (IEEE 802.1Qbb, IEEE 802.1Qaz, IEEE 802.1Qab) are all silicon ready and have been for some time. The other standards in CEE/DCB that are not as final such as IEEE 802.1Qau and TRILL are not at all required for FCoE.

      So to say that FCoE works on the N5K when you will have to turn around the REPLACE some of the hardware down the line seems a little premature.

      Actually, No! There will be no need for a customer to “REPLACE” the Nexus 5000 for any reason other than normal life cycling. FCoE works on the Nexus 5000 today. Funny you should use the word “REPLACE”, because that’s exactly what the HP Flex-10 customer will need to do in order to implement FCoE or what HP calls the “Converged Infrastructure”.

      And for those who say it will only be a software upgrade, ask your Cisco rep to put that in writing.

      Again, the Nexus 5000 doesn’t *need* any hardware upgrade. Can you say the same for Flex-10?

      This might seem like a knock on HP, but see my point above that FCoE isnt ratified yet.

      Uh, Wrong! You might want to get more informed about the status of FCoE. Take a look at http://www.fcoe.com and you will see that FCoE has been a standard since June 2009.

      10GB Ports on the N5K are free then? because your asking people to use 32 ports per chassis on your N5K. Using Flex-10 (or any 10GB Switch) would allow you to agregate your port usage a little.

      10GB ports on the Flex-10 are free then? Of course not. This is a ridiculous argument. You need to pay for the 10BG port a server connects to whether its on the Flex-10 or the Nexus 5000.

      As mentioned before, you dont really need 10Gbx2 per server

      That’s a bit presumptuous of you to say that.

      so not having a 1:1 server:uplink ratio doesnt really matter. And for those who would argue this, does that mean your N5K has 30+ 10GB uplinks to your core? I didnt think so.

      No. Customers are not asking for a 1:1 ratio of server to core. There will be oversubscription at the Nexus 5000 just like there is oversubscription at the Flex-10.

      not all Cisco customers have Nexus across the board. Most still have CAT6K and cannot move to an entirely Nexus solution.

      Yeah, and the good news is that HP Passthrough Module supports both 1G and 10G, which can connect to existing Catalyst switches that support 1G or 10G. In most cases when there is a project to enable 10G servers, it also coincides with a project to add 10G networking, and thats a perfect time to look at Nexus. And Nexus integrates nicely with existing Catalyst 6500 switches.

      Using Flex-10, at the last place I worked, we were able to have 4 chassis tied together with a 30Gbps private network for vMotion/HA/FT and then only using 4 x 10Gb uplinks to Cat6K Distribution layer for Management and VMs.

      Sweet. However its too bad that network has no FCoE readiness, no QoS, no In Service Software Upgrades (ISSU), no vPC, no network security, no network visibility, rate limited vMotion bandwidth, etc. All the things you can have with the HP Passthrough 10G module connected to Nexus.

      For example, using that same scenario, now you can connect those 4 chassis to (4) Nexus 2232 10G Fabric Extenders with the HP 10G Passthrough module. Connect the Nexus 2232′s to a pair of Nexus 5000′s, and connect the Nexus 5000′s with 4 x 10G uplinks to the Catalyst 6500.

      This design would provide you with the aforementioned FCoE readiness, QoS, full 10G bandwidth available to vMotion, In Service Software Upgrades (ISSU), 20G active/active bandwidth to each server with virtual Port Channels (vPC), proper network security features, and full traffic visibility. All of that even without the Nexus 1000V, this is just what the N5K/N2K brings. With the Nexus 1000V you have a comprehensive and consistent network operations model from the VM all the way through to the core of the network.

      Cheers,
      Brad

    • “No. Customers are not asking for a 1:1 ratio of server to core. There will be oversubscription at the Nexus 5000 just like there is oversubscription at the Flex-10.”
      Customers are asking for 8 uplinks on an hp chassis connecting to 4 different switches in the Data Center fabric. We are asking for very limited over subscription to the core, have a second switch between the cores and the blade chassis is just redundant.

  13. Sam says:

    “For example, using that same scenario, now you can connect those 4 chassis to (4) Nexus 2232 10G Fabric Extenders with the HP 10G Passthrough module. Connect the Nexus 2232’s to a pair of Nexus 5000’s, and connect the Nexus 5000’s with 4 x 10G uplinks to the Catalyst 6500.

    This design would provide you with the aforementioned FCoE readiness, QoS, full 10G bandwidth available to vMotion, In Service Software Upgrades (ISSU), 20G active/active bandwidth to each server with virtual Port Channels (vPC), proper network security features, and full traffic visibility.”

    At the cost of ~$150,000 for all that new network infrastructure…vs $1000 for some 10Gbe-CX4 cables.

    • Brad Hedlund says:

      Sam,
      Lets take a look at the math in each scenario based on (4) HP c7000 chassis. Lets start with the HP switches.

      Each HP c7000 will need (2) Flex-10 modules at $12,000 list price each, and (2) Virtual Connect FC modules at $9,000 list price each. That’s $42,000 in networking costs per HP c7000, and with (4) HP c7000′s thats $168,000.

      Now lets look at the Cisco Nexus solution with HP 10GE Pass-through modules. Each HP c7000 will need (2) 10GE passthrough modules at $5,000 list price each. So that’s $10,000 in networking costs per HP c7000, and with (4) HP c7000′s that’s $40,000. Now we need to connect those c7000′s to (4) Nexus 2232 Fabric Extenders at $10,000 list price each, totaling $40,000. And then we will connect the Nexus 2232′s to (2) Nexus 5010′s with FCoE licenses at $30,000 each, so that’s another $60,000.

      So in summary:

      HP networking solution = $168,000
      Cisco Nexus + 10GE Passthrough = $140,000

      Winner: Cisco Nexus + 10GE Passthrough

      Cheers,
      Brad

      • Ken Proulx says:

        Brad-

        You documented purchase cost. What about TCO? I know you’re an engineer type, but have you ever heard of a thing called Smartnet? Ouch. Add another $56,000 (minimum) in maintenance for years two & three on to the Cisco solution. What about power and cooling for the hardware components of the Cisco solution versus the HP Flex10 module?

        I’m sure there are maintenance costs on the HP side as well, but comparing “list” prices is hardly a valid comparison.

        KP

        • BT says:

          There is no way that Smartnet is going to cost 56k on 140,000 worth of equipment where 30k isnt equipment but is software licensing. The list price comparison is completely valid unless HP is giving away service for free. If you count HP incentives, then I would consider Cisco incentives as well

      • kevin says:

        you seemed to have forgotten transceivers. 10GB SFP= $750 list (approx) x 32 in the pass thru and 32 on the switch=48,000 x 4 chassis, HMM add another 192K to your little Nexus 5K config so that would be 140,000+192,000 = 322,000

        and to be fair for flex 10 there are 16 external ports and you need modules in the 5K too so 32 x$750= 24k. and 4 chassis that is 96K + 168= 264K

        SO you are right Ci$CO wins more money again

        Flex 10 option would be$264,000
        CIsco plus passthru would be $322,000

        I gotta pay for this stuff, both you and HP gotta remember we do have budgets. and truthfully the 2 Chassis I have with 15 blades each, ( about 45 various VMs each) work fine with 8 cables coming off the chassis( 4 per flex 10) the blades talk to eachother on the Vnets without having to jump all over the network. and I assign only .500mb for console nic. above you say 2GB for console?? why that makes no sense.

        • Brad Hedlund says:

          Kevin,
          You don’t need transceivers. The Nexus 5K supports the same HP DAC cables as you would use with Flex-10. The HP DAC cable provides 10GE for both ends of the link at approx $100 street price. It’s insignificant.
          Carve your Flex-10 up however you like in a way that makes sense for you. The point remains that you are providing much less than 10GE of available bandwidth to your virtual machines for I/O. With the Nexus setup, you don’t have to do that, because the bandwidth allocation works more intelligently.

          Cheers,
          Brad

      • Steve says:

        What you are comparing here is not even close to apples to apples. You are describing an HP solution with servers having 20Gb of IP bandwidth and 16Gb of storage bandwidth versus servers with 20Gb of combined traffic. Also in your QoS model and as you state, Cisco dedicates 50% for FCoE traffic. This leaves 5Gb for general VM, vMotion, and management traffic. You also mention that Qau is not required for FCoE while you are technically correct how do you scale without it lamost all storage manufactures are supporting single hop only without it. The silicon is just now hitting the street and products are still need. While I agree that HP has some work to do and that F10 is not for every site it is still a very good solution that fits a lot more sites than the Nexus solution.

  14. Steve Condas says:

    Did I miss something, or is the pricing comparing apples to oranges. It seems to me that that the quoted HP solution has 20Gb of network bandwidth and 16Gb of FC bandwidth to each blade, whereas the Cisco solution has on 20Gb of combined network and FCOE bandwidth (both are measured in a single direction). It is not suprising that the cost would be more for the HP solution if it gives you nearly 80% more bandwidth?? How does the pricing look if you use Cisco only for networking and use the HP VC FC for SAN?

    • Brad Hedlund says:

      Steve,
      Each solution is showing blade servers with both 10GE and FC connectivity, so in that sense it’s pretty close to “apples to apples”. Of course you can always pay for more bandwidth with more adapters, more switches, more things to manage, etc. But is there is a real requirement for that bandwidth? Maybe, maybe not.
      Also, the HP Virtual Connect FC modules only work if you have HP VC (Flex-10 / FlexFabric) Ethernet switches as well.

      Cheers,
      Brad

  15. Eugene says:

    Just wanted to add comments about QoS requirements. In many cases my customers wanted to have application backup within the same VMware infrastructure. This always demanded another pair of physical ports for bandwith separation. With QoS in mind one can take advantage of full 10Gb pipe but still keep this traffic as lower priority over other types such as vMotion, NFS, VM data, console, etc.

    Great post,
    Cheers

  16. Christer Hemgren says:

    Great Calc Brad

    But you miss the cost for HP SAN Switch, that is what we get with the Nexus.

    We have exact this discussion at work.
    Today we have two switch (c6500) and two SAN switch(MDS) in each DC server room and allot of HP and IBM blades with both Cisco and HP VC module.

    We are likely to go for Cisco Nexus 5K/N2K + 10GE Passthrough in HP blade. And force all new server to have CNA.

    We will then have two N5K in “DC com rooms” and many Nexus 2232 in the DC server rooms.

    Great post,

    Christer

  17. Jason says:

    I’m working with an organization that is looking at moving away from HP Virtual Connect in favor of the Nexus 5k / 2232 FEX / HP Pass-thru / Nexus 1000v model. They’re not looking at refreshing all their current blades and enclosures (so no UCS), but they have become discontented with the HP VC performance, architecture, and operational characteristics — and they like the prospect of a clean and comprehensive view of the network, and associated functionality down to the hypervisor. If they adopt such a model, it could mean junking their Flex-10 Ethernet & SAN I/O modules and migrating their blade servers to FCoE.

    I’ve only heard references to the Nexus w/HP Pass-thru as a good “migration path” to FCoE, but can it be done today? For example, can I take an existing c7000 enclosure and add ProLiant BladeSystem servers that either include integrated HP FlexFabric LOMs (for example, some G7 servers which are said to use HP Branded Emulex OneConnect-based LOMs), or add a OneConnect-based Mezz adapter to their existing ProLiant Blade Servers – and expect FCoE to work properly with the aforementioned FCoE-supporting pass-thru networking architecture?

    Is there any good information (architecture, design, whitepapers, etc.) that shows what some IT organizations are doing today to use FCoE w/HP Pass-thru and Nexus infrastructure?

    • Brad Hedlund says:

      Jason,
      I know of several customers who have successfully tested FCoE with Nexus 5000 / 2232 with HP c7000. Some might be in production now or close to it. If the solution did NOT work, it would mean that HP is selling CNA’s that are not FCoE standards compliant. HP would not want that egg on their face. But that’s not the case, because the HP FlexFabric CNA is an Emulex adapter, which we know is standards compliant, and therefore the solution works as expected.

      Here’s the whitepaper on Cisco.com that discusses the 10GE pass-through solution with FCoE for HP, IBM, Dell blade chassis:
      http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/white_paper_c11-594477.html

      Cheers,
      Brad

  18. Sean says:

    Brad, Thank you very much for your article. You made my job
    a lot easier today. I just had that same experience that you
    described in your first paragraph (me being the customer). Getting
    a straight answer from an SE is usually like pulling teeth. .
    .

  19. DirtiNetwork says:

    We are on the cusp of adding either the VC Flex 10 or the N1000V devices in our ESX infrastructure. We have over 40 enclosures with 200 vms on each host. My only concern is that even though we are a Cisco shop with 6500s… adding the Nexus 5k into our network would be difficult on the departments budget versus the N1000V would be on our Server Ops budget… so the question is: Would all the QoS policies apply, as stated above, work without the N5k and using a 6500?

    • Brad Hedlund says:

      Yes, the Nexus 1000V works with *any* upstream switch, even HP VC Flex 10. You should however question the QoS capabilities of the switch you are connecting to Nexus 1000V to insure your QoS policy is consistent throughout the network. The Catalyst 6500 has pretty good QoS, the HP Flex-10 has none.
      Keep in mind that Nexus 1000V and HP Virtual Connect Flex-10 are not competing products. One is a physical switch (Flex 10), the other is the software switch within ESX (Nexus 1000V). They are not mutually exclusive because they each work in different parts of the overall architecture.

      Make sense?

  20. Michael Muller says:

    Hi,

    I heard about virtualized switches. And I have to admit, that I really can´t see a real benefit. So our SANs are connected to our servers with 2x 24 port 10Gbe switches, stacked together, equipped with DAC cables where possible and some SFP+ optics, plus 5 year of manufacturer full service for ~32000,-€ incl. VAT.

    Talking about QOS, of course each basic layer 3 switch has QOS features nowadays.

  21. Giograves says:

    First, VMware ENT plus with dv-switches and network IO control (NIOC) play well with the “issue” of carving flexnic’s and losing max “burst” capability.

    Second , I need to understand the math better where 16×2 10Gb pass through will actually save over an HP solution using FLEX-10 (ah hem now also with the FLEX-FABRIC module available).

    10Gb ports aren’t exactly cheap. The X2 transceiver, the single/multi mode sc-lc cable, the 10Gb sfp…that’s between 3-4 grand per port alone.The whole point of HP’s solution is to save cabling. That has been their biggest selling point.

    Frankly I cant understand using c7000 blades if you are of the Cisco fabric extending, dynamic bandwidth allocation type of mindset. Weird.

    • Giograves says:

      HP/Cisco have collaborated and come out with a Nexus Interconnect for the c7000. Who would have thought a FEX sitting in an HP enclosure? That makes the prospect of a hybrid vendor solution much more enticing over 16×2 pass – through cabling.

  22. Cisco and HP have now released a Cisco Fabric Extender for HP BladeSystem.

    http://h18004.www1.hp.com/products/blades/components/ethernet/extender/
    http://www.cisco.com/en/US/products/ps11975/

    This means benefits of passthrough, without the extra cables

  23. Consultant says:

    I think of this situation as “grass is greener on other side”. If we set our mind set (as per suggested example) that server only has 3 NICs of 2Gb, 2Gb and 6Gb respectively, then things look simple. People (including myself) have burnt their hands by letting the things go out of the control i.e. automated management of bandwidth (or anything…). I am a consultant who manages ESX/Network and worked in several environment where educated customer had no issues coordinating between server/vmware/network teams.

    Bottom line, 10Gb bandwidth on blade server with ESX is giant and AFAIK HP blades can go upto 80Gb network bandwidth with 2 onboard and 6 mezzanine NIC cards. Very very unlikely you will find a good case of not having enough bandwidth.

    Finally, it’s all about three things… planning, planning and planning. In a planned environment we will never look at QOS a deal-breaker. And if we don’t have expertise to plan, we will always get into such mind boggling discussions.

  24. Nick says:

    Awesome dialogue and mind bending. At the end of the day you have to understand what you are installing to know how to support it. Go with what you know you can maintain. Simpler the better from the perspective of who is going to maintain it.

    • Haithem boukadida says:

      Folks,

      Maybe you should have a look at the new Virtual connect 4.01 FW. It has FCoE dual hope capabilities , QOEs support, Min and Max bandwidth allocation …
      Flex 10/10D supports up to 600Gb bandwidth, DCB support …. with the flexibility and simplicity of VC profiles management.

      Flex Fabric also supports FCOE and all these new features.

      @Brad : Being able to resume from failed server in about 2 minutes with server profile migration between servers in the same enclosure or event between enclosures (with VCEM) is not something you can consider as not important for high available environments.

      The real question is do you really need more that 2GB for VMotion network? In my own experience even with Enhanced VMotion (Live “VMotion + Storage VMotion”) with the maximum allowed concurrent operations we never needed as much bandwidth .

      Server administrators will really be happy I guess !

      Sheers ,

Leave a Reply

Your email address will not be published. Required fields are marked *