Nexus 1000V’s 17 load balancing algorithms, dedicated to Nicholas Weaver at EMC

Filed in Nexus, VMware by on August 31, 2010 9 Comments

I wanted to take a few minutes to point out the 17 different load balancing algorithms available when distributing traffic from a VMware vSphere ESX host to a network of clustered upstream switches. Normally I don’t write blogs on such short topics but this one has a little story behind it.

This week I am at VMworld 2010 attending a lot of great sessions, meeting new people, and reconnecting with some really awesome people I know in the industry.

I decided to drop into a session on Virtual Networking by Nicholas Weaver who is a vSpecialist at EMC, whom I know from conversations with him on twitter (@lynxbat) and his blog. One thing I learned today is that if you enter a session taught by Nicholas you had better be prepared to be called out to answer a question or provide commentary.

During the session Nick called me out several times in front of his packed audience to promote my blog. Thank you for that Nick, I really appreciate it, you didn’t need to do that. You did a great job with the session and deserve all the attention for it.

At one point Nick was presenting a slide on the Nexus 1000V and its capabilities to provide very granular load balancing. Nick stated to the audience that he believed the Nexus 1000V had 17 different possible load balancing algorithms, but had called me out again for verification. Caught a little off guard and unprepared, for some reason the number 15 came to my mind, so I responded: “Yeah, pretty close”. Not that a difference of 2 really matters, but nonetheless in such a technical session of paying participants you want to get every detail correct.

After Nick moved on to other slides I pulled out my laptop to double check, and sure enough, Nick was 100% correct that it is 17 different algorithms the Nexus 1000V can use to load balance traffic (when using a port channel uplink).

So, this post is dedicated to Nicholas Weaver and his packed VMworld session.  Great job with the session Nick.  I think its great to see non-Cisco presenters touting the virtues of Nexus 1000V and advocating its deployment.

Below are the *17* different load balancing algorithms you can choose from when using a port channel uplink from Nexus 1000V, preceded by a simple use case diagram.  Each algorithm is telling the Nexus 1000V which fields to look at to determine what constitutes a flow and calculate a hash that determines which physical port channel member link will carry that flow.

•dest-ip-port—Loads distribution on the destination IP address and L4 port.

•dest-ip-port-vlan—Loads distribution on the destination IP address, L4 port, and VLAN.

•destination-ip-vlan—Loads distribution on the destination IP address and VLAN

•destination-mac—Loads distribution on the destination MAC address.

•destination-port—Loads distribution on the destination L4 port.

•source-dest-ip-port—Loads distribution on the source and destination IP address and L4 port.

•source-dest-ip-port-vlan—Loads distribution on the source and destination IP address, L4 port, and VLAN.

•source-dest-ip-vlan—Loads distribution on the source and destination IP address and VLAN.

•source-dest-mac—Loads distribution on the source and destination MAC address.

•source-dest-port—Loads distribution on the source and destination L4 port.

•source-ip-port—Loads distribution on the source IP address.

•source-ip-port-vlan—Loads distribution on the source IP address, L4, and VLAN

•source-ip-vlan—Loads distribution on the source IP address and VLAN.

•source-mac—Loads distribution on the source MAC address.

•source-port—Loads distribution on the source port.

•source-virtual-port-id—Loads distribution on the source virtual port ID.

•vlan-only—Loads distribution on the VLAN only.

The algorithm that I recommend most is source-dest-ip-port, where the Nexus 1000V will look at the source and destination IP address and TCP/UDP port numbers to constitute a flow and make a hashing decision.  Given the inspection up to Layer 4, this generally provides the most granular flow definitions and therefore closer to 50/50 Even Steven load balancing than the other methods.  As shown in the diagram above, both VM1 and VM2 might have multiple flows each with different destination IP addresses or TCP/UDP port numbers, and therefore each flow could be distributed on separate physical links.

The default algorithm is source-mac.  This setting will take all flows from a VM (assuming a single source MAC address) for placement on a single physical link within a port channel.

Configuration Example:

Nexus-1000V(config)# port-channel load-balance ethernet src-dest-ip-port
Link to Configuration Documentation

About the Author ()

Brad Hedlund is an Engineering Architect with the CTO office of VMware’s Networking and Security Business Unit (NSBU), focused on network & security virtualization (NSX) and the software-defined data center. Brad’s background in data center networking begins in the mid-1990s with a variety of experience in roles such as IT customer, systems integrator, architecture and technical strategy roles at Cisco and Dell, and speaker at industry conferences. CCIE Emeritus #5530.

Comments (9)

Trackback URL | Comments RSS Feed

  1. Thanks Brad,

    Only called you out because you are THE MAN on this stuff.

    Honored that you took the time to drop by.

    .nick

  2. scott owens says:

    Have not yet worked with the 1000V in prod but why oh why is source-mac the default ?
    Kind of ditto with jumbo frames ( why is the default not the largest possible mtu ).

  3. Andy Daniel says:

    Brad, thought you might want to mention the command you can use to test the hashing results. It can be very useful when choosing IP addresses. Specifically, IPs to use for VMkernel networks like NFS and iSCSI…

    “show port-channel load-balance forwarding-path”

    I posted about the command a while back over at my site:

    http://www.vnephos.com/index.php/2010/07/nx-os-port-channel-hashing/

    Andy

    • Tom K says:

      While I wish this command was on the 1000v, it still didn’t make it there (as of 4.0.4.SV1.3a).

      It’s still only a 5K/7K deal (although extremely useful).

      Thanks!

  4. Tim Ward says:

    Can this load balancing behavior be replicated in the 2248/5000 FEX?

    • Brad Hedlund says:

      Tim,
      If a FEX uses a port channel fabric uplink it will load balance based on hashing as follows:

        For a Layer 2 frame (no IP header), the FEX uses the source and destination MAC addresses.

        For a Layer 3 frame, the FEX uses the source and destination MAC addresses and the source and destination IP addresses.

      Reference: http://bit.ly/96Uzu4

  5. Ian Erikson says:

    Brad,

    How do these load balance algorithms interact with the different uplink(channel group) modes, LACP , MAC-pinning, vPC-Hm?

    The command Andy describes above does not seem to be a command on the 1000v.

    thanks,
    Ian

  6. Mike says:

    Brad,

    Good info. How does this change with UCS in the picture?

Leave a Reply

Your email address will not be published. Required fields are marked *