Routing over Nexus 7000 vPC peer-link? Yes and No.

Filed in Design Diagrams, Nexus, Routing, vPC by on December 16, 2010 72 Comments

This is a Nexus 7000 design question that comes up from time to time:

In a Nexus 7000 Vpc environment, how can I form a layer 3 adjency between the two switches. Lets say I want to run OSPF and want to create two SVIs on the two switches connected via Vpc, Will the neighborship relation be formed over the Vpc Peer link or is the peer link only designed for control traffic for Vpc.

Some people believe that in order to form an L3 adjacency between two Nexus 7000 vPC peer switches you must provision a separate link (other than the peer link) to use for L3 routing.  This is not true.  You absolutely can use the existing vPC peer link to form a routing adjacency between two vPC peer Nexus 7000’s.

I believe the source of confusion comes from a vPC design caveat that gets condensed and passed on as simply: “No L3 routing over vPC”.

Not knowing any better, if you take that statement at face value its easy to see how one might believe that you cannot form a routing adjacency between the two Nexus 7000’s over the vPC peer link, and therefore seek to provision a new “non-vPC” inter-switch link for that purpose.

Let’s take a minute to look at what works, and what doesn’t.  I’m not going to bore you with all the technical details.  Instead we’ll take a look at six simple diagrams with brief explanations.

Diagram #1 below shows two Nexus 7000’s configured as vPC peers with a single inter-switch link between them, the vPC peer link.  The two Nexus 7000’s are configured for OSPF and are using an SVI associated to a VLAN on the peer-link to form the L3 adjacency.  This VLAN for the L3 adjacency should only be forwarded on the peer-link.  Do not forward this routing VLAN on the vPC member ports (such as toward the L2 Switch shown in the diagram).

L3 peering over vPC peer-link

In the diagram above, we have a Layer 3 switch or router upstream configured for OSPF.  This L3 switch is attached to each Nexus 7000 with two normal point to point Layer 3 interfaces, no port channels, no vPC.

This design works.  This design has been discussed in official Cisco design documents and is supported by Cisco TAC.


So where does the “No L3 routing over vPC” come from anyway? Simply put, this comes from the vPC design caveat that you should NOT have an external device (firewall, router, switch) forming a routing protocol adjacency with the Nexus 7000’s over the vPC peer link.

Diagram #2 below shows an L3 switch running OSPF attached with a vPC to the two Nexus 7000’s and attempting to form an OSPF adjacency with each Nexus 7000. This design does NOT work.

Router attached via vPC - peering with Nexus 7000

This design does NOT work because some of the OSPF routed traffic from the L3 switch to the Nexus 7000’s will traverse the vPC peer-link (even when no ports or links are failed). As a result, this traffic will be dropped as a result of loop prevention logic in the Nexus 7000 hardware.

Side note: For the more vPC savvy folks out there who might be wondering “Does the new vPC Peer Gateway” feature make this design work? The answer is No.


Let’s look at Diagram #3 below. Here’s another example of an external device building a routing protocol adjacency with the Nexus 7000’s, this time its firewalls. The firewalls are singly attached (no vPC) to a VLAN that is forwarded on the Nexus 7000’s vPC peer link. The firewalls are running OSPF and attempting for form an adjacency with the each Nexus 7000.  This design too does NOT work.

Firewalls running OSPF attached to Nexus 7000's running vPC

This design does NOT work for the same reason as Diagram #2. Each firewall will form an OSPF adjacency with both Nexus 7000’s. This means that some OSPF routed traffic will traverse the vPC peer-link (even when no ports or links are failed). As a result, this traffic will be dropped.

Each firewall see’s both 7K1 and 7K2 as directly adjacent OSPF neighbors. If Firewall-1 chooses 7K-2 as the next-hop, this traffic will traverse the peer-link with the loop prevention bit set. If 7K-2 realizes the packet is destined for a vPC member port it will drop the traffic if the corresponding vPC member port on 7K-1 is up. If the packet was not destined for a vPC member port it would be forwarded normally.


To make the above Diagram #3 work, we can provision a new inter-switch link between the Nexus 7000’s that is NOT a vPC peer-link. On this new link we will forward a VLAN that is not forwarding on the vPC peer-link. This makes it a non-vPC VLAN. From here we can attach our firewalls running OSPF to the Nexus 7000’s on the non-vPC VLAN. See Diagram #3B below.

Firewalls with OSPF attached to non-vPC VLANs

This design works because each firewall will form an adjacency with both Nexus 7000’s, however the OSPF routed traffic will not traverse the vPC peer-link and not be subject to any loop prevention logic.
Keep in mind that non-vPC VLANs cannot be forwarded on vPC member ports. So if you have another device that is vPC attached and needs to be Layer 2 adjacent to the firewalls, that will not work. You will need to get that other device attached to the non-vPC VLAN as well on a normal non-vPC connection.


If your firewalls are not running a routing protocol there is no problem at all. Simply have static routes, or a static default route, pointing to the HSRP VIP address on the Nexus 7000’s.
See Diagram 4 below. This design works.

Firewall with static routes connected to Nexus 7000's running vPC

In this case each Nexus 7000 will locally forward traffic sent to the HSRP VIP address (even if its the “Standby” switch). Therefore no static routed traffic will ever traverse the vPC peer-link, and there will be no problems.


Finally, let’s look at Diagram #5 below where we have external devices running a routing protocol attached to the Nexus 7000’s. Moreover, these devices are attached with a vPC. That shouldn’t work, right? Well, lets take a look. This design DOES work.

Routing peers using vPC for transit only

This design works because these devices are not attempting to form an OSPF adjacency with the Nexus 7000’s. Rather, these external routing devices are forming an adjacency only with each other, and simply using the vPC topology as a transit. Under normal conditions when no links or ports are failed, there will be no traffic traversing the peer link and no problems.


Summary

The Nexus 7000 hardware has loop prevention logic that drops traffic traversing the peer link (destined for a vPC member port) when there are no failed vPC ports or links.  Normally peer-link traffic is non-existent in a normal network and this is never a problem for attaching normal Layer 2 switches or servers.  However when external devices attempt to form a routing adjacency with the Nexus 7000’s over the vPC peer-link, some traffic destined for a vPC member port can be forced over the peer-link even when the network is healthy, causing traffic to be dropped by the loop prevention logic.

Best practice:

  • Attach external routers or L3 switches with L3 routed interfaces.
  • It’s OK to use the vPC peer-link to form a routing adjacency between the two Nexus 7000’s.  Use a VLAN dedicated to the routing adjacency and only forward this VLAN on the peer-link, not on the vPC member ports.
  • Use the ‘passive-interface default’ command in your routing protocol to prevent a routing adjacency on all the other VLANs.
  • If attaching external devices on a Layer 2 port running a routing protocol with the Nexus 7000’s (e.g. firewall running OSPF), provision a new non-vPC inter-switch link, and attach the device to non-vPC VLANs.
  • Use static routes to the HSRP gateway address on external devices such as firewalls and load balancers.  Do not run routing protocols on these devices unless absolutely necessary.
  • Read the Cisco vPC best practices design guides

About the Author ()

Brad Hedlund is an Engineering Architect with the CTO office of VMware’s Networking and Security Business Unit (NSBU), focused on network & security virtualization (NSX) and the software-defined data center. Brad’s background in data center networking begins in the mid-1990s with a variety of experience in roles such as IT customer, systems integrator, architecture and technical strategy roles at Cisco and Dell, and speaker at industry conferences. CCIE Emeritus #5530.

Comments (72)

Trackback URL | Comments RSS Feed

  1. Shahid Shafi says:

    Excellent Article Brad! Can you help me find other chapters on DC Design as the URL you posted is for Chapter 5

    thanks,

  2. ambi says:

    Brad

    great post

    what happens if passive interface is not configured in your first case and the routing vlan is also forwarded across the vpc memeber ports .. is it just not supported by Cisco TAC or it will not work at all.. if it doesnt work, it would be great if you could shed some light as to why

    Ambi

    • Brad Hedlund says:

      Ambi,
      As for not using the ‘passive interface’ … It will still “work” and Cisco TAC will still support it — its just not a recommended configuration. It doesn’t gain you anything other than a bunch of unnecessary OSPF adjacencies between the two 7Ks, that will A) consume CPU, and B) make troubleshooting more difficult.
      As for forwarding the routing VLAN on the member ports … you could get away with that too. Again, its just not recommended. You don’t gain anything from it and you just expose your servers to unnecessary traffic. Some might say: “but it makes my interface configuration easier to just forward all VLANs on all ports”. For those folks, I would recommend they use the Port Profile feature of the Nexus 7000. You can create an interface configuration profile just once, and apply it to many ports with a simple command that associates the physical port configuration with the profile.

      Cheers,
      Brad

      • Laurent says:

        Hello Brad,

        Excelent article.
        Regarding Diagram 2, you are stating the following:
        “Side note: For the more vPC savvy folks out there who might be wondering “Does the new vPC Peer Gateway” feature make this design work? The answer is No.”
        Before reading this side note, I was under the impression that this scenario should indeed be resolved by entering the Peer-gateway command.
        Could you maybe clarify what the peer-gateway command is allowing and why it will not fix the problem in this scenario?

        Thank you very much for your help.
        Laurent

  3. Andy says:

    For those of you who want the chapter by chapter of the document Brad references, the website he links to doesn’t have them all. But if you google the following you will find each one…

    Chapter 1 Data Center Design with Cisco Nexus Switches and Virtual PortChannel: Overview
    Chapter 2 Cisco NX-OS Software Command-Line Interface Primer
    Chapter 3 Cisco NX-OS Software Virtual PortChannel: Fundamental Concepts
    Chapter 4 Spanning Tree Design Guidelines for Cisco NX-OS Software and Virtual PortChannels
    Chapter 5 Data Center Aggregation Layer Design and Configuration with Cisco Nexus Switches and Virtual PortChannel
    Chapter 6 Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual PortChannels
    Chapter 7 10 Gigabit Ethernet Connectivity with Microsoft Windows Servers
    Chapter 8 Data Center Design with VMware ESX 4.0 and Cisco Nexus 5000 and 1000V Series Switches and 2000 Series Fabric Extenders

    Good reading… enjoy.
    Andy

  4. Mr_Mythbuster says:

    Scenario #2 and #3 are not possible because of peer link traffic filter, which is a requirement to avoid loop in the vPC system.
    See documentation here (explanation contain with figure 29 and 30 (left drawing)):
    http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html#wp1053500

    The IP connectivity is droped, but ARP mechanism passes. The para bellow figure 30 explains it all.

  5. Samer Labaky says:

    Excellent Post Brad

  6. mouse_support says:

    hi brad, thanks for the great article.
    i got a question for which i’m pretty sure but i would have a double check from you: may i apply the same concept of diagram 5 when interconnecting 2 pairs of Ne7k which are the core of 2 distinct datacenters?
    the 2 pairs are connected via a layer2 trunk done with a vpc, and by the way each chassis (in fact a vdc) is connected to its “brother” (another vdc) via a vpc-peer link.
    thanks for any advice you may had.

    • Brad Hedlund says:

      Good question. The Core 7Ks in each DC are presumably running L3 routing protocols, so you would NOT form L3 adjacencies over the vPC link between these two sets of Core 7Ks. You could either provision separate point-to-point links for L3, .. Or — have your Core 7Ks be an endpoint of a vPC domain provided by another pair of 7Ks or VDC dedicated to providing the data center interconnect.

      • mouse_support says:

        exactly.
        i will have no separate L3 link, so a new pair of Routing_VDC’s will form routing adjacencies in a shared vlan carried over the L2 VPC built by the core VDC’s. My Routing_VDC’s (4 in total) will behaves like your OSPF routers in figure 5, but of course the transport infrastructure will be different…

        thank you!

  7. Having_Issues says:

    Hi Brad,

    Thanks for the article. Would it possible to post some config examples from a command line point of view for design #1?

    Thanks!

  8. Blake Arnold says:

    Hi Brad,

    This is a great overview of how the vPC loop prevention works. I do have a few points of contention that I didn’t see you address. In order to clarify I need to give a different understanding of the vPC loop prevention process.

    The rule states: “If a frame arrives through a vPC and then traverses the vPC peer-link and is then destined to exit through a vPC the frame is discarded.”

    This all happens behind the scene and is a rule for using vPC’s. So, this issue can be seen if your IGP/EGP on one 7k choses a path through the vPC peer 7k to an OSPF neighbor. There are only a few situations that this presents itself and if is a best practice not to route over vPC peer links in most cases.

    Let me give a quick example using your last drawing. If bottom OSPF router sends traffic that arrives on left 7k then is sent across the peer-link and is destined to exit from the vPC link towards top OSPF router this traffic would be dropped. Now, I do realize that this is a layer 2 path only and the vPC should be intelligent enough for this not to happen but there are off cases where it could happen.

    Last example to provide further clarification. Lets say you have a L2 switch (SW1) connected to two 7k’s (R1 and R2) in a vPC and the 7k’s are connected to a second L2 switch (SW2). A host on SW1 sends a frame destined to a host on SW2 in two different broadcast domains (ie. different subnets). The traffic could leave SW1 toward R1 based on port-channel hashing, however the VRRP primary is on R2. So the frame from SW1 arrives in a vPC and is destined to R2 through the peer-link. At this point a flag is set in the frame and the traffic is then switched to the host on SW2 through a second vPC on R2. This traffic will be discarded.

    Please let me know what you think.

    • HeyNow says:

      Hey Blake.

      In your first example, the only reason the left 7K would pass the packet across to the right peer-link destined to the OSPF router on the top would be if it’s top vPC member port is down — in which case the peer 7K would pass the traffic normally. They share their mac tables, so under normal operations, 7K-left would know not to send it across the peer-link.

      In your second example, my understanding is VRRP is the same as HSRP, and is active/active. so even if the VRRP primary is on R2, R1 will route the traffic locally (which is destined to SW2), and will traverse down the vPC link to SW2.

      Thanks.

    • Brad Hedlund says:

      Blake,
      Your last example is not true. In this case both 7K’s will actively forward for the gateway virtual mac address. This is supported with HSRP (definitely) and VRRP (I think) .

      Brad

  9. Zg1004 says:

    Brad.
    Here is one question regarding vPC not related to L3 but related to non-vPC traffic traversing vPC peer link. On several Cisco design guides there is notice that non-vPC VLAN should not use vPC peer link for interconnection between two N7K since there is possibility that vPC peer link could becomes disabled and then devices is those non-vPC VLAN’s would remain orphaned. Can you confirm is this is true and if it is please explain failure scenario when vPC peers would disable vPC peer link.

  10. Michal says:

    Hi Brad ,
    thanks for sharing and I was recently checking some topologies were we have transit VPC domains an so when using the F1 modules for the VPC in the 5.1.2 NX-OS , the adjacency state of the OSPFv2 hits the FULL state for the diagram #2 for example ( the 3rd party switch/router wants the adjecency with one of the peer-vpc devices ( i.e. the SVI interface is enabled on one of the peer-vpc-devices).
    I have verified it also in terms of forwarding and with the 5.1.2 it doesn’t seem to pose the same problems , when VPC is enabled on the F1’s ( and with at least one M1 interface group allocated in order to get the SVIs in the UP state ).
    We can still however note that the VPC consistency check will report the SVI type-2 configuration incompatible state. I doesn’t mean however that you won’t establish your adjacencies or won’t be able to forward the traffic over it. Maybe there are some differences between the M1 and F1 for the anty loop checks that are enabled by default for the VPC .

    Cheers ,
    Michal
    ########################################################################
    NX7K-2-edge-2-2# sh vpc
    Legend:
    (*) – local vPC is down, forwarding via vPC peer-link

    vPC domain id : 1000
    Peer status : peer adjacency formed ok
    vPC keep-alive status : peer is alive
    Configuration consistency status: success
    Type-2 consistency status : failed
    Type-2 consistency reason : SVI type-2 configuration incompatible
    vPC role : secondary
    Number of vPCs configured : 2
    Peer Gateway : Disabled
    Dual-active excluded VLANs : –

    vPC Peer-link status
    ———————————————————————
    id Port Status Active vlans
    — —- —— ————————————————–
    1 Po1 up 100,122,126,200,226,300

    vPC status
    ———————————————————————-
    id Port Status Consistency Reason Active vlans
    — —- —— ———– —— ————
    10 Po10 up success success 100,122,126
    ,200,226
    20 Po20 up success success 100,122,126
    ,200,226

    NX7K-2-edge-2-2#

    • Matt Nichols says:

      Please note that in topology #2 it is possible to have OSPF neighbors come up in full state. Essentially there are two problems: first, the control plane, and second is the data forwarding plane.

      The reason this work in your case is that either you do not have peer-gateway enabled, or you have peer-gateway enabled and the unicast OSPF DBD packets hash to the local link of each peer, allowing the neighbor to come up.

      The second issue is that when you have these neighbors up, at later 3 you have 2 equal-cost routes (all metrics on both sides being equal) to the destination. If the next hop beyond the Nexus 7000 is attached via vpc, and you do not have peer-gateway configured, the peer-link can be part of the data forwarding path. If this is the case, packets that enter the peer-link ports cannot be forwarded out a vpc-enabled port, therefore dropping the packet. The fix for this would be to use peer-gateway, but the conundrum here is that it may introduce problem 1 again with the OSPF neighbors.

      Also keep in mind that it is possible to enter this state when you manipulate metrics with OSPF on either N7K leading to a sub-optimal forwarding path using the peer-link, or in some cases even with external neighbors.

      Brad’s statement for topology 2 as unsupported still stands, regardless of hardware combinations. It is possible to make this work under normal circumstances, but redundancy will most likely fail.

  11. HeyNow says:

    Hey,

    Would anyone mind clarifying why diagram #3 doesn’t work?

    Because each FW is singly attached (no vPC), traffic coming from the vPC peer link should not be dropped. I thought the rule stated:

    “Frame will only be forwarded if outgoing interface is NOT a vPC or if outgoing vPC doesn’t have active interface on other vPC peer”

    Well since the FW port is not configured as a vPC, I don’t see why the loop prevention would drop this traffic.

    This is similar to attaching two hosts, one to each 7K. If they’re on the same vlan, traffic will traverse the vPC peer-link fine, without being dropped.

    I think I’m missing something.

    • Brad Hedlund says:

      Each firewall see’s both 7K1 and 7K2 as directly adjacent OSPF neighbors. If Firewall-1 chooses 7K-2 as the next-hop, this traffic will traverse the peer-link with the loop prevention bit set. If 7K-2 realizes the packet is destined for a vPC member port it will drop the traffic if the corresponding vPC member port on 7K-1 is up. If the packet was not destined for a vPC member port it would be forwarded normally.

      • Eng Wee says:

        Hi Brad,

        I have the same question as “Hey” when reading your article on diagram #3. At the back of my mind, I have the same understanding as what you explain. So, I think you should put your explanation under diagram #3 as this will make things a lot clearer.

        Thanks for the excellent article.
        Rgds
        Eng Wee

      • Ryan says:

        The firewalls are connected to orphaned ports in diagram #3, per the manual: “One of the most important forwarding rules of vPC is the fact that a frame that entered the vPC peer switch from the peer link cannot exit the switch out of a vPC member port (except if this is coming from an orphaned port).”

        It looks like it should work, as the frame originated from an orphaned port prior to traversing the peer link.

        • Brad Hedlund says:

          Yep. The ability to define an ‘orphan port’ was made available some time after this post was published.

          • Tom K says:

            Would you mind elaborating that statement? The Cisco doc states “One of the most important forwarding rules of vPC is the fact that a frame that entered the vPC peer switch from the peer link cannot exit the switch out of a vPC member port (except if this is coming from an orphaned port).”

            I can’t seem to figure out a scenario when traffic originating from an orphaned port would need to traverse the peer-link and then traverse a vPC member port. It should, in theory, use the local vpc member port before sending any traffic over the peer-link — even when the traffic is “coming from an orphaned port”. Just kinda confused why they add different behavior for orphan ports sourcing traffic — I’d understand it better if was for traffic destined TO an orphaned port (of course it would have to traverse a peer-link to get there).

            Thanks in advance

          • Brad Hedlund says:

            Tom,
            You could have a device such as a router or firewall, running a routing protocol, attached to an orphan port. The device sees two neighbors on this link, the 7K its attached to, and the 7K on the other end of the peer-link. It may choose the far end 7K for routing a packet, and the destination of that packet may be a host attached to a vPC port on each 7K. In this case the packet will have traversed the peer-link and attempted to be delivered on a vpc member port.

          • Tom K says:

            Thanks Brad! That helps.

            In that scenario, wouldn’t the peer-gateway command kick in, if applied? I mean if the FW sends traffic to the far end 7K (ie, the dest MAC is the far end 7K’s SVI), the near end 7K will see the dst MAC is really the peer’s and handle that packet locally — routing it to the next hop IP (or client directly) on its local vpc member ports.

            Control plane might be an issue, I understand, since the dest IP for hello’s/etc could be the far end 7K, and if the vpc hashes it to the near end 7K, it still must traverse the peer-link, which decrements the ttl (or so I think), and that wouldn’t be good — based on the recent release note where they added the peer-gateway exclude vlan command for “backup router vlan”.

            Thanks for help. Much appreciated!

          • Tom K says:

            Unless vpc peer-gateway doesn’t kick in for orphan ports, even if it’s on a vpc vlan.

            (Couldn’t find the edit button.)

  12. mouse_support says:

    hi brad, you know that on ASA v8.4 it’s possible to aggregate interfaces with etherchannels.
    how diagram4 shall be modified? when using static routing, it is ok to connect ASA with 2 10-G ports and then creating a vPC towards two VDC’s?
    thank you

  13. Muhammad says:

    Hi Brad,

    Its an excellent link,thanks for sharing information. I have a scenario where two nexus are connected through vPC ,on the inside we have ASA,ACE with active/standby,meaning that N7K1 is active for all inbound and outbound traffic, now i want to implement mpls with provider and like to use eigrp as routing. The design is like N7K1 is connect to CE1 and CE2 and same as N7K2 is connecting to CE1 and CE2.All these four devices will run eigrp with same AS,also we want load balancing and HA. i like to show u my config, pls let me know if this looks good.

    Step 2: Configure EIGRP and Load Balancing

    N7K1#conf t
    Feature EIGRP
    Router eigrp DH1
    Autonomous-system 1000
    router-id 10.x.x.x………….(Loopback address)
    Log-adjacency-changes
    Log-neighbor-warnings
    maximum-paths 2

    Step 3: Configure EIGRP on interfaces
    Int E1/x
    Ip router eigrp DH1

    Int E1/y
    Ip router eigrp DH1

    Step 4: Configure redistribute connected routes
    Conf t
    ip access-list All-Routes
    permit ip 10.26.0.0/21 any

    route-map All-Routes permit 10
    match ip address All-Routes
    exit

    router eigrp Test1
    switch(config-router)# redistribute direct route-map All-Routes
    switch(config-router)# default-metric 100000 100 255 1 1500
    switch(config-router)# copy running-config startup-config

    Same config needs to apply on N7K2

    thanks
    Muhammad

  14. Marco Rizzi says:

    Thanks Brad,

    excellent and interesting post !

    Marco

  15. Tony says:

    Just an FYI, been working with TAC on this issue all day… apparently they do not want to support L3 across the vPC Peer Link.

  16. ERIC says:

    Hi Brad,

    Very nice explanation.
    Can you tell me if this rule: no traffic over peer link in normal VPC operation condition, applies with nexus 5500 equiped with L3 daughter card?

    Cisco tells me that in N5500 the routing decision is not made at the asic level as it is in 7K, but inside the L3 card which can be seen as a L3 router on stick so your scenario N°2 can be possible.

    Do you agree with this ?

    Thanks

    ERIC

    • Ted says:

      Just to reiterate this, I opened a TAC case to ask exactly this and got the response you proposed:

      Looking at the outputs you send me, everything looks as it should, the configuration you have seems to be really good to me. Now, regarding the layer 3 traffic over the peer-link, you are right but the information is based on the N7k which has a behavior a little bit different than the N5k. The design we talked on the phone will work fine on this switch you have so I will say that one is okay. As long as we allow all the VLANs you want to route on the physical link everything should be fine.

      So Design#2 should work in a 5548 pair with daughter cards, but not the 7k’s.
      -ted

  17. Ted says:

    Hi Brad,

    Is it possible to LAG or vPC between nexus 7000 over different media interface ? ( for example ethernet copper and fiber sx/lx interface) …. Thanks for replying in advance ….

    • Robert, note that #5 is not the same as Page 27 in the PPTX you linked to.

      – in #5 the N7k’s are in “L2 only” configuration, the other devices do not try to peer with N7k’s with a routing protocol

      – in Page 27 the N7k’s are L3 routing protocol peers for the other devices.

      Markku

  18. Network says:

    Hi,

    we have the following setup.

    ISP <——- Firewall <———– Switch

    (Vlan100)

    |

    |

    Nexus 2———– Nexus 1

    | (VPC) |

    | |

    User Switch stack – 3 cisco 2950

    (Vlan 101)

    Initally we just had one connection from the switch to the nexus. I configured vpc, thereby giving a secondary connection to each nexus. both nexus are connected using vpc. the end user switch is a stack of 3 2950 cisco switches. for some reason only for a few people, the internet does not seem to work (maybe a switch in the stack) – but when you remove the secondary connection the one of the nexus the internet seems to work for everyone.

    there is only one link from wan router connected to nexus 2. We have indivigual static routes on each nexus each pointing to the internet firewall. i understand that the edge switch can port channel load balance between the links to nexus 1 and nexus 2

    any thoughts on solving the problem above ?

    Thanks

  19. sidd says:

    Thanks Brad, its very infrmational

  20. Stuart says:

    Hi Brad,

    I have seen a number of the Cisco DCI designs that have a pair of Nexus switches at one data centre connected via vPC to another pair of Nexus switches at the second DC.

    Is it possible to route using SVIs over this vPC DCI?

    • Brad Hedlund says:

      Stuart,
      No. In this case you will need a separate set of links between each pair of 7Ks for the routed traffic. You cannot have a routing protocol run between the two 7K pairs using the vPC links.

  21. Rid says:

    Dear Brad
    In your article where you have said “it’s ok if you use firewall with static route” diagram(5th Diagram).

    Is it possible to have vPC with the firewall and run HSRP.

    Thanks

    • Brad Hedlund says:

      Yep, thats fine. As long as you are not running a routing protocol between the firewall and Nexus.
      Just point your firewall’s static route to the HSRP VIP.

      • Alex says:

        Dear Brad,
        I’m currently have 2 nexus 5k that will connect to N7K.. The N7k is not running VPC feature. Can i use vPC and non-vPC VLANs share the same VPL link in N5k? Do you think i can use “dual-active exclude interface-vlan ” for this? Because i need a non-vPC VLANs to form OSPF between N5k and N7K..Thanks for replying in advance.

        • Brad Hedlund says:

          Alex,
          As I no longer work at Cisco, I am no longer the ultimate authority on vPC best practices. That said, I’m pretty sure you’ll need to provision a separate ISL between the 5K’s for your non-VPC VLANs. Any VLAN forwarded on the vPC peer-link is by definition a vPC VLAN. The “dual-active exclude interface-vlan” command is only telling the vPC Secondary 5K which SVI interfaces it should NOT shutdown when the there is a peer-link failure.

          My best advice is to consult your Cisco SE, or try your question on Cisco’s support forums: http://supportforums.cisco.com

          Cheers,
          Brad

  22. jack says:

    who can tell me what means that peer-keepalive destnation 192.168.1.1 source 192.168.1.2 vrf management

  23. Aliaksandr says:

    What is about the idea to use the GRE tunnels over vPC member link with using dynamic routing inside GRE?

  24. Nav says:

    I have two questions regarding the Figure 2

    First case:-

    Can’t I have OSPF neighbor relationship on the HSRP VIP IP b/w the upstream layer 3 switch and the Two Nexus 7k1 and 7k2.

    And by utilizing the HSRP technology of the nexus (the MAC sharing by both the Active and Standby ) forwarding traffic layer 3 traffic at the same time.

    Second Case:

    Same figure and if use default route on the upstream switch towards the Nexus using the HSRP VIP as the next hop will this work.

    Mean to ask will both the N7K (Active and Standby) will serve the routing at the same time

    Your response will be highly appreciated.

    Thanx

  25. AndyO says:

    Nice article, man.
    But mistery remains, and it seems to be in the multicast treatment by N7K when it’s traversing vpc-peer-link, isnt it? because the rule of thumb for vpc insists “dont route traffic from one vpc to another vpc within vpc domain” and it is unclear why for the traffic L3-terminated on the N7K to be not forwarded down via vpc BO)
    Anyway gr8 10x for monolitic essencials!

  26. Jigar says:

    Hello Sir,
    I am much impressed with the kind of scenario explanation. however over and above to that I am also impressed by the tool you have used for presentation. I am sure its not visio tool that you have drawn diagrams on. could you share which graphical tool you are using to present.
    thanks much.

  27. Daniel says:

    Hi All,

    Cisco finally published some best practises for vPC where they talk about the above as well.. (Page 71 onwards)

    http://www.cisco.com/en/US/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

  28. Moti Amir says:

    Hi Brad,
    As this post was published 3 years ago, is it still relevant?
    I have just created on a LAB environment a topology similar to example 2 (diagram #2) where 2 Nexus 7Ks are connected as vPC peer and a Catalyst 6500 is connected to them with port-channel as vPC.
    EIGRP is enabled on a VLAN that is part of the vPC domain and VLAN interfaces on each of the 7Ks and 6500 is configured. The VLAN interfaces are forming adjacency and I can learn routes from the 6500 and from the 7Ks…

    Thanks,
    Moti

    • Dan Verzaal says:

      I don’t think it’s an issue of creating the peers and learning routes. It’s a problem when traffic is forwarded down to the southbound host/servers. Some traffic goes across the peer-link, n7k will see it’s destination is out a vpc southbound in which BOTH vpc member ports are up, so the vpc loop prevention rule says to drop the traffic since we assume n7k1 has already forwarded the traffic via it’s own vpc member port.

  29. Dani Petrov says:

    Hello Brad,

    before I raise my question – I’d like to say big thank you for the this extremely helpful blog!

    I have a question about Scenario #3.

    Am I right to consider that in case where we have “peer-switch” enabled in vpc domain – this option becomes invalid?

    Considering the fact that using peer-switch both nexus devices shall use the same MAC address for bpdu-bridge – the port will probably go into inconsistent/backup blocking state? (the switch will basically detect its “own” BPDU (that will probably be the secondary vpc neighbor as all the BPDUs are generated from the primary only).

    Thanks in advance!

    D.

  30. jean-christophe says:

    Hello Brad,

    Interesting post, I know I’m kinda late, but either there is a hidden rule regarding these use cases, or I do not understand some of what is stated here.

    Regarding the diagram #2, if we take a look at the official vPC loop avoidance rule which states:

    Traffic coming from vPC member port, then crossing vPC peer-link is NOT allowed to egress any vPC member port, including vPC member port that does not go back to the original device but is still in the same layer 2 domain; however it can egress any other type of port (L3 port, orphan port, …).

    Let’s suppose that the L3 device peers with both N7Ks over its vPC: that means potentially 4 neighborships.
    Let’s also suppose that a packet from the L3 device needs to reach the N7K on the right (N7K-2), and the L2 frame has to cross the vPC peer link from N7K-1 to N7K-2.
    If the frame arrives at N7K-2, it is then routed, which means that the L3 packet (corresponding to the original L3 packet, with a few different bits) which exits that N7K-2 is contained within a different L2 frame, with a different MAC enveloppe. Thus, the original L2 frame never egresses the N7K-2 or never “intends” to do so.

    This avoids the aforementionned official vPC loop avoidance rule, which is consistent with the fact that there is indeed no risk of loop there.

    Please put Cisco’s recommendations aside before answering ;)

    • Brad Hedlund says:

      A logical thought process, but unfortunately that scenario doesn’t work.
      In your description, when the frame went from N7K-1 to N7K-2, the frame header was marked in a proprietary way signalling that it had traversed a vPC peer-link. This tells the hardware on N7K-2 not to forward this frame on any vPC member port, no matter what.

      Cheers,
      Brad

  31. ArnisL says:

    Hello Brad, hello Jean,

    Despite Cisco recomendations, setup from diagram #2 works fine. At least with M modules and NX-OS 6.2(6a).
    I think – Jean’s thought process is fully understood for me and I agree with this too.
    Of course – It is not the best praxis to do non-recomended designs, for me it is temporally solution to migrate old network, but I don’t have other solution because lack of fibre connections.
    At least this unsupported solution works in my environment.
    Thanks, Arnis.

  32. Kim says:

    On Diagram #3 below, does it work if the firewalls are ASA active standby failover pair. EIGRP is only active on the active ASA. Each ASA only have single non VPC link to each 7k. I see the active ASA forming EIGRP relationships with both 7k.

    • Kim says:

      To follow up on my previous post, the ASAs are not connected to 7K directly. They are connected to layer 2 only 5k with VPC peer-link to each other.

  33. Roxy says:

    Hi Brad,
    Thanks for this article.

    In your Summary you list some bullet points.
    Referencing the 2 listed below, when attaching a FW running OSPF with the 7K, is it “better” to setup the FW on Layer 2 ports or treat it as a L3 and create routed interfaces?

    Best practice:

    – Attach external routers or L3 switches with L3 routed interfaces.

    – If attaching external devices on a Layer 2 port running a routing protocol with the Nexus 7000′s (e.g. firewall running OSPF), provision a new non-vPC inter-switch link, and attach the device to non-vPC VLANs.

    Thanks

  34. Raghav says:

    Is there any command to check drop on the VPC Peer links ?

  35. Pankaj says:

    I understand it’s an old post, but wondering if I can get some assistance for my query.

    Specifically for scenario 2 – What if we configure static OSPF neighbor between the router and Nexus OSPF vPC VLAN HSRP address? Is that expected to work?

    Thank you.

  36. Raje says:

    Hi Brad
    We looking for network device to manage our network.
    All The Router is from Same Channel or Network
    All we Need to be configured in same server, All routers
    Needed the same routed network- in the backgroud
    Route A,B,C and D connect same server.
    Our Application based on IP Binding .
    in the backgroud application server required connects
    Router Link A to same DB server -192.168.98.11
    Router Link B same time connect to-192.168.98.11
    Rouer Link C same time connect to -192.168.98.11
    Router Link D same time connect to 192.168.98.11
    Server and The other end of routers
    (192.168.98.11) authenticate
    the IP with specific user ids with DB and allow
    acess send confirmation specfic IP of Application
    Server , So its a two way link.(bidirectional)

  37. Andrew says:

    Great post, Brad… thanks for the explanation.

    One question we have is around the use of FabricPath without vPC(+). We’re hitting some issues with the dynamic peering limitations over vPCs one suggestion that has come up is to “just remove vPC from our core and run separate switches with FabricPath providing L2 redundancy/aggregation”. Obviously there’s some challenges there when we’re peering to a single device (which now will see two switches) however is this actually a supported/acceptable/useful topology? The comment was made today that a standard/best practice Cisco FabricPath deployment would always have vPCs deployed hand-in-hand.

    Any thoughts would be much appreciated.

Leave a Reply

Your email address will not be published. Required fields are marked *