vPC Technology has been introduced by Cisco Nexus OS as a key component of Cisco Data Center Networking architecture. The major benefits it brings are:
- Loop avoidance logic allows 2 physical connections from 1 device to 2 Nexus Switches to operate in active/active mode and eliminate STP blocked port to fully utilize the uplink bandwidth.
- Layer 2 Loop avoidance logics implemented directly in hardware while STP relies on control plane to do that, thus it provides faster convergence upon link or device failure.
- more efficient load balance than STP.
Most components or concepts of vPC technology are entailed in below picture.
Configuration Relevant Terminology
vPC domain ID
vPC domain ID should configured with the exactly same value on both peer devices.
- vPC domian can only be formed between Nexus 7000 peers, 5000 peers as well as 2000 peers, and only 2 peer devices in a single domain.
- vPC domain ID must be unique among vPC domains as it is used as part of LACP protocol as well as vPC System MAC which stands for virtual MAC address of logical vPC device.
vpc domain <domain id>
vPC Sytem-MAC and vPC Local System-Mac
vPC system-mac and vPC local system-mac are both used in the LACP protocol as the LACP system ID. However, vPC system-mac is used only with vPC attached (dual-attached with active/active mode) access devices while vPC local system-mac is used with single attached devices (orphan port or active/standby with or without STP).
- vPC system-mac: which are by default generated based on vpc domain-id.
vPC system-mac = 00:23:04:ee:be:<vpc domain-id in hex>; or you can manually configure the vPC system-mac value by below command.
- vPC local system mac: is owned by each peer devices so it is unique per device. vPC local system mac is derived from system or VDC mac address (show vdc command to view it). vPC local system mac is used whenever vPC systems do not need to present itself as a unique logical device. This is for instance the case with orphan ports.
vPC Role and vPC Operation Role
- vPC Role (primary & secodanry): defines which of the two vPC peer devices processes Bridge Protocol Data Units (BPDUs) and responds to Address Resolution Protocol (ARP). Use below command to customize the priority,
role priority <1-65535> //lowest priority wins, if ties, low local system MAC wins.
vPC role (primary or secondary) matters for the behavior in a peer-link failure. when a peer-link failure occurs, only the secondary peer device shuts its vPC member ports to down state and in addition shuts all its vPC VLAN interface (or SVIs – Switch Virtual Interface) [SVI associated to vPC VLAN].
- vPC Operational Role: primary or secondary is driven by the real-time behavior of peer devices, peer device’s vPC Operational Role can be different from vPC Role on very little occasions. vPC Role is non-preemptive while vPC Operational Role can be manually preempt by below procedures:
1. Configure a lower Role Priority for the peer devices who you want to be vPC Operational Primary mode.
2. Shutdown and no shutdown vPC peer-link.
(Be careful that this operation is disruptive as operational secondary peer device will shut its vPC member ports once peer-link is down).
Cisco Fabric Service (CFS) Protocol
- Both switches in the vPC domain maintain distinct control planes, thus CFS is required to provide reliable synchronization and consistency check mechanisms between the 2 peer devices and runs on top of vPC peer-link.
- CFS is enabled by default when vPC feature is turned on.
- CFS messages are encapsulated in standard Ethernet frames that are delivered between peers exclusively on the peer-link.
- Cisco Fabric Services messages are tagged with CoS = 4 for reliable communication.
- Since NX-OS version 5.2, graceful consistency check has been introduced to soften vPC system reaction for failure of CFS check.
Type 1 consistency check – Global Configuration parameters & vPC interface parameters: Configurations are manually done separately on each peer device. When vPC feature is enabled and vPC peer link configured on both peer devices, CFS service provides local peer device’s configuration copy to the other one, each peer device then determines whether any of the crucial configuration parameters differ on two peer devices.
- Type 1 global configuration consistency check: if inconsistent, all vPC member ports only on secondary peer device are set to down state.
- Type 1 vPC interface parameters consistency check: , only the inconsistent vPC member ports on secondary peer device are set to down state.
- Check results of Type 1 consistency check, issue below command
show vpc consistency-parameters [global | interface port-channel <id>]
Type 2 Consistency check – Global configuration & vPC interface parameters: Some global configuration consistency check parameters appear in sh vpc consistency-parameters global (mentioned above). However, most of vPC interface parameters (for type 2 consistency check) do not appear in sh vpc consistency-parameters interface port-channel command.
- When type 2 global configuration or vPC interface parameters are inconsistent, all vPC member ports remain in up state and vPC systems trigger to protective actions.
vPC Deployment Model
vPC domain deployment
vPC domain can be deployed in Single-sided vPC, Double-sided vPC (also called multilayer vPC) as well as Data Center Interconnect (DCI).
How devices attach to vPC domain
- Active/Active Dual-Attached: this is the scenario which full vPC benefits are provided.
- Active/Standby Dual-Attached with/without STP: 2 uplinks with only one of them is in forwarding state, STP could be running or not.
- Single Attached: port on peer devices which is connected to single-attached device using vPC VLAN is called Orphan ports. If that peer device port carries non-vPC VLAN, then it is not defined as Orphan port.
vPC Reactions to failure of peer-link or keepalive link
Both keepalive and peer-link are quite important for vPC system to work properly. Keepalive should be firstly established before procedures of Primary and Secondary role in vPC system can proceed. When only keepalive link is down, peer-link will carry keepalive heartbeats temporarily. Therefore, vPC almost reacts nothing to keepalive link failure. However, Cisco still recommends to fix this failure as soon as possible to avoid Split-Brain scenarios when both keepalive and peer-link are down. In summary, Cisco recommends spreading peer links and keepalive links to multiple ASICs or multiple modules and different cabling routes for keepalive and peer links to avoid a double failure.
vPC & STP Interoperation
Why STP is still required for vPC domain?
- STP prevents layer 2 loops for non-vPC attached devices In hybrid deployment where both vPC-attached and non-vPC attached devices co-exist.
- STP prevents layer 2 loops during addition & removal of a vPC or initialization & configuration change of a vPC domain.
Differences when enabling peer-switch (applies only to vPC ports)
Since NX-OS 4.2(6), 5.0(2a) and further, an enhancement, called vPC peer-switch, was brought to STP in the context of vPC.
- Peer-Switch Disabled:
Both peer devices generates and sends out BPDUs during STP topology initialization, but when STP topology is stable, only STP Root bridge (vPC primary swtich) will generates and sends out Config BPDU per 2 secs as keepalive message.
TCN BPDUs from vPC downstream devices are processed only by vPC primary switch, while Secondary swtich relays TCN BPDU to primary switch on peer-link.
- Peer-Switch Enabled:
Both vPC primary and secondary switch generate and send out BPDUs which has exactly the same content, during STP Initialization and stable STP topology.
Both vPC primary and secondary switch process locally received TCN BPDUs, and TCN BPDUs relay on vPC peer-link is no longer required.
It is meaningfully a macro command which consists of 2 sub-commands; Designated Priority & Root Priority.
- Designated Priority: dedicated for STP-attached devices, used to manually load balance the traffic from different VLANs across vPC peer switches.
- Root Priority: dedicated for vPC-attached devices, used to avoid STP topology change when a vPC peer switch recovers after failure or reload. During the recovery, due to the difference between the regular STP links (non-vPC) and vPC link bring-up. Regular STP link can be up prior to vPC, and hence the vPC peer-switch formation. Since vPC peer-switch is not formed, the other healthy vPC peer switch will use the local system MAC for STP bridge ID and if that local MAC address is better than the vPC system MAC then it will trigger STP topology change since the STP bridge priority is the same on both vPC peer devices.
Switch(config)# spanning-tree pseudo-information
Switch(config-pseudo)# vlan 1 designated priority 8192
Switch(config-pseudo)# vlan 2 designated priority 4096
Switch(config-pseudo)# vlan 1 root priority 4096
Switch(config-pseudo)# vlan 2 root priority 4096
Switch(config)# vpc domain 1
Bridge Assurance is a STP extension that prevents L2 loop in scenarios of unidirectional link event caused by physical cable failure or adjacent switch control plane failure but data plane still in forwarding state. Bridge Assurance BPDU are processed directly by SUPERVISOR CPU (Bridge Assurance BPDU is STP BPDU). It is not hardware off-loaded down to line card (like BFD for instance). Bridge Assurance BPDU are sent periodically. Default hello time is 2 seconds.【unidirectional BPDUs might cause transition to a forwarding state by error】
- How does it work: Bridge Assurance causes the switch to send BPDUs on all operational ports that carry a STP port type setting of “network”, including alternate and backup ports for each hello time period. If a neighbor port stops receiving BPDUs, the port is moved into the blocking state. If the blocked port begins receiving BPDUs again, it is removed from bridge assurance blocking state, and goes through normal Rapid-PVST transition.
- Why Bridge Assurance required on Peer-Link: When BPDUs not exchanged on peer-link due to unidirectional link or adjacent switch control plane failure, chances are that L2 loop might be formed provided that all vPC member ports are preferred to be in Forwarding state because vPC peer switches are preferred to be STP Root & Secondary bridge.
- Why Bridge Assurance should be disabled on vPC member ports: When peer-switch is not enabled, primary peer device processes the STP BPDU in a steady state. If the primary peer device fails over, the secondary peer device needs to start sending BPDUs. As the primary peer device was also the Spanning Tree Protocol root, the secondary also has to take over the STP role as root. If this process lasts too long, the uplink ports on access devices may go into Bridge Assurance inconsistent (BA_Inconsistent) state. This can occur in specific conditions of intense CPU utilization. To avoid this unreasonable blocking of uplink ports on access device, Bridge Assurance should be disabled on vPC member ports.
- Configuration Tips: Bridge Assurance is enabled by default on vPC peer-link (at the creation of the link). Bridge assurance on the peer-link is fine so there is no need to disable it.
Best Practices for Design & Deployment: