EtherChannel
Companies require greater and cheaper bandwidth to run their networks. Users are becoming more impatient with any sort of latency that occurs in the network. The insatiable appetite of customers for faster networks and higher availability of the networks has made the competition intense between vendors. A few years ago, Cisco came up with a method to not only provide substantially higher bandwidth but with lower cost overhead.
EtherChannel is a technology originally developed by Cisco Systems as a LAN switch-to-switch technique of inverse multiplexing of multiple Fast or Gigabit Ethernet switch ports into one logical channel. Its benefit is that it is effectively cheaper than higher-speed media while using existing switch ports, as shown in Figure 4-9.
Figure 4-9. 4-Port EtherChannel
EtherChannel has developed into a cross-platform method of load balancing between servers, switches, and routers. EtherChannel can bond two, four, or eight ports (Catalyst 6500) to develop one logical connection with redundancy. The three major aspects to EtherChannel are
Frame distribution
Management of EtherChannel
Logical port
EtherChannel does not do frame-by-frame forwarding on a round-robin fashion on each of the links. The load-balancing policy or frame distribution used is contingent upon the switch platform used. For instance, in a Catalyst 5500 switch platform, the load-balancing operation performs an X-OR calculation on the two lowest-order bits of the source and destination MAC address. An X-OR operation between a given pair of addresses will use the same link for all frames. One of the primary benefits of the X-OR operation is to prevent out-of-order frames on the downstream switch. The other advantage is redundancy. If the active channel used by a connection is lost, the existing traffic can traverse over another active link on that EtherChannel. The one disadvantage to X-OR operation is the load on the channels might not be equal because the load-balancing policy is done on a specific header as defined by the platform or user configuration. On a Catalyst 6500 switch, the load-balancing operation can be performed on MAC address, IP address, or IP + TCP/UDP depending on the type of Supervisor/PFC used. Use the show port capabilities command to check the module for EtherChannel feature.
The default frame distribution behavior for the Catalyst 6500 is IP. Example 4-14 is based off a Supervisor II/PFC2 card (WS-X6K-SUP2-2GE/WS-F6K-PFC2). This specific Supervisor can support the load-balancing policy up to Layer 4. Some of the older Catalyst 6500 Supervisors do not have this feature available. It is worth noting that most enterprise customers have deployed Catalyst 6500 Supervisor IIs in their networks. Load-balancing policies cannot be configured on Catalyst 5000, and the older Catalyst 4000 Supervisors. The newer Catalyst 4000s with Supervisor II-plus and higher do support load-balancing policies.
Example 4-14. Output of the set port channel Command
Switch1 (enable) set port channel all distribution ?
ip Channel distribution ip
mac Channel distribution mac
session Channel distribution session
The management of the EtherChannel is done by Port Aggregation Protocol (PAgP). PAgP packets are sent every 30 seconds using multicast group MAC address 01-00-0C-CC-CC-CC with protocol value 0x0104. PAgP checks for configuration consistency and manages link additions and failures between two switches. It ensures that when EtherChannel is created that all ports have the same type of configuration. In EtherChannel, it is mandatory that all ports have the same speed, duplex setting, and VLAN information. Any port modification after the creation of the channel will also change all the other channel ports.
Finally, the last component of EtherChannel is creation of the logical port. The logical port, or Agport, is composed of all the links that make up the EtherChannel. The actual functionality and behavior of the Agport is no different than any other port. For instance, the spanning-tree algorithm treats Agport as a single port.
Example 4-15 shows the recommended steps for configuring EtherChannel on a Catalyst 6500, as shown in Figure 4-9. The ports used by Switch1 for EtherChannel are 10/13-16, and are configured with desirable mode. The desirable mode stipulates that a port actively initiate a channel setup.
Example 4-15. Output of the show port channel group Command
Switch1 (enable) show channel group
Admin Group Ports
----------- -----------------------------------------------
272 10/1-4
273 10/5-8
274 10/9-12
275 10/13-16
276 10/17-20
277 10/21-24
278 10/25-28
279 10/29-32
280 10/33-36
281 10/37-40
282 10/41-44
283 10/45-48
The Admin Group defines the range of the ports that are going to be used by the EtherChannel. In Example 4-15, the Admin Group is 275, which covers the ports that fall in the range of 13-16. Ports 10/9-12 are configured as a channel using Admin Group 274. If a new EtherChannel needs to be configured on ports 10/11-14, two separate channels would form because the ports are part of two different Admin Groups. The Admin Group for an EtherChannel needs to be the same. The Admin Group can be reassigned with the following command in Example 4-16 to allow ports 10/11-14 to be in a single EtherChannel.
Example 4-16. Output of the set port channel Command
Switch1 (enable) set port channel 10/10-14 ? <admin_group> Admin group mode Channel mode
Example 4-17 shows how to configure ports 13-16 on module 10 for EtherChannel.
Example 4-17. Enabling a 4-Port EtherChannel
Switch1 (enable) set port channel 10/13-16 mode desirable
The show port channel command shown in Example 4-18 shows ports that are configured for channeling. The Admin Group is 275, and the Channel ID for the EtherChannel is 871.
Example 4-18. Viewing EtherChannel Configuration
Switch1 (enable) show port channel
Port Status Channel Admin Ch
Mode Group Id
----- ---------- -------------------- ----- -----
10/13 connected desirable silent 275 871
10/14 connected desirable silent 275 871
10/15 connected desirable silent 275 871
10/16 connected desirable silent 275 871
----- ---------- -------------------- ----- -----
Port Device-ID Port-ID Platform
----- ------------------------------- ------------------------- ---------------
10/13 TBA04081025(Switch#2) 3/13 WS-C6506
10/14 TBA04081025(Switch#2) 3/14 WS-C6506
10/15 TBA04081025(Switch#2) 3/15 WS-C6506
10/16 TBA04081025(Switch#2) 3/16 WS-C6506
Channel ID distinguishes between different EtherChannels on the switch. Silent/Non-Silent modes are involved with unidirectional link failures. It is possible for a fiber connection to remain up even if one of its transceivers has become faulty. Non-Silent mode detects faulty RX/TX transceivers on a fiber port. PAgP will reset the port for 1.6 seconds to force the other side to shut down as well. A unidirectional link can cause black holing of traffic because the return traffic is not received by the RX transceiver. The detection of unidirectional link by a PAgP is about 3.5 * 30 seconds. On the other hand, UniDirectional Link Detection (UDLD) can detect the failure less than 50 seconds versus PAgP. UDLD will be discussed later in Chapter 10, "Implementing and Tuning Spanning Tree." In Silent mode, PAgP does not look for faulty transceivers. The recommendation is to leave Silent/Non-Silent modes at their default values because UDLD better addresses this problem. However, for devices that do not support UDLD, configure Non-Silent mode. Example 4-19 provides useful information about the channel that was created.
Example 4-19. Output from the show port channel information Command
Switch1 (enable) show port channel information
Switch Frame Distribution Method: ip both
Port Status Channel Admin Channel Speed Duplex VLAN
mode group id
----- ---------- -------------------- ----- ------- ----- ------ ----
10/13 connected desirable silent 275 871 a-100 a-full 1
10/14 connected desirable silent 275 871 a-100 a-full 1
10/15 connected desirable silent 275 871 a-100 a-full 1
10/16 connected desirable silent 275 871 a-100 a-full 1
----- ---------- -------------------- ----- ------- ----- ------ ----
Port ifIndex Oper-group Neighbor Oper-Distribution PortSecurity/
Oper-group Method Dynamic port
----- ------- ---------- ---------- ----------------- -------------
10/13 132 49 65 ip both
10/14 132 49 65 ip both
10/15 132 49 65 ip both
10/16 132 49 65 ip both
----- ------- ---------- ---------- ----------------- -------------
The key point here is the load-balancing policy used by the switch. According to Example 4-19, the method used for load balancing is IP. To find out which IP address pairing is using a specific link on the EtherChannel, use the hidden command, as shown in Example 4-20. The show bundle hash provides the same information in the newer codes.
Example 4-20. Output of show bundle hash Command
Switch1 (enable) show bndlhash 871 10.1.11.3 10.1.34.4
Selected port: 10/14
As noted, for source 10.1.11.3 to get to destination 10.1.34.4, it must use the 10/14 link of the EtherChannel.
The show channel traffic command shown in Example 4-21 provides utilization information on each of the EtherChannel links.
Example 4-21. Output of the show channel traffic Command
Switch1 (enable) show channel traffic ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst ------ ----- ------- ------- ------- ------- ------- ------- 869 10/11 26.08% 0.00% 20.10% 51.70% 0.00% 26.13% 869 10/12 17.39% 40.00% 19.57% 25.69% 0.00% 5.68% 869 10/13 30.43% 60.00% 40.21% 11.14% 100.00% 64.77% 869 10/14 26.10% 0.00% 20.12% 11.45% 0.00% 3.40%
For troubleshooting purposes, it is important to note if the switch is sending and receiving PAgP packets on the wire as revealed in Example 4-22. This is one of the first commands that needs to be looked at to ensure that adjacent devices configured for EtherChannel support PAgP, and/or the devices are configured correctly.
Example 4-22. Output of the show port channel statistics Command
Switch1 (enable) show port channel statistics
Port Admin PAgP Pkts PAgP Pkts PAgP Pkts PAgP Pkts PAgP Pkts PAgP Pkts
Group Transmitted Received InFlush RetnFlush OutFlush InError
----- ------- ----------- --------- --------- --------- --------- ---------
10/13 275 180 149 0 0 0 0
10/14 275 181 150 0 0 0 0
10/15 275 148 130 0 0 0 0
10/16 275 152 133 0 0 0 0
----- ------- ----------- --------- --------- --------- --------- ---------
Example 4-23 is a hidden command on the Catalyst switch. The show agport command provides the assignment of the logical port, 14/39.
Example 4-23. Output of the show agport Command
Switch1 (enable) show agport
--- 14/39 ---
old_mem_cnt = 0; path_cost = 8; path_VLAN_cost = 0
trunk_id = 870, time_stamp = 233242506, agifindex = 132
chnl_list = 10/13-16
agport_list = 10/13-16
bndlctrl: prtcnt = 4, num_map = f0, dist_req = 2, dist_port = 0
mod port bndl_port bndl_sel bndl_sel* act_flag no_bits
10 13 0 c0 c0 2 2
10 14 1 30 30 2 2
10 15 2 0c 0c 2 2
10 16 3 03 03 2 2
0 0 0 00 00 0 0
0 0 0 00 00 0 0
0 0 0 00 00 0 0
0 0 0 00 00 0 0
The agport_list parameter shows the active ports on the channel. For instance, if the 10/13 link is lost, the agport_list will take the port out from the list. Notice that in Example 4-24, agport_list does not have port 10/13 as member of the channel.
Example 4-24. Output of show agport with Link 10/13 Nonfunctional
Switch1 (enable) show agport --- 14/39 --- old_mem_cnt = 0; path_cost = 8; path_VLAN_cost = 0 trunk_id = 870, time_stamp = 233242506, agifindex = 132 chnl_list = 10/13-16 agport_list = 10/14-16 bndlctrl: prtcnt = 4, num_map = f0, dist_req = 2, dist_port = 0 mod port bndl_port bndl_sel bndl_sel* act_flag no_bits 10 13 0 00 00 0 0 10 14 1 30 30 2 2 10 15 2 8c 8c 2 3 10 16 3 43 43 2 3 0 0 0 00 00 0 0 0 0 0 00 00 0 0 0 0 0 00 00 0 0 0 0 0 00 00 0 0
In On mode, the agport_list field is never adjusted because PAgP is disabled. Remember that PAgP is responsible for the addition and deletion of links on the channel.
From spanning tree's perspective, the EtherChannel is seen as a single port, as shown in Example 4-25.
Example 4-25. Output of the show spantree Command Using an EtherChannel
Switch1 (enable) show spantree
VLAN 1
Spanning tree mode PVST+
Spanning tree type ieee
Spanning tree enabled
Designated Root 00-05-74-18-04-80
Designated Root Priority 4097
Designated Root Cost 0
Designated Root Port 1/0
Root Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec
Bridge ID MAC ADDR 00-05-74-18-04-80
Bridge ID Priority 4097 (bridge priority: 4096, sys ID ext: 1)
Bridge Max Age 20 sec Hello Time 2 sec Forward Delay 15 sec
Port VLAN Port-State Cost Prio Portfast Channel_id
------------------------ ---- ------------- --------- ---- -------- ----------
10/13-16 1 forwarding 8 32 disabled 871
The valid EtherChannel configurations are
Desirable-Desirable
Desirable-Auto
On-On
Off-Off
Cisco recommends Desirable-Desirable mode configuration for EtherChannel. This is beneficial because ports will actively negotiate setting up a channel and will allow the operation of PAgP. It is also recommended to leave Silent/Non-Silent parameters to their default values if UDLD is supported.
Table 4-3 describes the type of channel states that will develop depending on the configuration of the adjacent switches. Spanning tree shuts down (errdisable) channels that are misconfigured, as noted in Table 4-3.
Switch-A Channel Mode | Switch-B Channel Mode | Channel State |
|---|---|---|
On | On | Channel |
On | Off | Not Channel (errdisable) |
On | Auto | Not Channel (errdisable) |
On | Desirable | Not Channel (errdisable) |
Off | On | Not Channel (errdisable) |
Off | Off | Not Channel |
Off | Auto | Not Channel |
Off | Desirable | Not Channel |
Auto | On | Not Channel (errdisable) |
Auto | Off | Not Channel |
Auto | Auto | Not Channel |
Auto | Desirable | Channel |
Desirable | On | Not Channel (errdisable) |
Desirable | Off | Not Channel |
Desirable | Auto | Channel |
Desirable | Desirable | Channel |
