Understanding the Need for Cisco Express Forwarding

To appreciate the advantages offered by Cisco Express Forwarding (CEF), it is important to understand some of the shortfalls of flow-based switching mechanisms, such as MLS, described in the previous sections. MLS works well for the most part. After the first packet, all other packets are hardware switched. The switching of the packet at hardware provides higher performance many times over than software-based switching done by a router. The problem with MLS implementation is that the first packet always hits the router. Any network changes caused by neighbor resets, route flaps, and aging timers can cause the MLS table for those specific entries to get flushed. This puts unnecessary burden on the router to perform the initial packet switching again. If the numbers of these network changes are high enough in a short period of time, there would be a massive performance hit on the network. The router will be churning to repopulate the MLS table. The aging timers can also affect the performance of MLS. If there are too many flows to different destinations, many MLS entries are created. A problem occurs when these flows are short lived, therefore causing some flows to be in the MLS table for a longer period than they need to. At the same time, new flows are constantly being created, causing a bigger MLS table.

Another issue with MLS-based switching is limited storage space. Depending on what type of flow mask is used, the MLS entries might get filled up. A ceiling issue exists with enabling MLS. Also, the algorithms used for lookups are not as efficient, which is why CEF was introduced.

CEF-based switching does not rely on the router to handle the first packet. In the Catalyst 6500 with a Supervisor II/MSFC2, for the most part, the MSFC2 (router component) does not software switch any data packets (with the exception of subnet broadcasts, NAT'd packets, unsupported route-map statements, and a few other cases). All data packets are handled by the PFC2 hardware. The only packets that hit the MSFC are those that are destined to the router, such as routing updates, SNMP queries, and any other control traffic.

The two major components of CEF are Forwarding Information Base (FIB) and the adjacency table. These two tables make CEF scalable and robust. The FIB table is an exact replica of the routing table. Any additions or deletions in the routing table are also changed on the FIB table. The FIB table consists of four levels of hierarchy to correspond with the 4 bytes of an IPv4 address. CEF searches for the longest match in the hierarchical structure to switch the packet. The Catalyst 6500 with Supervisor II can store up to 256 K FIB entries, plus 16 K multicast entries. The adjacency table maps Layer 2 MAC addresses to the associated Layer 3 IP addresses. The combination of the two allows for packets to be switched at a high rate.

The MSFC2 in Supervisor II obtains routing information from its routing protocol peers, and it creates the appropriate adjacencies for the corresponding IP address. After having built these two tables, it pushes them down to the PFC2 card. The PFC2 card now handles all data switching for the MSFC2. The MSFC2 job at this point is minimal. It primarily handles control traffic that is essential for network connectivity. The Supervisor II/PFC2/MSFC2 only operates in CEF mode.

The following process outlines a packet as it traverses a Supervisor II using CEF (see Figure 6-10):

Host1 sends traffic to Host2 that resides on a separate VLAN.
The Layer 2 and Layer 3 forwarding engines receive packet header from the dBus.
The Layer 2 engine does a lookup for the 6-byte destination MAC address, checks for input quality of service (QoS)/ACL/Security ACL. The Layer 3 engine simultaneously does a FIB and NetFlow table lookup. Some of the tasks are undone simultaneously by the engines to reduce packet latency on the switch.
The results from QoS/ACL/Security ACL are forwarded to Layer 3 by the Layer 2 engine. In parallel, the Layer 3 engine sends to Layer 2 the destination VLAN for the packet.
Layer 3 performs adjacency lookup. Layer 2 does any outbound QoS/ACL/Security ACL.
Layer 2 QoS/ACL/Security ACL information is sent to the Layer 3 engine. The Layer 3 engine will implement specific policy such as filter, QoS, and so on for the packet.
Layer 3 computes the rewrite result and forwards to Layer 2. Layer 3 updates adjacency and NetFlow table statistics. These steps occur in parallel.
The Layer 2 engine does a lookup on the destination MAC address that it receives from Layer 3. Layer 2 chooses between its result and the one it received from Layer 3 and forwards on the rBUS.

Figure 6-10. Catalyst 6500 with Supervisor II

[View full size image]

There are a handful of commands available both on the MSFC2 and Supervisor II to view CEF-related information. For example, the adjacency command provides information about the IP address and the corresponding MAC address. Most of the output in Example 6-11 is obvious with the exception of adjacency type.

Example 6-11. CEF Adjacency


Switch1 (enable) show mls enable cef adjacency

Mod:                 16

Destination-IP:      10.1.3.40          Destination-Mask:   255.255.255.255

FIB-Type:            resolved

AdjType  NextHop-IP      NextHop-Mac       Vlan Encp Tx-Packets   Tx-Octets

-------- --------------- ----------------- ---- ---- ------------ -------------

connect  10.1.3.40       00-10-f6-b3-48-00    3 ARPA            5           500

There are five different types of adjacency states, as shown in Table 6-1.

Table 6-1. Five Types of Adjacency States
Adjacency Type	Description
Connect	Complete rewrite information. Most entries will be connected.
Punt	Traffic is software switched by the MSFC2.
No R/W	Rewrite information is incomplete and must be handled by MSFC2.
FRC DRP	Entry used to drop packets because of ARP throttling.
Drop, Null, Loopbk	Entries used to drop packets.

NOTE

Do not get confused with the word MLS on the Catalyst 6500 with Supervisor II/MSFC2. The switching is CEF not MLS.

The FIB table is listed in Example 6-12 with relevant next hops.

Example 6-12. MLS Table


Switch1 (enable) show mls entry

Mod FIB-Type  Destination-IP  Destination-Mask NextHop-IP      Weight

--- --------- --------------- ---------------- --------------- ------

16 resolved  10.1.3.40       255.255.255.255  10.1.3.40            1

16 resolved  10.1.3.2        255.255.255.255  10.1.3.2             1

 16 resolved  10.1.3.3        255.255.255.255  10.1.3.3             1

The volume of traffic through the switch can be ascertained through the show mls cef command. As noted in Example 6-13, IPX and IP multicast are also hardware switched.

Example 6-13. MLS Statistics


Switch1 (enable) show mls cef

Total L3 packets switched:              123504682

Total L3 octets switched:            183111880220

Total route entries:                           38

IP route entries:                            37

IPX route entries:                            1

IPM route entries:                            0

IP load sharing entries:                        0

IPX load sharing entries:                       0

Forwarding entries:                            11

Bridge entries:                                24

Drop entries:                                   3

The output in Example 6-14 was gathered from the MSFC2. The MAC address, 0004C0D0AC38, is the actual Layer 2 address of the host machine. The router MAC address along with 0800 makes the number 0005741804BC0800 as listed in the output. All this information is needed for the rewrite information. The Cisco IOS CEF commands have information on CEF summary information, next hop addresses, and so on.

Example 6-14. Adjacency Information Off the Router


msfc_15#show adjacency detail

Protocol Interface                 Address

IP       Vlan3                     10.1.3.2(7)

0 packets, 0 bytes

0004C0D0AC38

0005741804BC0800

                                   ARP        02:57:46

CEF addresses numerous issues that MLS could not. First and foremost, it is more scalable than MLS. Enterprise networks have grown in size and the types of services it has made available to its users in the past three years. The creation and deletion of a high number of flows affected the performance of the switch, and subsequently created greater latency on the network. Therefore, CEF had to address scalability issues. The second major problem was the flow-based switching algorithm's inefficient cache table lookup. CEF, as mentioned earlier, offers a better mechanism to do lookups and allows for latency to be low. The obvious advantage was that most traffic never hit the router; the packets were hardware switched. CEF, as a result, addressed issues with scalability, latency, and overall robustness.