Checking Hardware Components
After verifying that no issues can be found in software, check the hardware modules to ensure proper functioning. A physical check of the hardware can be helpful in situations where the hardware has failed. Sometimes, a faulty module may still show its LED indicator as green even though the switch software does not see the device; such inconsistency can help pinpoint the problem. It is very difficult to rely on physically checking devices because the switches are spread across the network. In most cases, the show module command is used to check the status of all the linecards in the Catalyst chassis. The status column displays whether the card is functioning, as shown in Example 12-2. If it is not, the status will indicate faulty. In older software versions, the Supervisor will show faulty if there is only one power supply in the chassis.
Example 12-2. Modules Currently Installed on the Switch
Switch2 (enable) show module Mod Slot Ports Module-Type Model Sub Status --- ---- ----- ------------------------- ------------------- --- -------- 1 1 2 1000BaseX Supervisor WS-X6K-SUP1A-2GE yes ok 15 1 1 Multilayer Switch Feature WS-F6K-MSFC no ok 3 3 48 10/100BaseTX Ethernet WS-X6248-RJ-45 no ok Mod Module-Name Serial-Num --- -------------------- ----------- 1 SAD04310JC3 15 SAD04281ARM 3 SAD03408164 Mod MAC-Address(es) Hw Fw Sw --- -------------------------------------- ------ ---------- ----------------- 1 00-01-64-71-d9-6a to 00-01-64-71-d9-6b 3.4 5.3(1) 7.6(3a) 00-01-64-71-d9-68 to 00-01-64-71-d9-69 00-01-63-29-bc-00 to 00-01-63-29-bf-ff 15 00-30-b6-3e-53-8c to 00-30-b6-3e-53-cb 2.1 12.1(8b)E1 12.1(8b)E15 3 00-30-b6-d1-5d-e8 to 00-30-b6-d1-5e-17 1.1 4.2(0.24)V 7.6(3a) Mod Sub-Type Sub-Model Sub-Serial Sub-Hw Sub-Sw --- ----------------------- ------------------- ----------- ------ ------ 1 L3 Switching Engine WS-F6K-PFC SAD04170HC1 1.1
If a problem exists with the module, the show test command provides post-diagnostic information on the module in question. Example 12-3 shows the results for module 3. The 48 ports associated with module 3 all have passed the post-diagnostic test that was performed by the switch. In situations were a port becomes faulty, a failed status will show under the appropriate port.
Example 12-3. Checking Test Results for Module 3
Switch2 (enable) show test 3 Diagnostic mode: minimal (mode at next reset: complete) Module 3 : 48-port 10/100BaseTX Ethernet Line Card Firmware Status for Module 3 : PASS Port Status : Ports 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 ----------------------------------------------------------------------------- . . . . . . . . . . . . . . . . . . . . . . . . 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ------------------------------------------------------------------------ . . . . . . . . . . . . . . . . . . . . . . . . Line Card Diag Status for Module 3 (. = Pass, F = Fail, N = N/A) Loopback Status [Reported by Module 1] : Ports 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 ----------------------------------------------------------------------------- . . . . . . . . . . . . . . . . . . . . . . . . Ports 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ----------------------------------------------------------------------------- . . . . . . . . . . . . . . . . . . . . . . . . InlineRewrite Status : InlineRewrite Test skipped as Minimal diagnostics selected
The Test feature has three modes as listed in the output from Example 12-4. By default, the switch does minimal diagnostics on each of the modules.
Example 12-4. Diagnostic Options
Switch2 (enable) set test diaglevel ?
complete Complete diagnostics
minimal Minimal diagnostics
bypass Bypass diagnostics
Sometimes, hardware goes faulty because of environmental issues such as lack of power to the switch, fan flow problems, and so on. The show environment all command outputs information on how many power supplies currently are on the chassis and the amount of amps the cards are drawing. It also provides information on the temperature of the modules in the chassis. All in all, this command can be very helpful, especially when deploying these devices in new data centers or closets. Example 12-5 shows the output from the show environment all command.
Example 12-5. Switch Environmental Levels
Switch2 (enable) show environment all
Environmental Status (. = Pass, F = Fail, U = Unknown, N = Not Present)
PS1: . PS2: N PS1 Fan: . PS2 Fan: N
Chassis-Ser-EEPROM: . Fan: .
Clock(A/B): A Clock A: . Clock B: .
VTT1: . VTT2: . VTT3: .
Intake Exhaust Device 1 Device 2
Slot Temperature Temperature Temperature Temperature
--------------- --------------- --------------- --------------- ---------------
1 23C(50C,65C) 34C(60C,75C) 26C 34C
3 26C(50C,65C) 28C(60C,75C) 29C 34C
1 (Switch-Eng) 24C(50C,65C) 29C(60C,75C) N/A N/A
1 (MSFC) 29C(50C,65C) 34C(60C,75C) N/A N/A
Chassis Modules
---------------------
VTT1: 24C(85C,100C)
VTT2: 24C(85C,100C)
VTT3: 25C(85C,100C)
PS1 Capacity: 852.60 Watts (20.30 Amps @42V)
PS2 Capacity: none
PS Configuration : PS1 and PS2 in Redundant Configuration.
Total Power Available: 852.60 Watts (20.30 Amps @42V)
Total Power Available for Line Card Usage: 852.60 Watts (20.30 Amps @42V)
Total Power Drawn From the System: 390.18 Watts ( 9.29 Amps @42V)
Remaining Power in the System: 462.42 Watts (11.01 Amps @42V)
Configured Default Inline Power allocation per port: 7.00 Watts (0.16 Amps @42V)
Slot power Requirement/Usage :
Slot Card Type PowerRequested PowerAllocated CardStatus
Watts A @42V Watts A @42V
---- ------------------- ------- ------ ------- ------ ----------
1 WS-X6K-SUP1A-2GE 138.60 3.30 138.60 3.30 ok
2 0.00 0.00 138.60 3.30 none
3 WS-X6248-RJ-45 112.98 2.69 112.98 2.69 ok
Slot Inline Power Requirement/Usage :
Slot CardType Total Allocated Max H/W Supported Max H/W Supported
To Module (Watts) Per Module (Watts) Per Port (Watts)
The next commands provide further proof if there is anything wrong with the hardware, software, and the traffic that is being handled by the switch. Most of these commands are self-explanatory. The commands mentioned in this book help form a basic template of what to look at first when troubleshooting Catalyst switches.
When verifying the functionality of the hardware, show process cpu is another helpful command that shows the current load on the Supervisor CPU and also displays what process has invoked the CPU for how long and percentage utilized. If CPU load is above 85 percent for a extended period of time, check if the switch and the network are stable.
The show scp (serial line communication) command provides information about communication issues between the line card and the Supervisor module. This is a hidden command. The output in Example 12-6 shows some of the options that are currently available on a Catalyst 6500 switch using SCP. Typically, the show scp command is used when a module has disappeared, reloaded, or crashed. You can also use the show scp command to check the reliability of communication between redundant Supervisors on the same Catalyst switch. If, for example, a standby Supervisor has disappeared, check the show scp command to see if the active Supervisor lost communication with the standby Supervisor.
Example 12-6. show scp Command Output
Switch2 (enable) show scp
Show scp commands:
----------------------------------------------------------------------------
show scp failcnt Show SCP fail count
show scp mc Show SCP Multicast groups and their members
show scp memory Show SCP memory contents
show scp module Show SCP module based statistics
show scp process Show SCP process based statistics
show scp registration Show SCP registration table
show scp traceinfo Show SCP trace
show scp statistics Show SCP statistics
The show top command can be used to analyze data collected for each port by the switch. Example 12-7 shows the type of data gathered by the show top command. In a situation where excessive traffic is seen on the network, it might be helpful to find the user responsible for the high volume of traffic. The show top command can also be used in situations where users complain about applications not working correctly or slow network response time by examining packet errors, overflow, and utilization, as shown in Example 12-7.
Example 12-7. The show top Options
Switch2 (enable) show top ?
<N> Number of physical ports (default 20)
util Sort the report based on utilization
bytes Sort the report based on tx/rx bytes
pkts Sort the report based on tx/rx packets
bcst Sort the report based on tx/rx broadcast packets
mcst Sort the report based on tx/rx multicast packets
errors Sort the report based on rx errors
overflow Sort the report based on overflow
interval Show Topn report in an interval
all Show all port type
eth Show ethernet port type
l0e Show 10 ethernet port type
fe Show fast ethernet port type
ge Show gigabit ethernet port type
10ge Show 10 gigabit ethernet port type
background Running TOPN task in background
report Show report for TOPN
The show system command displays the switch backplane utilization and its peak. It also indicates if Coredump, a log file generated when the switch crashes, has been enabled on the switch. In most circumstances, switch backplane utilization should not be higher than 7 percent. If it is higher, a closer examination of the network and the switch is required. A common source of high backplane utilization has been spanning tree. Make sure that there is no spanning-tree loop occurring in the network.
The next section looks at some of the more common problems seen on Catalyst switches, starting with Supervisor missing, MSFC missing, and specific port issues.
Supervisor Missing
One of the more frustrating troubleshooting events is when a Supervisor disappears from the chassis. It is usually the standby Supervisor. To fix the problem, you must have console access to the Catalyst switch. Typically, a bad configuration is involved; for example, the boot variable field may have an incorrect software filename. As a result, the Supervisor is in ROMMON mode. To help mitigate boot issues, first do a dir bootflash to see what Catalyst OS software is available on the box and then select and boot the file using the boot bootflash command.
A supervisor can also be forced into ROMMON mode if the Catalyst OS image is corrupted. Two solutions can be implemented to fix this problem:
Load the desired image on a PCMCIA card from a different switch and then place the PCMCIA card on the bad supervisor. Next, attempt to boot the image using the boot disk command.
Move the Supervisor to a working Catalyst switch and let it synchronize with that switch's supervisor.
Software caveats in rare situations can also cause a Supervisor to disappear from the chassis. A quick search through available Catalyst OS caveats can help determine potential switching problems.
Like any other hardware product, sometimes the device breaks down. A quick check of the light emitting diodes (LEDs) provides the status of the module. If no LED lights, the Supervisor more than likely is faulty. Table 12-1 shows LED states and their meaning.
LED | Color/State | Description |
|---|---|---|
STATUS | Green | All diagnostics pass. The module is operational (normal initialization sequence). |
Orange | The module is booting or running diagnostics (normal initialization sequence). An overtemperature condition has occurred. (A minor temperature threshold has been exceeded during environmental monitoring.) | |
Red | The diagnostic test failed. The module is not operational because a fault occurred during the initialization sequence. An overtemperature condition has occurred. (A major temperature threshold has been exceeded during environmental monitoring.) | |
SYSTEM[1] | Green | All chassis environmental monitors are reporting OK. |
Orange | The power supply has failed or the power supply fan has failed. Incompatible power supplies are installed. The redundant clock has failed. | |
Red | Two VTT modules fail or the VTT module temperature major threshold has been exceeded.[2] The temperature of the Supervisor engine major threshold has been exceeded. | |
ACTIVE | Green | The Supervisor engine is operational and active. |
Orange | The Supervisor engine is in standby mode. | |
PWR MGMT[3] | Green | Sufficient power is available for all modules. |
Orange | Sufficient power is not available for all modules. | |
SWITCH LOAD | If the switch is operational, the switch load meter indicates (as an approximate percentage) the current traffic load over the backplane. | |
PCMCIA | The PCMCIA LED is lit when no Flash PC card is in the slot, and it goes off when you insert a Flash PC card. | |
LINK | Green | The port is operational. |
Orange | The link has been disabled by software. | |
Flashing orange | The link is bad and has been disabled because of a hardware failure. | |
Off | No signal is detected. |
[1] The SYSTEM and PWR MGMT LED indications on a redundant Supervisor engine are synchronized to the active Supervisor engine.
[2] VTT = voltage termination module. The VTT module terminates signals on the Catalyst switching bus.
[3] If no redundant Supervisor engine is installed and there is a VTT module minor or major overtemperature condition, the system shuts down.
The show test command, as mentioned earlier in the "Checking Hardware Components" section, will also provide information about the status of the hardware. A failed status appears if the Supervisor is faulty, as shown in Example 12-8.
Example 12-8. Supervisor Failure
Module 1
Earl VI Status :
NewLearnTest: F
IndexLearnTest: F
DontForwardTest: U
DontLearnTest: U
ConditionalLearnTest: F
BadBpduTest: U
TrapTest: U
MatchTest: U
PortSpanTest: U
CaptureTest: U
ProtocolMatchTest: U
ChannelTest: F
IpFibScTest: F
IpxFibScTest: F
L3DontScTest: F
L3Capture2Test: F
L3VlanMetTest: F
AclPermitTest: F
AclDenyTest: .
InbandEditTest: F
RWEngineSpanTest: F
ForwardingEngineTest: F
Loopback Status [Reported by Module 1] :
Ports 1 2
-----------
F F
MSFC Missing
The same logic and reasoning that was used in troubleshooting Supervisor modules applies here as well. The first thing to do when the MSFC disappears is to console into it. The switch console command, running Catalyst OS software, allows the console screen to switch from the Supervisor to the local MSFC. In other words, if you are consoled in Supervisor module 1, you can console to MSFC 15, but not MSFC 16, which is on Supervisor module 2. Execute Ctrl^C^C^C to switch back to the Supervisor. The following steps outline how to recover an MSFC missing from the chassis:
- Step 1. Switch from the Supervisor console to the MSFC console:
Switch1 (enable) switch console Trying Router-15... Connected to Router-15. Type ^C^C^C to switch back... rommon 8 > - Step 2. Check what IOS images are currently available on the bootflash:
rommon 3 > dir bootflash: File size Checksum File name 1688336 bytes (0x19c310) 0x786a3e5f c6msfc2-boot-mz.121-8b.E11 12278476 bytes (0xbb5acc) 0x91074cd4 c6msfc2-jsv-mz.121-8b.E11 - Step 3. After selecting which IOS image to use, use the boot bootflash command to load the image on the MSFC:
rommon 4 > boot bootflash:c6msfc2-jsv-mz.121-8b.E11
Use the show module command to verify that the MSFC does, in fact, appear in the chassis.
