Checking Hardware Components

After verifying that no issues can be found in software, check the hardware modules to ensure proper functioning. A physical check of the hardware can be helpful in situations where the hardware has failed. Sometimes, a faulty module may still show its LED indicator as green even though the switch software does not see the device; such inconsistency can help pinpoint the problem. It is very difficult to rely on physically checking devices because the switches are spread across the network. In most cases, the show module command is used to check the status of all the linecards in the Catalyst chassis. The status column displays whether the card is functioning, as shown in Example 12-2. If it is not, the status will indicate faulty. In older software versions, the Supervisor will show faulty if there is only one power supply in the chassis.

Example 12-2. Modules Currently Installed on the Switch

Switch2 (enable) show module

Mod Slot Ports Module-Type               Model               Sub Status

--- ---- ----- ------------------------- ------------------- --- --------

1   1    2     1000BaseX Supervisor      WS-X6K-SUP1A-2GE    yes ok

15  1    1     Multilayer Switch Feature WS-F6K-MSFC         no  ok

3   3    48    10/100BaseTX Ethernet     WS-X6248-RJ-45      no  ok

Mod Module-Name          Serial-Num

--- -------------------- -----------

1                        SAD04310JC3

15                       SAD04281ARM

3                        SAD03408164

Mod MAC-Address(es)                        Hw     Fw         Sw

--- -------------------------------------- ------ ---------- -----------------

1   00-01-64-71-d9-6a to 00-01-64-71-d9-6b 3.4    5.3(1)     7.6(3a)

    00-01-64-71-d9-68 to 00-01-64-71-d9-69

    00-01-63-29-bc-00 to 00-01-63-29-bf-ff

15  00-30-b6-3e-53-8c to 00-30-b6-3e-53-cb 2.1    12.1(8b)E1 12.1(8b)E15

3   00-30-b6-d1-5d-e8 to 00-30-b6-d1-5e-17 1.1    4.2(0.24)V 7.6(3a)

Mod Sub-Type                Sub-Model           Sub-Serial  Sub-Hw Sub-Sw

--- ----------------------- ------------------- ----------- ------ ------

1   L3 Switching Engine     WS-F6K-PFC          SAD04170HC1 1.1

If a problem exists with the module, the show test command provides post-diagnostic information on the module in question. Example 12-3 shows the results for module 3. The 48 ports associated with module 3 all have passed the post-diagnostic test that was performed by the switch. In situations were a port becomes faulty, a failed status will show under the appropriate port.

Example 12-3. Checking Test Results for Module 3

Switch2 (enable) show test 3

Diagnostic mode: minimal   (mode at next reset: complete)

Module 3 : 48-port 10/100BaseTX Ethernet

Line Card Firmware Status for Module 3 : PASS

Port Status :

  Ports 1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21 22 23 24


        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

        25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48


        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

Line Card Diag Status for Module 3  (. = Pass, F = Fail, N = N/A)

 Loopback Status [Reported by Module 1] :

  Ports 1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19 20 21 22 23 24


        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

  Ports 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48


        .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

 InlineRewrite Status :

        InlineRewrite Test skipped as Minimal diagnostics selected

The Test feature has three modes as listed in the output from Example 12-4. By default, the switch does minimal diagnostics on each of the modules.

Example 12-4. Diagnostic Options

Switch2 (enable) set test diaglevel ?

  complete                   Complete diagnostics

  minimal                    Minimal diagnostics

  bypass                     Bypass diagnostics

Sometimes, hardware goes faulty because of environmental issues such as lack of power to the switch, fan flow problems, and so on. The show environment all command outputs information on how many power supplies currently are on the chassis and the amount of amps the cards are drawing. It also provides information on the temperature of the modules in the chassis. All in all, this command can be very helpful, especially when deploying these devices in new data centers or closets. Example 12-5 shows the output from the show environment all command.

Example 12-5. Switch Environmental Levels

Switch2 (enable) show environment all

Environmental Status (. = Pass, F = Fail, U = Unknown, N = Not Present)

  PS1: .     PS2: N     PS1 Fan: .     PS2 Fan: N

  Chassis-Ser-EEPROM: .     Fan: .

  Clock(A/B): A         Clock A: .     Clock B: .

  VTT1: .    VTT2: .    VTT3: .

                  Intake          Exhaust         Device 1        Device 2

Slot            Temperature     Temperature     Temperature     Temperature

--------------- --------------- --------------- --------------- ---------------

1               23C(50C,65C)    34C(60C,75C)    26C             34C

3               26C(50C,65C)    28C(60C,75C)    29C             34C

1  (Switch-Eng) 24C(50C,65C)    29C(60C,75C)    N/A             N/A

1  (MSFC)       29C(50C,65C)    34C(60C,75C)    N/A             N/A

Chassis Modules


VTT1: 24C(85C,100C)

VTT2: 24C(85C,100C)

VTT3: 25C(85C,100C)

PS1 Capacity: 852.60 Watts (20.30 Amps @42V)

PS2 Capacity: none

PS Configuration : PS1 and PS2 in Redundant Configuration.

Total Power Available: 852.60 Watts (20.30 Amps @42V)

Total Power Available for Line Card Usage: 852.60 Watts (20.30 Amps @42V)

Total Power Drawn From the System: 390.18 Watts ( 9.29 Amps @42V)

Remaining Power in the System: 462.42 Watts (11.01 Amps @42V)

Configured Default Inline Power allocation per port: 7.00 Watts (0.16 Amps @42V)

Slot power Requirement/Usage :

Slot Card Type           PowerRequested PowerAllocated CardStatus

                         Watts   A @42V Watts   A @42V

---- ------------------- ------- ------ ------- ------ ----------

1    WS-X6K-SUP1A-2GE    138.60   3.30  138.60   3.30  ok

2                          0.00   0.00  138.60   3.30  none

3    WS-X6248-RJ-45      112.98   2.69  112.98   2.69  ok

Slot Inline Power Requirement/Usage :

Slot CardType            Total Allocated   Max H/W Supported  Max H/W Supported

                         To Module (Watts) Per Module (Watts) Per Port (Watts)

The next commands provide further proof if there is anything wrong with the hardware, software, and the traffic that is being handled by the switch. Most of these commands are self-explanatory. The commands mentioned in this book help form a basic template of what to look at first when troubleshooting Catalyst switches.

When verifying the functionality of the hardware, show process cpu is another helpful command that shows the current load on the Supervisor CPU and also displays what process has invoked the CPU for how long and percentage utilized. If CPU load is above 85 percent for a extended period of time, check if the switch and the network are stable.

The show scp (serial line communication) command provides information about communication issues between the line card and the Supervisor module. This is a hidden command. The output in Example 12-6 shows some of the options that are currently available on a Catalyst 6500 switch using SCP. Typically, the show scp command is used when a module has disappeared, reloaded, or crashed. You can also use the show scp command to check the reliability of communication between redundant Supervisors on the same Catalyst switch. If, for example, a standby Supervisor has disappeared, check the show scp command to see if the active Supervisor lost communication with the standby Supervisor.

Example 12-6. show scp Command Output

Switch2 (enable) show scp

Show scp commands:


show scp failcnt                Show SCP fail count

show scp mc                     Show SCP Multicast groups and their members

show scp memory                 Show SCP memory contents

show scp module                 Show SCP module based statistics

show scp process                Show SCP process based statistics

show scp registration           Show SCP registration table

show scp traceinfo              Show SCP trace

show scp statistics             Show SCP statistics

The show top command can be used to analyze data collected for each port by the switch. Example 12-7 shows the type of data gathered by the show top command. In a situation where excessive traffic is seen on the network, it might be helpful to find the user responsible for the high volume of traffic. The show top command can also be used in situations where users complain about applications not working correctly or slow network response time by examining packet errors, overflow, and utilization, as shown in Example 12-7.

Example 12-7. The show top Options

Switch2 (enable) show top ?

  <N>                        Number of physical ports (default 20)

  util                       Sort the report based on utilization

  bytes                      Sort the report based on tx/rx bytes

  pkts                       Sort the report based on tx/rx packets

  bcst                       Sort the report based on tx/rx broadcast packets

  mcst                       Sort the report based on tx/rx multicast packets

  errors                     Sort the report based on rx errors

  overflow                   Sort the report based on overflow

  interval                   Show Topn report in an interval

  all                        Show all port type

  eth                        Show ethernet port type

  l0e                        Show 10 ethernet port type

  fe                         Show fast ethernet port type

  ge                         Show gigabit ethernet port type

  10ge                       Show 10 gigabit ethernet port type

  background                 Running TOPN task in background

  report                     Show report for TOPN

The show system command displays the switch backplane utilization and its peak. It also indicates if Coredump, a log file generated when the switch crashes, has been enabled on the switch. In most circumstances, switch backplane utilization should not be higher than 7 percent. If it is higher, a closer examination of the network and the switch is required. A common source of high backplane utilization has been spanning tree. Make sure that there is no spanning-tree loop occurring in the network.

The next section looks at some of the more common problems seen on Catalyst switches, starting with Supervisor missing, MSFC missing, and specific port issues.

Supervisor Missing

One of the more frustrating troubleshooting events is when a Supervisor disappears from the chassis. It is usually the standby Supervisor. To fix the problem, you must have console access to the Catalyst switch. Typically, a bad configuration is involved; for example, the boot variable field may have an incorrect software filename. As a result, the Supervisor is in ROMMON mode. To help mitigate boot issues, first do a dir bootflash to see what Catalyst OS software is available on the box and then select and boot the file using the boot bootflash command.

A supervisor can also be forced into ROMMON mode if the Catalyst OS image is corrupted. Two solutions can be implemented to fix this problem:

  • Load the desired image on a PCMCIA card from a different switch and then place the PCMCIA card on the bad supervisor. Next, attempt to boot the image using the boot disk command.

  • Move the Supervisor to a working Catalyst switch and let it synchronize with that switch's supervisor.

Software caveats in rare situations can also cause a Supervisor to disappear from the chassis. A quick search through available Catalyst OS caveats can help determine potential switching problems.

Like any other hardware product, sometimes the device breaks down. A quick check of the light emitting diodes (LEDs) provides the status of the module. If no LED lights, the Supervisor more than likely is faulty. Table 12-1 shows LED states and their meaning.

Table 12-1. Supervisor LED






All diagnostics pass. The module is operational (normal initialization sequence).


The module is booting or running diagnostics (normal initialization sequence).

An overtemperature condition has occurred. (A minor temperature threshold has been exceeded during environmental monitoring.)


The diagnostic test failed. The module is not operational because a fault occurred during the initialization sequence.

An overtemperature condition has occurred. (A major temperature threshold has been exceeded during environmental monitoring.)



All chassis environmental monitors are reporting OK.


The power supply has failed or the power supply fan has failed.

Incompatible power supplies are installed.

The redundant clock has failed.


Two VTT modules fail or the VTT module temperature major threshold has been exceeded.[2]

The temperature of the Supervisor engine major threshold has been exceeded.



The Supervisor engine is operational and active.


The Supervisor engine is in standby mode.



Sufficient power is available for all modules.


Sufficient power is not available for all modules.



If the switch is operational, the switch load meter indicates (as an approximate percentage) the current traffic load over the backplane.



The PCMCIA LED is lit when no Flash PC card is in the slot, and it goes off when you insert a Flash PC card.



The port is operational.


The link has been disabled by software.

Flashing orange

The link is bad and has been disabled because of a hardware failure.



No signal is detected.

[1] The SYSTEM and PWR MGMT LED indications on a redundant Supervisor engine are synchronized to the active Supervisor engine.

[2] VTT = voltage termination module. The VTT module terminates signals on the Catalyst switching bus.

[3] If no redundant Supervisor engine is installed and there is a VTT module minor or major overtemperature condition, the system shuts down.

The show test command, as mentioned earlier in the "Checking Hardware Components" section, will also provide information about the status of the hardware. A failed status appears if the Supervisor is faulty, as shown in Example 12-8.

Example 12-8. Supervisor Failure

Module 1

  Earl VI Status :

        NewLearnTest:             F

        IndexLearnTest:           F

        DontForwardTest:          U

        DontLearnTest:            U

        ConditionalLearnTest:     F

        BadBpduTest:              U

        TrapTest:                 U

        MatchTest:                U

        PortSpanTest:             U

        CaptureTest:              U

        ProtocolMatchTest:        U

        ChannelTest:              F

        IpFibScTest:              F

        IpxFibScTest:             F

        L3DontScTest:             F

        L3Capture2Test:           F

        L3VlanMetTest:            F

        AclPermitTest:            F

        AclDenyTest:              .

        InbandEditTest:           F

        RWEngineSpanTest:         F

        ForwardingEngineTest:     F

 Loopback Status [Reported by Module 1] :

  Ports 1  2 


        F  F 

MSFC Missing

The same logic and reasoning that was used in troubleshooting Supervisor modules applies here as well. The first thing to do when the MSFC disappears is to console into it. The switch console command, running Catalyst OS software, allows the console screen to switch from the Supervisor to the local MSFC. In other words, if you are consoled in Supervisor module 1, you can console to MSFC 15, but not MSFC 16, which is on Supervisor module 2. Execute Ctrl^C^C^C to switch back to the Supervisor. The following steps outline how to recover an MSFC missing from the chassis:

Step 1. Switch from the Supervisor console to the MSFC console:

Switch1 (enable) switch console 

Trying Router-15...

Connected to Router-15.

Type ^C^C^C to switch back...

rommon 8 > 

Step 2. Check what IOS images are currently available on the bootflash:

rommon 3 > dir bootflash:

         File size           Checksum   File name

   1688336 bytes (0x19c310)   0x786a3e5f    c6msfc2-boot-mz.121-8b.E11

  12278476 bytes (0xbb5acc)   0x91074cd4    c6msfc2-jsv-mz.121-8b.E11

Step 3. After selecting which IOS image to use, use the boot bootflash command to load the image on the MSFC:

rommon 4 > boot bootflash:c6msfc2-jsv-mz.121-8b.E11

Use the show module command to verify that the MSFC does, in fact, appear in the chassis.