IPSec Connection Troubleshooting

Probably one of the most difficult things to troubleshoot on a router is IPSec connections that just do not want to work, no matter what you try to do. In my upcoming book The Complete Cisco VPN Configuration Guide (Cisco Press, 2005), I devote a separate chapter for troubleshooting for Cisco IOS routers, PIX firewalls, and the 3000 series concentrators. In this section, I cover only the very basics of troubleshooting IPSec VPN connections.

TIP

To troubleshoot IPSec connection problems, you must be familiar with how IPSec connections are set up and the negotiation process that occurs between peers. Many times I have used show and debug commands on Cisco devices to troubleshoot problems, only to find out that the problem I was experiencing was not showing up in the output of these commands. Of course, this has happened only a handful of times. In most instances, the output of debug commands will pinpoint the problem. Only through an intimate knowledge of IPSec was I able to diagnose these oddball problems correctly. Plus, to understand the output of the debug commands, which is quite copious, you must understand the process that IPSec goes through in establishing connections. Therefore, if you will be working with IPSec and VPNs quite a bit, I definitely recommend that you pick up a book on VPNs that explains in detail not only how IPSec connections are set up, but also how to read diagnostic information from commands such as debug.

Examining SAs

As you recall from the "Defining IKE Phase 1 Policies" section, earlier in this chapter, three connections are set up in IPSec: one bidirectional management connection during IKE Phase 1, and two unidirectional data connections during IKE Phase 2.

To view the IKE Phase 1 management connections, use the show crypto isakmp sa command. Example 19-12 shows sample show crypto isakmp sa output.

Example 19-12. Viewing the IKE Phase 1 Management Connection


Router# show crypto isakmp sa

dst         src         state      conn-id   slot

200.1.1.1   192.1.1.1   QM_IDLE          3      0

When troubleshooting, this is the first command that you should use to determine whether you have an IKE Phase 1 management connection to the remote peer. The state column in this output displays the status of the management connection. If it says QM_IDLE, this indicates that the router was successful in establishing a connection to the remote peer. Table 19-2 describes the various states of the connection that you can see in this column.

Table 19-2. IPSec Connection States
ISAKMP/IKE Mode	State	Description
IKE Phase 1 Main Mode	MM_NO_STATE	During IKE Phase 1 main mode, the management SA was created on the router, but nothing has been negotiated with the remote peer.
	MM_SA_SETUP	During IKE Phase 1 main mode, both IPSec peers successfully negotiated the IKE policy parameters.
	MM_KEY_EXCH	During IKE Phase 1 main mode, the DH exchange occurred, and a shared secret key was generated.
	MM_KEY_AUTH	During IKE Phase 1 main mode, the authentication of the identity of both peers was successful, and IKE Phase 2 now can begin.
IKE Phase 1 Aggressive Mode	AG_NO_STATE	During IKE Phase 1 aggressive mode, the management SA was created on the router, but nothing has been negotiated with the remote peer.
	AG_INIT_EXCH	During IKE Phase 1 aggressive mode, both IPSec peers successfully negotiated the IKE policy parameters, and the DH exchange occurred, with a shared secret key being generated.
	AG_AUTH	During IKE Phase 1 aggressive mode, the authentication of the identity of both peers was successful, and IKE Phase 2 can begin.
IKE Phase 2 Quick Mode	QM_IDLE	This completes IKE Phase 1 and the beginning, and possibly the ending of IKE Phase 2.

To view the IPSec data that SAs built in IKE Phase 2, use the show crypto ipsec sa command. Example 19-13 shows sample output from this command.

Example 19-13. Viewing the IKE Phase 2 Data Connections


Router# show crypto ipsec sa

interface: Ethernet1                                              (1)

    Crypto map tag: IPSECMAP, local addr. 192.1.1.1



   local  ident (addr/mask/prot/port):                            (2)

                        (192.1.1.1/255.255.255.255/0/0)

   remote ident (addr/mask/prot/port):

                        (200.1.1.1/255.255.255.255/0/0)

   current_peer: 200.1.1.1

     PERMIT, flags={origin_is_acl,}

    #pkts encaps: 20, #pkts encrypt: 20, #pkts digest 20          (3)

    #pkts decaps: 20, #pkts decrypt: 20, #pkts verify 20

    #send errors 0, #rcv errors 0

     local crypto endpt.: 192.1.1.1,                              (4)

        remote crypto endpt.: 200.1.1.1

     path mtu 1500, media mtu 1500

     current outbound spi: 20890A6E



     inbound esp sas:                                             (5)

      spi: 0x257A1038(628756536)

        transform: esp-sha-hmac, esp-3des,

        in use settings ={Tunnel, }

        slot: 0, conn id: 14, crypto map: IPSECMAP

        sa timing: remaining key lifetime (k/sec): (4607999/3600)

        IV size: 8 bytes

        replay detection support: Y

     inbound ah sas:

     outbound esp sas:

      spi: 0x20890A6E(545852014)

        transform: esp-sha-hmac, esp-3des,

        in use settings ={Tunnel, }

        slot: 0, conn id: 15, crypto map: IPSECMAP

        sa timing: remaining key lifetime (k/sec): (4607999/3600)

        IV size: 8 bytes

        replay detection support: Y

     outbound ah sas:

The following is a brief explanation of the output in Example 19-13, with reference to the numbering on the right side:

On Ethernet1, the crypto map named IPSECMAP is enabled, and the IP address that this router uses for IPSec communications is 192.1.1.1.
The local and remote ident lines display the peers involved in the connection.
If the connection successfully has been set up, you should see the number of packets that were encapsulated and encrypted, that had an ICV value attached to them when set to the remote peer, and the opposite when received from the remote peer.
These are the local and remote endpoints for the IPSec connection, as well as the MTU used.
There will be two sets of SAs for each tunnel: inbound and outbound. Inbound is from the remote peer to this router. Outbound is from this router to the remote peer. For both inbound and outbound, there are two sections: one for ESP and one for AH. In this example, there is nothing listed under AH, indicating that AH is not used to protect the connection. I examine one of these sections, inbound esp sas, to explain some of the information found for a connection. First, notice that each of these connections has a unique SPI value. For the inbound ESP SA, the transform parameters used to protect it are ESP SHA and 3DES encryption. The connection mode is tunnel mode.

The show crypto engine connections command is also a useful troubleshooting tool. It enables you to examine information about the encryption and decryption process. Here are some parameters that you can use with this command:

active? View the active connections using encryption/decryption.
dh? View the DH connections.
dropped-packet? View information about dropped packets.
flow? View flow table entries.

Using debug Commands

You can use many debug commands to troubleshoot the setup and maintenance of IPSec SAs. Here is a basic list of debug commands:

debug crypto engine? Troubleshoots the encryption and decryption process by the router
debug crypto isakmp? Troubleshoots IKE Phase 1 connections
debug crypto key-exchange? Troubleshoots key exchange problems, including DH
debug crypto pki transactions? Troubleshoots public key infrastructure (PKI) certificate problems, including the enrollment and verification process
debug crypto ipsec? Troubleshoots IKE Phase 2 connections

Most IPSec problems are related to the negotiation process in IKE Phase 1, so I briefly look at the output of the debug crypto isakmp command. Example 19-14 shows sample output.

Example 19-14. Using the debug crypto isakmp Command


Router# debug crypto isakmp

15:31:51: ISAKMP (8): beginning Main Mode exchange

15:31:51: ISAKMP (8): processing SA payload. message ID = 0

15:31:51: ISAKMP (8): Checking ISAKMP transform 1 against         (1)

  priority 100 policy

15:31:51: ISAKMP: encryption 3DES-CBC

15:31:51: ISAKMP: hash SHA

15:31:51: ISAKMP: default group 2

15:31:51: ISAKMP: auth pre-share

15:31:51: ISAKMP (8): atts are acceptable. Next payload is 0      (2)

15:31:51: ISAKMP (8): SA is doing pre-shared key authentication   (3)

15:31:52: ISAKMP (8): processing KE payload. message ID = 0

15:31:52: ISAKMP (8): processing NONCE payload. message ID = 0

15:31:52: ISAKMP (8): SKEYID state generated

15:31:52: ISAKMP (8): processing ID payload. message ID = 0

15:31:52: ISAKMP (8): processing HASH payload. message ID = 0

15:31:52: ISAKMP (8): SA has been authenticated                   (4)

15:31:52: ISAKMP (8): beginning Quick Mode exchange,              (5)

  M-ID of 767162845

15:31:52: ISAKMP (8): processing SA payload.

  message ID = 767162845

15:31:52: ISAKMP (8): Checking IPSec proposal 1                   (6)

15:31:52: ISAKMP: transform 1, ESP_3DES

15:31:52: ISAKMP: attributes in transform:

15:31:52: ISAKMP: encaps is 1

15:31:52: ISAKMP: SA life type in seconds

15:31:52: ISAKMP: SA life duration (basic) of 3600

15:31:52: ISAKMP: SA life type in kilobytes

15:31:52: ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0

15:31:52: ISAKMP: authenticator is HMAC-SHA

15:31:52: ISAKMP (8): atts are acceptable.                        (7)

15:31:52: ISAKMP (8): processing NONCE payload.

  message ID = 767162845

15:31:52: ISAKMP (8): processing ID payload.

  message ID = 767162845

15:31:52: ISAKMP (8): processing ID payload.

  message ID = 767162845

15:31:52: ISAKMP (8): Creating IPSec SAs

15:31:52: inbound SA from 200.1.1.1 to 192.1.1.1

  (proxy 200.1.1.1 to 192.1.1.1 )

15:31:52: has spi 134123537 and conn_id 10 and flags 4

15:31:52: lifetime of 3600 seconds

15:31:52: lifetime of 4608000 kilobytes

15:31:52: outbound SA from 192.1.1.1 to 200.1.1.1

  (proxy 192.1.1.1 to 200.1.1.1)

15:31:52: has spi 88123189 and conn_id 11 and flags 4

15:31:52: lifetime of 3600 seconds

15:31:59: lifetime of 4608000 kilobytes

The following is a brief explanation of the output in Example 19-14, with reference to the numbering on the right side:

In this reference, the two peers have exchanged their IKE Phase 1 policies, and this router is comparing the remote peer's first policy (policy 1) to the local policy 100.
In this example, I was lucky and the first policy matched (atts are acceptable).
Because identity authentication was specified as preshare in the matching ISAKMP policy, this is being performed now.
The preshared key authentication was successful, completing IKE Phase 1.
IKE Phase 2 begins.
The crypto map transform sets are being compared between the two peers.
A match was found on the first pair of transform sets, and now one data SA is created in each direction.

Clearing Connections

Two basic clear commands exist: One deals with IKE Phase 1, and the other deals with IPSec SAs. To clear your active IKE Phase 1 management connections, use the clear iskamp sa command:


Router# clear crypto isakmp [connection_ID]

If you omit the connection_ID, all management connections are deleted. To delete a specific connection, specify the connection's connection_ID. This can be found in the conn-id column of the output of the show crypto isakmp sa command.

To remove all IPSec connections on your router, use the privileged EXEC clear crypto sa command. You should clear your connections any time you make a policy change to your IPSec configuration. Instead of deleting all of your IPSec SAs, you can modify this command by adding another parameter to restrict the connections that are deleted. Here are the variations of this command:

clear crypto sa peer {IP_address | host_name}? Removes all SAs specific to a single IPSec peer
clear crypto sa map crypto_map_name? Removes all SAs specific to the named crypto map
clear crypto sa entry dest_IP_address protocol spi_#? Removes a specific SA to a particular IPSec peer
clear crypto sa counters? Clears your SA traffic counters only (not the actual SAs)