Five Steps of IPSec Revisited

Five Steps of IPSec Revisited

This section discusses the individual steps required for a successful IPSec data exchange in greater detail. While IPSec incorporates many component technologies and offers multiple encryption options, the basic operation can be broken down into the following five main steps. Figure 9-17 show a graphical representation of the IPSec process.

Step 1

Interesting traffic initiates an IPSec session—Access lists interpret IPSec security policy to determine which traffic will be protected by IPSec.

Step 2

IKE Phase One—IKE authenticates peers and negotiates IKE SAs to determine if a secure channel can be established between the peers.

Step 3

IKE Phase Two—IKE negotiates the stricter IPSec SA parameters between the peers.

Step 4

IPSec data transfer—Qualifying data is transferred between IPSec peers.

Step 5

IPSec tunnel termination—IPSec session terminates through deletion or by timing out.

Click To expand
Figure 9-17: IPSec session five steps

Step 1—Determine Interesting Traffic

Data communications covers a wide gamut of topics, sensitivity, and security requirements. Just as all rumors you hear aren’t worth repeating, often much network traffic isn’t worth securing, such as an employee’s personal web browsing. Any security measures can come at the expense of network resources and performance, so a method must exist to determine what traffic is protected and what traffic travels normally through the public network.

The VPN security policy generally determines which type of traffic is considered “interesting” and, therefore, protected. How the VPN security policy gets implemented depends on the device platform. Cisco routers and the PIX Firewall use access lists to define the traffic to secure. The access lists are then incorporated in a crypto policy, which causes traffic associated with permit statements to be encrypted, while traffic associated with deny statements is sent unencrypted.

Cisco VPN Client and some VPN management software tools allow the use of menu windows to define connections to be secured by IPSec.

Both methods of identifying interesting traffic are covered in Chapters 10 through 16.

Step 2—IKE Phase One

IKE is a key management protocol standard used in conjunction with IPSec. While IPSec can be configured without IKE, the use of IKE enhances the IPSec with additional features and makes it scalable. IKE authenticates each peer in an IPSec session, automatically negotiates two levels of SAs, and handles the exchange of session keys. This is accomplished in two phases: Phase One and Phase Two.


IKE was formerly known as the Internet Security Association Key Management Protocol/Oakley (ISAKMP/Oakley) or ISAKMP. In Cisco documents, it’s typically referred to as IKE, but configuration commands still use the ISAKMP keyword. Just recognize that the two mean the same thing, but might not be interchangeable.

IKE Phase One’s main purpose is to authenticate the IPSec peers and to set up a secure channel between the peers. To accomplish this, IKE Phase One performs the following functions:

  • Authenticates while protecting the identities of the IPSec peers

  • Negotiates a common IKE SA policy between peers to protect the IKE exchange

  • Performs a DH exchange to create shared secret keys

  • Sets up a secure tunnel to negotiate IKE Phase Two parameters

IKE Phase One occurs in two modes: Main mode and Aggressive mode.

Main Mode

Main mode has three two-way exchanges between the peers to create the secure connection and develop the common SAs, while protecting the identities of the IPSec peers.

  • First exchange The security algorithms and hash methods to be used to secure the IKE exchanges are agreed on to create the common IKE SA for each peer.

  • Second exchange A DH exchange is performed to generate shared secret keying material to be used by each peer to generate shared secret keys. Nonces, pseudorandom numbers, are sent to the other peer, signed, and returned to prove their identity.

  • Third exchange The peer’s identity is verified using the peer’s IP address or fully qualified domain name (FQDN), such as, in encrypted form.

The resulting IKE SA in each peer is bidirectional and specifies IKE exchange choices for the authentication method, encryption and hash algorithms, DH group, the lifetime of the IKE SA in seconds or kilobytes, and the shared secret key values for the encryption algorithms.

Aggressive Mode

As the name implies, in the Aggressive mode, only a single three-way exchange is performed. In the initial exchange, the sender defines the proposed IKE SA values and adds their DH public key, a nonce to be signed by the other party, and an identity packet to verify their identity via a third party.

The peer sends everything back that’s required to complete the exchange, plus their DH public key. The only thing left is for the initiator to confirm the exchange.

While Aggressive mode is faster than Main mode, it exposes the peers to discovery because both sides have exchanged information before you have a secure channel. The possibility exists to sniff the wire and discover who formed the new SA.

Peer Authentication

Would-be IPSec peers must authenticate themselves to each other before IKE can proceed. IKE Phase One has three methods to authenticate IPSec peers in Cisco products. The two peers must negotiate a common authentication protocol from the following choices:

  • Preshared keys—A key value entered into each peer manually (out of band) and used to authenticate the peer.

  • RSA signatures—Uses a digital certificate authenticated by an RSA signature.

  • RSA encrypted nonces—Uses RSA encryption to encrypt a nonce value (a random number generated by the peer) and other values.

A common value used by all authentication methods to help identify the peer is the peer identity (ID). Some ID values include the peer’s IP address or their FQDN, such as

Preshared Key Authentication

With this method, the same preshared key is configured on each IPSec peer. These IKE peers can authenticate each other by generating a hash of their ID, plus the key appended to the ID before transmission. If the receiving peer can create the same hash value using its preshared key, it authenticates the source peer. Figure 9-18 shows the preshared key authentication exchange.

Click To expand
Figure 9-18: Preshared key authentication exchange

While preshared keys are easier to configure than configuring the IPSec policy values on each IPSec peer, they don’t scale well and potential security risks can be associated with getting the key to the peer.

RSA Signature Authentication

RSA Signature Authentication is a public-key cryptosystem supported by IPSec for IKE Phase One authentication. This technology was developed in 1977 by Ron Rivest, Adi Shamir, and Leonard Adleman. RSA is the first letter of each developer’s last name.

RSA signatures use a CA to generate a unique digital-identity certificate for each peer for authentication. The digital-identity certificate is similar in function to the pre-shared key, but provides stronger security.

To authenticate for an IKE session, the initiator and responder send each other their ID values, identity digital certificates, and an RSA signature value consisting of various IKE values, all encrypted using the negotiated IKE encryption method (DES or 3DES). Figure 9-19 shows an RSA signature authentication exchange.

Click To expand
Figure 9-19: RSA signature authentication exchange
RSA Encryption Authentication

The RSA-encrypted nonces authentication method uses the RSA encryption public key cryptography algorithm. This technology requires that each party generate a pseudorandom number (a nonce) and encrypt it (and possibly other publicly and privately available information), using the other party’s RSA public key. Authentication occurs when each party decrypts the other party’s nonce with its local private key, and then uses the decrypted nonce to compute a keyed hash.

A drawback to this method is this: either side of the exchange can plausibly deny that it took part in the exchange. Cisco IOS software is the only Cisco product that supports this authentication method. Figure 9-20 shows an RSA encryption authentication exchange.

Click To expand
Figure 9-20: RSA encryption authentication exchange
CAs and Digital Certificates

CAs and Digital Certificates are covered in greater detail in Chapter 11, but for our purposes here, they represent a digital identification system whereby an independent third party vouches for them. Conceptually, this is similar to most state driver’s licenses when used as identification. The merchant accepting the ID accepts that the state has performed adequate verification to be reasonably sure the identity is valid.

The distribution of security keys through an untrusted network, such as the Internet, involves questionable levels of risk.

CAs are trusted third-party organizations, such as Verisign, Entrust, and Netscape, which provide digital certificates. The peers wanting to use digital certificates register with a CA. Once the CA verifies the client’s credentials, a digital certificate is issued. The digital certificate contains the certificate bearer’s identity (name or IP address), the certificate serial number, the certificate expiration date, and a copy of the certificate bearer’s public key.

The digital certificate standard format is defined in the X.509 specification. Cisco supports X.509 version 3.

Step 3—IKE Phase Two

IKE Phase Two has only one mode, Quick mode, which occurs after IKE has established the secure tunnel in Phase One. In Quick mode, IKE

  • Negotiates a shared IPSec policy

  • Establishes IPSec SAs

  • Derives shared secret keys used for the IPSec security algorithms

To negotiate the IPSec SAs, the sender forwards one or more transformsets, including transform combinations and related settings that represent the sender’s security requirements/preferences for the new IPSec session. The receiving peer compares these requirements to its own transform sets (requirements/preferences). If one matches, then the recipient returns that single transform set, indicating a mutually agreed on transform and algorithms for the IPSec session.

Quick mode is also used to renegotiate a new IPSec SA any time the IPSec SA lifetime expires. If PFS (next section) isn’t specified, Quick mode refreshes the key generation material used to create the shared secret keys derived from the DH exchange in Phase One.

Perfect Forward Secrecy (PFS)

When Perfect Forward Secrecy (PFS) is specified in the IPSec policy, a new DH exchange is performed for each Quick mode, providing new key material and, thereby, greater resistance to cryptographic attacks. Because each DH exchange requires large exponentiations, this option increases CPU use and can reduce performance.


Quick mode exchanges nonces to provide replay protection. These nonces are used to generate new, shared, secret key material and to prevent replay attacks from generating bogus SAs.

Step 4—IPSec Data Transfer

Information is exchanged via the IPSec session based on the method for defining interesting traffic. Packets are encrypted and decrypted at the IPSec peers using any encryption specified in the IPSec SA.

Step 5—Session Termination

The IPSec session can be terminated because the traffic ended and the IPSec SA was deleted or the SA can time –out based on either SA lifetime setting. The SA timeout can be after a specified number of seconds or a specified number of bytes passed through the connection. The keys are discarded when SAs terminate, requiring IKE to perform a new Phase Two and, possibly, a new Phase One negotiation. New SAs can be established before the current ones expire, maintaining uninterrupted data flows.

Part III: Virtual Private Networks (VPNs)