Cisco IOS IPSec Technologies

Cisco IOS IPSec Technologies

IPSec acts at the network layer, protecting and authenticating IP packets between participating IPSec peers, such as VPN software clients, VPN 3002 hardware clients, Cisco routers, PIX Firewalls, and the VPN 3000 Series Concentrators. Cisco IPSec VPNs provide the following network services:

  • Data integrity The IPSec receiver can authenticate packets sent by an IPSec sender to make sure the data hasn’t been modified during transmission.

  • Data origin authentication The IPSec receiver can authenticate the source of any IPSec packets received. This service is dependent on the data integrity service.

  • Anti-replay The IPSec receiver can detect and reject replayed packets.

  • Data confidentiality IPSec can encrypt data packets before transmitting them across a network in a form that can only be unencrypted by the destination target.

IPSec Security Overview

IPSec is a framework of several open standards that provides data confidentiality, data integrity, and data authentication in exchanges between participating VPN peers at Layer 3. IPSec technologies can be used to authenticate and protect data flows between IPSec peers. The two main IPSec security protocols are the following:

  • Authentication Header (AH)

  • Encapsulating Security Payload (ESP)

In addition, IPSec uses other existing encryption standards to complete the protocol suite. These encryption standards are covered in the section “Other IPSec Encryption Standards.”

Authentication Header (AH)

The Authentication Header (AH) security protocol provides only data authentication and integrity for IP packets forwarded between two systems. It does not provide encryption services for data confidentiality.

The AH authentication is provided by both ends of the tunnel performing a one-way hash calculation on the packet using a shared key value. The function of a message digest hash is never to be decrypted, but to provide a check value for the receiving party to verify data hasn’t been modified. While performing a hash calculation isn’t difficult, trying to modify the packet and end up with exactly the same resulting message digest is difficult. The hashing algorithm was designed to be nonpredictable and, therefore, hard to duplicate.

The fact that creating the two one-way hash calculations involves using a shared secret key known to the two systems means authenticity is guaranteed.

AH can also help to implement antireplay protection by requiring the receiving host to set the replay bit in the header to indicate the packet was seen.

AH Authentication and Integrity

The AH authentication and integrity function is applied to the entire IP packet, except for those IP header fields that must change in transit; such as the Time To Live (TTL) field, which is decremented by the routers along the path. Figure 9-6 shows the AH process diagrammatically. The AH process works as follows:

Step 1

The IPSec source device performs a hash function on the IP header and data payload using a key value known also by the destination device.

Step 2

The resulting message digest is appended to the original packet as a part of creating a new packet for transit.

Step 3

The new packet is transmitted to the IPSec peer device.

Step 4

The receiving device separates the original IP header and data payload from the message digest and it performs a hash function on the IP header and data using the key value shared with the source. If the resulting message digest doesn’t exactly match the one received from the source, it’s rejected by the destination device.

Click To expand
Figure 9-6: AH authentication and integrity process

Even a single bit change or substitution in the transmitted packet creates a different hash result and can cause the packet to be discarded. Even if the packet were captured in transit and the IP packet information was in Cleartext, the hacker doesn’t know the shared key.

Encapsulating Security Payload (ESP)

The ESP security protocol provides confidentiality via encryption, data origin authentication, data integrity, optional antireplay protection, and limited traffic flow confidentiality by defeating traffic flow analysis. Authentication and integrity can be provided via the same algorithms used by AH. Confidentiality can be implemented independent of the other services.

The ESP confidentiality is accomplished by performing encryption at the IP packet layer. ESP supports a variety of symmetric encryption algorithms, but the default for IPSec is 56-bit DES. This particular cipher must be implemented to conform to the IPSec standard and to ensure interoperability with other vendor IPSec products. Cisco products support DES plus 3DES for even stronger encryption.

DES Encryption Algorithm

Data Encryption Standard (DES) is a popular symmetric-key encryption method that uses a 56-bit key to ensure secure, high-performance encryption. The first public encryption standard, DES is based on an algorithm developed by IBM. DES is used to encrypt and decrypt packet data turning cleartext into ciphertext via an encryption algorithm. The receiving device uses a decryption algorithm and the same shared key value to restore the cleartext. Figure 9-7 shows the encryption process.

Click To expand
Figure 9-7: DES encryption process

Specialized “DES cracker” machines, while uncommon, can recover a DES key after only a few hours, so Cisco recommends 3DES as the main encryption algorithm for VPN.

Triple DES Algorithm (3DES)

Cisco products implementing IPSec can use the Triple DES (3DES) algorithm as a much stronger encryption method. 3DES is a variation of the 56-bit DES that breaks the data up into 64-bit blocks, and then processes each block three times, each time with an independent 56-bit key. This process effectively doubles encryption strength over 56-bit DES.

Both DES and 3DES offer adequate performance for production network applications. Now that DES/3DES encryption is available in ASIC hardware in products, such as the VPN 3002 Hardware Client Device and VPN 3000 Series Concentrators, you can add encryption to a VPN with little impact on overall system performance.

Advanced Encryption Standard (AES)

AES encryption technique was recently approved as a Federal Information Processing Standard (FIPS)-approved cryptographic algorithm (FIPS PUB 197). AES is based on the Rijndael (pronounced Rhine Dahl or Rain Doll) algorithm, which defines how to use 128-, 192-, or 256-bit keys to encrypt 128-, 192-, or 256-bit source blocks (all nine combinations of key length and block length are possible). AES offers greater flexibility than even 3DES because it supports multiple key sizes and multiple encoding passes.

Release 3.6 of Cisco VPN products introduce support for AES (128 and 256 bit), providing a stronger encryption standard option and improved remote access performance for both software and hardware clients. Cisco is working with the IETF IPSec Working Group to push for a new specification outlining how AES will work within the IPSec framework.

Choosing Between AH and ESP

Deciding to use AH or ESP in a particular situation might seem confusing, but remember these two basics: any time more services are required, you can expect additional resources will be consumed. The same can be expected of throughput so, if a service isn’t needed, don’t use it. Follow both of these rules:

  • If you need to make sure that data from an authenticated source is transferred with integrity, but confidentiality isn’t necessary, then use the AH protocol. This saves the additional overhead and bandwidth associated with ESP encryption.

  • If data confidentiality (privacy through encryption) is required, then ESP must be used. ESP can provide authentication and integrity for the packets via the same algorithms used by AH.

Other IPSec Encryption Standards

IPSec employs several different technologies to provide a complete system of confidentiality, integrity, and authenticity. These technologies are as follows:

  • Message Digest 5 (MD5)–A hash algorithm is used to authenticate packet data.

  • Secure Hash Algorithm-1 (SHA-1)–A hash algorithm is used to authenticate packet data.

  • Diffie-Hellman (DH)–A key exchange standard allows two parties to establish a shared secret key to be used by encryption algorithms.

  • Rivest, Shamir, and Adelman Signatures (RSA)–A public-key cryptographic system used for authentication.

  • Internet Key Exchange (IKE)–A hybrid protocol that provides setup utility services for IPSec, including authentication of the IPSec peers, negotiation of IKE and IPSec security associations (SAs), and establishment of keys for encryption algorithms used by IPSec.

  • Certificate authority (CA)–Cisco router and PIX Firewall support of CAs allows the IPSec-protected network to scale by providing the equivalent of a digital ID card for each device.


    Internet Security Association Key Management Protocol (ISAKMP) is synonymous with IKE in Cisco router or PIX Firewall configuration commands. The keyword ISAKMP is always used instead of IKE.

Transport and Tunnel Mode

In configuring IPSec, one of the early decisions that must be made is whether the session is a Tunnel or a Transport mode connection. This distinction impacts other configuration decisions that have to be made.

Transport Mode

Transport mode is used between two end-host devices or between a remote host and a gateway device, where the gateway is the actual destination device. An example of a gateway device being the target destination would involve an encrypted Telnet session to configure a router or a PIX Firewall. In either case, this is basically a one-device to one-device connection. Figure 9-8 shows two possible examples of a Transport mode connection. In VPN 1, administrator Nancy must be able to access the perimeter router from home to check the status and make any configuration changes. In VPN 2, Nancy needs to access a server to make user or group account changes. In each case, a host-to-host connection exists.

Click To expand
Figure 9-8: VPN Transport mode connections

Both examples are offered with the warning that these practices might be banned by a security policy. Allowing a VPN to pass through any perimeter router and/or firewall to get directly into the protected LAN is an especially risky proposition.

Transport Mode Encryption

In Transport mode, if encryption is performed, only the upper-layer IP protocol fields (IP packet payload) are encrypted, leaving the IP header untouched. The IP header must be left unencrypted, so the packet can be routed through the network. Any device recording a packet in transit would be unable to read the data, but could easily determine the source and destination information.

Tunnel Mode

Tunnel mode is the most common mode involving a VPN connection between two gateway devices or a connection between an end-station and a gateway device. The Tunnel mode connection would be common between a branch office and the main office providing multiple hosts on the branch LAN to multiple shared resources on the main network. An example of the end-host to gateway VPN tunnel would be a traveling employee or telecommuter connecting to the company network to access shared resources, such as e-mail, files, printing, and so forth.

In Figure 9-9, each of the connections to the main office would normally be a Tunnel mode connection between the two routers. The mobile user or telecommuter would typically have a VPN tunnel connection from their workstation to the Main Office router.

Click To expand
Figure 9-9: Typical Tunnel mode VPN connections
Tunnel Mode Encryption

The more secure Tunnel mode encrypts both the IP header and the payload. This is possible because while the packet is in transit through the tunnel, it’s fully encapsulated in a packet that uses the tunnel endpoints as the source and destination address. Any device recording a packet in transit would be unable to read any part of the original packet and could only determine the end points of the tunnel.

Tunnel Mode Benefits

Tunnel mode allows a router or VPN hardware host device to act as an IPSec proxy, which means the device performs encryption services for the hosts. The Tunnel mode endpoint device is used to protect datagrams that originate from or are destined to non-IPSec host systems, making the process invisible to end users. Another great advantage is the source and the destination addresses are invisible while encrypted.

AH Transport and Tunnel Mode

Figure 9-10 compares AH Transport mode versus AH Tunnel mode. In AH Transport mode, AH authentication and integrity services protect the original IP packet by hashing the header fields that don’t change in transit through the network, plus the data payload. Fields like TTL aren’t included.

Click To expand
Figure 9-10: AH Transport mode versus AH Tunnel mode

The AH header, which is the result of the hashing algorithm, is inserted after the IP header and before the data payload (higher layer fields) in the original packet. Because no encryption is involved, the IP header destination address is readable by any Layer 3 device encountered in transit and any fields, such as TTL, are still available for required changes. One advantage of Transport mode is it only adds a few bytes to each packet.

In AH Tunnel mode, the entire original IP header and data becomes the “payload” for the new packet: a new IP header reflecting the end points of the VPN tunnel is added and the hash is performed on the resulting packet. The new IP header is protected exactly the same as the IP header in Transport mode. The fields that change in transit are excluded. The hash result again becomes the AH header following the new IP header.

Remember, both AH methods provide sender authentication and integrity verification that nothing in the data has been changed, but neither mode prevents the information from being read by an eavesdropper or a packet capture device.

A second, more serious concern is this: AH is incompatible with NAT. This is an issue if NAT occurs after the AH packet has been built because NAT changes the source IP address. This can cause the hash value stored in the AH header not to match the one generated at the destination peer and cause the packets to be rejected. This would always be a problem with AH Transport mode because the host computer is the source and, therefore, NAT would have to occur in transit. With planning, Tunnel mode would work if NAT were performed before the AH packet was built, for example, if NAT was performed at the firewall and IPSec at the perimeter router. Figure 9-11 shows how AH and NAT can be implemented to work together.

Click To expand
Figure 9-11: AH implementation to work with NAT

ESP Transport and Tunnel Mode

Figure 9-12 compares ESP Transport mode versus ESP Tunnel mode. In ESP Transport mode, the IP payload is encrypted, while the original IP header is left intact. An ESP header is inserted after the IP header and before the upper-layer protocol header (original Layer 3 data), while an ESP trailer is appended after the original data. The inserted ESP header contains a Security parameter index (SPI) value to identify the VPN security association, a sequence number field, and authentication data to verify packet authenticity.

Click To expand
Figure 9-12: ESP Transport mode versus AH Tunnel mode

In Transport mode, only the original data and ESP trailer fields are encrypted. The IP header isn’t encrypted because some fields will be required by a Layer 3 device encountered in transit. The ESP header field isn’t encrypted because it’s needed by the destination peer to decipher the payload.

If ESP authentication is used, the upper-layer protocols (original data payload) are hashed with the ESP header and trailer, and then appended to the packet as the ESP Authentication field. Cisco IOS software and the PIX Firewall refer to this authentication service as ESP HMAC. ESP Transport mode doesn’t authenticate any portion of the IP header itself.

In ESP Tunnel mode, as in AH, a new IP header reflecting the end points of the VPN tunnel is added. Both encryption and authentication incorporate the entire original IP header.

When both authentication and encryption are selected, encryption is performed before authentication. This order facilitates rapid detection and rejection of replayed or bogus packets by the receiving peer, potentially reducing the impact of denial-of-service (DoS) attacks.

Choosing AH versus ESP

Both AH and ESP support both MD5 and SHA-1 hashing algorithms for authentication. The main difference between ESP and AH authentication is this: ESP doesn’t protect any IP header fields in Transport mode. Both ESP and AH authenticate all IP header fields in Tunnel mode.

IPSec Transforms and Transform Sets

One set of decisions that must be made early in the IPSec implementation is whether to use AH or ESP, or MD5 or SHA-1 hashing, combined with Transport or Tunnel mode. These decisions create what are called transforms.

An IPSec transform defines a single IPSec security protocol—AH or ESP—with its associated security algorithms and mode. You can see the possible transform choices in Figure 9-13.

Click To expand
Figure 9-13: IPSec transform options

Two possible transform choices might include the following:

  • AH protocol with HMAC-MD5 authentication algorithm in Transport mode, where authentication and performance are important, but encryption security isn’t required.

  • ESP protocol, 3DES encryption algorithm, and HMAC-SHA-1 authentication algorithm in Tunnel mode, where both data confidentiality and authentication are critical. Performance will be traded away for improved security.

The actual transforms supported might vary by device type. VPN devices support a slightly different group than the PIX Firewall. The following are transforms supported by the IOS-based devices.

AH Transforms—Choose up to one




AH with the MD5 (HMAC variant) authentication algorithm


AH with the SHA (HMAC variant) authentication algorithm


Older version of the AH protocol (RFC 1828)

ESP Encryption Transforms—Choose up to one




ESP with the 56-bit DES encryption algorithm


ESP with the 168-bit DES (Triple DES) encryption algorithm


Older version of the ESP protocol (RFC 1829). Doesn’t support using ESP authentication transform


ESP without cipher. Can be used with esp-md5-hmac or esp-sha-hmac for ESP authentication without encryption. Shouldn’t be used in production network because of the lack of security

ESP Authentication Transform—Choose up to one, only if you also selected the esp-des or esp-3des transform (not esp-rfc1829)




ESP with the MD5 (HMAC variant) authentication algorithm


ESP with the SHA (HMAC variant) authentication algorithm

Transform Sets

A transform set is a combination of up to three individual IPSec transforms designed to implement a specific security policy for secure data transmission. The transform sets represent the choices available during IPSec security negotiation between two IPSec peers. The peers must agree to use a particular transform set for protecting a particular data flow or the exchange can’t occur. Transform sets are limited to no more than one AH transform, plus no more than two ESP transforms: one for encryption and one for authentication.

Some possible examples of acceptable transform combinations include the following:

  • ah-md5-hmac AH protocol with MD5 authentication

  • esp-des ESP protocol with DES encryption

  • esp-3des and esp-md5-hmac ESP protocol with DES encryption, plus ESP MD5 authentication

  • ah-sha-hmac and esp-des and esp-sha-hmac AH protocol with SHA-1 authentication, ESP DES encryption, plus ESP SHA-1 authentication

  • ah-rfc1828 and esp-rfc1829 Legacy AH protocol with ESP encryption

When configuring transform sets, the parser prevents you from entering invalid combinations. Transform sets are discussed in greater detail in Chapters 10 and 11 when configuring IPSec is covered.

Part III: Virtual Private Networks (VPNs)