IPSec Foundation

IPSec is a suite of protocols operating at the network layer to secure communication between IPSec peers, either client to gateway, host to host, or gateway to gateway. IPSec tunnel connections can originate from gateways (routers, firewalls), hosts, or IPSec hardware/software clients and can terminate at VPN concentrators, hosts, or gateways (routers, firewalls).

IPSec provides both authentication and encryption services. Encryption renders intercepted payload over intrinsically hostile or untrusted infrastructures useless, and authentication ensures that packets are from a legitimate peer and have not been altered in transit or maliciously injected (man-in-the-middle attacks).

Figure 11-4 demonstrates the most popular applications for IPSec. Although IPSec introduces some caveats with regard to NAT and MTU issues, it is still the approach of choice to secure IP datagram transport transparently over hostile or untrusted (shared) environments, such as the Internet per se or wireless or powerline infrastructures. A lot of work is going on right now to make IPSec more "NAT-friendly" (NAT-Traversal). For further details about current work in the IPSec area, visit the IPSec Charter of the IETF (http://www.ietf.org/html.charters/ipsec-charter.html).

Figure 11-4. IPSec Applications



IPSec is an extension of IPv4 and a mandatory part of IPv6.

The protocol suite consists of the following major building blocks:

  • The IPSec security architecture for the Internet (RFC 2401)

  • IKE key management (RFC 2409)

  • ESP (RFC 2406) and AH (RFC 2402) to protect IP traffic

  • ISAKMP security association (SA) management (RFC 2408)

  • IPSec DOI for ISAKMP (RFC 2407)

  • The OAKLEY key determination protocol (RFC 2412)

  • PKI X.509v3 certificate management protocol (RFC 2510, RFC 2527)

IPSec is a complex architecture because it has to handle complex tasks. This makes the concepts difficult to grasp. It would require hundreds of pages to cover IPSec appropriately, and even more for an introduction to cryptography and PKI (Public Key Infrastructure); therefore, you are referred to the standard documents and Internet resources.

IPSec ESP/AH and Tunnel and Transport Mode

IPSec offers two services: authentication and encryption. These can be used separately, but they often are used in combination. The protocols Encapsulating Security Payload (ESP) and Authenticated Header (AH) form the heart of the IPSec architecture. They provide authenticity, integrity, and replay protection and, in the case of ESP, confidentiality (based on payload encryption). Central to the ESP and AH concept is an abstraction referred to as security association (SA), which is explained later. ESP/AH protocol information is inserted between the IP header and the payload portion of the IP datagram. This example is referred to as IPSec transport mode, protection of payload, and transport of self-originated IP datagrams. In the case of a VPN setup, IPSec tunnel mode is used. This means that IPSec gateways deliver IP datagrams on behalf of other hosts/secure network segments. In this case, the gateway adds a completely new outer transport header, and the entire original IP datagram is encrypted. ESP most often is the approach of choice because it supports authentication and encryption and has a better chance of working with NAT gateways.

Manual/Automatic Keying, Preshared Secrets, and Certificates

Manual key management is tedious, less secure (no rekeying), more error prone, and (even worse) does not scale at all. It requires pregenerated keys to be distributed securely. If no key management daemon (such as racoon or isakmpd) is running (with the added benefit of automatic rekeying), manual keys have to suffice. This also means that SA setup has to be handled manually. The following scenarios are deployed commonly:

  • IPSec with manual key. (IPSec secret key will not change over time.)

  • IPSec with IKE with preshared secret. (IPSec secret key changes over time.)

  • IPSec with IKE with X.509v3 certificates. (IPSec secret key changes over time, requires a PKI.)

IKE Phase 1 and 2: Main Mode and Aggressive Mode

IKE uses a two-phase process for establishing IPSec parameters between two IPSec peering nodes that would otherwise have to be configured manually (for instance, by OpenBSD ipsecadm(8)). IKE phase 1 deals with initially establishing a secure and authenticated channel (a security association = SA) between IPSec peers with either preshared keys or X.509v3 certificates to ensure that the other side is who it claims to be. This can work either in main mode or aggressive mode and forms the foundation for phase 2. That is, "main mode sends the various authentication information in a certain sequence, providing identity protection. Aggressive mode does not provide identity protection because all of the authentication information is sent at the same time."[7]

In phase 2, unidirectional endpoint SAs are negotiated on behalf of IPSec and deal with the actual key exchange necessary for data encryption. Because phase 1 has already established and verified the credentials of the peers, Quick mode suffices in phase 2.

Resolving the IKE, PKI, SA, ISAKMP, and Oakley Confusion

Internet Key Exchange (IKE) is a somewhat generic hybrid protocol that approaches the issue of negotiation and provisioning of keying material for security associations in a protected manner. It does not per se claim to be compatible with Oakley or SKEME. IKE takes into consideration client mode negotiation (negotiation proxies) independent of the actual endpoints of SAs. An SA qualitatively is a logical point-to-point security channel between two security entities (peer IP addresses).

Security associations are by their nature connection-oriented and unidirectional. From the view of IKE, an SA is nothing more than "a set of policies and key(s) used to protect information. The ISAKMP SA is the shared policy and key(s) used by the negotiating peers in this protocol to protect their communication" (quoted from RFC 2409). IKE negotiations have two phases: UDP/500 (phase 1 and phase 2).

ISAKMP provides a framework for authentication, SA management, and cryptographic key exchange, but it was designed to be intrinsically key exchange independent and thus not defining a specific key exchange. Oakley specifically describes a series of key exchanges. SKEME describes a specific, versatile key exchange technique. Oakley and SKEME are designed to be compatible with the ISAKMP framework; however, they do not form a hierarchy.

A key exchange specification essentially defines a protocol by which two already authenticated parties or peers can agree on secure and secret keying material. Public Key Infrastructure (PKI) essentially brings into play X.509v3 certificates (digital signatures), frameworks, and architectures to deal with certificates (certificate authorities [CAs], dissemination and revocation of certificates, tokens, and so on). Certificates identify organizations, sites, entities, accounts, services, and individuals. LDAP interfaces to PKI are the most common way of interfacing with other systems.

What Is Opportunistic Encryption (OE)?

You can find the basic principles of OE in context with IPSec, IKE, and DNSsec in draft-richardson-ipsec-opportunistic-12.txt (available from the http://www.ietf.org drafts archive). Linux FreeS/WAN was among the first strong supporters and contributors to this idea. The general idea is to provide a method for ad-hoc encryption for secure communication without a lot of effort for prearrangements involving the end systems/parties. The approach relies on DNS (DNS TXT resource records [RRs] in the case of FreeS/WAN) for public key distribution and can be secured via DNSsec. This requires intervention from system administrators to add ancillary DNS records for opportunistic encryption support. This approach certainly has architectural drawbacks. For further information, consult the work of the IETF IPSec and ISAKMP/Oakley Work Group and the IETF DNSEXT Work Group, where DNSsec work is done.

What Is NAT-Traversal (NAT-T)?

NAT in context with IPSec is almost always a source for trouble. Try to avoid this combination, or at least perform NAT and IPSec tunnel termination on the same box and always use ESP. See draft-ietf-ipsec-nat-reqts-04.txt for an excellent introduction as to why and in what way NAT breaks IPSec in most cases. STUN (RFC 3489) offers another solution to the NAT problem. On the plus side, tunnels in general can be the solution for NAT-ignorant protocols such as H.323 (for example, Microsoft NetMeeting).

You will find that draft-ietf-ipsec-nat-t-ike-06.txt "describes how to detect one or more Network Address Translation devices (NATs) between IPSec hosts, and how to negotiate the use of UDP encapsulation of the IPSec packets through the NAT boxes in Internet Key Exchange (IKE)." The NAT-T capability negotiation with remote gateways is done in IKE phase 1 and allows safe IPSec traversal through firewalls performing NAT. In addition, the path between the two gateways is checked for NAT presence. The negotiation of UDP-encapsulated IPSec packets is performed in IKE Quick mode.

DHCP Provisioning over IPSec Tunnel Mode

RFC 3456, "Dynamic Host Configuration Protocol (DHCPv4) Configuration of IPSec Tunnel Mode," describes a mechanism that allows remote hosts to be assigned addresses of a corporate address space. Therefore, the host would appear to be part of the intranet it securely connects to via IPSec.

IPSec Implementations

The VPNC IPSec/IKE conformance test suite uses KAME/racoon and OpenBSD's isakmpd as reference code. This should demonstrate the maturity, interoperability, and reliability of the open-source code base. It works extremely well with Cisco IPSec implementations and IPSec client software. Nevertheless, it took the IPSec vendors a long time to work out interoperability issues. Occasionally I get the impression that some vendors, for awkward commercial reasons, deliberately break what took them years to fix. This is an especially big problem with IPSec intervendor client-to-gateway interoperability. (You are invited to draw your own conclusion.)

Linux IPSec

Until recently, Linux FreeS/WAN (http://www.freeswan.org) was the most popular implementation of IPSec and IKE for the Linux 2.2/2.4 kernel (IPv4 only). A related project called Super FreeS/WAN (http://www.freeswan.ca) included all the latest-and-greatest patches and early development features (NAT-Traversal, AES, X.509, ALG [all ciphers/hashes as modules], Dead Peer Detection IETF Draft, and so on). Because of disagreements about the evolution of FreeS/WAN and some peculiar points of view (removal of AH, focus on opportunistic encryption, and lack of interest in IPSec standard compliance and interoperability), the project has been discontinued and is no longer actively maintained.

The OpenSWAN Project (http://www.openswan.org) has started as a code fork of FreeS/WAN and will continue the work. The FreeS/WAN maintainer had a strategic interest in pushing a new approach to IPSec tunnel/VPN provisioning referred to as "opportunistic encryption" involving advanced DNS features, plug-and-play friendliness at the cost of IPSec generic compliance, and reduced security. It escaped the attention of some people that FreeS/WAN gateways do not necessarily talk only to FreeS/WAN gateways. This led to a misjudgment of the community support of that strategic direction.

OpenSWAN is a huge kernel patch and module with accompanying user-space tools that never made it into the mainstream kernel. KLIPS is the kernel IPSec module, and pluto is the IKE daemon. The new 2.6 Linux kernel drastically changed this picture. The current state of affairs is this: "As of Linux 2.5.47, there is a native IPSec implementation in the kernel. It was written by Alexey Kuznetsov and Dave Miller, inspired by the work of the USAGI IPv6 group. With its merge, James Morris' CrypoAPI also became part of the kernel?it does the actual crypting. As of 2.5.49, IPSec works without further patches."[8]

Linux native IPSec for the new 2.6 Linux kernel is based on the KAME Project's IKE daemon racoon, which is a big leap forward for consistent IPSec interoperability between BSD platforms and Linux. OpenBSD's isakmpd also has been ported to Linux, and OpenSWAN is completing 2.6 integration support of their own implementation. OpenSWAN is easy to install; just follow the ReadMe that comes with the sources. IPSec-Tools is a port of KAME's IPSec utilities to the Linux 2.6 including racoon/setkey. isakmpd and racoon/setkey can be retrieved from the following resources:

  • http://bender.thinknerd.de/~thomas/IPsec/isakmpd-linux.html

  • http://ipsec-tools.sourceforge.net/


The KAME Project is a software kit that provides reference implementations of IPv6 and IPSec (both IPv4/IPv6) for BSD operating systems as well as the alternate queuing framework (ALTQ) discussed in Chapter 13, "Policy Routing, Bandwidth Management, and QoS." KAME's IKE daemon is called racoon.

During IPSec setup, the kernel needs to know what traffic to encrypt or secure and how. This is referred to as establishing an IPSec security policy. On BSD systems, this information is stored in a table in the kernel known as the security policy database (SPD).

The setkey(8) utility is the tool to manipulate the SPD. Another table (the security association database [SAD]) stores the various encryption keys needed to secure communications between hosts, clients, and gateways. In case of manual keying, the setkey(8) program also is used to configure the manual keys in the SAD. With an IKE approach, racoon will handle the adding and deleting of entries from the SAD automatically.


To add IPSec support to your FreeBSD kernel, add the code in Example 11-14 to your kernel config file, and then recompile and install the new kernel. You essentially need the two utilities setkey and the IKE daemon racoon.

Example 11-14. FreeBSD IPSec Kernel Configuration Options

options         IPSEC                   #IP security

options         IPSEC_ESP               #IP security

options         IPSEC_DEBUG             #debug for IP security

device          gif 4                   #For IPSec Tunnel Mode

FreeBSD IPSec configuration includes the following steps:

  1. Retrieve and install the racoon daemon from ports by using cd /usr/ports/security/racoon followed by make all, make install, make clean.

  2. Define the IPSec security policy via setkey -DFP (for manual policy management) and setkey -f /etc/ipsec.conf.

  3. You also can do that dynamically via the IKE daemon racoon. It requires two configuration files: /usr/local/etc/racoon/psk.txt (for preshared keys) and /usr/local/etc/racoon/racoon.conf.

  4. Then start racoon via racoon -l /var/log/racoon -f usr/local/etc/racoon/racoon.conf.


IPSec tunnel connections between two FreeBSD gateways require the configuration of the gif pseudo-interface for proper routing between the protected private (RFC 1918) networks behind these gateways (Example 11-15).

Example 11-15. Castor gif Pseudo-Interface Setup

[root@castor:~#] gifconfig gif0

[root@castor:~#] ifconfig gif0 inet netmask

graphics/ccc.gif netmask

[root@castor:~#] gifconfig gif0

[gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280

        inet6 fe80::210:5aff:fec4:2c04%gif0  prefixlen 64

        inet --> netmask 0xffffff00

        physical address inet -->

[root@castor:~#] netstat -rn -f inet

Routing tables


Destination        Gateway            Flags    Refs      Use  Netif Expire

default        UGSc        3      121    xl0            UH          0        0    lo0          UH          1       32    lo0

192.168.2          link#1             UC          2        0    xl0        00:10:5a:c4:2c:04  UHLW        3        4    lo0      52:54:05:e3:e4:2f  UHLW        5      446    xl0    402

192.168.7          link#2             UC          0        0    ed0        UH          0        0   gif0

192.168.80         link#15            UC          0        0  vlan8


Consult http://www.freebsd.org/doc/en_USISO88591/books/handbook/ipsec.html for further details.


isakmpd is the OpenBSD ISAKMP/Oakley (a.k.a. IKE) key management daemon. The isakmpd daemon is responsible for the establishment of management of security associations for encrypted or authenticated network traffic. It is the only scalable alternative to manual keying with tedious and error-prone manual SA setups and ESP/AH flows on top. isakmpd is also available for NetBSD. There you have the luxury to choose between isakmpd and racoon. Example 11-16 shows the kernel configuration options required by OpenBSD and related sysctl parameters.

Example 11-16. OpenBSD IPSec Kernel Configuration Options

### kernel config:

option    CRYPTO          # Cryptographic Framework

option    IPSEC           # IPSec VPN

#option   KEY             # KEY implied by IPSec

pseudo-device enc 4       # Encapsulation device used by IPSec

### sysctl:

[root@ganymed:~#] sysctl -w net.inet.esp.enable=1

[root@ganymed:~#] sysctl -w net.inet.ah.enable=1

[root@ganymed:~#] sysctl -w net.inet.ip.forwarding=1

[root@ganymed:~#] sysctl -w net.inet6.ip6.forwarding=1

[root@ganymed:~#] sysctl -w net.inet.ipcomp.enable = 1

An OpenBSD IPSec setup consists of these components:

  • /etc/rc.vpn (for manual keying only, manual SAs, and IPSec flows)

  • /etc/isamkmpd/isakmpd.conf

  • /etc/isamkmpd/isakmpd.policy

  • isakmpd (not required for manual keying)

  • The keynote utility (optional, man pages 1, 3, 4, 5; RFC 2704), public/private key generation, and trust management

  • ipsecadm (generic admin interface for manual setup)

  • The enc(4) pseudo-interface for encryption (adjust your firewall filters accordingly)


photurisd is an alternative key management daemon for OpenBSD. It is based on RFC 2522 and RFC 2523. A word of wisdom: Stay away from the OpenBSD photurisd. It has major limitations, and the whole community is using ISAKMP anyway. photurisd was removed from the OpenBSD tree with version 3.3 for good reason and can be considered deprecated.