8.2 How Does IPSec Work?

IPSec isn't one single component. The designers of IPSec chose to create it as a number of separate components that work together similarly to TCP/IP, which was created from a number of specifications that are integrated into one large solution, or suite. Also like TCP/IP, IPSec has a large number of components involved and features a very complex set of interactions among those components. IPSec's design provides the benefit of modularity: when a change is made to one component, the others are not necessarily changed. The drawback of this modularity is that IPSec is exceedingly difficult to explain without the use of diagrams, simply because so many different components must work together to make IPSec operate.

The number of components and their interaction is somewhat complex. Administrators, such as yourself, benefit from some discussion of the IPSec architecture, since it's good to know how it works on your network and what burden it will place on your servers and clients. But because there are numerous dedicated references already available, I will not analyze IPSec completely down to the bit level.

8.2.1 IPSec Components

At the core of IPSec are four major components:

IPSec driver or core engine

This is the component of IPSec that does the actual encryption (application of the Encapsulating Security Payload?ESP?protocol), decryption, signing (application of the Authenticated Header?AH?protocol), and signature verification. The driver is also responsible for coordinating the effort of the other IPSec components to ensure that it can complete the necessary tasks. This is the workhorse of IPSec.

IPSec Policy Agent

This is the cognitive function of IPSec. The policy module examines the IPSec settings of a system and determines which traffic should be protected and some generic settings for that protection. It does not do the actual work of protecting the data, it simply alerts the IPSec driver that the traffic must be protected. When the Policy Agent isn't running, no new IPsec policies can be downloaded to the client.

Internet Security Association and Key Management Protocol (ISAKMP, pronounced EYE-suh-kamp)

ISAKMP is the negotiator for Internet security settings. When the computer receives a request to establish a shared secret key with a target computer, it must learn the IPSec configuration on that remote computer to ensure that its settings are acceptable. ISAKMP connects between the computers to negotiate an agreed-upon group of settings to use for encryption and signing. However, ISAKMP does not establish the data encryption keys itself. It uses another component, the Oakley Key Determination Protocol, to establish those data encryption keys.

ISAKMP is a subcomponent of IKE.

ISAKMP connects from a source computer to a destination computer using user datagram protocol (UDP) port 500. If IPSec is not running on the destination computer, it is likely that no service is listening on UDP port 500. The source computer will send several requests to establish an ISAKMP session. If nothing is listening on the destination computer, there will be no response and the source computer will assume that IPSec is unavailable. Whether the connection is made after this failure depends on the IPSec policy, as you will see later in this chapter.

Internet Key Exchange (IKE)

Because IPSec uses shared secret key encryption, there must be some mechanism for the computers to contact each other and agree on a key. IKE manages this by providing generic key agreement services for IPSec. It also stores shared secret keys and configuration information to manage secure connections. IKE can use a myriad of key agreement and encryption protocols, depending on the settings it is provided by ISAKMP.

8.2.2 IPSec Component Interaction

From the information I've presented so far, it's a bit difficult to visualize how IPSec works. Numerous components provide specific functions, but there seems to be no single piece that you can point at and identify as IPSec. The easiest way to explain how all the components of IPSec come together to provide a security solution is to demonstrate how they interact, the order in which they operate, and how an IPSec connection is established. Figure 8-1 illustrates this flow of data between IPSec components.

Figure 8-1. The IPSec components and the flow of data between them

The IPSec components work in this fashion:

  1. Local or group policy is applied to a computer during startup and periodically while the computer is on. This is not IPSec-specific, but rather the normal operation of Group Policy as described in Chapter 5.

  2. Any IPSec policies are retrieved by the IPSec Policy Agent.

  3. When one or more IPSec policies exist, the IPSec Policy Agent monitors communication to the TCP/IP protocol from all applications. It's watching for traffic that matches the policy it is configured with?that is, network traffic that it must protect.

  4. When network traffic that needs protection is identified, the IPSec Policy Agent communicates with the IPSec driver. It informs the IPSec driver of the type of protection required.

  5. The IPSec driver then determines whether a Security Association (SA) exists that can be used to protect the traffic. For the purposes of this discussion, an SA is a set of IPSec settings and key material that is shared between this computer and the destination computer.

  6. If no SA exists, the IPSec driver contacts the IKE service. IKE is responsible for negotiating settings between the computers, performing mutual authentication, and establishing shared secret keys that conform to the security policy. IKE uses ISAKMP for this task.

  7. IKE provides the SA to the IPSec driver, which then protects the network traffic.

  8. The driver returns the protected traffic to the TCP/IP protocol for continued processing.

Whew! This process is complex and can take time to carry out. You might not feel confident explaining this after your first read-through of this explanation. Consider bookmarking this page and referring back to it when you need to review IPSec data flow and components.