8.3 Microsoft's Implementation of IPSec in Windows Server 2003

The Microsoft implementation of IPSec in Windows Server 2003 adheres to the appropriate RFCs and is compatible with other hosts running RFC-compliant IPSec software. This includes other computers running Windows 2000 and Windows XP Professional, computers running other operating systems with an IPSec component, and intermediate network devices such as routers. In the next few sections, I'll discuss the specific software components that implement IPSec in Windows Server 2003.

8.3.1 Microsoft IPSec Components

In the Microsoft implementation of IPSec, the IPSec driver acts as a filter driver on top of the TCP/IP protocol stack. The IPSec driver is initialized at the same time that other network services are initialized at system bootup, and the driver receives its policy information from the IPSec Policy Agent. The agent examines local and domain policies (which are set through Group Policy) that apply to IPSec to determine exactly how IPSec should behave. The appropriate configuration information is then delivered to the IPSec driver as a policy list. The Policy Agent does little else, but does periodically check for changed IPSec policy settings and delivers those to the IPSec driver as necessary. So the core IPSec work is done by the driver, but the agent is necessary to deliver the driver's configuration information.

The IPSec driver examines all data sent through the TCP/IP protocol stack and destined to be sent over the network. This examination occurs before the protocol stack processes the data, allowing IPSec the opportunity to modify (encrypt or sign) the data prior to transmission. The driver compares all traffic to its policy list to determine if the traffic is supposed to go out encrypted or signed, or possibly not go out at all. When the IPSec driver finds that some data matches one or more of its IPSec policies, the driver does its job and protects that data?signing it, encrypting it, or blocking it as defined in the policies.

Assuming the traffic isn't blocked, if no shared secret keys already exist between the local computer and the destination, ISAKMP and IKE negotiate the keys and provide them to the IPSec driver. These keys are generated only after mutual authentication takes place between the computers. Windows Server 2003 supports three types of authentication for IPSec communication:


Each party in the IPSec communication must have a certificate and a corresponding private key. This certificate must chain to a specified trusted root. For more information on certificate chaining, see Chapter 9.


Each end of the IPSec communication must authenticate using Kerberos, which is explained in Chapter 7.

Shared secret key

A shared secret key is specified on each host. The hosts must use the same shared secret key to set up communication. When both hosts have the same shared secret key, it indicates that they have prearranged authentication through some other means (such as human interaction). This option is by far the least desirable since the shared secret key is itself a vulnerability.

After authentication, a shared secret key (also known as a session key) is established between the parties and provided to the IPSec driver. Once the IPSec driver has secured the data appropriately using the specified key and rules, the data is passed to the TCP/IP protocol stack and sent over the network.

When the data is received on the destination computer, the packet is examined by the TCP/IP stack, which determines that the data is protected by IPSec. The stack hands the data to the local IPSec driver for decryption, digital signature verification, or both. Because the receiving IPSec driver gets its shared secret key and security settings information from its own ISAKMP and IKE components, it already has all the information it needs to decrypt the data or verify its digital signature. The data is decrypted and/or verified and returned to the TCP/IP stack for distribution to its intended application?such as a web browser, FTP application, or other application.

Network traffic is processed in this fashion until the network communication is terminated. However, continuing to use the same symmetric session key over a long period would present a security risk. If an attacker managed to crack one key, she could decrypt enormous amounts of data. Therefore, IPSec can be configured to periodically generate and use new keys. It does this on the fly without interrupting data transmissions. Normally, the new key is based on the previous authentication and key exchange to save CPU and network utilization and is configured by default to happen infrequently enough to avoid taxing system resources. But you can configure an element of IPSec called perfect forward secrecy (PFS) to require IPSec to carry out authentication and rekey from scratch every time. Although PFS tends to consume a lot of computer resources, its cryptographic protection of data is slightly more secure.

You can see how IPSec adds extra work to the tasks of sending and receiving data. Entire additional software modules are loaded into memory, data is encrypted and decrypted, encryption keys are negotiated, and so forth.

IPSec Overhead and Offloading

I recently attended a computer conference in Orlando where I delivered a presentation on how to correctly deploy IPSec. This session was attended primarily by network administrators with varying degrees of experience in both security and general network operation.

I got no further than the first technical slide when I saw a hand go up. The question was: "I've heard that IPSec presents a tremendous overhead and will drag my network down to its knees." Well, that's not really a question, but it brought up an excellent point. Most administrators are under the assumption that IPSec is some kind of network Sword of Damocles?always ready to destroy everything. Because this is such a common theme, it's helpful to address it specifically here.

Two aspects of IPSec can have an impact on your systems: network bandwidth and computer performance. Network performance is not significantly impacted by IPSec deployment. The amount of traffic that IPSec adds to the network is minimal. ISAKMP negotiations are very small, and the addition of the AH or ESP headers to the TCP/IP packets is negligible. If network bandwidth consumption is within desired performance guidelines before deploying IPSec, it will almost certainly remain at the same level after IPSec deployment. If, however, the network is already performing improperly due to saturation or other issues, nothing should be added (including IPSec) until that preexisting condition is addressed. IPSec could fail or perform poorly in those conditions or potentially aggravate the existing condition.

Computer performance with IPSec is usually a more realistic concern than network performance. The encryption, decryption, signing, and signature verification functions all take processor and memory resources to complete. In addition, SA and key management on computers with large numbers of simultaneous IPSec sessions can consume even more resources. This resource consumption can range in severity from barely measurable to severe enough to prevent network communication and local console operation.

IPSec-enabled network cards, as mentioned earlier in this chapter, help with the computer performance issues. However, they're not always necessary. A server running with an average of 1% CPU utilization and plenty of available RAM probably doesn't need an IPSec-capable network card and will probably not realize any benefit from such an upgrade. The most appropriate application of such hardware is in marginal performance cases or when IPSec has been demonstrated to deteriorate computer performance.

One other question comes up frequently on this IPSec topic: should I buy an IPSec-capable NIC if I'm planning to purchase a system that will use IPSec? When planning ahead, an IPSec card may be a good idea. IPSec network cards are often a minor change in cost when planned as part of a system purchase. After installation, though, you can forget about the card and use the system as normal. The card may not provide a huge benefit over time, but if it's a small additional cost, it should certainly be considered.

8.3.2 Deployment of IPSec to Windows Computers

Computers running Windows 2000, Windows XP Professional, and Windows Server 2003 can all use IPSec to secure network communications. If the client computers are members of an Active Directory domain, they can receive their IPSec settings through Group Policy. As discussed in Chapter 5, Group Policy is a method used to distribute a group of settings to groups of computers. IPSec settings are easily deployed using this mechanism to as many computers as you desire. IPSec settings can also be configured on a per-computer basis using the computers' Local Security Policy snap-in. Because IPSec configuration can be complex, however, local configuration can be arduous and error-prone unless you have a small number of computers to configure. When dealing with a large number of computers, I don't recommend implementing IPSec unless you have an Active Directory domain with which you can centrally configure IPSec settings.

At the beginning of this chapter, I explained that IPSec is a transparent security mechanism, so users cannot easily determine if and when they are using IPSec. Because Group Policy is also transparent and automatic, the combination allows you to deploy and enforce security settings without the cooperation or awareness of your users. This is often a highly desired solution, as users have the propensity to circumvent or simply ignore security requirements when they feel no threat. Users in a physically secured building may feel safe and assume that the path their data takes must also be safe. Conversely, users in a branch office may automatically assume all their data could be intercepted and that they should use extravagant security means to protect it. Using Group Policy and IPSec allows you to make these decisions and removes control over data security from your users' hands.