8.1 What Is IP Security?

Simply put, IP Security (IPSec) is a method to cryptographically process IP packets by encrypting the data, signing the packet, or both. This cryptography is done on the computer before the data is transmitted on the network.

IPSec is standardized in a series of lengthy Internet RFC documents. These include RFCs 2401 through 2412, which define its core functionality, plus another half-dozen or so additions to IPSec functionality or features. These documents describe in painstaking detail exactly how IPSec functions and communicates.

The impact of IPSec is profound. Insecure intermediate networks can easily transport secure data. Because IPSec is an extension of the standard TCP/IP suite, routers and other internetworking devices work well with it. They usually don't know that the traffic is protected, and the protection has no effect on routing (with some minor exceptions I'll discuss later).

8.1.1 Benefits of IPSec

IPSec provides a number of important security benefits.

First, it helps protect data on physical LANs. I've already discussed the need for physical security in an earlier chapter. When the wires that connect your computers are insecure, the data that travels on them is insecure. This happens more often than most people realize. Common walls in a shared office building, network connections in unsecured locations, shallow buried cable in remote locations?any of these will compromise the security of the data on the LAN. An intruder with basic tools and hardware can capture all the data transmitted over these insecure lines. IPSec cannot stop an intruder from capturing this data, but it can protect it in such a way that it is useless to the intruder. Without the cryptographic key, the intruder has a great deal of work to do to decrypt the information.

IPSec and SMB signing sound as if they do the same thing. In a sense, both can provide the authentication of packets on the network. However, SMB signing is very limited in its scope and flexibility. It can only authenticate packets, and only SMB packets at that. It is also limited in its authentication mechanisms. IPSec has the ability to both encrypt and sign packets, and it can be used with any type of data that travels over the TCP/IP protocol suite. You can implement both SMB signing and IPSec simultaneously, but there is no advantage and the overhead may be detrimental to performance.

IPSec also provides protection from eavesdropping when using the Internet. The Internet is an inherently insecure medium for data transfer. However, many corporations use it as a cheap backbone to connect remote office and field personnel. Although a few security technologies?such as SSL?can encrypt data across the Internet, these are most often application-layer technologies. This means the application itself must encrypt and decrypt the data. SSL, for example, is useful only for web browsers and a handful of email applications; you can't use it to protect FTP connections, since FTP applications don't implement SSL or anything like it. IPSec operates at a lower layer, allowing any application to communicate securely. The application simply uses the TCP/IP communications suite as usual, unaware that the data is transparently being protected. The Windows operating system handles the encryption, making it accessible to any and all applications running on the computer.

IPSec provides strong protection from data-insertion attacks. Intruders don't always seek to capture data. Instead, they may want to replace some data on your LAN with their own data. This type of attack may take the form of a virus delivery vehicle or a corporate espionage need, in which intruders profit from your data being replaced with theirs. IPSec provides protection against these attacks through its ability to digitally sign each packet sent on the network, ensuring that each packet is received unchanged from how it was originally sent. Packet signing can be used independently of IPSec's data encryption functionality so that the appropriate amount of cryptography is used.

Both encrypting and signing the same packet may, on the surface, seem redundant. However, they're not. Encryption protects only the payload of the packet, while signing guarantees the authenticity of the whole packet?including the header. Together they provide very strong data security.

IPSec is based on a set of Internet Requests for Comments (RFCs). The RFC process allows peers from around the world to collaborate and create a widely accepted standard that has been discussed and examined in great detail. This often means many corporations and governments accept the RFC standard and that products from different manufacturers can interoperate if they adhere to the standard.

The main body of IPSec RFCs are RFCs 2401-2412. They constitute the basic set of documentation that virtually all implementations are based on. Many additional RFCs add to or change parts of IPSec, but 2401-2412 are considered the core elements. All these RFCs can be downloaded from http://www.rfc-editor.org.

The implementation of IPSec in Windows Server 2003 adheres to the standards, making it interoperable with IPSec implementations running on other operating systems. There have been some concerns in the past about this, which I'll address later in the chapter.

Just because two implementations of IPSec are RFC-compliant doesn't mean that interoperability is a gimme. Often some configuration and testing work must be done before two vendors' IPSec products will work together well. But RFC compatibility is a great indicator that the products will cooperate?eventually.

8.1.2 Drawbacks of IPSec

No matter how it may have sounded up to this point, IPSec isn't the Holy Grail of network security. IPSec has a few drawbacks. None of these should stop you from considering IPSec for your systems, but they are important points to keep in mind.

First, IPSec requires significant system resources of each computer that uses it. If you consider the scale of what IPSec does, you may be surprised that your computer can do it at all. It has to do cryptographic operations on every single packet that is sent. For example, a 1MB file sent on an Ethernet network creates about 750 packets of data. The corresponding decryption or signature verification occurs when each packet is received. In other words, IPSec usage can result in a very large number of cryptographic operations. In addition, IPSec clients have to change encryption keys periodically, which is an even more resource-intensive operation. All this work requires system resources, which can be depleted quickly if IPSec is not implemented correctly. For example, if you decide to use IPSec to encrypt and decrypt every packet of data sent to and from your computer, expect your computer to slow down significantly.

Manufacturers like 3Com and Intel offer IPSec-capable network adapters. With one of these adapters, Windows' own IPSec capabilities aren't used. Instead, dedicated hardware on the adapter is used to perform all IPSec operations. Because the hardware can work hundreds of times faster than Windows' built-in IPSec software, the adapters make it more feasible to use IPSec on a large scale. Check with your network adapter vendor to see what IPSec solutions they offer and how they work with Windows Server 2003.

IPSec makes it nearly impossible to analyze data packets via network capture. When you use IPSec to encrypt the data portion of IP packets, you're protecting them from everyone except the sender and receiver. If problems occur on the network and you want to use a network traffic analyzer, such as Microsoft's Network Monitor, to capture and examine packets, the tool will see the IPSec packets. But because the data portion is encrypted, the network traffic analyzer will be unable to interpret those packets. It will still show the packet's addressing information and that the packet is encrypted with IPSec, but nothing more. To successfully analyze network operations at this level, you may need to temporarily disable IPSec.

Unfortunately, some operating systems do not support IPSec. If you restrict sensitive computers to communicate only using IPSec, you are effectively limiting which clients can connect to those computers. Limiting the computers that can connect to one another may be either desired or undesired, depending on the diversity of your network clients. You should ensure that all clients that must connect to this resource support IPSec.

Finally, IPSec is easy to mess up. It has the power to stop all network communication in your organization. You cannot just sit down and hack together an effective IPSec policy on your production domain. The policies that define how IPSec behaves are complex and must be carefully planned, based on a risk analysis, and then tested prior to any deployment. Shortcuts or omissions during this process have led to many people regretting their decision to explore IPSec.