Two parties in a network communication want to communicate using symmetric encryption. At least one party has the RSA public key of the other, which was either transferred in a secure manner or will be validated by a trusted third party.
You want to do authentication and key exchange without any of the information leakage generally associated with password-based protocols.
Depending on your authentication requirements, you can do one-way authenticating key transport, two-way authenticating key transport, or two-way authenticating key agreement.
With key transport, one entity in a system chooses a key and sends it to the entity with which it wishes to communicate, generally by encrypting it with the RSA public key of that entity.
In such a scenario, the sender has to have some way to ensure that it really does have the public key of the entity with which it wants to communicate. It can do this either by using a trusted third party (see Chapter 10) or by arranging to transport the public key in a secure manner, such as on a CD-R.
If the recipient can send a message back to the sender using the session key, and that message decrypts correctly, the sender can be sure that an entity possessing the correct private key has received the session key. That is, the sender has authenticated the receiver, as long as the receiver's public key is actually correct.
Such a protocol can be modified so that both parties can authenticate each other. In such a scheme, the sender generates a secret key, then securely signs and encrypts the key.
Assuming that the recipient has some way to receive and validate the sender's public key, the recipient can now validate the sender as well.
The major limitation of key transport is that the machine initiating a connection may have a weak source of entropy, leading to an insecure connection. Instead, you could build a key agreement protocol, where each party sends the other a significant chunk of entropy and derives a shared secret from the information. For example, you might use the following protocol:
The client picks a random 128-bit secret.
The client uses a secure technique to sign the secret and encrypt it with the server's public key. (See Recipe 7.14 for how to do this securely.)
The client sends the signed, encrypted key to the server.
The server decrypts the client's secret.
The server checks the client's signature, and fails if the client isn't authenticated. (The server must already have a valid public key for the client.)
The server picks a random 128-bit secret.
The server uses a secure technique to sign the secret and encrypt it with the client's public key (again, see Recipe 7.14).
The server sends its signed, encrypted secret to the client.
The client decrypts the server's secret.
The client checks the server's signature, and fails if the server isn't authenticated. (The client must already have a valid public key for the server.)
The client and the server compute a master secret by concatenating the client secret and the server secret, then hashing that with SHA1, truncating the result to 128 bits.
Both the client and the server generate derived keys for encryption and MAC'ing, as necessary.
The client and the server communicate using their new agreed-upon keys.
Incorporating either key transport or key exchange into a protocol that involves algorithm negotiation is more complex. In particular, after keys are finally agreed upon, the client must MAC all the messages received, then send that MAC to the server. The server must reconstruct the messages the client received and validate the MAC. The server must then MAC the messages it received (including the client's MAC), and the client must validate that MAC.
This MAC'ing is necessary to ensure that an attacker doesn't maliciously modify negotiation messages before full encryption starts. For example, consider a protocol where the server tells the client which encryption algorithms it supports, and the client chooses one from the list that it also supports. An attacker might intercept the server's list and instead send only the subset of algorithms the attacker knows how to break, forcing the client to select an insecure algorithm. Without the MAC'ing, neither side would detect the modification of the server's message.
The SSL/TLS protocol handles all of the above concerns for you. It provides either one-way or two-way authenticating key exchange. (Note that in one-way, the server does not authenticate the client using public key cryptography, if at all.) It is usually much better to use that protocol than to create your own, particularly if you're not going to hardcode a single set of algorithms.
If you do not want to use a PKI, but would still like an easy off-the-shelf construction, combine PAX (Recipe 8.15) with the secure channel from Recipe 9.12.
Recipe 4.10, Recipe 5.10, Recipe 7.14, Recipe 8.15, Recipe 9.1, Recipe 9.2, Recipe 9.12