Using OpenSSL, you want to adjust a configurable parameter of a cipher other than the key length.
OpenSSL provides an obtuse, ioctl()-style API for setting uncommon cipher parameters on a context object:
int EVP_CIPHER_CTX_ctrl(EVP_CIPHER_CTX *ctx, int type, int arg, void *ptr);
OpenSSL doesn't provide much flexibility in adjusting cipher characteristics. For example, the three AES configurations are three specific instantiations of a cipher called Rijndael, which has nine different configurations. However, OpenSSL supports only the three standard ones.
Nevertheless, there are two cases in which OpenSSL does allow for configurability. In the first case, it allows for setting the "effective key bits" in RC2. As a result, the RC2 key is crippled so that it is only as strong as the effective size set. We feel that this functionality is completely useless.
In the second case, OpenSSL allows you to set the number of rounds used internally by the RC5 algorithm. By default, RC5 uses 12 rounds. And while the algorithm should take absolutely variable-length rounds, OpenSSL allows you to set the number only to 8, 12, or 16.
The function EVP_CIPHER_CTX_ctrl( ) can be used to set or query either of these values, given a cipher of the appropriate type. This function has the following arguments:
Pointer to the cipher context to be modified.
Value indicating which operation to perform (more on this a little later).
Numerical value to set, if appropriate (it is otherwise ignored).
Pointer to an integer for querying the numerical value of a property, if appropriate (the result is placed in the integer being pointed to).
The type argument can be one of the four macros defined in openssl/evp.h:
EVP_CTRL_GET_RC2_KEY_BITS EVP_CTRL_SET_RC2_KEY_BITS EVP_CTRL_GET_RC5_ROUNDS EVP_CTRL_SET_RC5_ROUNDS
For example, to set an RC5 context to use 16 rounds:
EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_SET_RC5_ROUNDS, 16, NULL);
To query the number of rounds, putting the result into an integer named r:
EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GET_RC5_ROUNDS, 0, &r);