On-the-fly email encryption/signature

Question

Background: I've inherited an embedded linux-based system that contains an SMTP proxy and some wacky constraints that I just have to live with. It sits between an SMTP client and server. When an SMTP client connects, the proxy opens a connection to the server and passes the client's data to the server after some on-the-fly processing.

The challenge: I need to sign and/or encrypt the email on its way to the server using standard PKI techniques and S/MIME formats (see RFC2311, for example). I have access to all the required public keys from the appropriate certificates.

The wacky constraints (please just accept them, as they're far beyond my control):

1. I can't store the email; it must be processed on-the-fly.
2. I can do the encryption locally using the public keys, but I cannot access the private keys directly, which means the digital signature must be done by a "signing device" through a 9600bps connection.
3. Typical email messages are tens or hundreds of MB in size. (The email server and recipients can handle those sizes; the only issue is the unacceptable delay when signing.)
4. Any new code should be in C, but it's acceptable, for example, to pipe the data to a stand-alone utility for encryption/signature as long as the data is never stored (e.g. no temporary files).
5. Delivery is in 14-21 days.

Questions:

1. I was hoping to find an open-source utility or library that would generate the appropriate MIME headers and encrypt/sign a blob of data, but I didn't find that on Sourceforge, Google code, etc. Have you used one that you could recommend?
2. I was desperately hoping to find an RFC that says it's acceptable to hash the 100MB of data and then sign the hash, as that would mitigate the 9600bps bottleneck. But again, no luck. Is there an industry-standard "shortcut" (RFC?) that would be compatible with typical email clients?

Thanks for your thoughts.

Answer 1

Question 1:

OpenSSL is both a utility and a library that can both create and verify S/MIME messages, including the MIME headers. See the smime(1) man page for usage of the utility version - this is all built using the library version, so it can do it too.

Question 2:

Not only is that acceptable, it's the way S/MIME signatures are always done. You will presumably be creating a signed message using the multipart/signed format (see section 3.4.3 of RFC2311). This multipart MIME type contains the detached signature as an object with a MIME type of application/pkcs7-signature. Section 3.4.3.1 tells us that this contains a PKCS #7 signedData object. PKCS #7 is described in RFC2315, and the signedData object is described in section 9. This section tells us that we create a message digest of the message to be signed (S/MIME says that implementations have to understand at least MD5 and SHA1 message digests, so you would use SHA1 as the interoperable option with best security), and encrypt it with the signer's private key.

As long as the signing device is happy to take a SHA1 hash from you and encrypt it with the signer's private key, then you can do all the rest of the signature generation yourself.

You'd then take the multipart/signed MIME object and encrypt it as per the S/MIME specs, and then sign the whole ball of wax again (Sign-Encrypt-Sign mode), so that ultimately you have:

• multipart/signed object, where the first part is:
• an application/pkcs7-mime object, which when decrypted by following PKCS #7 contains:
• another multiplart/signed object, where the first part is:
• a MIME object representing the original email (or just the body; whatever you need...)

Addendum:

OpenSSL supports pluggable cryptographic "engines" that can perform cryptographic operations on behalf of the library. The best way to implement this would probably be to create an OpenSSL engine for your external signing device, and just call the regular S/MIME OpenSSL functions with that engine enabled. If your external signing device is "off-the-shelf", there might already be OpenSSL an engine wrapper for it.