How to encrypt small data block with only RSA public key using Microsoft ECSP?

Question

I need to encrypt a small block of data (16 bytes) using 512 bit RSA public key -- quite an easy task for most cryptography libraries known to me, except for MS CSP API, as it seems. Documentation for CryptEncrypt function states that

The Microsoft Enhanced Cryptographic Provider supports direct encryption with RSA public keys and decryption with RSA private keys. The encryption uses PKCS #1 padding.

It didn't work to me though. Well, my code works and produces encrypted block of data with correct size, but openssl fails to decypher it. It looks much like CryptEncrypt still uses symmetric cypher.

Unfortunately all the examples I've found refer to combined cryptography with symmetric cypher, so I don't have a working example on hands which definitely would make things easier.

Could please anyone point me to such an example or let me know if there are some not that obvious pitfalls I've missed?

Thank you.

Answer 1

This sounds like an endianness issue. Microsoft's CryptEncrypt function returns the ciphertext in little-endian format, while OpenSSL expects its data to be in big-endian format. You'll need to reverse the encrypted data before passing it to OpenSSL.

Answer 2

Here's the code (just in case someone has googled this topic out):

BYTE *spkiData = SPKI; // X.509 ASN.1 encoded SubjectPublicKeyInfo
DWORD dwSPKISize = SPKI_SIZE; // 94 bytes for RSA

DWORD dwBufSize = 0;
// Get buffer size for decoded spki structure
CryptDecodeObject(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, spkiData, dwSPKISize, 0, NULL, &dwBufSize);
BYTE* decBuf = new BYTE[dwBufSize];
CryptDecodeObject( X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, spkiData, dwSPKISize, 0, decBuf, &dwBufSize);
// Now decode the RSA Public key itself
CERT_PUBLIC_KEY_INFO * spki = (CERT_PUBLIC_KEY_INFO *) decBuf;
// Get buffer size for decoded public key structure
CryptDecodeObject( X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB, spki->PublicKey.pbData, spki->PublicKey.cbData, 0, 0, &dwBufSize);
// Get the RSA public key blob
BYTE *blobBuf = new BYTE[dwBufSize];
CryptDecodeObject(X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB, spki->PublicKey.pbData, spki->PublicKey.cbData, 0, blobBuf, &dwBufSize);
// Acquire crypto provider context
HCRYPTPROV hCryptProv = NULL;
CryptAcquireContext(&hCryptProv, 0, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT);
// Import key
HCRYPTKEY key = NULL;
CryptImportKey(hCryptProv, blobBuf, dwBufSize, 0, 0, &key);
// Get the key size
DWORD dwKeySize;
DWORD dwParamSize = sizeof(DWORD);
CryptGetKeyParam(key, KP_KEYLEN, (BYTE*) &dwKeySize, &dwParamSize, 0);
// we need it in bytes for convenience
dwKeySize /= 8;
// Now the fun
// allocate a buffer of key size
BYTE *data = new BYTE[dwKeySize];
// Copy data need to be encrypted
// With PKCS#1 padding data length can not exceed keysize - 11 bytes
DWORD dataLen = 16;
memcpy(data, "0123456789012345", dataLen);
CryptEncrypt(key, 0, TRUE, 0, data, &dataLen, dwKeySize)
// now convert it to big endian (for the rest of the world)
for (int i = 0; i < (dwKeySize / 2); i++) {
    BYTE c = data[i];
    data[i] = data[dwKeySize - 1 - i];
    data[dwKeySize - 1 - i] = c;
}
// now data points to a dwKeySize length block of RSA PKCS#v1.5 encrypted data