SIMPLE OpenSSL RSA Encryption in C/C++ is causing me headaches

Question

SOLVED: I was dumb. First argument of encrypt should have been key.size() and first argument of decrypt should have been RSA_size(myKey).

Hey guys, I'm having some trouble figuring out how to do this.

Basically I just want a client and server to be able to send each other encrypted messages.

This is going to be incredibly insecure because I'm trying to figure this all out so I might as well start at the ground floor.

So far I've got all the keys working but encryption/decryption is giving me hell.

I'll start by saying I am using C++ but most of these functions require C strings so whatever I'm doing may be causing problems.

Note that on the client side I receive the following error in regards to decryption.

error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed

I don't really understand how padding works so I don't know how to fix it.

Anywho here are the relevant variables on each side followed by the code.

Client:

RSA *myKey; // Loaded with private key
// The below will hold the decrypted message
unsigned char* decrypted = (unsigned char*) malloc(RSA_size(myKey));
/* The below holds the encrypted string received over the network. 
Originally held in a C-string but C strings never work for me and scare me 
so I put it in a C++ string */
string encrypted;

// The reinterpret_cast line was to get rid of an error message.
// Maybe the cause of one of my problems?
if(RSA_private_decrypt(sizeof(encrypted.c_str()), reinterpret_cast<const unsigned char*>(encrypted.c_str()), decrypted, myKey, RSA_PKCS1_OAEP_PADDING)==-1)
    {
        cout << "Private decryption failed" << endl;
        ERR_error_string(ERR_peek_last_error(), errBuf);
        printf("Error: %s\n", errBuf);
        free(decrypted);
        exit(1);
    }

Server:

RSA *pkey; // Holds the client's public key
string key; // Holds a session key I want to encrypt and send
//The below will hold the encrypted message
unsigned char *encrypted = (unsigned char*)malloc(RSA_size(pkey));

// The reinterpret_cast line was to get rid of an error message.
// Maybe the cause of one of my problems?
if(RSA_public_encrypt(sizeof(key.c_str()), reinterpret_cast<const unsigned char*>(key.c_str()), encrypted, pkey, RSA_PKCS1_OAEP_PADDING)==-1)
        {
            cout << "Public encryption failed" << endl;
            ERR_error_string(ERR_peek_last_error(), errBuf);
            printf("Error: %s\n", errBuf);
            free(encrypted);
            exit(1);
        }

Let me once again state, in case I didn't before, that I know my code sucks but I'm just trying to establish a framework for understanding this.

I'm sorry if this offends you veteran coders.

Thanks in advance for any help you guys can provide!

Answer 1

Maybe not the only problem but: The first argument to RAS_xxxcrypt functions is the number of bytes of the buffers. sizeof(key.c_str()) does not yield the number of bytes in key, it yields the size of the type of key.c_str()'s result type, i.e. sizeof(const char*). You probably want to pass the number of chars in the string instead, which can be obtained with the size() member function.