views:

481

answers:

1

SOLVED: I was dumb. First argument of encrypt should have been key.size() and first argument of decrypt should have been RSA_size(myKey).

Hey guys, I'm having some trouble figuring out how to do this.

Basically I just want a client and server to be able to send each other encrypted messages.

This is going to be incredibly insecure because I'm trying to figure this all out so I might as well start at the ground floor.

So far I've got all the keys working but encryption/decryption is giving me hell.

I'll start by saying I am using C++ but most of these functions require C strings so whatever I'm doing may be causing problems.

Note that on the client side I receive the following error in regards to decryption.

error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed

I don't really understand how padding works so I don't know how to fix it.

Anywho here are the relevant variables on each side followed by the code.

Client:

RSA *myKey; // Loaded with private key
// The below will hold the decrypted message
unsigned char* decrypted = (unsigned char*) malloc(RSA_size(myKey));
/* The below holds the encrypted string received over the network. 
Originally held in a C-string but C strings never work for me and scare me 
so I put it in a C++ string */
string encrypted;

// The reinterpret_cast line was to get rid of an error message.
// Maybe the cause of one of my problems?
if(RSA_private_decrypt(sizeof(encrypted.c_str()), reinterpret_cast<const unsigned char*>(encrypted.c_str()), decrypted, myKey, RSA_PKCS1_OAEP_PADDING)==-1)
    {
        cout << "Private decryption failed" << endl;
        ERR_error_string(ERR_peek_last_error(), errBuf);
        printf("Error: %s\n", errBuf);
        free(decrypted);
        exit(1);
    }

Server:

RSA *pkey; // Holds the client's public key
string key; // Holds a session key I want to encrypt and send
//The below will hold the encrypted message
unsigned char *encrypted = (unsigned char*)malloc(RSA_size(pkey));

// The reinterpret_cast line was to get rid of an error message.
// Maybe the cause of one of my problems?
if(RSA_public_encrypt(sizeof(key.c_str()), reinterpret_cast<const unsigned char*>(key.c_str()), encrypted, pkey, RSA_PKCS1_OAEP_PADDING)==-1)
        {
            cout << "Public encryption failed" << endl;
            ERR_error_string(ERR_peek_last_error(), errBuf);
            printf("Error: %s\n", errBuf);
            free(encrypted);
            exit(1);
        }

Let me once again state, in case I didn't before, that I know my code sucks but I'm just trying to establish a framework for understanding this.

I'm sorry if this offends you veteran coders.

Thanks in advance for any help you guys can provide!

+1  A: 

Maybe not the only problem but: The first argument to RAS_xxxcrypt functions is the number of bytes of the buffers. sizeof(key.c_str()) does not yield the number of bytes in key, it yields the size of the type of key.c_str()'s result type, i.e. sizeof(const char*). You probably want to pass the number of chars in the string instead, which can be obtained with the size() member function.

Éric Malenfant
Holy crap that helped though I'm still getting some problems.
Josh