BN_hex2bn magically segfaults in openSSL

Question

Greetings, this is my first post on stackoverflow, and i'm sorry if its a bit long.

I'm trying to build a handshake protocol for my own project and am having issues with the server converting the clients RSA's public key to a Bignum. It works in my clent code, but the server segfaults when attempting to convert the hex value of the clients public RSA to a bignum.

I have already checked that there is no garbidge before or after the RSA data, and have looked online, but i'm stuck.

header segment:

typedef struct KEYS {  
    RSA *serv;  
    char* serv_pub;  
    int pub_size;  
    RSA *clnt;  
} KEYS;

KEYS keys;

Initializing function:

// Generates and validates the servers key

/* code for generating server RSA left out, it's working */
//Set client exponent
keys.clnt = 0;
keys.clnt = RSA_new();
BN_dec2bn(&keys.clnt->e, RSA_E_S); // RSA_E_S contains the public exponent

Problem code (in Network::server_handshake):

// *Recieved an encrypted message from the network and decrypt into 'buffer' (1024 byte long)*
cout << "Assigning clients RSA" << endl;
// I have verified that 'buffer' contains the proper key
if (BN_hex2bn(&keys.clnt->n, buffer) < 0) { 
    Error("ERROR reading server RSA");
}
cout << "clients RSA has been assigned" << endl;

The program segfaults at

BN_hex2bn(&keys.clnt->n, buffer)

with the error (valgrind output)

Invalid read of size 8 at 0x50DBF9F: BN_hex2bn (in /usr/lib/libcrypto.so.0.9.8) by 0x40F23E: Network::server_handshake() (Network.cpp:177) by 0x40EF42: Network::startNet() (Network.cpp:126) by 0x403C38: main (server.cpp:51) Address 0x20 is not stack'd, malloc'd or (recently) free'd

Process terminating with default action of signal 11 (SIGSEGV) Access not within mapped region at address 0x20 at 0x50DBF9F: BN_hex2bn (in /usr/lib/libcrypto.so.0.9.8)

And I don't know why it is, Im using the exact same code in the client program, and it works just fine. Any input is greatly appriciated!

Answer 1

RSA_new() only creates the RSA struct, it does not create any of the bignum objects inside that struct, like the n and e fields. You must create these yourself using BN_new(), or more likely you need to find the right openssl function to generate or read in your RSA key.