SSL_CTX_use_PrivateKey_file fail under Linux

Question

I'm trying to use the SSL_CTX_use_PrivateKey_file function in OpenSSL under Linux, but it returns false. The surrounding code has been ported from Windows, where everything runs fine. Is there something that must be done differently depending on system?

I've compiled the OpenSSL library myself (default config etc) under Ubuntu and am using pre-compiled binaries for Windows (linked from the OpenSSL site).

The certificates are in .pem files as well as the key. Also, there's a password established.

The following is basically what's done;

SSL_CTX_set_default_passwd_cb( pContext, passwdCallback );
SSL_CTX_set_default_passwd_cb_userdata( pContext, (void*)this );
SSL_CTX_use_certificate_file( pContext, strCertificateFile, SSL_FILETYPE_PEM );
SSL_CTX_use_Privatekey_file( pContext, strPrivateKeyFile, SSL_FILETYPE_PEM ); // fail in Linux but work fine in Windows

Does anyone have an idea?

Answer 1

To keep things simple, I removed all code from my password callback, and had simple pBuf = "mypass"; return 6; This would be the bare-minimum of the callback function. This worked fine.

So what was different between the Windows code and the Linux code? Well, a call to strcpy_s and strcpy, respectively, was the only difference in the code. What's different between those two (except additonal validation parameters)?

To validate the string copy operation's success, the code simply checked for equality to 0. However, the two copy functions have different specifications for their return values. Microsoft changed "strcpy"'s return behaviour from "0 means error" to "0 means success". Sigh...