views:

52

answers:

2

Hey all,

Having some issues with C. I have this is my code:

// First line works, second line does not! 
char outbuf[1024];
// char *outbuf = (char *) malloc(1024);  // char is always 1

I am passing this outbuf to a method called PK11_CipherOp(), declared in the NSS library. The Documentation for this method can be found here, you can see that it expects a char * for the outbuf param.

I cannot understand why the first line works reliably, and the second line does not!

To see the full code, please see File Source

Thanks,

Hamilton

+3  A: 

Your problem appears to be a missing declaration for malloc - you haven't included <stdlib.h>.

This means that your compiler is assuming the function returns int, which is then being explicitly cast to (unsigned char *). If you are on a platform where sizeof(int) != sizeof(void *), like x86-64, then this will mangle the pointer value.

This illustrates perfectly why in C (unlike C++) you should not cast the result of malloc() - doing so is unnecessary, and hides this exact bug from you. It also illustrates why you should turn on compiler warnings.

The fix is to #include <stdlib.h>, and remove the cast from malloc() while you're there.

Addendum:

Your other issue is that you're passing sizeof(outbuf) to PK11_CipherOp(). This will be correct (equal to out_buf_size) when outbuf is an array, but is incorrect (the size of the pointer itself) when outbuf is a pointer.

Replace each sizeof(outbuf) with out_buf_size and you should be good to go (there's another one later too)...

caf
caf, Thanks for the help. Unfortunately, that did not fix the issue. I have warnings on (and didnt get one about declaring malloc), but I implemented your solution just to be sure. You can see the updated source at http://code.google.com/p/sysecure/source/browse/trunk/session_keys.c?r=27#119When run, this prints: sysecure: Error when attempting to encrypt messagesysecure: Failure to perform cipher operation (err -8189)sysecure: Error Name - ((null))sysecure: Error Message - (Unknown code ___f 3)and to stderrEncrypted Data: Data length 0 Any thoughts would really help me out.
Hamy
Ah yes, there's a second issue there too - see my updated answer.
caf
A: 

Found the issue. Turns out that I was using sizeof(outbuf) as one of the parameters. When the outbuf is declared on the stack, as char outbuf[1024]; this returns 1024. However, when outbuf is on the heap, as char * outbuf = malloc(1024); the size of outbuf is just 4. This parameter told the function how much room was available in the outbuf, so the function thought that it only had 4 bytes of room. It needs 16 to do anything so it just returned without performing any work.

Hamy