It is not really malloc/free which causes the exception, it is "new" which is definitely in C++ part of your application. It looks like you are providing a parameter which is too big for "new" to allocate.
'std::bad_alloc' is caused by the following code for example:
int * p = new int[50000000];
What does backtrace says when you load crash dump into gdb?
If you cannot generate dump, you can ask GDB to stop when exception is thrown or caught.
Unfortunately, some versions of GDB support only the following syntax:
catch throw
which allows you to break application when any exception is thrown.
However, in help you see that it should be possible to run
catch throw std::bad_alloc
in newer versions.
And don't forget that:
(gdb) help catch
is a good source for other useful information.