Attempted to read or write protected memory when calling native C DLL.

Question

I have a native C dll that exports one function besides DllEntryPoint, FuncX. I'm trying to find out how FuncX communicates with it's caller, because it has a void return type and no parameters. When I call it from a C# harness, I get an AccessViolationException - Attempted to read or write protected memory.

I have a hunch that its client application may allocate a buffer for sending or receiving values from the dll. Is this a valid hunch?

I can't debug the client application because for some reason it doesn't run, so I can't start it and attach to the process. I can, however, disassemble it in IDA Pro, but don't know how to, if I can, try and debug it in there.

Answer 1

If the DLL in question has any static or global symbols, it's possible that all communication is done via those symbols. Do you have any API code that looks like it might be doing this?

It is unlikely that the DLL is using a client supplied buffer, as both client and server would need to know the base address of that buffer, and you can't ask calloc or malloc for a "preferred" address at call time.

You might also try running link /dump /symbols and point it at your DLL. That will show you the list of exported symbols in your DLL. Good luck!

Answer 2

I would try loading the DLL itself into IDA Pro. Hopefully C# preserves the native call stack, and you can look at the code around where the DLL crashes.

Side note: the Decompiler plugin is pretty awesome.