Can I programatically deduce the calling convention used by a C++ dll?

Question

Imagine you'd like to write a program that tests functions in a c++ dll file. You should enable the user to select a dll (we assume we are talking about c++ dlls). He should be able to obtain a list of all functions exported by the dll. Then, the user should be able to select a function name from the list, manually input a list of arguments ( the arguments are all basic types, like int, double, bool or char arrays (e.g. c-type strings) ) and attempt to run the selected function with the specified arguments. He'd like to know if the function runs with the specified arguments, or do they cause it to crash ( because they don't match the signature for example ).

The main problem is that C++, being a strongly typed language, requires you to know the number and type of the arguments for a function call at compile time.And in my case, I simply don't know what these arguments are, until the user selects them at runtime.

The only solution I came up with, was to use assembly to manually push the arguments on the call stack.

However, I've come to understand that if I want to mess with assembly, I'd better make damn sure that I know which calling convention are the functions in the dll using.

So (finally:) here's my question: can I deduce the calling convention programmaticaly? Dependency Walker won't help me, and I've no idea how to manually read PE format.

Answer 1

The compiled code does not just say 'Here this function is a fastcall, and this one here is stdcall' unfortunately.

Not even modern disassemblers like IDA try to deduce call types by default (there might be a plugin or an option somewhere idk).

Basically if you are a human you cn look at the first few instructions and tell 90% of the time. If they are pop and push, its stdcall, if its passing params through the registers (especially ecx) then its cdecl. Fastcall also uses the registers but does something special.. dunno off the top of my head. But all this info is useless because your program obviously will not be a human.

If you are doing testing, dont you at least have the header files?? This is an awfully hard way to skin a cat..

Answer 2

If you want to know what calling convention a C++ function uses, your best hope is to study

1. The header that declares that function, and
2. The documentation for the compiler that compiled your particular DLL.

But this whole thing sounds like a bit of a mess, honestly. Why does your friend want to be able to do this, and why can't he get the information he needs by parsing a header that declares the relevant functions?

Answer 3

The answer is maybe.

If the functions names are C++ decorated, then you can determine the argument count and types from the name decoration, this is your best case scenario, and fairly likely if MSVC was used to write the code in the first place.

If the exported functions are stdcall calling convention (the default for windows api), you can determine the number of bytes to be pushed, but not the types of the arguments.

The bad news is that for C calling convention, there isn't any way to tell by looking at the symbol names. You would need to have access to the source code or the debug info.

http://en.wikipedia.org/wiki/X86_calling_conventions

The name that a function is given as an export is not required to have any relationship with the name that the linker sees, but most of the time, the exported name and the symbol name that the linker sees are the same.

Answer 4

You didn't specify whether you're talking 32-bit or 64-bit here, and the difficulties outlined by you and the other posters mainly apply to 32-bit code. On 64-bit Windows, there's essentially only one calling convention (it's in also in the wikipedia article linked by John Knoeller), which means that you do know the calling convention (of course with the exception of anybody cooking up their own).

Also, with the Microsoft x64 calling convention, not knowing the number of parameters of the function to be called does not stop you from calling it, providing as many parameters as you wish/the user wishes to. This is because you as a caller set aside stack space and clean it up afterwards. -- Of course, not providing the right [number of] parameters may still have the called function do silly things because you're providing invalid input, but that's another story.

Answer 5

This page describes the way VC++6 encodes parameter and calling convention info into a symbol name: http://www.bottledlight.com/docs/mangle.html

I suspect that later versions of VC++ will be compatible but I haven't confirmed this.

There are also some tools that automate this which accompany the compiler: http://msdn.microsoft.com/en-us/library/5x49w699.aspx

The name mangling only applies for C++ functions; if a function is 'extern "C"' then this won't work.