Custom X86_64 calling convention to C function call

Question

Hi stackoverflow

I've to do an interface (say, a wrapper) that allow a call from X86_64 assembly code using his calling convention to C function, with other calling convention. The best thing would be to be pretty "compiler independant" (juste modifying the wrapper), so i'm looking for something that puts registers/stack stuff on compiler back. I've already look here : http://stackoverflow.com/questions/390798/custom-calling-convention-for-p-invoke-and-c and it's come close from what i've to do. Currently, I'm using GCC, but hints from other compilers are welcome !

So, here is the thing, for a best view of the problem (the custom coding convention is strange ) :

pushq  %r11    # saves r11 for call
movq 64bits_address %r11 # move a 64 bits address that points on a data structure
callq *8(%r11) # calls an address in the data structure
popq %r11      # restores %r11 ; return value is in the structure

I need to be able to call a "special" (wrapper) C function ; here job will be to dispatch calls between other C functions. So this wrapper needs to find %r11, save all registers and prepare the stack for further call. Is there a proper way to do this in C (with some inline asm) ?

Thanks a lot

Answer 1

For documentation about calling conventions and how are the parameter passed to a function (in registers? which? what's on the stack etc) have a look at Agner Fog's document.

Then, you can have a look at libffi's source code to see how they do it.