how do procedure calls work in assembler?

Question

I just started tinkering with ASM and I'm not sure if my understanding of procedure calls is correct.

say at some point in the code there is a procedure call

call dword ptr[123]

and the procedure consists of only one command, ret:

ret 0004

what would be the effect of this procedure call, and where would the return value be stored? I read somewhere that a return value of 2 bytes would be stored in AX, but when I replace the procedure call by

mov AX, 0004

(together with the necessary NOPs) the program crashes.

Answer 1

I don't think the return value is stored in the register AX

Answer 2

// possibly there are arguments pushed here
...
call dword ptr[123] // push next OP code offset in the stack and jump to procedure

// procedure
...
ret 0004 // pop offset, set EIP to that offset and decrease ESP by 4

we additionally decrease the ESP if we had pushed arguments in the stack prior calling the procedure.

If there are pushed arguments, your program crashes because you don't pop them. The return offset for the current procedure will be wrong since it will get a value from one of the pushed arguments as the offset.

Answer 3

in x86 assembler the parameter to the ret instruction means:

RET immediate

Return to calling procedure and pop immediate bytes from the stack.

(quoting from Intel® 64 and IA-32 Architectures Software Developer's Manuals Vol 2B)

So when you type:

ret 0004

You're telling the CPU to return to the instruction immediately after the call, and to pop 4 bytes off the stack. This is great if you pushed 4 bytes onto the stack before the call.

push eax
call dword ptr[123]

Note that this has nothing to do with the return value. In fact, a procedure in Assembly has no way of specifying that a value is a return value. This is all done by convention. Most compilers of which I am aware will use EAX to hold the return value, but this is true only because the calling function will expect the result there.

So your calling code would be:

call dword ptr [123]
mov dword ptr [result], eax

and your function that returns the value 4 would be:

mov eax, 4
ret

Answer 4

It all depends on the calling convention being used. I won't repeat the Wikipedia article here, just read the definition.

In the C calling convention, for example, the return value would be in EAX/AX/AL. Your single-instruction does not have one: It is a void function taking around 4 bytes of parameters (possibly a single int) that does nothing. As it is the callee's duty to clean up the stack in this calling convention, ignoring to do that and replacing the call with a 'mov ax' does not work.

Also I suspect you may be tinkering with 32-bit assembly while reading a 16-bit document. It's not a big problem, but you should be aware of the differences.