C run time stackoverflow

Question

#include <stdio.h>

int doHello(){
    doHello();
}

int main(){
    doHello();
    printf("\nLeaving Main");
    return 0;
}

When you run the program quits without printing the message "Leaving Main" on the screen. This is a case of Stack Overflow and because of which program is terminating but I don't see any error messages on the command window. (Ran on Windows/Cygwin/)

Q1. I have not declared any local variables in the doHello function but still stack is getting used. Is this is because of

• return values
• information stored about the function calls?

---

Clarification

Q2. How to debug such cases in your program? I am not asking to debug an infinite loop which I mentioned above.

for example:

#define SIZE 512*1024
void doOVerflow(){
   char str[SIZE];
   doHello();
}

void doHello(){
   char strHello[256];  // stack gets filled up at this point
   doNothing();         // program terminates and function doNothing does not get called
}

EDIT:

Q3. What information is stored in run time stack?

Answer 1

is this is because of a. return values b. information stored about the function calls?

In your case, it is the return addresses that are consuming the stack.

How to debug such cases in your program?

Don't write them in the first place. A recursive function must always have a terminating condition.

Q3. What information stored in run time stack?

For most languages, local variables for each function invocation, and the return address for each function invocation.

When a function call is made, the current execution point is pushed omto the stack and the code for the function is executed. When that execution ends, the value on the stack is popped and the execution path returns to that point. In your code, the return address is pushed onto the stack, the function is called - the first thing that call of the function does is call itself, which means the return address is pushed onto the stack, and the function is called, and so on, ad infimnitum, or untill you get a stack overflow.

Answer 2

The address where the call is made is stored on the stack for each time you call the function.

Answer 3

Even though you don't have any local variables, the stack frame will still contain the return address.

Note: Modern compilers on sufficiently high optimization levels should be able to perform tail call optimization, thus avoiding the stack overflow.

Answer 4

The other guys already addresses why you are getting a stack overflow.

Just an extra note on debugging. Most modern debuggers will break on stack overflow and let you look at the call stack in the debugger. At that point, you would see that same function over and over. If you're not using a modern debugger to diagnose this type of issue, you're making things needlessly tough on yourself.

Answer 5

It's frame pointers and return addresses, usually. See e.g. wikipedia's "call stack" article. In case you're curious:

$ gcc -S test.c  # <--- assembles, but does not compile, test.c; output in test.s
$ cat test.s
// [some contents snipped]
_doHello:
        pushl   %ebp        // <--- pushes address of stack frame onto stack
        movl    %esp, %ebp  // <--- sets up new stack frame
        call    _doHello    // <--- pushes return value onto stack, makes call
        popl    %ebp        // <--- pops address of stack frame off stack
        ret                 // <--- pops return value off stack, returns to it

For fun, try "-fomit-frame-pointers":

$ gcc -fomit-frame-pointers -S test.c
$ cat test.s
// [some contents snipped]
_doHello:
        call    _doHello   // <--- pushes return value onto stack, makes call
        ret                // <--- pops return value off stack, returns to it

For even more fun, let's see what happens when we turn on optimizations:

$ gcc -fomit-frame-pointers -O4 -S test.c # <--- heavy optimization
$ cat test.s
// [some contents snipped]
_doHello:
L2:
        jmp     L2         // <--- no more stack operations!

The last version will run forever, rather than quitting, at least on my setup (cygwin, at the moment).

To diagnose issues like this, you can run in your favorite debugger (e.g. Microsoft Visual C++ or gdb) or examine stackdumps that are usually generated by most systems (.core or .stackdump files) using those debuggers.

You may also be able to set a hardware breakpoint near the top of your stack, if your debugger supports this -- any attempt to write to this variable, and your stack may be getting full. Some operating systems have additional mechanisms to alert you to stack overflows.

Finally, debugging environments such as valgrind or Application Verifier may help.

Answer 6

Disassemble and analyze the function.

Let's take your first code (saved as so.c):

int doHello(){
    doHello();
}

int main(){
    doHello();
    printf("\nLeaving Main");
    return 0;
}

Compile and disassemble doHello():

$ gcc -Wall -ggdb3 -O0 so.c -o so
$ gdb --annotate=3 so
(gdb) disassemble doHello
Dump of assembler code for function doHello:
0x080483e4 <doHello+0>: push   %ebp
0x080483e5 <doHello+1>: mov    %esp,%ebp
0x080483e7 <doHello+3>: sub    $0x8,%esp
0x080483ea <doHello+6>: call   0x80483e4 <doHello>
0x080483ef <doHello+11>:        leave
0x080483f0 <doHello+12>:        ret
End of assembler dump.

Now, we have the assembly code for the function and we can see exactly what it's doing. As it happens, the answer to the question is staring us in the face: first instruction pushes an dword (the return pointer) onto the stack.

Hope this clarifies things.

Cheers.