Recovering from stack overflow on Mac OS X

Question

Hi all,

I am implementing a cross platform scripting language for our product. There is a requirement to detect and properly handle stack overflow condition in language VM. Before you jump in and say make sure there is no stack overflow in the first place, re-read my first sentence - this is a scripting language and end users may write incorrect programs in this language, which may overflow the stack via for example endless recursion.

Now I know how to detect and recover from stack overflow in Windows (see http://support.microsoft.com/kb/315937). However I am unable to find any solution for Mac OS X.

The VM is implemented in C++: MSVC++ on Windows, GCC on Mac OS X.

Ideally the mechanism must be based on UNIX capabilities since we also plan to port to Linux.

Thanks.

Answer 1

OCaml has the same constraints as you ("scripting" language where the programmer may cause a stack overflow). Its native compiler uses the system stack for function calls -- as you do -- and it handles stack overflows (materializing them as exceptions).

If you do not receive a more explicit answer, I suggest you look at how it's done in the OCaml sources.

~/ppc $ cat >> t.ml

let rec f x = (f x) + (f x) ;;

f 0 ;;

~/ppc $ ocamlopt t.ml
~/ppc $ ./a.out 
Fatal error: exception Stack_overflow

The above is on Mac OS X Leopard. Search for #ifdef HAS_STACK_OVERFLOW_DETECTION in the source files.

Answer 2

Can't you determine a max stack size for your program, the start of the stack and then compute the end of the stack adress? Before each operation in the vm, you can then check if the stack will overflow (increase over the max adress) and then raise the exception. Another way would be writing an identifier after the end of stack (like DEADBEEF or so) and check for it's value after each stack operation.

---

I still don't see why it isn't possible for you to add checks (this is how stack-manipulation is normally detected). But you could still add a hardware breakpoint (on write) at the end of your stackarea. When it breaks, you know you have a stackoverflow. I wouldn't advice it though since it may mess up your debugging environment.

But basically - if you have a VM, you also have place that reads the code (e.g. bytecode) and compiles it / interprets it. That's the place where you can check for the mark (comparable to the checks to detect stack buffer overflows).

Answer 3

Have a look at libsigsegv. It's a C library with routines to help you implement a stack overflow handler, among other things.