Vectored Exception Handling During StackOverflowException

Question

If I've registered my very own vectored exception handler (VEH) and a StackOverflow exception had occurred in my process, when I'll reach to the VEH, will I'll be able to allocate more memory on the stack? will the allocation cause me to override some other memory? what will happen?
I know that in .Net this is why the entire stack is committed during the thread's creation, but let's say i'm writing in native and such scenario occurs ... what will i able to do inside the VEH? what about memory allocation..?

Answer 1

Stacks need to be contiguous, so you can't just allocate any random memory but have to allocate the next part of the address space.

If you are willing to preallocate the address space (i.e. just reserve a range of addresses without actually allocating memory), you can use VirtualAlloc. First you call it with the MEM_RESERVE flag to set aside the address space. Later, in your exception handler you can call it again with MEM_COMMIT to allocate physical memory to your pre-reserved address space.

Answer 2

In the case of a stack overflow, you'll have a tiny bit of stack to work with. It's enough stack to start a new thread, which will have an entirely new stack. From there, you can do whatever you need to do before terminating.

You cannot recover from a stack overflow, it would involve unwinding the stack, but your entire program would be destroyed in the progress. Here's some code I wrote for a stack-dumping utility:

// stack overflows cannot be handled, try to get output then quit   
set_current_thread(get_current_thread());
boost::thread t(stack_fail_thread);

t.join(); // will never exit

All this did was get the thread's handle so the stack dumping mechanism knew which thread to dump, start a new thread to do the dumping/logging, and wait for it to finish (which won't happen, the thread calls exit()).

For completeness, get_current_thread() looked like this:

const HANDLE process = GetCurrentProcess();
HANDLE thisThread = 0;
DuplicateHandle(process, GetCurrentThread(), process,
        &thisThread, 0, true, DUPLICATE_SAME_ACCESS);

All of these are "simple" functions that don't require a lot of room to work (and keep in mind, the compiler will inline these msot likely, removing a function call). You cannot, contrarily, throw an exception. Not only does that require much more work, but destructors can do quite a bit of work (like deallocating memory), which tend to be complex as well.

Your best bet is to start a new thread, save as much information about your application as you can or want, then terminate.