Catching "Run-Time Check Failure #0 - The value of ESP was not properly "

Question

So, I am calling a function from an unmanaged .dll file from my C# code. Depending on the arguments passed to that function, it can cause "Run-Time Check Failure #0 - The value of ESP was not properly " error.This is completely normal behavior for the function ( yes , I know this sounds VERY strange, but bear with me ). However, if this happens I want to be able to handle it , like I would handle an exception: just fail gracefully , output a message to the user and avoid crashing my entire program. Is this possible?

Answer 1

If you are using the GS flag you can override the behavior by calling the function __set_security_error_handler. This allows you to specify the function that is called when the GS flag, and other security errors, is tripped.

I highly advise against this though. Any function that regularly corrupts ESP sounds like a real problem. You also risk hiding other real security issues by overriding this function.

A much better solution would be to mark the individual method as naked and prevent the GS check from occuring.

• http://msdn.microsoft.com/en-us/library/h5w10wxs(VS.80).aspx