strange assembler... could this be the cause of my apps crash?

Question

I have a program in native C++ under windows that uses a VB6 DLL. The C++ app crashes with a "Floating Point In-exact Result" error when using that DLL, for a certain action.

When I go into debug and do view assembler I get this line:

75A0FB7C  je          759E8797 

When you hover over the right most address it shows a tooltip with this in:

1.#INF0000000000

Could this be why I get this error?

Does anyone know why this is caused? Running this VB6 DLL under the VB6 IDE does not produce such an error...

Answer 1

I suspect the debugger is misinterpreting that address as a float, and that bit representation happens to be #INF. At least in some cases x86 hardware will crash on the floating point statement AFTER the one that actually caused the problem. I think we need more context, but was one of the preceding statements a floating point comparison in preparation for the conditional jump?

It seems possible that the C++ program is doing some floating point math that's setting one of the FP status registers, and then the VB6 DLL is purely a victim that the FPU is in an unstable state when it's called.

Have you determined precisely what inputs introduce the issue?

Answer 2

No, this has nothing to do with the FPU. The VB6 runtime is ab/using that floating point exception to implement its own exception handling. The On Error statement, I'm sure you're familiar with it.

Why on Earth they chose to re-use a hardware exception code has always been mysterious to me, that tidbit is lost in 15 odds years of Visual Basic design. It doesn't byte because the FPU is initialized by the VB6 runtime with that exception masked.

Anyhoo, the diagnostic is that your VB6 code is crashing with an unhandled exception. To find out what is going on, be sure to run the code from the VB6 debugger. Also make sure you check your Debug + Exceptions dialog, the Thrown boxes should be off. Press F5 to let the exception be handled the normal way. It ought to go kaboom after that.

From MarkJ's helpful link:

Another exception that you will commonly see is c000008f. If you look the number up then you will find that it is a floating point inexact result exception. It is used in a different meaning here – since we don’t generate real floating point inexact result exceptions, they can safely be thrown to indicate VB errors of the normal trappable type