x86: ZF not always updated by AND?

Question

Hello! I'm debugging my code on x86 and the problem tracks down to AND instruction sometimes does not clear ZF flag if the result is not zero. Here is the problematic piece of code:

0257A9F9  mov         edx,dword ptr [ecx+18h] 
0257A9FC  and         edx,80000h 
0257AA02  int         3    
0257AA03  je          0257AA2A 

I added a breakpoint after AND for debugging. When it stops on the breakpoint EDX==0x80000 and ZF==1. But ZF should be cleared if EDX!=0. The code works perfectly fine when single stepped in debugger, but it fails consistently during normal run.

Here is a screenshot of debugger session.

Any hints?

If that matters the code is generated by JIT, so I'm executing data.

Thank you in advance.

Answer 1

According to the Intel instruction set reference, ZF is always set according to the result. Could something in the int 3 handler be manipulating this?

Edit: After further digging through the manuals (thank god for Intel sending out free copies!), my only ideas are that it's either the int 3 handler setting it somehow, or the processor only looking at dx instead of edx when setting flags. Both seem unlikely, but the latter seems completely implausible. What mode are you running in? (Real, protected, unreal, long?)

Answer 2

You can easily examine the int 3 handler to see if it's returning with a iret (i.e. pop back the callers flags) or if it's returning with retf 2 (i.e. preserve the flags from the handler).

Answer 3

It could be your debugger is doing something special, such as syncing the memeroy and registers. When you run it without the debugger it fails you say?

Answer 4

Thanks everyone. It was my fault, sorry to bother you. There is a branch to 'int 3' from another place. That's why the flags are inconsistent with instructions before 'int 3'. I was confused by always having edx==0x80000 at this point. Sorry again.