_commonBeginModalSessionForWindow:relativeToWindow:modalDelegate:didEndSelector:contextInfo hangs

Question

Hi, i've been experiencing a weird bug for days. In my program i use a sheet attached to the main window and at random it hangs without displaying the sheet itself.

Here's the trace:

#0  0x95af0266 in mach_msg_trap
#1  0x95af7a5c in mach_msg
#2  0x951e6e7e in CFRunLoopRunSpecific
#3  0x951e7aa8 in CFRunLoopRunInMode
#4  0x944e00ce in -[NSMoveHelper _doAnimation]
#5  0x944deb8d in -[NSMoveHelper(NSSheets) _moveParent:andOpenSheet:]
#6  0x944de47c in -[NSWindow(NSSheets) _orderFrontRelativeToWindow:]
#7  0x9433a6d7 in -[NSWindow _reallyDoOrderWindow:relativeTo:findKey:forCounter:force:isModal:]
#8  0x9437c567 in -[NSApplication _orderFrontModalWindow:relativeToWindow:]
#9  0x9437bfd1 in -[NSApplication _commonBeginModalSessionForWindow:relativeToWindow:modalDelegate:didEndSelector:contextInfo :]

After doing a lot of tests i found it doesn't happen when passing nil to modalForWindow parameter, perhaps because there will be no showing animation and _doAnimation never gets called.

Digging a little bit i compared the CPU register contents before and after the hangs. Well, it appears to me that after an hang some x87 registers have a NaN:

$fctrl    0x37f  
$fstat    0x1061  
$ftag     0xf00f  
$fiseg    0x17  
$fioff    0x943830c5  
$foseg    0x1f  
$fooff    0xbfffe368  
$fop      0x59d  
$st0      <a float number>  
$st1      <a float number>  
$st2      <a float number>  
$st3      <a float number>  
$st4      1  
$st5      -nan(0xc000000000000000)  
$st6      -nan(0xc000000000000000)  
$st7      <a float number>

Maybe this is the cause of the msg_trap.

I'm on an Intel machine running osx 10.5.8 and this problem still exists on 10.6.4. I haven't a PPC machine, so i cannot test on this arch.

I don't know what to do, maybe _doAnimation uses $stn registers and hangs because they are dirty. Any suggestions?

Thanks Marco

UPDATE: i've corrected some compiler warnings concerning implicit casts and the problem seems to be solved. Perhaps some random junk/corrupted data in the sheet RunLoop thread stack that were not cleaned by the beginSheet method caused the hang.