+1  A: 

I think the message on your second trace is the most instructive

A native exception has occurred on BbCore.exe

It appears that it's a native exception not a managed exception that is taking down your product. Native exceptions are as a general rule not catchable by managed code. It is possible in certain cases but generally speaking native exceptions are fatal.

You can try to use the catch block for SEH exceptions and see if that works.

try { 
  ...
} catch { 

}

But in general if the native code is throwing your application is unstable and it should crash.

JaredPar
So, putting in a catch with no (Exception ex) might be able to catch unmanaged exceptions? I didn't realize that. It makes sense since the (Exception ex) would only trap errors derived from Exception. Great tip!
Steve Hiner
I guess I'm ok if it crashes but I'd at least like to log it so I can find out what the input was that caused it to crash. The times this has happened I have no reason to believe the input would have been anything other than something we create in code and no different than every other time we call the method. I'm starting to wonder if memory is getting corrupted. If I can see the input I'll find out if the input even should cause it to crash and if the input is something different than what the other code sent.
Steve Hiner
+1  A: 

Turned out it was a buffer overrun error in native code. the C# code was calling into a method and passing a byte array with 8 elements. The C++ code was filling 6 bytes then overwriting another 6 with zeroes. This method got called quite a lot and every time it was overwriting 4 bytes of memory with zeroes. Doh!

That explains the utterly strange errors, probably was overwriting bits of the .Net framework in memory.

Gotta watch out for those interactions between managed and unmanaged code. Fortunately for me, the code in question wasn't mine.

(Not sure if I should accept my own answer as the answer since no one could have answered this without looking at our code base. Likewise I shouldn't mark JaredPar's answer as "the answer" since the problem was memory corruption, not really a native exception. I guess I'll post this anyway since maybe someone else will have a similar situation and maybe they'll look into interactions with native code. Admins: feel free to delete this thread if you think that's best.)

Steve Hiner
You should accept your own answer ;) Btw, I think i'm having the same issues, this confirms some suspicions I had. Thanks.
Stormenet