I'm using an ASIC simulator (vcs) that takes a bunch of different languages (Verilog, VHDL, Vera, C and Tcl in our case) and compiles them all into a C executable.
Which is all great, right until it core dumps with a cryptic stack trace. What's the best way to approach debugging this?
If you've already looked at the stack trace then I suspect you've gleaned most of the information that you can out of the core file. Especially if you're not familiar with how the simulator is implemented, and it doesn't sound like you are.
I would say that the tool is buggy, and that you ought to hold the vendor responsible. Dumping core is never a good response, even if there is a problem with the input you gave it.
If you want to try analyzing the problem further, however, one suggestion is to start with some simpler version of your ASIC design that did work, and then make changes incrementally, making sure that it still works after every set of changes. This way you can narrow down the specific part of your system that's causing the problem, and it might allow you to fix it or to file a more specific bug report with your vendor.
Of course, that assumes that you can go back to something that works, but I'm going to hope that you tested things at least a few times over the course of the design process, and that you have some sort of version control (or at least backups).