Yongwei,
Colin is right; in effect, you're racing against IIS as it is recylcing the application pool. As you're snapping your process snapshot, you're either hitting a memory-threshold for recycling, or health checks are perceiving the process to be hung and instituting a recycle (possibly due to ADPlus locking the process)
Here's how I would modify your application pool characteristics prior to attempting your next capture. You only need these changes in effect for as long as it takes to capture your dump:
- Turn off memory-based recycling limits (physical and virtual)
- Turn off the idle timeout limit (if it's on)
- Disable both Pinging and Rapid Fail Protection
In effect: you need to turn off all of the features that try to keep your app pools running well. Capturing a memory snapshot takes time (as you know).
I would also recommend checking out ProcDump (http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx) from the SysInternals guys. It was just released last month, and it makes process memory captures a bit easier. An article on using it to capture the W3WP is here: http://blogs.msdn.com/webtopics/archive/2009/08/08/using-procdump-exe-to-monitor-w3wp-exe-for-cpu-spikes.aspx
I hope this helps!