Recommendations for IPC between parent and child processes in .NET?

Question

My .NET program needs to run an algorithm that makes heavy use of 3rd party libraries (32-bit), most of which are unmanaged code. I want to drive the CPU as hard as I can, so the code runs several threads in parallel to divide up the work.

I find that running all these threads simultaneously results in temporary memory spikes, causing the process' virtual memory size to approach the 2 GB limit. This memory is released back pretty quickly, but occasionally if enough threads enter the wrong sections of code at once, the process crosses the "red line" and either the unmanaged code or the .NET code encounters an out of memory error. I can throttle back the number of threads but then my CPU usage is not as high as I would like.

I am thinking of creating worker processes rather than worker threads to help avoid the out of memory errors, since doing so would give each thread of execution its own 2 GB of virtual address space (my box has lots of RAM). I am wondering what are the best/easiest methods to communicate the input and output between the processes in .NET?

The file system is an obvious choice. I am used to shared memory, named pipes, and such from my UNIX background. Is there a Windows or .NET specific mechanism I should use?

Answer 1

If you use WPF, many common forms of IPC are available. This can be done in such a way that config file changes are all that is required, and leaves room for communicating across machine boundaries as well.

Answer 2

IPC in .NET 3.0+ can be done with WCF and named pipes binding. The most obvious benefit is that you can split your heavy tasks across machines by simply switching the binding. More on this here.

Regarding memory footprint you may try to get your 3rd party libraries in 64-bit and run your stuff on 64-bit OS to get more memory available to your process.