C# - Multicore threading for database intensive app

Question

I have an application, which need to do some validation on the database end. Each process goes through database validation (search on the table that has around 6 million records), then goes through some Lucene Index searcher (the index is built off this 6 million record table). As these steps are disjoint for each line items being passed, I am thinking of utilizing multicore threading. (each of these lines take around 1/2 minute on a single thread).

What are my options with multicore in C#? Is there some good resources / third party library (I looked a bit at PowerThreading by Jeff Ritcher), some good tutorials.

I assume I need to do some thread pools in N core machines.

Currently, it takes around 40 secs to process 100 lines, looking to get this done to around 10 secs.

Thanks...

Answer 1

Have you looked into F#?

It is designed from the ground up for parallelizing tasks.

Answer 2

If you want a factor of four speed increase and have four cores, all you need to do is avoid dependencies among tasks, which sounds feasible. I think you'll find that you'll actually want to run more threads than cores because, at any given time, a number of threads will be blocked waiting for I/O. Therefore, I suggest that, whichever method you use, ensure that it's easy to benchmark it with different thread counts.

Answer 3

Simple threading should give you access to the multicore. You will have to play around with the size of the thread pool, as your tasks look to have a lot of IO as well.