Database operation via Threadpool or Asyn programming

Question

Hi guys,
i have database with larger number of records.(database values are updated via a webservice)

Each record/row is of type (id,xmlfilename,operation,parameters,operationId,status) ie. we have perform operation as specified by'operation' on a xmlfile specified by "xmlfilename" with parameters for operation specified by "parameters".. status part is "intially" free as is updated as when reqd.

I have to do these (each row) operation using Threads..ie one thread per 'id' number of entries.as long as there are 'id' rows in db fetch and perform "operation"

how can i do this efficiently with maximum parallelism and/or concurrency.

Is there a better way?

What is best suited Threadpool or custom threads or asyn programming?

Edit Added:Here is pseudo code i tried

string operationId=null;

while(operationId = DBWorker.getNextFreeOperationId ())
//how do i check this?another query??untill there are operations with status "Free" 
//,keep selecting operationids 
//note db is updating asynchronously.    
{ 
  //retrieve all rows with operationid=operationId eg:800 . 1 thread/qid???? and
  // status="free" ...
//there are multiple operations with same operationIds 

  DataRowCollection dbRows=DBWorker.retrieveQueuedEntries(operationId); 
  MyWorkItem workItem = new DBWorker.MyWorkItem();
  workItem.DataRows = dbRows; 
  workItem.Event = new AutoResetEvent(false); 
 //MyWorkItem.DoWork will do the necessary "Operation" 
 ThreadPool.QueueUserWorkItem(new WaitCallback(workItem.DoWork),workItem); 
} 
--------------
 MyWorkItem.DoWork(obj x){ 
//for brevity 
for each DataRow row in this.DataRows
  {
    performOperation(row);//use row["operation"] ..
  }
---------------
bool performOperation(DataRow row)
{
  string operation = (string)row["operation"];//retrieve other similarly
  switch(operation)
  {
    case Operations.Add: //call Add operation ..
    ...
  }
}
------------

above code is using.net2.0.. doesnt achieve multithreading .. scenario may be the database is updating from one end asynchronously,while above code will be running as windows service at the same time.

thx
Amit

Answer 1

If you're not using .net 4.0 - Use ThreadPool I would create a function that would receive the row (event better if you convert the rows to strongly type objects in DAL) it has to process, and do what it has to do with it - something like

using System;
using System.Threading;

namespace SmallConsoleAppForTests
{
class Program
{

    private static AutoResetEvent[] events;

    static void Main(string[] args)
    {

        int dataLength = 3;
        // creating array of AutoResetEvent for signalling that the processing is done
        AutoResetEvent[] events = new AutoResetEvent[dataLength];

        // Initializing the AutoResetEvent array to "not-set" values;
        for (int i = 0; i < dataLength; i++)
            events[i] = new AutoResetEvent(false);

        //Processing the data
        for (int i = 0; i < dataLength; i++)
        {
            var data = new MyWorkItem { Event = events[i], Data = new MyDataClass() };

            ThreadPool.QueueUserWorkItem(x =>
            {
                var workItem = (MyWorkItem)x;
                try
                {
                    // process the data

                }
                catch (Exception e)
                {
                    //exception handling
                }
                finally
                {
                    workItem.Event.Set();
                }
            }, data);
        }

        //Wait untill all the threads finish
        WaitHandle.WaitAll(events);
    }




}

public class MyWorkItem
{
    public AutoResetEvent Event { get; set; }
    public MyDataClass Data { get; set; }
}

// You can also use DataRow instead
public class MyDataClass
{
    //data
    //
}
}

If you are using .net 4.0 - look into Tasks and Parallel Extensions (PLINQ)