views:

210

answers:

1

JI have written a .NET C# Windows Form app in Visual Studio 2008 that uses a Semaphore to run multiple jobs as threads when the Start button is pressed.

It’s experiencing an issue where the Form goes into a comma after being run for 40 minutes or more. The log files indicate that the current jobs complete, it picks a new job from the list, and there it hangs.

I have noticed that the Windows Form becomes unresponsive when this happens. The form is running in its own thread.

This is a sample of the code I am using:

protected void ProcessJobsWithStatus (Status status)
        {
int maxJobThreads = Convert.ToInt32(ConfigurationManager.AppSettings["MaxJobThreads"]);
            Semaphore semaphore = new Semaphore(maxJobThreads, maxJobThreads);  // Available=3; Capacity=3


            int threadTimeOut = Convert.ToInt32(ConfigurationManager.AppSettings["ThreadSemaphoreWait"]);//in Seconds
            //gets a list of jobs from a DB Query.
List<Job> jobList = jobQueue.GetJobsWithStatus(status);
            //we need to create a list of threads to check if they all have stopped.
            List<Thread> threadList = new List<Thread>();
            if (jobList.Count > 0)
            {
                foreach (Job job in jobList)
                {
                    logger.DebugFormat("Waiting green light for JobId: [{0}]", job.JobId.ToString());
                    if (!semaphore.WaitOne(threadTimeOut * 1000))
                    {
                        logger.ErrorFormat("Semaphore Timeout. A thread did NOT complete in time[{0} seconds]. JobId: [{1}] will start", threadTimeOut, job.JobId.ToString());

                    }

                    logger.DebugFormat("Acquired green light for JobId: [{0}]", job.JobId.ToString());
                    // Only N threads can get here at once    
                    job.semaphore = semaphore;
                    ThreadStart threadStart = new ThreadStart(job.Process);

                    Thread thread = new Thread(threadStart);
                    thread.Name = job.JobId.ToString();
                    threadList.Add(thread);
                    thread.Start();


                }

                logger.Info("Waiting for all threads to complete");

                //check that all threads have completed.
                foreach (Thread thread in threadList)
                {
                    logger.DebugFormat("About to join thread(jobId): {0}", thread.Name);
                    if (!thread.Join(threadTimeOut * 1000))
                    {
                        logger.ErrorFormat("Thread did NOT complete in time[{0} seconds]. JobId: [{1}]", threadTimeOut, thread.Name);
                    }
                    else {
                        logger.DebugFormat("Thread did complete in time. JobId: [{0}]", thread.Name);
                    }
                }                   

            }

            logger.InfoFormat("Finished Processing Jobs in Queue with status [{0}]...", status);

}

//form methods

private void button1_Click(object sender, EventArgs e)
        {
            buttonStop.Enabled = true;
            buttonStart.Enabled = false;

            ThreadStart threadStart = new ThreadStart(DoWork);
            workerThread = new Thread(threadStart);

            serviceStarted = true;
            workerThread.Start();



        }

private void DoWork()
        {
            EmailAlert emailAlert = new EmailAlert ();
            // start an endless loop; loop will abort only when "serviceStarted" flag = false
            while (serviceStarted)
            {  
                emailAlert.ProcessJobsWithStatus(0);

                // yield
                if (serviceStarted)
                {
                    Thread.Sleep(new TimeSpan(0, 0, 1));
                }
            }

            // time to end the thread
            Thread.CurrentThread.Abort();
        }

//job.process()

 public void Process()
        {
             try
            {

                //sets the status, DateTimeStarted, and the processId
                this.UpdateStatus(Status.InProgress);

                //do something

                logger.Debug("Updating Status to [Completed]");

                //hits, status,DateFinished
                this.UpdateStatus(Status.Completed);

            }
            catch (Exception e)
            {
                logger.Error("Exception: " + e.Message);
                this.UpdateStatus(Status.Error);

            }
            finally {
                logger.Debug("Relasing semaphore");
                semaphore.Release();
            }

I have tried to log what I can into a file to detect where the problem is happening, but so far I haven't been able to identify where this happens. Losing control of the Windows Form makes me think that this has nothing to do with processing the jobs. Any ideas?

Solution: Profiling it with RedGate ANTS was producing the issue. It does not happen when run directly.

+2  A: 

The first thing I see is your Thread.CurrentThread.Abort() call. This is unnecessary. Let the function exit and the thread will shut down gracefully.

The second thing I noticed is that even if a timeout occurs for acquiring a semaphore a thread will be created anyway. Your form could be hanging as a result of too many threads.

logger.DebugFormat("Waiting green light for JobId: [{0}]", job.JobId.ToString());
if (!semaphore.WaitOne(threadTimeOut * 1000))
{
   logger.ErrorFormat("Semaphore Timeout. A thread did NOT complete in time[{0} seconds]. JobId: [{1}] will start", threadTimeOut, job.JobId.ToString());
   // Should have exit here.
}
logger.DebugFormat("Acquired green light for JobId: [{0}]", job.JobId.ToString());

The third thing is your worker thread is calling a function which simply creates threads for every job in the jobs loop. If a job doesn't complete (assuming this causes a job to be removed from the queue since I don't see that code anywhere.) before DoWork wakes up from it's sleep it's going to iterate over the same Job and try to create another thread for it.

Spencer Ruport
For what I can see in the logs the threads are terminating in time for the semaphore. And the number of total threads remains more a less constant during the execution of the app.
Benjamin Ortuzar
How are you determining the thread count?
Spencer Ruport
If you're using ThreadList, an object that is local to ProcessJobsWithStatus it will be constantly 30 but the actual count will be much higher. Check task manager to see what I mean.
Spencer Ruport
I only allow 3 jobs at a time into the Semaphore. I was using RedGate: ANTS, the product displays a graph of the number of threads used by the app. It was most of the time between 18 - 20 threads.
Benjamin Ortuzar
Okay well if you look at the code I've pointed out it's clearly possible for multiple threads to be created for the current job and for a thread to be created despite being unable to acquire a semaphore.
Spencer Ruport
Fix those two issues and you can move on from there.
Spencer Ruport