Threaded code execution time rises slowly. How to determine the culprit?

Question

I have some code in a thread. The code's main function is to call other methods, which write stuff to a SQL database like this:

private void doWriteToDb()
    {
        while (true)
        {
            try
            {
                if (q.Count == 0) qFlag.WaitOne();

                PFDbItem dbItem = null;
                lock (qLock)
                {
                    dbItem = q.Dequeue();
                }
                if (dbItem != null)
                {
                    System.Diagnostics.Stopwatch sw = System.Diagnostics.Stopwatch.StartNew();
                    //write it off
                    PFResult result = dbItem.Result;
                    double frequency = dbItem.Frequency;
                    int i = dbItem.InChannel;
                    int j = dbItem.OutChannel;
                    long detail, param, reading, res;
                    detail = PFCoreMethods.AddNewTestDetail(DbTables, _dicTestHeaders[result.Name.ToString().ToLower()]);

                    param = PFCoreMethods.AddNewTestParameter(DbTables, detail, "Frequency");
                    PFCoreMethods.AddNewTestParameterValue(DbTables, param, frequency.ToString());

                    param = PFCoreMethods.AddNewTestParameter(DbTables, detail, "In channel");
                    PFCoreMethods.AddNewTestParameterValue(DbTables, param, i.ToString());

                    param = PFCoreMethods.AddNewTestParameter(DbTables, detail, "Out channel");
                    PFCoreMethods.AddNewTestParameterValue(DbTables, param, j.ToString());

                    param = PFCoreMethods.AddNewTestParameter(DbTables, detail, "Spec");
                    PFCoreMethods.AddNewTestParameterValue(DbTables, param, result.Spec);


                    dbItem.Dispose();
                    dqcnt++;
                    sw.Stop();
                }
            }
            catch (Exception ex)
            {


            }

        }
    }

The AddNewTestParameter method is using a 3rd party class which has the SQL code. Currently I have no access to its internals.

DbTables is a collection object, whose properties are table objectscreated by the 3rd party program. There is only one DbTable object which the program uses.

The problem is that as time passes (couple of hours) the AddNewTestParameter method call takes longer and longer, starting from about 10ms to about 1sec.

The q, is a queue with objects that contain the necessary information to write into the database. The items are added to this queue by the main thread. THe thread simply takes them out, writes them, and disposes of them. The q.Count is no more than 1, although in time as the database writes become slower, the q.Count rises since dequeueing cannot catch up. At its worst, the q.Count was over 30,000. I write over 150,000 entries to the database in total.

On the SQL end, I ran some traces on the server, and the trace shows that internally SQL always takes about 10ms, even during the time the C# code itself takes 1sec.

So, currently, I have 2 suspicions:

1. My code is the problem. The thread is low-priority, perhaps this might affect performance. Also, after watching the memory usage for 20 minutes, I see that it rises at about 100K/min, CPU usage seems constant around %2-5. How can I figure out where the memory leak happens? Can I pinpoint it to a specific part of the code?

2. The 3rd party code is the problem. How could I go about proving this? What methods are there to watch and confirm that the problem lies in 3rd party code?

Answer 1

Anyway, if I had to make a suggestion I would look at DBTables ... if that's a collection maybe you're forgetting to reset it so everytime you call it it has one more element... so after a while the 3rd party routine that's O(n^2), or something like that, starts to degrade because it's expecting a worst case scenario of 20 tables and you're providing 1000.

Edit: Ok, I would discard the problem being in the queue as dequeuing should be a really fast operation (you can measure it anyway). It still points to the DBTables collection growing bigger and bigger, have you check its size after the first x iterations?

Edit2: Ok, another approach, let's say the AddNewTestParameter does exactly what is says it does... ADD a new parameter that then gets added to an internal collection. Now, there are two options, if that's the case, either you're supposed to clear that collection by calling the "ClearParameters" function after each iteration and then it will be your fault, or you have not such functionality and then it's 3rd code fault. That would explain also your memory loses (altough that can also be related to the growing queue)