Multiple Producers, Multiple Consumers and Store Problem

Question

I have a scenario where Multiple Producers are running on different machines in a Network. These are Singleton WCF Services which are queuing the Products (Output) in the Central Store which is also a Singleton WCF Service.

There are consumers who dequeue the product from the Central Store by calling the Central Store via a JSON Request. The products are delivered by resolving certain priority constraints. The Producers produce the products at very high rate around 10000 in a minute the aim is to serve them at the same speed to the consumers and not keep them waiting.

Everything works fine as long as I have 3-4 Producers and upto 10 consumers. But as I increase the Producers and Consumers everything Freezes.

I was using TimedLock which is a wrapper to Monitor.TryEnter. I have tried all Types of Synchronization Techniques like ReaderWriterLockSlim and other posts on the Web but the result is same. I was avoiding ReaderWriterLockSlim as I've large number of writes than reads.

As I don't have control over the Consumer and Producer threads which are spawned by WCF as Singleton Store is accessed by them. I was not able to implement the Produer/Consumer samples available on Web. Following is the Sample Code of the Data Store.

public class Store:IEnumerable<Product>
{
    private List<Product> _Products;
    private object _MonitorLock;

    public Store()
    {
        _Products = new List<Product>();
        _MonitorLock = new object();
    }
    public void Add(Product Product)
    {
        lock (_MonitorLock)
        {
            _Products.Add(Product);
        }
    }

    public void Remove(Product Product)
    {
        lock (_MonitorLock)
        {
            _Products.Remove(Product);
        }
    }

    public int Count
    {
        get
        {
            lock (_MonitorLock)
            {
                return _Products.Count;
            }
        }
    }

    public IEnumerator<Product> GetEnumerator()
    {
        List<Product> ProductsCopy;

        lock (_MonitorLock)
        {
            ProductsCopy = new List<Product>(_Products);
        }

        foreach (Product oEntry in ProductsCopy)
            yield return oEntry;
    }


    public Product GetHighestPriorityProduct()
    {
        Product oProduct;
        lock (_MonitorLock)
        {
            //Some Logic to Resolve the Priority

            _Products.Remove(oProduct);
        }
        return oProduct;
    }


    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

Answer 1

I'm not sure if this will be any help at all, but in this sort of environment I would investigate whether using a Windows Workflow (i.e. handing scheduling issues off to a part of the framework) would help. The WF runtime can be fed with a queue of items to work on; and will green-thread activities within a single workflow on a single OS thread for you (for more OS threads, just add more workflows).

Answer 2

Are you saying that it freezes because of a deadlock on _MonitorLock? Are you using ThreadPool threads, and if so, does freeze happen about the time the number of threads hits the ThreadPool limit?

Have you done varied the producer and consumer mix to see what's driving the lock-up?

Have you added logging to your code to see if its freezing because it's taking to long to grab the next highest priority object? If that's the case, then perhaps implementing a prioritized queue would help (http://www.vcskicks.com/priority-queue.php).