Can I remove items from a ConcurrentDictionary from within an enumeration loop of that dictionary?

Question

So for example:

ConcurrentDictionary<string,Payload> itemCache = GetItems();

foreach(KeyValuePair<string,Payload> kvPair in itemCache)
{
    if(TestItemExpiry(kvPair.Value))
    {   // Remove expired item.
        Payload removedItem;
        itemCache.TryRemove(kvPair.Key, out removedItem);
    }
}

Obviously with an ordinary Dictionary this will throw an exception because removing items changes the dictionary's internal state during the life of the enumeration. It's my understanding that this is not the case for a ConcurrentDictionary as the provided IEnumerable handles internal state changing. Am I understanding this right? Is there a better pattern to use?

Answer 1

Edit, after checking Dan Tao solution and testing independently.

Yes, is the short answer. It won't except, it does seem to use fine grained locking, and works as advertised.

Bob.

Answer 2

It's strange to me that you've now received two answers that seem to confirm you can't do this. I just tested it myself and it worked fine without throwing any exception.

Below is the code I used to test the behavior, followed by an excerpt of the output (around when I pressed 'C' to clear the dictionary in a foreach and S immediately afterwards to stop the background threads). Notice that I put a pretty substantial amount of stress on this ConcurrentDictionary: 16 threading timers each attempting to add an item roughly every 15 milliseconds.

It seems to me this class is quite robust, and worth your attention if you're working in a multithreaded scenario.

Code

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;

namespace ConcurrencySandbox {
    class Program {
        private const int NumConcurrentThreads = 16;
        private const int TimerInterval = 15;

        private static ConcurrentDictionary<int, int> _dictionary;
        private static WaitHandle[] _timerReadyEvents;
        private static Timer[] _timers;
        private static volatile bool _timersRunning;

        [ThreadStatic()]
        private static Random _random;
        private static Random GetRandom() {
            return _random ?? (_random = new Random());
        }

        static Program() {
            _dictionary = new ConcurrentDictionary<int, int>();
            _timerReadyEvents = new WaitHandle[NumConcurrentThreads];
            _timers = new Timer[NumConcurrentThreads];

            for (int i = 0; i < _timerReadyEvents.Length; ++i)
                _timerReadyEvents[i] = new ManualResetEvent(true);

            for (int i = 0; i < _timers.Length; ++i)
                _timers[i] = new Timer(RunTimer, _timerReadyEvents[i], Timeout.Infinite, Timeout.Infinite);

            _timersRunning = false;
        }

        static void Main(string[] args) {
            Console.Write("Press Enter to begin. Then press S to start/stop the timers, C to clear the dictionary, or Esc to quit.");
            Console.ReadLine();

            StartTimers();

            ConsoleKey keyPressed;
            do {
                keyPressed = Console.ReadKey().Key;
                switch (keyPressed) {
                    case ConsoleKey.S:
                        if (_timersRunning)
                            StopTimers(false);
                        else
                            StartTimers();

                        break;
                    case ConsoleKey.C:
                        Console.WriteLine("COUNT: {0}", _dictionary.Count);
                        foreach (var entry in _dictionary) {
                            int removedValue;
                            bool removed = _dictionary.TryRemove(entry.Key, out removedValue);
                        }
                        Console.WriteLine("COUNT: {0}", _dictionary.Count);

                        break;
                }

            } while (keyPressed != ConsoleKey.Escape);

            StopTimers(true);
        }

        static void StartTimers() {
            foreach (var timer in _timers)
                timer.Change(0, TimerInterval);

            _timersRunning = true;
        }

        static void StopTimers(bool waitForCompletion) {
            foreach (var timer in _timers)
                timer.Change(Timeout.Infinite, Timeout.Infinite);

            if (waitForCompletion) {
                WaitHandle.WaitAll(_timerReadyEvents);
            }

            _timersRunning = false;
        }

        static void RunTimer(object state) {
            var readyEvent = state as ManualResetEvent;
            if (readyEvent == null)
                return;

            try {
                readyEvent.Reset();

                var r = GetRandom();
                var entry = new KeyValuePair<int, int>(r.Next(), r.Next());
                if (_dictionary.TryAdd(entry.Key, entry.Value))
                    Console.WriteLine("Added entry: {0} - {1}", entry.Key, entry.Value);
                else
                    Console.WriteLine("Unable to add entry: {0}", entry.Key);

            } finally {
                readyEvent.Set();
            }
        }
    }
}

Output (excerpt)

cAdded entry: 108011126 - 154069760   // <- pressed 'C'
Added entry: 245485808 - 1120608841
Added entry: 1285316085 - 656282422
Added entry: 1187997037 - 2096690006
Added entry: 1919684529 - 1012768429
Added entry: 1542690647 - 596573150
Added entry: 826218346 - 1115470462
Added entry: 1761075038 - 1913145460
Added entry: 457562817 - 669092760
COUNT: 2232                           // <- foreach loop begins
COUNT: 0                              // <- foreach loop ends
Added entry: 205679371 - 1891358222
Added entry: 32206560 - 306601210
Added entry: 1900476106 - 675997119
Added entry: 847548291 - 1875566386
Added entry: 808794556 - 1247784736
Added entry: 808272028 - 415012846
Added entry: 327837520 - 1373245916
Added entry: 1992836845 - 529422959
Added entry: 326453626 - 1243945958
Added entry: 1940746309 - 1892917475

Also note that, based on the console output, it looks like the foreach loop locked out the other threads that were trying to add values to the dictionary. (I could be wrong, but otherwise I would've guessed you would've seen a bunch of "Added entry" lines between the "COUNT" lines.)

Answer 3

Additional information on this behavior can be found here:

MSDN Blog

Snippet:

The biggest change is that we're iterating over what's returned by the "Keys" property, which returns a snapshot of the keys in the dictionary at a given point. This means that the loop will not be affected by subsequent modifications to the dictionary, as it is operating on a snapshot. Without going into too many details, iterating over the collection itself has subtely different behavior that may allow subsequent modifications to be included in the loop; this makes it less deterministic.

If items are added by other threads after the loop begins, they will be stored in the collection, but they will not be included in this update operation (incrementing the Counter properties).

If an item is removed by another thread before the call to TryGetValue, the call will fail and nothing will happen. If an item is removed after the call to TryGetValue, the "tmp.