Map and Reduce in .NET

Question

What scenarios would warrant the use of the "Map and Reduce" algorithm?

Is there a .NET implementation of this algorithm?

Answer 1

http://en.wikipedia.org/wiki/MapReduce#Uses

Answer 2

If you were trying to write your own version of Google then that might warrant it..!!!

Seriously though, if you've got a problem that you can decompose into several smaller problems then a Map-Reduce solution would work. The Google document on MapReduce has a number of good examples, including how to process thousands of web pages, count words in document etc etc.

Answer 3

See this other question regarding map/reduce:

Generic List Extensions in C#

Answer 4

The classes of problem that are well suited for a mapreduce style solution are problems of aggregation. Of extracting data from a dataset. In C#, one could take advantage of LINQ to program in this style.

From the following article: http://codecube.net/2009/02/mapreduce-in-c-using-linq/

the GroupBy method is acting as the map, while the Select method does the job of reducing the intermediate results into the final list of results.

var wordOccurrences = words
                .GroupBy(w => w)
                .Select(intermediate => new
                {
                    Word = intermediate.Key,
                    Frequency = intermediate.Sum(w => 1)
                })
                .Where(w => w.Frequency > 10)
                .OrderBy(w => w.Frequency);

For the distributed portion, you could check out DryadLINQ: http://research.microsoft.com/en-us/projects/dryadlinq/default.aspx

Answer 5

There's a pretty cute MapReduce implementation for .NET at: http://mapsharp.codeplex.com/