LINQ: take a sequence of elements from a collection

Question

Hi there,

I have a collection of objects and need to take batches of 100 objects and do some work with them until there are no objects left to process.

Instead of looping through each item and grabbing 100 elements then the next hundred etc is there a nicer way of doing it with linq?

Many thanks

Answer 1

static void test(IEnumerable<object> objects)
{
    while (objects.Any())
    {
        foreach (object o in objects.Take(100))
        {
        }
        objects = objects.Skip(100); 
    }
}

:)

Answer 2

int batchSize = 100;
var batched = yourCollection.Select((x, i) => new { Val = x, Idx = i })
                            .GroupBy(x => x.Idx / batchSize,
                                     (k, g) => g.Select(x => x.Val));

// and then to demonstrate...
foreach (var batch in batched)
{
    Console.WriteLine("Processing batch...");

    foreach (var item in batch)
    {
        Console.WriteLine("Processing item: " + item);
    }
}

Answer 3

I don't think linq is really suitable for this sort of processing - it is mainly useful for performing operations on whole sequences rather than splitting or modifying them. I would do this by accessing the underlying IEnumerator<T> since any method using Take and Skip are going to be quite inefficient.

public static void Batch<T>(this IEnumerable<T> items, int batchSize, Action<IEnumerable<T>> batchAction)
{
    if (batchSize < 1) throw new ArgumentException();

    List<T> buffer = new List<T>();
    using (var enumerator = (items ?? Enumerable.Empty<T>()).GetEnumerator())
    {
        while (enumerator.MoveNext())
        {
            buffer.Add(enumerator.Current);
            if (buffer.Count == batchSize)
            {
                batchAction(buffer);
                buffer.Clear();
            }
        }

        //execute for remaining items
        if (buffer.Count > 0)
        {
            batchAction(buffer);
        }
    }
}

Answer 4

This will partition the list into a list of lists of however many items you specify.

public static IEnumerable<IEnumerable<T>> Partition<T>(this IEnumerable<T> source, int size)
{
    int i = 0;
    List<T> list = new List<T>(size);
    foreach (T item in source)
    {
        list.Add(item);
        if (++i == size)
        {
            yield return list;
            list = new List<T>(size);
            i = 0;
        }
    }
    if (list.Count > 0)
        yield return list;
}