compare all rows in DataTable - identify duplicate records

Answer 1

I am not sure if I understand the question correctly, but when dealing with System.Data.DataTable the following should work.

for (Int32 r0 = 0; r0 < dataTable.Rows.Count; r0++)
{
   for (Int32 r1 = r0 + 1; r1 < dataTable.Rows.Count; r1++)
   {
      Boolean rowsEqual = true;

      for (Int32 c = 0; c < dataTable.Columns.Count; c++)
      {
         if (!Object.Equals(dataTable.Rows[r0][c], dataTable.Rows[r1][c])
         {
            rowsEqual = false;
            break;
         }
      }

      if (rowsEqual)
      {
         Console.WriteLine(
            String.Format("Row {0} is a duplicate of row {1}.", r0, r1))
      }
   }
}

Answer 2

I'm not too knowledgable about LINQ, but can you use the .Distinct() operator?

http://blogs.msdn.com/charlie/archive/2006/11/19/linq-farm-group-and-distinct.aspx

Your question doesn't make clear whether you need to specifically identify duplicate rows, or whether you're just looking to remove them from your query. Adding "Distinct" would remove the extra instances, though it wouldn't necessarily tell you what they were.

Answer 3

The easiest way to get this done without O(n²) complexity is going to be using a data structure that efficiently implements Set operations, specifically a Contains operation. Fortunately .NET (as of 3.0) contains the HashSet object which does this for you. In order to make use of this you're going to need a single object that encapsulates a row in your DataTable.

If DataRow won't work, I recommend converting relevant records into strings, concatenating them then placing those in the HashSet. Before you insert a row check to see if the HashSet already contains it (using Contains). If it does, you've found a duplicate.

Edit:

This method is O(n).