Counting occurrences of a string in an array and then removing duplicates

Question

I am fairly new to C# programming and I am stuck on my little ASP.NET project.

My website currently examines Twitter statuses for URLs and then adds those URLs to an array, all via a regular expression pattern matching procedure. Clearly more than one person will update a with a specific URL so I do not want to list duplicates, and I want to count the number of times a particular URL is mentioned in, say, 100 tweets.

Now I have a List<String> which I can sort so that all duplicate URLs are next to each other. I was under the impression that I could compare list[i] with list[i+1] and if they match, for a counter to be added to (count++), and if they don't match, then for the URL and the count value to be added to a new array, assuming that this is the end of the duplicates.

This would remove duplicates and give me a count of the number of occurrences for each URL. At the moment, what I have is not working, and I do not know why (like I say, I am not very experienced with it all).

With the code below, assume that a JSON feed has been searched for using a keyword into srchResponse.results. The results with URLs in them get added to sList, a string List type, which contains only the URLs, not the message as a whole.

I want to put one of each URL (no duplicates), a count integer (to string) for the number of occurrences of a URL, and the username, message, and user image URL all into my jagged array called 'urls[100][]'. I have made the array 100 rows long to make sure everything can fit but generally, this is too big. Each 'row' will have 5 elements in them.

The debugger gets stuck on the line: if (sList[i] == sList[i + 1]) which is the crux of my idea, so clearly the logic is not working. Any suggestions or anything will be seriously appreciated!

Here is sample code:

  var sList = new ArrayList();

    string[][] urls = new string[100][];

    int ctr = 0;
    int j = 1;

    foreach (Result res in srchResponse.results)
    {           

        string content = res.text;
        string pattern = @"((https?|ftp|gopher|telnet|file|notes|ms-help):((//)|(\\\\))+[\w\d:#@%/;$()~_?\+-=\\\.&]*)";
        MatchCollection matches = Regex.Matches(content, pattern);

      foreach (Match match in matches)
      {

        GroupCollection groups = match.Groups;

                    sList.Add(groups[0].Value.ToString());
      }
    }

    sList.Sort();    
    foreach (Result res in srchResponse.results)
    {
        for (int i = 0; i < 100; i++)
        {
            if (sList[i] == sList[i + 1])
            {
                j++;
            }
            else
            {
                urls[ctr][0] = sList[i].ToString();
                urls[ctr][1] = j.ToString();
                urls[ctr][2] = res.text;
                urls[ctr][3] = res.from_user;
                urls[ctr][4] = res.profile_image_url;
                ctr++;
                j = 1;
            }
        }



    }

The code then goes on to add each result into a StringBuilder method with the HTML.

Is now edite

Answer 1

I'd recommend using a more sophisticated data structure than an array. A Set will guarantee that you have no duplicates.

Looks like C# collections doesn't include a Set, but there are 3rd party implementations available, like this one.

Answer 2

The description of your algorithm seems fine. I don't know what's wrong with the implementation; I haven't read it that carefully. (The fact that you are using an ArrayList is an immediate red flag; why aren't you using a more strongly typed generic collection?)

However, I have a suggestion. This is exactly the sort of problem that LINQ was intended to solve. Instead of writing all that error-prone code yourself, just describe the transformation you're interested in, and let the compiler work it out for you.

Suppose you have a list of strings and you wish to determine the number of occurrences of each:

var notes = new []{ "Do", "Fa", "La", "So", "Mi", "Do", "Re" };

var counts = from note in notes 
             group note by note into g
             select new { Note = g.Key, Count = g.Count() }

foreach(var count in counts)
    Console.WriteLine("Note {0} occurs {1} times.", count.Note, count.Count);

Which I hope you agree is much easier to read than all that array logic you wrote. And of course, now you have your sequence of unique items; you have a sequence of counts, and each count contains a unique Note.

Answer 3

Your loop fails because when i == 99, (i + 1) == 100 which is outside the bounds of your array.

But as other have pointed out, .Net 3.5 has ways of doing what you want more elegantly.

Answer 4

If you don't need to know how many duplicates a specific entry has you could do the following:

LINQ Extension Methods

.Count()   
.Distinct()  
.Count()