Regex : how to get words from a string (C#)

Question

My input consists of user-posted strings.

What I want to do is create a dictionary with words, and how often they've been used. This means I want to parse a string, remove all garbage, and get a list of words as output.

For example, say the input is : "#@!@LOLOLOL YOU'VE BEEN *PWN3D* ! :') !!!1einszwei drei !"

The output I need is the list : "LOLOLOL", "YOU'VE", "BEEN", "PWN3D", "einszwei", "drei"

I'm no hero at regular expressions and I've been Googling but my Google-kungfu seams to be weak...

How would I go from input to the wanted output ?

Answer 1

You should look into Natural Language Processing (NLP), not regular expressions, and if you are targeting more than one spoken language, you need to factor that in as well. Since you're using C#, check out the SharpNLP project.

Edit: This approach is only necessary if you care about the semantic content of the words you're trying to split up.

Answer 2

You don't necessarily need a regex for this, if tokenizing is all you're doing. First you could sanitize the string by removing all non-letter characters except for spaces and then do a Split() on the space character. That will work for most everything, although contractions may be tough. That should get you started at least.

Answer 3

My gut feeling would not be to use regular expressions, but just do a loop or two.

Iterate over each char in the string, if not a valid char, replace it with a space Then use String.Split() and split over spaces.

Appostrophes and hyphens may be a little more tricky to determine if they are junk characters or legite ones. But if you are using a for loop to iterate over the string then looking backwards and forwards from the current character should help you.

Then you will have a list of words - for each of these words check if they are valid in your dictionary. If you want this to be fast, performing somekind of binary search would be best. But just to get it working a linear search would be easier to start with.

EDIT: I only mentioned the dictionary thing because I thought you might be interested only in legitimate words, ie not "asdfasdf" but ignore that last statement if that's not what you need.

Answer 4

Simple Regex:

\w+

This matches a string of "word" characters. That is almost what you want.

This is slightly more accurate:

\w(?<!\d)[\w'-]*

It matches any number of word characters, ensuring that the first character was not a digit.

Here are my matches:

1 LOLOLOL
2 YOU'VE
3 BEEN
4 PWN3D
5 einszwei
6 drei

Now, that's more like it.

EDIT:
The reason for the negative look-behind, is that some regex flavors support Unicode characters. Using [a-zA-Z] would miss quite a few "word" characters that are desirable. Allowing \w and disallowing \d includes all Unicode characters that would conceivably start a word in any block of text.

EDIT 2:
I have found a more concise way to get the effect of the negative lookbehind: Double negative character class with a single negative exclusion.

[^\W\d][\w'-]*(?<=\w)

This is the same as the above with the exception that it also ensures that the word ends with a word character. And, finally, there is:

[^\W\d](\w|[-']{1,2}(?=\w))*

Ensuring that there are no more than two non-word-characters in a row. Aka, It matches "word-up" but not "word--up", which makes sense. If you want it to match "word--up", but not "word---up", you can change the 2 to a 3.

Answer 5

Using the following

var pattern = new Regex(
  @"( [^\W_\d]              # starting with a letter
                            # followed by a run of either...
      ( [^\W_\d] |          #   more letters or
        [-'\d](?=[^\W_\d])  #   ', -, or digit followed by a letter
      )*
      [^\W_\d]              # and finishing with a letter
    )",
  RegexOptions.IgnorePatternWhitespace);

var input = "#@!@LOLOLOL YOU'VE BEEN *PWN3D* ! :') !!!1einszwei drei foo--bar!";

foreach (Match m in pattern.Matches(input))
  Console.WriteLine("[{0}]", m.Groups[1].Value);

produces output of

[LOLOLOL]
[YOU'VE]
[BEEN]
[PWN3D]
[einszwei]
[drei]
[foo]
[bar]