Can Regular Expressions Achieve This?

Question

Hi, I'm trying to split a string into tokens (via regular expressions) in the following way:

Example #1
input string: 'hello'
first token: '
second token: hello
third token: '

Example #2
input string: 'hello world'
first token: '
second token: hello world
third token: '

Example #3
input string: hello world
first token: hello
second token: world

i.e., only split up the string if it is NOT in single quotation marks, and single quotes should be in their own token.

This is what I have so far:

string pattern = @"'|\s";
Regex RE = new Regex(pattern);
string[] tokens = RE.Split("'hello world'");

This will work for example #1 and example #3 but it will NOT work for example #2. I'm wondering if there's theoretically a way to achieve what I want with regular expressions

Answer 1

'[^']+' will match text inside single quotes. If you want it grouped, (')([^']+)('). If no matches are found, then just use a regular string split. I don't think it makes sense to try to do the whole thing in one regular expression.

EDIT: It seems from your comments on the question that you actually want this applied over a larger block of text rather than just simple inputs like you indicated. If that's the case, then I don't think a regular expression is your answer.

Answer 2

You could build a simple lexer, which would involve consuming each of the tokens one by one. So you would have a list of regular expressions and would attempt to match one of them at each point. That is the easiest and cleanest way to do this if your input is anything beyond the very simple.

Answer 3

Use a token parsor to split into tokens. Use regex to find a string patterns

Answer 4

You can first split on quoted string, and then further tokenize.

foreach (String s in Regex.Split(input, @"('[^']+')")) {
    // Check first if s is a quote.
    // If so, split out the quotes.
    // If not, do what you intend to do.
}

(Note: you need the brackets in the pattern to make sure Regex.Split returns those too)

Answer 5

Try this Regular Expression:

([']*)([a-z]+)([']*)

This finds 1 or more single quotes at the beginning and end of a string. It then finds 1 or more characters in the a-z set (if you don't set it to be case insensitive it will only find lower case characters). It groups these so that group 1 has the ', group 2 (or more) has the words which are split by anything that is not a character a - z and the last group has the single quote if it exists.

Answer 6

Not exactly what you are trying to do, but regular expression conditions might help out as you look for a solution:

(?<quot>')?(?<words>(?(quot)[^']|\w)+)(?(quot)')

If a quote is found, then it matches until a non-quote is found. Otherwise looks at word characters. Your results are in groups named "quot" and "words".

Answer 7

While it would be possible to match ' and the text inside separately, and also alternatively match the text alone, RegExp does not allow an indefinite number of matches. Or better said, you can only match those objects you explicitely state in the expression. So ((\w+)+\b) could theoretically match all words one-by-one. The outer group will correctly match the whole text, and also the inner group will match the words separately correctly, but you will only be able to reference the last match.

There is no way to match a group of matched matches (weird sentence). The only possible way would be to match the string and then split it into separate words.

Answer 8

You'll have hard time using Split here, but you can use a MatchCollection to find all matches in your string:

string str = "hello world, 'HELLO WORLD': we'll be fine.";
MatchCollection matches = Regex.Matches(str, @"(')([^']+)(')|(\w+)");

The regex searches for a string between single quotes. If it cannot find one, it takes a single word.
Now it gets a little tricky - .net returns a collection of Matchs. Each Match has several Groups - the first Group has the whole string ('hello world'), but the rest have sub-matches (',hello world,'). Also, you get many empty unsuccessful Groups.
You can still iterate easily and get your matches. Here's an example using LINQ:

var tokens = from match in matches.Cast<Match>()
             from g in match.Groups.Cast<Group>().Skip(1)
             where g.Success
             select g.Value;

tokens is now a collection of strings:
hello, world, ', HELLO WORLD, ', we, ll, be, fine