Hidden Features of RegEx

Question

Continuing the tradition of 'the great hidden features threads seen here'

Show us your best / coolest / craziest / "usefulest" RegEx

EDIT: It was foolish of me to not be more specific. My intent was to focus on the lesser known or esoteric features of RegEx implementations in any language. For example, .NET supports a variant of named groups like this: (?<first>group)(?'second'group) so that when doing a search and replace you can reference the named group like so: ${name}

Answer 1

I won't try, I can't write anything more useful than this : Crucial Concepts Behind Advanced Regular Expressions.

Answer 2

Backreference construct For eg. to find doubled word characters use

(?<char>\w)\k<char>

Here wherever <char> occurs in regex, it matches \w

Backreferencing uses named groups to allow you to search for other instances of characters that match a wildcard. Backreferences provide a convenient way to find repeating groups of characters. They can be thought of as a shorthand instruction to match the same string again.

For example, the regular expression (?<char>\w)\k<char>, using named groups and backreferencing, searches for adjacent paired characters. When applied to the string "I'll have a small coffee," it finds matches in the words "I'll", "small", and "coffee". The metacharacter \w finds any single-word character. The grouping construct (?<char>) encloses the metacharacter to force the regular expression engine to remember a subexpression match (which, in this case, will be any single character) and save it under the name "char". The backreference construct \k<char> causes the engine to compare the current character to the previously matched character stored under "char". The entire regular expression successfully finds a match wherever a single character is the same as the preceding character.

Added:

An eg of usage of backreference would be to check if a particular tag is xml. Here it is mandatory that the start tag and end tag should match. Hence backreference is handy. Check out my example on expression to match xml tag.

Answer 3

Atomic grouping. (And, if your engine has it, possessive quantifiers, which are a notational convenience for the commonest use case of Atomic grouping)

This is very often the answer to making your regular expressions fast. Anyone who's struggled with regular expressions knows how easy it is to accidentally write a regular expression that takes forever on long input. As a simple example, this regex:

([+\w-]+)@foo.com

Takes time proportional to n² to run against a string of n "x" characters. (with most engines; perl does some fancy optimization) It's easy to construct a regular expression that takes O(n³), O(n⁴), etc. time when failing to match, and even ones that take O(2ⁿ) time. Often, the way out of this is to signal to the regular expression engine that it shouldn't backtrack through certain constructs. That's where atomic grouping comes in.

This regular expression matches exactly the same things as the other one, but only takes O(n) time:

((?>[+\w-]+))@foo.com

The atomic group construct (?> ) tells the regular expression engine to not backtrack back inside the parentheses if subsequent characters fail to match. Instead, the entire expression should fail to match. This is what keeps the behavior linear.

You do have to be a little bit careful with atomic groups - sometimes you really do want backtracking - but it's a nice thing to start thinking of whenever you notice performance issues in your regular expressions.

(It's a shame Python doesn't support this feature - yet)

Answer 4

Isn't everything in RegEx a "hidden" feature? How does the joke go?

I had a problem that I tried to solve with regular expressions. Now I have two problems.

:)