RegexOptions.IgnoreCase is more expensive than I would have thought (eg, should be barely measurable)
Assuming that this applies to PHP, Python, Perl, Ruby etc as well as C# (which is what I assume Jeff was using), how much of a slowdown is it and will I incur a similar penalty with "/[a-zA-z]/" as I will with "/[a-z]/i" ?
Yes, [A-Za-z] will be much faster than setting the RegexOptions.IgnoreCase, largely because of Unicode strings. But it's also much more limiting -- [A-Za-z] does not match accented international characters, it's literally the A-Za-z ASCII set and nothing more.
I don't know if you saw Tim Bray's answer to my message, but it's a good one:
One of the trickiest issues in internationalized search is upper and lower case. This notion of case is limited to languages written in the Latin, Greek, and Cyrillic character sets. English-speakers naturally expect search to be case-insensitive if only because they’re lazy: if Nadia Jones wants to look herself up on Google she’ll probably just type in nadia jones and expect the system to take care of it.
So it’s fairly common for search systems to “normalize” words by converting them all to lower- or upper-case, both for indexing and queries.
The trouble is that the mapping between cases is not always as straightforward as it is in English. For example, the German lower-case character “ß” becomes “SS” when upper-cased, and good old capital “I” when down-cased in Turkish becomes the dotless “ı” (yes, they have “i”, its upper-case version is “İ”). I have read (but not verified first-hand) that the rules for upcasing accented characters such “é” are different in France and Québec. One of the results of all this is that software such as java.String.toLowerCase() tends to run astonishingly slow as it tries to work around all these corner-cases.
http://www.tbray.org/ongoing/When/200x/2003/10/11/SearchI18n
If you can tolerate having numbers and underscores in that regex, you can e.g. use the \w modifier (Perl syntax). I believe some engines support [:alpha:], but that is not pure Perl. \w takes into account the locale you are in, and matches both uppercase and lowercase, and I bet it is faster than using [A-Z] while ignoring case.
Hey guys, just as a little helper, you can express a case-insensitive a-z range easily doing this:
[A-z]
If you want to include numbers in that, I believe you can even either begin or end with the lowest or highest number (depending on whether it's beginning or ending).
@Brian Warshaw:
[A-z] includes a few more things:
>>> ''.join([ chr(c) for c in range(ord('A'), ord('z')+1)])
'ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz'
If you're concerned about this, it may be worthwhile to set the case to all upper or all lower before you check.
For instance, in Perl:
$x = "abbCCDGBAdgfabv";
(lc $x) =~ /bad/;
May in some cases be better than
$x = "abbCCDGBAdgfabv";
$x =~ /bad/i;