tags:

views:

538

answers:

7

Having read a number of questions/answers over the past few weeks, I have seen the use of \d in perl regular expressions commented on as incorrect. As in the later versions of perl \d is not the same as [0-9], as \d will represent any Unicode character that has the digit attribute, and that [0-9] represents the characters '0', '1', '2', ..., '9'.

I appreciate that in some contexts [0-9] will be the correct thing to use, and in others \d will be. I was wondering which people feel is the correct default to use?

Personally I find the \d notation very succinct and expressive, whereas in comparison [0-9] is somewhat cumbersome. But I have little experience of doing multi-language code, or rather code for languages that do not fit into the ASCII character range, and therefore may be being naive.

I notice

$find /System/Library/Perl/5.8.8/ -name \*pm | xargs grep '\\d' | wc -l
  298
$find /System/Library/Perl/5.8.8/ -name \*pm | xargs grep '\[0-9\]' | wc -l
  26

Thanks Beano

+3  A: 

I feel both must have there place however 99.999% of the time (especially in my closed big American cooperation world) they are interchangeable. I use perl to manipulate data every day and in none of the data sets I deal with are their numbers that don't fit in [0-9]. However I do appreciate that there is an important distinction between \d and [0-9] and its good to be aware of that difference. I use \d because it seems more succinct (as you said) and would never be "wrong" in my small world of data manipulation.

Copas
You want \d not /d - if you want it at all.
Telemachus
+9  A: 

For maximum safety, I'd suggest using [0-9] any time you don't specifically intend to match all unicode-defined digits.

Per perldoc perluniintro, Perl does not support using digits other than [0-9] as numbers, so I would definitely use [0-9] if the following are both true:

1) You want to use the result as a number (such as performing mathematical operations on it or storing it somewhere that only accepts proper numbers (e.g. an INT column in a database)).

and

2) It is possible non-[0-9] digits would be present in the data in such a way that the regular expression could match them. (Note that this one should always be considered true for untrusted/hostile input.)

If either of these are false, there will only rarely be reason to specifically not use \d (and you'll probably be able to tell when that is the case), and if you're trying to match all unicode-defined digits, you'll definitely want to use \d.

Nicholas Knight
\d can indeed match more than 10 different characters, if applied to Unicode strings.
pts
A: 

If you apply \d to a Unicode string (such as in "\X{660}" =~ /\d/), it will match a Unicode digit. If you apply \d to a binary string (such as the UTF-8 equivalent of the above: "\xd9\xa0" =~ /\d/), it will match only the 10 ASCII digits. Perl 5.8 doesn't create Unicode strings by default (unless you specifically ask for it, such as in "\X{...}" or use utf8; etc.).

So my advice is: only pay attention to the difference between \d and [0-9] if your application uses Unicode strings.

pts
+5  A: 

According to perlreref, '\d' is locale-aware and Unicode aware.

However, if the codeset you are using is not Unicode, then you don't need to worry about the Unicode digits, and if the codeset you are using is something like Latin-1 (ISO 8859-1, or 8859-15), then the locale-awareness won't hurt you either because the codeset does not include any other digit characters.

So, for many people, much of the time, you can use '\d' without concern. However, if Unicode data is part of your work, then you need to consider what you are after more carefully.

Jonathan Leffler
+1  A: 

Just like nuking the site from orbit, [0-9] is the only way to be sure. Yeah, it is ugly. Yeah, the choice to make \d be UNICODE and locale aware was stupid. But this is our bed and we have to lie in it.

As for the people ducking their heads in the sand saying it doesn't effect the character set they are using today, well you may be using that character set today, but the rest of the world is using UTF-8 now and you will be using it soon as well. Remember to code like the guy who maintains your code is a homicidal maniac who knows where you live.

Oh, and as for Perl modules using \d vs [0-9], even the core still has UNICODE problems.

If you do in fact mean any digit, but want to be able to do math with the results, you can use Text::Unidecode:

#!/usr/bin/perl

use strict;
use warnings;

use Text::Unidecode;

my $number = "\x{1811}\x{1812}\x{1813}\x{1814}\x{1815}";
print "$number is ", unidecode($number), "\n";

After some more testing it looks like Text::Unidecode doesn't handle all digit characters correctly. I am writing a module that will work.

Chas. Owens
+5  A: 
mirod
+1 for that list! I was beginning to wonder which other number characters there were.
nickf
If Perl has embraced UNICODE this far, then it seems like it should go the rest of the way and handle all the digits. Of course, that way lies madness, but isn't madness the fate of all Perl programmers ;-) ?
RBerteig
there are still more characters, but I only included the ones that I could display on my system. I used the unicode data from http://www.unicode.org/Public/UNIDATA/UnicodeData.txt, and extracted the character info from there.
mirod
I understand and appreciate that if you are dealing with UNICODE you need to handle numbers that perl cannot do arithmetic with. If I use \d I may end up with numbers that I cannot do arithmetic on, but if I use [0-9] I may miss out on numbers that I wanted to capture....so which is right - it's all down to the context of the input. I suppose I find it non-intuitive that perl decided to have the shorthand \d mean any number character and not any number character that I can do arithmentic on, or at least not provide another suitable shorthand.
Beano
@nickf At my current count there are 61 sets of digits, see the module link in my answer for the list.
Chas. Owens
@Beano I am not saying don't use \d; I am saying don't use \d when you mean [0-9]. It is similar to not using \s when you mean [ ]. The question comes down to do you mind matching ⑤ as well as 5?
Chas. Owens
A: 

If [0-9] feels clunky perhaps you could define: $d=qr/[0-9]/; and use that instead of \d.