perl removing stop words from a big file

Question

Hello everyone,

I have a billion word corpus which I have collected in a scalar. I have a .regex file that contains all the stop words that I want to eliminate from my data (text).

I dont know how to use this .regex file so I have made an array and stored all the stop words of the .regex file in my stop word array.

To remove the stop words I do something like this :

grep { $scalarText =~ s/\b\Q$_\E\b/ /g } @stopList;

This takes a long time to execute. I was wondering if someone knows how to use the .regex file in my perl program to remove the stop words ? Or if there is any faster way to remove the stop words ?

Thanks so much for the help!

Answer 1

You may want to use Regexp::Assemble to compile a list of Perl regexes into one regex.

Answer 2

I found a faster way to do it. Saves me around 4 seconds.

my $qrstring = '\b(' . (join '|', @stopList) . ')\b'; $scalarText =~ s/$qrstring/ /g;

where stopList is the array of all my words and scalarText is my whole text.

Can anyone please tell me a faster way if you know any? Thanks!

Answer 3

Yes, I imagine what you're doing there is extremely slow, albeit for a couple of reasons. I think you need to process your stopwords regex before you build up your string of a billion words from your corpus.

I have no idea what a .regex file is, but I'm going to presume it contains a legal Perl regular expression, something that you can compile using no more than:

$stopword_string = `cat foo.regex`;
$stopword_rx     = qr/$stopword_string/;

That probably presumes that there's a (?x) at the start.

But if your stopword file is a list of lines, you will need to do something more like this:

chomp(@stopwords = `cat foo.regex`);

# if each stopword is an independent regex:
$stopword_string = join "|" => @stopwords;

# else if each stopword is a literal
$stopword_string = join "|" => map {quotemeta} @stopwords;

# now compile it (maybe add some qr//OPTS)
$stopword_rx     = qr/\b(?:$stopword_string)\b/;

WARNING

Be very careful with \b: it's only going to do what you think it does above if the first character in the first word and the last character in the last word is an alphanumunder (a \w character). Otherwise, it will be asserting something you probably don't mean. If that could be a possibility, you will need to be more specific. The leading \b would need to become (?:(?<=\A)|(?<=\s)), and the trailing \b would need to become (?=\s|\z). That's what most people think \b means, but it really doesn't.

Having done that, you should apply the stopword regex to the corpus as you're reading it in. The best way to do this is not to put the stuff into your string in the first place that you'll just need to take out later.

So instead of doing

$corpus_text = `cat some-giant-file`;
$corpus_text =~ s/$stopword_rx//g;

Instead do

my $corpus_path = "/some/path/goes/here";
open(my $corpus_fh, "< :encoding(UTF-8)", $corpus_path)
    || die "$0: couldn't open $corpus_path: $!";

my $corpus_text = q##;

while (<$corpus_fh>) {
    chomp;  # or not
    $corpus_text .= $_ unless /$stopword_rx/;
}

close($corpus_fh)
    || die "$0: couldn't close $corpus_path: $!";

That will be much faster than putting stuff in there that you just have to weed out again later.

My use of cat above is just a shortcut. I don't expect you to actually call a program, least of all cat, just to read in a single file, unprocessed and unmolested. ☺