How can I extract URLs from plain text with Perl?

Question

How can I use Perl regexps to extract all URLs of a specific domain (with possibly variable subdomains) with a specific extension from plain text. I have tried:

my $stuff = 'omg http://fail-o-tron.com/bleh omg omg omg omg omg http://homepage.com/woot.gif dfgdfg http://shomepage.com/woot.gif aaa';
while($stuff =~ m/(http\:\/\/.*?homepage.com\/.*?\.gif)/gmsi)
{
print $1."\n";
}

It fails horribly and gives me:

http://fail-o-tron.com/bleh omg omg omg omg omg http://homepage.com/woot.gif
http://shomepage.com/woot.gif

I thought that shouldn't happen because I am using .*? which ought to be non-greedy and give me the smallest match. Can anyone tell me what I am doing wrong? (I don't want some uberly complex, caned regexp to validate URLs; I want to know what I am doing wrong so I can learn from it)

Answer 1

Visit CPAN: Regexp::Common::URI

Edit: Even if you don't want a canned regular expression, it may help you to look at the source of a tested module that works.

If you want to find URLs that match a certain string, you can easily use this module to do that.

#!/usr/bin/env perl
use strict;
use warnings;
use Regexp::Common qw/URI/;

while (<>) {
  if (m/$RE{URI}{HTTP}{-keep}/) {
    print $_ if $1 =~ m/what-you-want/;
  }
}

Answer 2

URI::Find is specifically designed to solve this problem. It will find all URIs and then you can filter them. It has a few heuristics to handle things like trailing punctuation.

Answer 3

i thought that shouldn't happen because i am using .*? which ought to be non-greedy and give me the smallest match

It does, but it gives you the smallest match going right. Starting from the first http and going right, that's the smallest match.

Please note for the future, you don't have to escape the slashes, because you don't have to use slashes as your separator. And you don't have to escape the colon either. Next time just do this:

m|(http://.*?homepage.com\/.*?\.gif)|

or

m#(http://.*?homepage.com\/.*?\.gif)#

or

m<(http://.*?homepage.com\/.*?\.gif)&gt;

or one of lots of other characters, see the perlre documentation.

Answer 4

URLs aren't allowed to contain spaces, so instead of .*? you should use \S*?, for zero-or-more non-space characters.