Emulation of lex like functionality in Perl or Python.

Question

Here's the deal. Is there a way to have strings tokenized in a line based on multiple regexes?

One example:

I have to get all href tags, their corresponding text and some other text based on a different regex. So I have 3 expressions and would like to tokenize the line and extract tokens of text matching every expression.

I have actually done this using flex (not to be confused with Adobe), which is an implementation of the good old lex. lex provides an elegant way to do this by executing "actions" based on expressions. One can control the way lex reading a file too (block / line based read).

The problem is that flex actually produces C/ C++ code which actually does the tokenizing job. I have a make file which wraps all these things. I was wondering if perl /python can in some way do the same thing. Its just that I would like to do everything I like in a single programming language itself.

Tokenizing is just one of the things that I want to do as part of my application.

Apart from perl or python can any language (functional also) do this?

I did read about PLY and ANTLR here (http://stackoverflow.com/questions/34081/parsing-where-can-i-learn-about-it#34085).

But is there a way to do it naturally in python itself? pardon my ignorance, but are these tools used in any popular products / services?

Thank you.

Answer 1

Have you looked at PyParsing?

From their homepage:

Here is a program to parse "Hello, World!" (or any greeting of the form ", !"):

from pyparsing import Word, alphas
greet = Word( alphas ) + "," + Word( alphas ) + "!" # <-- grammar defined here
hello = "Hello, World!"
print hello, "->", greet.parseString( hello )

The program outputs the following:

Hello, World! -> ['Hello', ',', 'World', '!']

Answer 2

Sounds like you really just want to parse HTML, I recommend looking at any of the wonderful packages for doing so:

• BeautifulSoup
• lxml.html
• html5lib

Or! You can use a parser like one of the following:

• PyParsing
• DParser - A GLR parser with good python bindings.
• ANTLR - A recursive decent parser generator that can generate python code.

This example is from the BeautifulSoup Documentation:

from BeautifulSoup import BeautifulSoup, SoupStrainer
import re

links = SoupStrainer('a')
[tag for tag in BeautifulSoup(doc, parseOnlyThese=links)]
# [<a href="http://www.bob.com/"&gt;success&lt;/a&gt;, 
#  <a href="http://www.bob.com/plasma"&gt;experiments&lt;/a&gt;, 
#  <a href="http://www.boogabooga.net/"&gt;BoogaBooga&lt;/a&gt;]

linksToBob = SoupStrainer('a', href=re.compile('bob.com/'))
[tag for tag in BeautifulSoup(doc, parseOnlyThese=linksToBob)]
# [<a href="http://www.bob.com/"&gt;success&lt;/a&gt;, 
#  <a href="http://www.bob.com/plasma"&gt;experiments&lt;/a&gt;]

Answer 3

Look at documentation for following modules on CPAN

HTML::TreeBuilder

HTML::TableExtract

and

Parse::RecDescent

I've used these modules to process quite large and complex web-pages.

Answer 4

If you're specifically after parsing links out of web-pages, then Perl's WWW::Mechanize module will figure things out for you in a very elegant fashion. Here's a sample program that grabs the first page of Stack Overflow and parses out all the links, printing their text and corresponding URLs:

#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;

my $mech = WWW::Mechanize->new;

$mech->get("http://stackoverflow.com/");

$mech->success or die "Oh no! Couldn't fetch stackoverflow.com";

foreach my $link ($mech->links) {
    print "* [",$link->text, "] points to ", $link->url, "\n";
}

In the main loop, each $link is a WWW::Mechanize::Link object, so you're not just constrained to getting the text and URL.

All the best,

Paul

Answer 5

If your problem has anything at all to do with web scraping, I recommend looking at Web::Scraper , which provides easy element selection via XPath respectively CSS selectors. I have a (German) talk on Web::Scraper , but if you run it through babelfish or just look at the code samples, that can help you to get a quick overview of the syntax.

Hand-parsing HTML is onerous and won't give you much over using one of the premade HTML parsers. If your HTML is of very limited variation, you can get by by using clever regular expressions, but if you're already breaking out hard-core parser tools, it sounds as if your HTML is far more regular than what is sane to parse with regular expressions.

Answer 6

Also check out pQuery it as a really nice Perlish way of doing this kind of stuff....

use pQuery;

pQuery( 'http://www.perl.com' )->find( 'a' )->each( 
    sub {
        my $pQ = pQuery( $_ ); 
        say $pQ->text, ' -> ', $pQ->toHtml;
    }
);

# prints all HTML anchors on www.perl.com
# =>  link text -> anchor HTML

However if your requirement is beyond HTML/Web then here is the earlier "Hello World!" example in Parse::RecDescent...

use strict;
use warnings;
use Parse::RecDescent;

my $grammar = q{
    alpha : /\w+/
    sep   : /,|\s/
    end   : '!'
    greet : alpha sep alpha end { shift @item; return \@item }
};

my $parse = Parse::RecDescent->new( $grammar );
my $hello = "Hello, World!";
print "$hello -> @{ $parse->greet( $hello ) }";

# => Hello, World! -> Hello , World !

Probably too much of a large hammer to crack this nut ;-)

/I3az/

Answer 7

From perlop:

A useful idiom for lex -like scanners is /\G.../gc . You can combine several regexps like this to process a string part-by-part, doing different actions depending on which regexp matched. Each regexp tries to match where the previous one leaves off.

 LOOP:
    {
      print(" digits"),    redo LOOP if /\G\d+\b[,.;]?\s*/gc;
      print(" lowercase"),    redo LOOP if /\G[a-z]+\b[,.;]?\s*/gc;
      print(" UPPERCASE"),    redo LOOP if /\G[A-Z]+\b[,.;]?\s*/gc;
      print(" Capitalized"),  redo LOOP if /\G[A-Z][a-z]+\b[,.;]?\s*/gc;
      print(" MiXeD"),     redo LOOP if /\G[A-Za-z]+\b[,.;]?\s*/gc;
      print(" alphanumeric"), redo LOOP if /\G[A-Za-z0-9]+\b[,.;]?\s*/gc;
      print(" line-noise"),   redo LOOP if /\G[^A-Za-z0-9]+/gc;
      print ". That's all!\n";
    }

Answer 8

Modifying Bruno's example to include error checking:

my $input = "...";
while (1) {
    if ($input =~ /\G(\w+)/gc) { print "word: '$1'\n"; next }
    if ($input =~ /\G(\s+)/gc) { print "whitespace: '$1'\n"; next }

    if ($input !~ /\G\z/gc)  { print "tokenizing error at character " . pos($input) . "\n" }
    print "done!\n"; last;
}

(Note that using scalar //g is unfortunately the one place where you really can't avoid using the $1, etc. variables.)

Answer 9

Oh, yes. Look here.