is there a good web crawler library available for PHP or Ruby?

Question

is there a good web crawler library available for PHP or Ruby? a library that can do it depth first or breadth first... and handle the links even when href="../relative_path.html" and base url is used.

Answer 1

Check this page out for a Ruby library: Ruby Mechanize

I'd like to mention that you would still be responsible for the way in which your crawler traverses sites.

Answer 2

http://phpcrawl.cuab.de/

Answer 3

you can go for webrat or watir in ruby, much easier than mechanize