robots.txt parser java

Question

hi

i want to know how to parse the robots.txt in java.

already any code is there?

thanks in advance

Answer 1

Heritrix is an open-source web crawler written in Java. Looking through their javadoc, I see that they have a utility class Robotstxt for parsing the robots.txt file.

Answer 2

There's also jrobotx library hosted at SourceForge.

(Full disclosure: I spun off the code that forms that library.)