Is there any robots.txt parser written in JavaScript? I'm usually coding with python and there's robotparser: http://docs.python.org/library/robotparser.html, which is very easy to use.
Is there any robots.txt parser written in JavaScript? I'm usually coding with python and there's robotparser: http://docs.python.org/library/robotparser.html, which is very easy to use.