Robots.txt to disallow everything and allow only specific parts of the site/pages. Is "allow" supported by crawlers like Ultraseek and FAST?

Question

Hi,

Just wanted to know if it is possible to disallow the whole site for crawlers and allow only specific webpages or sections? Is "allow" supported by crawlers like FAST and Ultraseek?

Kind regards,

Answer 1

There is an Allow Directive however there's no guarantee that a particular bot will support it (much like there's no guarantee a bot will even check your robots.txt to begin with). You could probably tell by examining your weblogs whether or not specific bots were indexing only the parts of your website that you allow.

The format for allowing just a particular page or section of your website might look like:

Allow: /public/section1/
Disallow: /

This (should) prevent bots from crawling or indexing anything except for content under /public/section1