views:

24

answers:

1

In the robots.txt file, I am about to disallow some sections of my site.

For instance, I don't want my "terms and conditions" to be indexed by search engines.

 User-agent: *
 Disallow: /terms

The real path to the file is actually

 /data/terms_and_conditions.html

But I have used .htaccess to rewrite the URL.

Now to my Q, should I specify the rewritten url in the robots.txt or the actual url?

Follow-up question: Do I need to have an "allow" line too, or will the search engines assume all other is allowed which isn't in the robots.txt file?

Thanks

+1  A: 

Search engines will assume that all other is allowed which isn't in the robots.txt. In your case it will disallow path /term.

pyfunc