tags:

views:

26

answers:

1

I want to disallow robots from crawling any folder, at any position in the url with the name: this-folder.

Examples to disallow:

http://mysite.com/this-folder/
http://mysite.com/houses/this-folder/
http://mysite.com/some-other/this-folder/
http://mysite.com/no-robots/this-folder/

This is my attempt:

Disallow: /.*this-folder/

Will this work?

+1  A: 

Officially globbing and regular expressions are not supported:

http://www.robotstxt.org/robotstxt.html

but apparently some search engines support this.

DmitryK
OK, that's what I needed to know. Thank you for the link, it seems my Google is broken.
Michael Robinson