I want to disallow robots from crawling any folder, at any position in the url with the name: this-folder
.
Examples to disallow:
http://mysite.com/this-folder/
http://mysite.com/houses/this-folder/
http://mysite.com/some-other/this-folder/
http://mysite.com/no-robots/this-folder/
This is my attempt:
Disallow: /.*this-folder/
Will this work?