I am modifying my robots.txt file as follows:
I want to prevent the robots from accessing urls that end with /new
My question is:
is this the correct pattern to use to disallow access to all urls terminating in '/new' ?
Disallow: /*/new
I am modifying my robots.txt file as follows:
I want to prevent the robots from accessing urls that end with /new
My question is:
is this the correct pattern to use to disallow access to all urls terminating in '/new' ?
Disallow: /*/new
Yes seems google supports this
Try https://www.google.com/webmasters "Site Configuration > Crawler access" to test your robots.txt file for given URLs
Here is the result of this tool
http://www.***.com/new/asdasd
Allowed by line 6: Allow: /
http://www.***.com/asdasd/new/
Blocked by line 8: Disallow: /*/new
Detected as a directory; specific files may have different restrictions
http://www.***.com/asdasd/new
Blocked by line 8: Disallow: /*/new