views:

42

answers:

2

I am modifying my robots.txt file as follows:

I want to prevent the robots from accessing urls that end with /new

My question is:

is this the correct pattern to use to disallow access to all urls terminating in '/new' ?

Disallow: /*/new
A: 

Yes seems google supports this

Try https://www.google.com/webmasters "Site Configuration > Crawler access" to test your robots.txt file for given URLs

Here is the result of this tool

http://www.***.com/new/asdasd
Allowed by line 6: Allow: /

http://www.***.com/asdasd/new/
Blocked by line 8: Disallow: /*/new
Detected as a directory; specific files may have different restrictions

http://www.***.com/asdasd/new   
Blocked by line 8: Disallow: /*/new
Ergec
A: 
Disallow: /new$ 

will check for URLs terminating in /new

JoseK