Specifying variables in robot.txt

Question

My URL structure is set up in two parallels (both lead to the same place ):

• www.mydomain.com/subname
• www.mydomain.com/123

The trouble is is that, the spiders are crawling into things like:

• www.mydomain.com/subname/default_media_function
• www.mydomain.com/subname/map_function

Note that the name "subname" represents thousands of different pages on my site that all have that same function.

And they are erroring out because those links are strictly for JSON or AJAX purposes and not actual links. I would like to block them from accessing those pages, but how would I do that if the URL contains a variable?

Would this work in robots.txt?

Disallow: /map_function

Answer 1

you are gonna have to do Disallow: /subname/map_function

the robots will look for the robot.txt at root level, also there they evaluation urls left to right with no wildcards.

so you will either need to make one location for all the map_function and exclude that or exclude all locations.