Or even forbid indexing the whole site?
UPDATE
Is the space after : mandatory in robots.txt?
views:
64answers:
2
+1
A:
Create a robots.txt file and add the appropriate Disallow directives, such as:
User-agent: *
Disallow: /CRM
klausbyskov
2010-05-17 14:45:05
How to write the `Disallow` s?
2010-05-17 14:46:52
@user198729: I have updated my answer.
klausbyskov
2010-05-17 14:47:45
So I can use `Disallow: /` to forbidden indexing the whole site,right?
2010-05-17 14:49:39
@user198729: That is correct.
klausbyskov
2010-05-17 14:50:07
Is the space after `:` mandatory?
2010-05-17 14:53:31
Keep in mind that the robots.txt is case-sensitive, so if the folder is called /crm, you need to "disallow: /crm". If the server is not case-sensitive, you may need to disallow multiple versions of the path.
John Mueller
2010-06-01 12:29:30