Is there a way to stop Google from indexing a site?
views:
596answers:
4What is the robots.txt file?
Nick
2008-12-23 23:31:00
Added link to the wikipedia article on robots.txt
tvanfosson
2008-12-23 23:32:13
+17
A:
robots.txt
User-agent: *
Disallow: /
this will block all search bots from indexing.
for more info see: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
Unkwntech
2008-12-23 23:32:23
Actually, to be precise, this will block all *legitimate* bots from crawling the site. Malicious ones will still attempt to do so, just in case that matters.
Software Monkey
2008-12-24 06:18:01
That is correct however if the "spider" does not check robots.txt then it is likely malicious, which from my experience means that they will also spoof the user-agent which makes it ridiculously hard to stop.
Unkwntech
2009-01-05 10:01:22
+4
A:
FYI - Google has a Webmaster Tools site that is worth at least checking out .. .. http://www.google.com/webmasters/start/
Scott Evernden
2008-12-23 23:39:55