I have a website which has two domains added. Both domains point to the root of the website. Is it possible to alter the robots.txt so that one of the domains doesn't get crawled, while the other still does?
+1
A:
You ought to perform a 301 redirect to your preferred domain, which tells robots what URL you want crawled.
For example
www.domain1.com
Should 301 redirect to
www.domain2.com
Similarly
www.domain1.com/Contact-Us/
Should 301 redirect to
www.domain2.com/Contact-Us/
Sohnee
2010-05-06 07:03:21
I thought about this as well, but hoped I could do it the other way around, since the users still wants to be on www.domain1.com/contact-us, but the search engines should only crawl domain1.com/contact-us
Dofs
2010-05-06 07:16:54
You should try to use the same preferred main address for both to be honest. If you want your users to see the www, you should make the robots use www.
Sohnee
2010-05-06 15:28:34