I want my robot.txt to allow indexing of only my index file in google. How does this look? Will the following do the trick?
User-agent: Google
Disallow: /_/
Disallow: /library/
Disallow: /media/
Disallow: /www/
User-agent: *
Disallow: /
views:
40answers:
1
A:
Try this
User-agent: Google
Disallow: /
Allow: /index.php
Allow: /$ #this will allow your homepage as well**
**but its better to use htaccess redirect , to redirect example.com to example.com/index.php. To avoid duplication of url.
JapanPro
2010-09-22 00:40:39
What about denying all the other robots except google?
Wilson
2010-09-22 00:44:14
Also, will google index http://example.com/ or http://example.com/index.php or both?
Wilson
2010-09-22 00:49:05
it will work only for index.php witch is best practice , coz you are avoiding duplication. check the edited one now
JapanPro
2010-09-22 01:11:06