does anyone know that it is possible to setup any property in order to inform googlebot to just come and crawl the site during specific day or time period (eg. during the weekend only)?
thanks,
A:
Don't think so, but googlebot does re-read your robots.txt quite frequently, so I wonder if it would work if you swapped in an alternative robots.txt at those times, e.g. with a script?
Jeffrey Kemp
2009-07-07 08:07:07
The swapped alternative here , you refer to turn on and off googlebot depends on day?
pang
2009-07-07 08:21:15
Yeah - say, you have a script that runs on Friday night that switches to a robots.txt that says "ok, scan me", then on Sunday night another script swaps it for a restrictive robots.txt that says "go away". It might not be perfect, i.e. there might be some lag time before googlebot picks up the change, but it wouldn't be too long I'd expect.
Jeffrey Kemp
2009-07-07 08:30:32
I'd be VERY careful about that - the bot might interpret a "Disallow" as an order to remove the site from the index entirely.
Michael Borgwardt
2009-07-07 08:37:06
Very good point Michael, I think you're right. -1 to my answer :)
Jeffrey Kemp
2009-07-07 09:04:27
A:
You can advise googlebots that you prefer a slower crawl rate (if your site is being crawled at faster than the lowest rate) but this takes effect for 90 days (see http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620).
Changing the robots.txt could be problematic as that is cached by Google, so disallowing crawling could result in the site not being crawled for far longer than intended.
Google has more than one bot type so you may be able to be selective over which parts of the site are appropriate for each of them to crawl, using robots.txt as it is intended. See http://www.google.com/support/webmasters/bin/answer.py?answer=40360.
mas
2009-07-07 08:34:06
+2
A:
You can use an XML sitemap to give a hint about the appropriate crawling frequency, but it's only a hint, and requesting specific days is not possible.
Michael Borgwardt
2009-07-07 08:40:20
A:
what is the time of Google crawler on 1 site my site http://www.w3in.in
deepak
2010-03-02 12:48:58