Hi there,
With all HTTP data available,
What 'signs' can you look for to recognize Google's search engine robots?
views:
84answers:
3
+1
A:
As far as I know, Google's crawlers have the user-agent set to "Googlebot".
Other search engine providers typically stick to a recognisable name in the user-agent; there are various lists of well-known agents, such as that on http://www.jafsoft.com/searchengines/webbots.html.
Rob
2009-05-29 18:22:10
It is the link i was looking for, thank you !
Boris Guéry
2009-05-29 18:26:41
+1
A:
The User-Agent header should be enough to detect the Google bot
Check out user-agents.org website to get a list of known se bot
By the you would like to want to be sure that's a true googlebot from google, then you can check out the ip/host which is always
c[nn].googlebot.com
Where [nn] is a number.
Boris Guéry
2009-05-29 18:24:53