How to know whether an incoming request is from a search engine robot?

Question

Does anyone know how to detect whether an incoming request is from a search engine robot? Do HTML headers contain any specific info for that?

Answer 1

You can usually use a combination of IP and the user agent in the HTTP headers to figure it out. There is an article all about it here: http://www.jafsoft.com/searchengines/spider_hunting.html

Answer 2

You can use jQuery to check the user agent from the request.

You can get to this information through $.browser

Here is an article that has a list of known bots.

Keep in mind, though, that this is easily spoofed.