Is there a list of known web crawlers?

Question

I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know.

Is there somewhere a list of know web crawlers with some documentation like user agent, IPs, behavior, etc?

I'm not interested in the official ones, like Google's, Yahoo's, or Microsoft's. Those are generally well behaved and self-indentified.

Answer 1

http://www.robotstxt.org/db.html is a good place to start. They have an automatable raw feed if you need that too. http://www.botsvsbrowsers.com/ is also helpful.

Answer 2

I asked question some thing like this before some time Please if it can help you ?

http://stackoverflow.com/questions/1350884/what-is-a-good-web-search-and-web-crawling-engine-for-java

Answer 3

I'm using http://www.user-agents.org/ usually as reference, hope this helps you out.

You can also try http://www.robotstxt.org/db.html or http://www.botsvsbrowsers.com.

Answer 4

Unfortunately we've found that bot activity is too numerous and varied to be able to accurately filter it. If you want accurate download counts, your best bet is to require javascript to trigger the download. That's basically the only thing that is going to reliably filter out the bots. It's also why all site traffic analytics engines these days are javascript based.

Answer 5

There is an API available at www.atlbl.com that will identify web crawlers based on their useragent and source IP address. It manages to catch stealth webcrawlers, google-impersonators, and other nefarious bots, by their IP.