Is there a way to tell when googlebot/bingbot/yahoobot is crawling my site in asp.net 2005 IIS6?

Question

I want to know when google is crawling the site, preferably by sending myself an email.

Is there any way to do this that won't adversely effect performance?

Answer 1

The user agent identifies itself as gsa-crawler/Googlebot depending if a Google appliance or Google. You could write some code (Request.ServerVariables("HTTP_USER_AGENT")) that watches for that or analyses IIS log files.

You might want to verify that the bot really is Google- that is something to watch out for.

I'd choose the Web app root if you need an email to sent out ASAP and the file analyse root if you can wait a little longer.

Answer 2

Your IIS logfiles (usu. in c:\windows\system32\logfiles...), will show you when the spiders have crawled your site. Just be sure you've enabled logging and that you log the browser/user agent. The browser agent is a string that contains, among other things, the name of the web browser/bot/whatever that requested the page. Most entries are obvious on visual inspection (e.g., googlebot contains the string "googlebot").

If you want to be notified, I'd have a task that runs every fifteen or twenty minutes, scans through your the log entries since the last scan, and emails you on a pattern match, as the previous respondent has suggested.