Detect if visitor is a browser and not a crawler

Question

Hi

How would I server side detect if the user of the webpage is a browser?

The reason for detecting this is, that I'm storing statistics about visited pages and I don't want to store any statistics when the user is a crawler. So I'm not trying to detect which browser vendor. I only want a boolean answer, is the user a browser.

I assume the answer is connected with the user-agent header, but the numerous possibly values is too overwhelming for me to figure out. How can I detect it? A 90% solution where I only detect the most Top5 popular browser would be good enough.

I'm using C# ASP.Net, but I guess most solutions in other languages and frameworks could be translated.

Answer 1

If you're using PHP, try $_SERVER['HTTP_USER_AGENT'] or the get_browser() function.

Here's a class that's available for download that makes this process incredibly easy. Example usage:

$browser = new Browser();
if( $browser->getBrowser() == Browser::BROWSER_FIREFOX && $browser->getVersion() >= 2 ) {
 echo 'You have FireFox version 2 or greater';
}

Answer 2

If you need to decide whether to return (say) raw text for a script versus HTML for a browser, perhaps you should inspect the Accept-Encoding request header?

Answer 3

Perl has a module HTTP::BrowserDetect. According to the docs, this has a method which checks whether a user agent is a robot. Anything which isn't a robot is probably a browser, I guess.

Answer 4

This code is by no means exhaustive, but gives you a basic founding in PHP. I can't guarantee a few won't slip through the net, but this should catch most browsers and ignore most bots.

<?php
$browserlist = '/(opera|aol|msie|firefox|chrome|konqueror|safari|netscape|navigator|mosaic|lynx|amaya|omniweb|avant|camino|flock|seamonkey|mozilla|gecko)+/';
if (preg_match($browserlist,$_SERVER['HTTP_USER_AGENT'])){
    // is a browser
    }
else{
    // is not a browser
    }
?>

Answer 5

I am not sure why you are doing this.

However, you do not want to even pretend you can tell between a human versus a robot based on the User-agent header. You might be able to get some success analyzing the pattern and timing of requests.