tags:

views:

38

answers:

2
+2  Q: 

The best and most efficient methods of detecting web crawlers

There are many ways to pretend to be a human being. So what are the best methods to see past it?

A: 

Configure your web server to include a Voight-Kampff response header

paintcan  2010-06-29 19:12:08
A: 

See http://stackoverflow.com/questions/233192/detecting-stealth-web-crawlers

Rob  2010-06-29 19:15:49

related questions

Web crawlers and Google App Engine Hosted applications
Detecting CacheBuster querystrings when crawling a page
How to prevent robots.txt passing from staging env to production?
Detecting honest web crawlers
How to force a page to be removed from the search engine index?
How to best develop web crawlers
robots.txt: disallow all but a select few, why not?
Asp.net Request.Browser.Crawler - Dynamic Crawler List?
What are the best prebuilt libraries for doing Web Crawling in Python
Anyone know of a good Python based web crawler that I could use?
Web crawler links/page logic in PHP
Crawler/parser for Xapian
Protect Email on Web Site From Robots and Crawlers
Recommendations for a spidering tool to use with Lucene or Solr?
Detecting 'stealth' web-crawlers
Can I block search crawlers for every site on an Apache web server?
HttpBrowserCapabilities.Crawler property .NET
Prevent site data from being crawled and ripped
What's a good Web Crawler tool
Building a web crawler - using Webkit packages
Is there a .NET equivalent of Perl's LWP / WWW::Mechanize?
How do you turn a dynamic site into a static site that can be demo'd from a CD?
keep rsync from removing unfinished source files
How to set up a robot.txt which only allows the default page of a site
What are the key considerations when creating a web crawler?