tags:

views:

106

answers:

1
+4  Q: 

Why is Yahoo Indexing Bot considered as "evil"?

After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow's sitemap. This returned a 404 error which I am guessing is just a protected page by determining if your are a Index Bot or simply doesnt exists.

This then lead me to look at the robots.txt for StackOverFlow. I was surprised to see the comment "Yahoo bot is evil" along with a couple other Indexing bots (Spinn3r and KSCrawler) . I am unfamilular with Spinn3r and KSCrawler but my question is, why are these bots (particular Yahoo) considered as evil? Surely any and all indexing of any Search Engine is a good thing?

+2  A: 

These are very aggressive bots that do not throttle requests and can bring a site down.

See this forum thread for an example for the yahoo bot.

Oded  2010-04-01 18:43:39
It ignores crawl-delay?
Brian  2010-04-01 18:51:36

related questions

How do you know what a good index is?
What datatype should be used for storing phone numbers in SQL Server 2005?
What is the best way to build an index to get the fastest read response?
How would I go about creating a custom search index much like Lucene?
What is a Covered Index?
When should you use full-text indexing?
SQL/Oracle: when indexes on multiple columns can be used
Best use of indices on temporary tables in T-SQL
On Disk Substring index
How to implement a "related" degree measure algorithm?
Non-Clustered Index on a Clustered Index column improves performance?
HTML Help keyword location
One or Two Primary Keys in Many-to-Many Table?
Strategies for keeping a Lucene Index up to date with domain model changes
Do indexes work with "IN" clause
Table Scan vs. Add Index - which is quicker?
Why does SQL Server work faster when you index a table after filling it?
PostgreSQL: GIN or GiST indexes?
What's the difference between a Table Scan and a Clustered Index Scan?
Database Case Insensitive Index?
Testing for inequality in T-SQL
Tables with no Primary Key
How do I index a database field
How does database indexing work?
Why doesn't SQL Full Text Indexing return results for words containing #?