is there any point in having a robots.txt with just one line of content like below
User-agent: *
compared to not having it at all ?
A:
No, google (and let's not beat around the bush, Google is the search engine you should care about) by default will crawl everything unless you tell it not to.
If you want to help Google, use google webmaster tools and submit a sitemap in the correct format
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184
Nick Allen - Tungle139
2010-02-24 14:28:05
the google crawler would avoid an http:404 however.I wonder if that would mean that my site is more reliable.
lorenzo 72
2010-02-24 14:29:23
google webmaster tools will identify 404 pages it is coming across allowing you to fix them or setup 301 re-directs, which you should be doing anyway. This in turn will make your site more reliable, plus being a bit more careful : )
Nick Allen - Tungle139
2010-02-24 14:31:28
-1 Google is only (roughly) 66% of all search engine usage. So what you are recommending is that the OP just disregard 33% of his possible customer base? That sounds like a bad decision to me.
amischiefr
2010-02-24 14:31:46
66% where?! Google is responsible for around 98% of searches (in the UK anyway, it was roughly 90% in 2007). In any case the point stands true for any search engine, what the OP is suggesting is pointless. What other search engine also provides webmaster tools like googles. IMHO worrying about search engines outside of google is a waste of time because anything you do for google will probably benefit other search engines anyway
Nick Allen - Tungle139
2010-02-24 14:34:03
Lol. Google has a mamoth share of the search engine market. If you exclude China then Nick is correct. amischiefr is totally wrong.
Laykes
2010-02-24 14:35:36
Oh wait hold on, maybe we should be optimizing for wolfram alpha, I heard about 1000 people used that last year LOL
Nick Allen - Tungle139
2010-02-24 14:37:45
http://www.ahfx.net/weblog/154, http://seo.zunch.com/search_engine_usage_statistics.htm, http://searchenginewatch.com/2156451 Really? I'm totally off huh? Show me some research that shows Google has 98% of all internet searches. I'ld love to see it.
amischiefr
2010-02-24 14:52:09