Do I absolutely need a robots.txt file for my public-facing web sites?

Question

I'm reviewing a couple of my web sites to make sure my SEO bases are covered. There are no private pages on the site in question, and we want all pages to be indexed. Does including a robots.txt file anyway, even if it isn't needed, make my site look more reputable and important to Google?

Answer 1

no, it won't.

google has its own rules to decide and won't rely on the existance of robots.txt.

Answer 2

Only google can answer that for you.

According to wikipedia, robots.txt is only for blacklisting, rate limiting, and sitemaps.

Answer 3

I would just include a blank one regardless because without it your server is still spitting out a 404 message to the requester. Which is probably bigger then a blank robots.txt file.

Answer 4

As per Google, it's a good idea. Their instructions / guidelines are actually pretty thorough.

Answer 5

Search engines will look for a robots.txt file. If it's not there, the search engine will get a 404. It's a good idea to have one even if it's blank.