views:

266

answers:

7

Whats the best recommended way yo hide my staging website from search engines, i Googled it and found some says that i should put a metatag, and some said that i should put a text file inside my website directory, i want to know the standard way.

my current website is in asp.net, while i believe that it must be a common way for any website whatever its programming language.

+1  A: 

http://www.robotstxt.org/robotstxt.html

Chris
+5  A: 

You could also use your servers robots.txt:

User-agent: *
Disallow: /

Google's crawler actually respects these settings.

amphetamachine
+7  A: 

Use a robots.txt file.

see here http://www.robotstxt.org/robotstxt.html

NinjaBomb
A: 

There are search engines / book marking services which do not use robots.txt. If you really don't want it to turn up ever I'd suggest using capcha's just to navigate to the site.

steve
Good luck getting any visitors with that.
grawity
Just out of curiosity, which specific search engines / book marking services are you referring to?
NinjaBomb
or put a login page
Amr ElGarhy
@Grawity - he doesnt want to be in a search engine, therefore I assume he isn't looking for mainstream people. @NinjaBomb - delicious,a group of p2p search engines which have people going around the internet index the site automatically instead of google , yahoo etc.
steve
+5  A: 

Really easy answer; password protect it. If it’s a staging site then it quite likely is not intended to be publicly facing (private audience only most likely). Trying to keep it out of search engines is only treating a symptom when the real problem is that you haven’t appropriately secured it.

Troy Hunt
+2  A: 

Keep in mind that you can't hide a public-facing unprotected web site from a search engine. You can ask that bots not index it (through the robots.txt that my fine colleagues have brought up), and the people who write the bots may choose not to index your site based on that, but there's got to be at least one guy out there who is indexing all the things people ask him not to index. At the very least one.

If this is a big requirement, keeping automated crawlers out, some kind of CAPCHA solution might work for you.

quillbreaker
A: 

Whats the best recommended way yo hide my staging website from search engines

Simple: don't make it public. If that doesn't work, then only make it public long enough to validate that it is ready to post live and then take it down.

However, all that said, a more fundamental question is, "Why care?". If the staging site is really supposed to be the live site one step before pushing live, then it shouldn't matter if it is indexed.

Thomas