How-To get private pages being crawled by google

Question

How can i get private pages of my web site being crawled and indexed by google ?

maybe it's not very "conventionnal", but i want my private page "links" displayed in google index, but next require a registration to display the page.

Answer 1

They'll get crawled if and only if they're publicly accessible and your robots.txt file allows it. That's pretty much all you need to do.

Answer 2

Are you asking how to get Google to index your pages?

There are a couple of ways. You need to ensure that you have SEO'd, or Search Engine Optimisation, the pages properly with title text and description key words in your meta data.

You can also submit your site to Google, it's a free service, and it'll be placed in a queue of things that Google will index. May take some time though.

By far the best way to get your pages indexed is using the meta data in the pages themselves.

Answer 3

EDIT: Based on the addition of "maybe it's not very "conventionnal", but i want my private page "links" displayed in google index, but next require a registration to display the page." To the question: You can check the User Agent in your php code to basically allow google to see pages if it was a registered user (google's user agent is "Googlebot/1.0" and you can search to find user agents for other common engines).

However, this behavior is specifically against google's rules and they can and will remove your site from the index if they catch you doing it. Their policy is you should not treat googlebot any differently than you treat any random person who visits your site.

(Original Answer) One way is to use a sitemap to show google how to find all of your pages. In general, and even in the case of sitemaps, if the content you want indexed is not linked to from a page that can be found through the "root" (/) (i.e. there is no way for the public to find it), then it probably won't get indexed. The only way to get it indexed is to link it in someplace.

The question is though, why do you want your private pages in google anyway?

Answer 4

Google will only index what is

• linked from somewhere already in Google's index
• accessible to its crawler via normal (unauthenticated) HTTP

It will also

• make the contents available in search results to anyone.

This may conflict with your idea of a "private" page.

Answer 5

I'm going to assume that all the other previous answerers are misunderstanding you. As I read it, you aren't asking how to get Google to index your pages, but rather how to get a list of all the pages that Google currently has already indexed on your site? If that is true, you should have a look at the Google Webmaster Tools.