Why doesn't Googlebot index pages it crawls?

views:

200

answers:

Why doesn't Googlebot index pages it crawls?

Three months ago I published my small personal website (~10 pages), submitted the URL to Google, and a few days later Googlebot showed up. Over the course of the last couple of weeks, Googlebot visits my website approximately twice a week and crawls maybe every other page.

Ever since Googlebot first crawled my website, whenever I run a search for site:example.com Google returns only my homepage. (Interestingly, so does Bing, so maybe the problem isn't specific to Google.)

I built the website with CodeIgniter mainly to familiarize myself with it. It's really simple, only a couple of pages about me and my projects. I am not using any black-hat SEO techniques, JavaScript, or anything like that.

What could be possible reasons why Googlebot would crawl my pages but not index them?

EDIT: I do have a Webmaster Tools account. There are no crawl errors, internal links are listed, but listed keywords come only from my homepage.

+4 A:

Dewfy 2009-12-25 16:54:50

+1 for mentioning the Webmaster tool. It's the most reliable way to tell whether Google has a problem indexing a site.

Pekka 2009-12-25 16:58:33

Googlebot has no problem finding the pages. According to my logs, it does visit other pages on the website -- for some reason it just doesn't index them.(1) AFAIK this shouldn't be necessary if links to other pages are easily discoverable, as they are in my case, but I guess submitting a sitemap won't hurt.(2) Individual pages are very different. (3) I have the same navigation menu on all pages.

MiseryIndex 2009-12-25 16:59:59

@MiseryIndex I think googleboot indexes +9 pages without any problem, but it "think" that these pages has no unique content in comparison with home. So you have solve: why I need 10 pages but not 1? If you will use advanced search option with google, you can see "Show with duplicated result" - try to expand this.

Dewfy 2009-12-25 17:31:38

MiseryIndex 2009-12-25 17:42:17

If so, there is no such link when I search for my website.

MiseryIndex 2009-12-25 17:44:00

@MiseryIndex definitly not. I'm using signed in version of google under the Chrome. In the top there is "+" to show duplicated, and at left there is a tree of advanced options of search.

Dewfy 2009-12-25 21:37:20

I see no plus sign to show duplicated at the top even if I sign in to Google. If it's not too much trouble could you post a screenshot?

MiseryIndex 2009-12-25 22:53:54

Dewfy 2009-12-28 10:33:23

Hmm... I see no such links when I search for my site, but they do show up when I use your query. Must be something else. Thanks for trying though. :-)

MiseryIndex 2009-12-30 03:42:28

ansaurus

tags:

views:

answers:

Why doesn't Googlebot index pages it crawls?

related questions