Google SEO - Get a list of my pages in Google's index?

Question

I'm attempting to diagnose an issue wherein Google reduced my site:mysite.com page count from 86,400 to 17,500 over the course of three days.

Is there any way I can get a complete list of the pages that Google has in its index for my domain?

Details

I normally wouldn't post actual links for my site, but here's a good example: our Device Database (list of devices and related content that the tools we sell support) has over 2200 devices published:

(index at http://www.keil.com/dd/

However, the Google query site:www.keil.com/dd/chip site:keil.com/dd returns only 1,620 results.

Each of the device pages also links to example code, PDFs, and other assets that are also supposed to be indexed, and were until very recently.

This isn't a huge difference. My discussion forum, however, has around 16,627 threads, each with their own pseudo-static page that is updated whenever anyone posts a new thread or a response. The site: query for http://www.keil.com/forum/docs returns only around 5,000 pages.

There are numerous other areas on this site (it's huge, and has been incrementally developed over the past 15 years) that are suffering from the same effect. On April 1st, I was seeing the 86k number from the site: query, where it has been for the past several years (+- based on forum activity). By April 4th, it was in its current state.

Our SEO expert left awhile back. Help.

Answer 1

Use the Webmaster Tools to list links, report errors, choose which query string parameters are/aren't safe to ignore, and add a sitemap if necessary.

(Did you really have 86,400 real pages of actual unique content?)

Answer 2

There are three elements involved in a change like that:

1. You need to make sure that you have minimal duplication through URLs if you want to have a somewhat correct count to work with. See http://googlewebmastercentral.blogspot.com/2009/10/reunifying-duplicate-content-on-your.html for some ideas on that.

2. The site:-query count is a very, very rough approximation that is not worth tracking.

3. Instead of the site:-query, submit Sitemap files with all of your preferred URLs. For those URLs, you will see the indexed URL count in Webmaster Tools (it will only count if the listed URL is indexed exactly like you have included it, so working on fixing duplication through irrelevant URLs is vital).

If you want to know more about which URLs are actually indexed, I'd create separate Sitemap files for logical parts of your site and check the indexed URL count there. It might be that the non-indexed URLs are not so important (eg detail pages might not matter so much if a higher-level page is indexed).