Are AJAX sites crawlable by search engines?

Question

I had always assumed that AJAX-driven content was invisible to search engines.

(i.e. content inserted into the DOM via XMLHTTPRequest)

For example, in this site, the main content is loaded via AJAX request by the browser:

http://www.trustedsource.org/query/terra.cl

...if you view this page with Javascript disabled, the main content area is blank.

However, Google cache shows the full content after the AJAX load:

http://74.125.155.132/search?q=cache:JqcT6EVDHBoJ:www.trustedsource.org/query/terra.cl+http://www.trustedsource.org/query/terra.cl&cd=1&hl=en&ct=clnk&gl=us

So, apparently search engines do index content loaded by AJAX.

Questions:

• Is this a new feature in search engines? Most postings on the web indicate that you have to publish duplicate static HTML content for search engines to find them.
• Are there any tricks to get an AJAX-driven content to be crawled by search engines (besides creating duplicate static HTML content).
• Will the AJAX-driven content be indexed if it is loaded from a separate subdomain? How about a separate domain?

Answer 1

Content that gets loaded immediately (say with a secondary HTTP request as in your example after the initial pageload) is usually visible to the search engine crawler.

However, if you have content that beyond this gets loaded via ajax following a user action, e.g. clicking a tab or button and such, won't be seen or indexed. Those will only be seen or indexed if they have 'real' anchor links.

Answer 2

AJAX-driven are not crawled by search engines (or at least, not by Google).

The reason you can see the page in the google cache is because in the cache, there is the full page, including .js file. So when you see the page, your browser use the google cached .js file.

I don't think there is any trick to make it crawled by search engine, except using a static .html.

Edit at April, 27th 2010 : Google published a way to make AJAX crawlable

Google webmaster toolkit might help.

Answer 3

I have NoScript installed and active. Both links show the same content (+/- the google header bar). Therefore, the Google cache shows only what is statically there.

Answer 4

If you're using something like jQuery tabs, even if you're linking to HTML files within the same directory, it degrades nicely back to normal without the javascript, and the tabs just become likes to the actual pages. It's ugly, but it works. You can also style these versions, too.

Answer 5

Following this guide from Google, AJAX sites may be made crawlable:

http://code.google.com/intl/sv-SE/web/ajaxcrawling/docs/getting-started.html