Does Google crawl AJAX content?

Question

On the home page of my site I use JQuery's ajax function to pull down a list of recent activity of users.

The recent activity is displayed on the page, and each line of the recent activity includes a link to the user profile of the user who did the activity.

Will Google actually make the ajax call to pull down this info and use it in calculating page relevancy / link juice flow?

I'm hoping that it does not because the user profile pages are not very Google index worthy, and I don't want all those links to the User profile pages diluting my home page's link juice flow away from other more important links.

Answer 1

No, it won't make an Ajax call.

But if it discovers the Url in plain text your Ajax function is calling - it will definitely try to look what's there.

You may wish to add to those links "noindex" meta tag.

Answer 2

Maybe. If you want to guarantee Google won't spider your JSON responses, put them in robots.txt. That isn't security, though; indeed, it's the first place a cracker will look for "interesting" pages. And other crawlers may ignore it.

Answer 3

No, it will not crawl AJAX content by default.

http://code.google.com/web/ajaxcrawling/ has instructions on how to make AJAX content crawlable, but those are explicit steps you need to take, it isn't automatic

Answer 4

Here is some info on AJAX crawling from the makers of JQuery Address plugin.

Answer 5

Google is definitely crawling content in our page that is only referenced within an AJAX request.

I'm wondering if Google is going through the page source, and looking for potentially valid URLs, and testing to see if they've got content.

Here is what our request looks like... which might offer some insight into what's going on.

'$(document).ready(function() {

    $("#theDiv").block({ message: 'Getting latest content...' });
    $.ajax({
        url: '/content/pages/articles?count=4&part=true',
        success: function(data) {
            $('#theDiv').html(data);
            $("#theDiv").unblock();
        }
    });
});`

Answer 6

Google now has a way for you to enable crawling on AJAX pages. If your links contain "#!", then google will change that to "?_escaped_fragment_=" and request that document from your server. However, when it shows that page in the search results, it will show the original URL with the "#!".

http://code.google.com/web/ajaxcrawling/docs/learn-more.html