Does googlebot crawl urls in jQuery $.get() calls and can it be prevented?

Question

Hi,

I have a page that has a form using this ajaxForm jQuery plugin. The form submits, and when it's complete, there is a call using $.get() to load some new content to the page.

My problem is, the Googlebot "appears" to be indexing the url in the $.get() method.

My first question is, is that even possible? I was under the impression the Googlebot didn't evaluate javascript for the most part (I read something about it being able to index content on urls with !#).

My second question is, if Google is indexing this call to that url, is there a way to prevent it?

Thanks in advance.

Answer 1

You could robots.txt the file specifically, googlebot will should honor it.

From robotstxt.org:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

You can also look at Google's Webmaster Central to remove the file from the listing.

Answer 2

First of all you need to check that that is really the GoogleBot because anyone can pretend being GoogleBot, even a legitimate user.

The recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP lookup using that googlebot.com name.

Sourced from Official Google Webmaster Central Blog: How to verify Googlebot.