I have some div on page loaded from server by ajax, but in the scenario google and other search engine don't index the content of this div. The only solution I see, it's recognize when page get by search robot and return complete page without ajax. 1) Is there more simple way? 2) How distinguish humans and robots?
+3
A:
You could also provide a link to the non-ajax version in your sitemap, and when you serve that file (to the robot), you make sure to have included a canonical link-element to the "real" page you want users to see:
<html>
<head>
[...]
<link rel="canonical" href="YOUR_CANONICAL_URL_HERE" />
[...]
</head>
<body>
[...]
YOUR NON_AJAX_CONTENT_HERE
</body>
</html>
edit: if this solution is not appropriate (some comments below points out that that this solution is non-standard and only supported by the "big-three"), you might have to re-think whether you should make the non-ajax version the standard solution, and use JavaScript to hide/show the information instead of fetching it via AJAX. If it is business critical information that is fetched, you have to realize that not all users have JavaScript enabled, and thus they won't be able to see this information. A progressive enhancement approach might be more appropriate in this case.
PatrikAkerstrand
2009-05-23 12:26:07
+1, That's really useful.
karim79
2009-05-23 12:44:25
Interesting idea, but canonical isn't a standart and support only google, yohoo and microsoft.
dotneter
2009-05-23 13:05:54
Darn, only google, yahoo and msft, that's less than 99.99% of all search engines by traffic, how could a site possibly survive on so little...?-)
Alex Martelli
2009-05-23 15:57:52
I think each not english speaking country have some specific national search engine, like Baidu in China.
dotneter
2009-05-23 16:45:42
@ais: link rel='canonical' is a w3c standard
ilya n.
2009-06-20 00:41:42
+2
A:
Google gets antsy if you are trying to show different things to you users than to crawlers. I suggest simply caching your query or whatever it is that needs AJAX and then using AJAX to replace only what you need to change. You still haven't really explained what's in this div that only AJAX can provide. If you can do it without AJAX then you should be, not just for SEO but for braille readers, mobile devices and people without javascript.
SpliFF
2009-05-25 10:10:20
A:
You can specify a sitemap in your robots.txt. That sitemap should be a list of your static pages. You should not be giving to Google a different page at the same URL, so you should have a different URL with static and dynamic content. Typically, the static URL is .../blog/03/09/i-bought-a-puppy and dynamic URL is something like .../search/puppy.
ilya n.
2009-06-20 00:40:39