ansaurus

Question

Any way to NOT index certain keywords on page?

Answer 1

+1 A:

You can detect the incoming user agent and if it is a bot user agent render the page with the words you don't want to index.

However, chances are good that such keywords are going to be much more popular on other sites than your blog - is it really a concern?

If so, the simple fact is that "the Internet doesn't forget" - it is better not to post something compromising to begin with.

Oded 2010-10-15 20:09:26

Whats the most accurate way of detecting the user agent as a spider? using either PHP, JS, HTML? Thanks!

John Isaacks 2010-10-15 20:16:02

@John Isaacks - looking at the user-agent header. However, this (like everything else on the web) is not 100% reliable and can be spoofed. See this: http://www.user-agents.org/

Oded 2010-10-15 20:22:51

I'd be careful with that. Google (and probably other search engines) doesn't like you sending it something that regular users won't see. http://www.google.com/support/webmasters/bin/answer.py?answer=66355

Adam Jaskiewicz 2010-10-20 03:06:02

Answer 2

+1 A:

Look into the following tag:

<!--googleoff: all-->

http://www.geekzilla.co.uk/ViewC8614968-56ED-4729-9C12-F01677DAC412.htm

Sabeen Malik 2010-10-15 20:10:29

That is very cool, however, I don't think it works for regular Google indexing. Would be very nice if it did.

John Isaacks 2010-10-15 20:14:58

it does work with regular Google indexing as well, we have tried successfully in the past :)

Sabeen Malik 2010-10-15 20:16:42

this is for google enterprise search, not for the real google. see: http://code.google.com/apis/searchappliance/documentation/46/admin_crawl/Preparing.html#pagepartno, it does not work. sabeen, please point to a page where this was implemented. i tested it too a long time ago - it does not work, never did.. but i would be very happy to get falsified.

Franz 2010-10-19 10:21:47

@Franz ... i can not directly tell you the URL for corporate reasons but i can tell you what happened to us. Google was showing our pages with description which was in hidden divs, we asked our frontend guy about it and how to fix it and then he went ahead and used these tags and a week or so later the description changed on google results. From that i am assuming it does work.

Sabeen Malik 2010-10-19 10:33:57

@sabeen the snippet logic / description logic is far removed from the indexing logic. there are numerous reasons (i.e. the term you entered via search) what snippet gets shown. the best way to see what gets indexed is to click on "cache" -> "text version only" next the a search listing on the google SERPs ... trust me,  does not work, otherwise you would have discovered the golden grail of SEO.

Franz 2010-10-19 10:42:36

@Franz .. fair enough, though we did look at the results for the exact same search term etc but i guess you are right, it could be something else that triggered the change, i remember the guy showing me that even adobe.com uses in their site, even the home page has it , well I guess this is pure speculation.

Sabeen Malik 2010-10-19 10:47:58

Answer 3

A:

implement it as an image
implement it as an iframe which is blocked via robots.txt
implement it as a post load external javascript

i recommend not doing it via user agent cloaking as this might be seen from google side as misleading and can lead to a penalty or demotion

Franz 2010-10-19 10:27:04

Answer 4

A:

Google doesn't support no indexing portions of a page. You can watch a video of Google's Matt Cutts talk specifically about this issue: http://www.youtube.com/watch?v=_p37hQhHDTQ

Yahoo back in 2007 came out with a robots=nocontent which allowed you to noindex portions of your page. However, it's unclear whether or not Bing will continue to support it ... many guess that Bing won't.

There are HTML techniques that you can use like implementing the content as an image, iframe, or javascript, but those are fairly "spammy" hacks and most likely one of your competitors will report you.

phaithful 2010-10-20 04:31:44

ansaurus

tags:

views:

answers:

Any way to NOT index certain keywords on page?

related questions