My rating system allows anonymous users to add ratings, but Google's crawler is rating things.
How can I ensure that Googlebot won't follow the link?
+3
A:
Use a robots.txt to point out links that bots shouldn't follow. For example, put the following in http://example.com/robots.txt
User-Agent: *
Disallow: /vote.php
You can read more about robots.txt here: http://www.robotstxt.org/
Google and every other well-behaved bot will read and follow directions in robots.txt.
If you also have problems with bots not following those directions you will have to code some logic to block bots, or at least to decrease their impact. You can for example log how many votes you've got from an IP address in a certain time frame and block votes above that level. Another solution can be to only allow posts, and also have some JS logic (or similar) to block out spam bots, but that's much more work than robots.txt so only put time into it if it becomes a problem.
You can also block bad-behaving bots entirely by blocking their IPs in your web server. There are a few lists of bad-behaving bots out there you can try if you prefer the block solution.
Emil Vikström
2010-10-20 13:47:32
+5
A:
You shouldn't accept a GET request for any action that modifies data (voting, editing a post, etc.). Your voting should be done via a POST request, which Googlebot won't perform.
More information in this SO post: http://stackoverflow.com/questions/46585/when-do-you-use-post-and-when-do-you-use-get
ceejayoz
2010-10-20 13:48:41
i want google dont followlink.
meotimdihia
2010-10-20 13:52:03
Yes, we know. Google isn't the only bot you should be afraid of, though - spambots will trigger this link, even if you use the `robots.txt` approach in @Emil Vikström's post. If you want to truly fix this, you should fix the underlying problem, which is that you're changing data in your application via a GET request - something that shouldn't done.
ceejayoz
2010-10-20 13:54:17
ceejayoz, note that using POST often gets you a lot of spam bots instead, since those are really fast to jump on web forms.
Emil Vikström
2010-10-20 13:56:07
@Emil Vikström Having the links trigger a POST via AJAX would be a good way to prevent that.
ceejayoz
2010-10-20 13:57:15