I have heard that web crawlers are supposed to follow only GET requests and not POST ones.
In the real world is this a valid assumption?
+1
A:
Are you asking if crawlers will parse out URLs from form tags and follow them? Some will. It's a pretty safe assumption they will always use the GET verb though.
Spam bots are a different story.
Spencer Ruport
2009-06-19 19:07:55
My primary question was if they would ever use the POST verb.
Dana Holt
2009-06-19 19:23:39
No one with good intentions will. But it's just a request header, and a lot of people have bad intentions. Design for evil.
marr75
2009-06-20 18:03:51
+2
A:
Generally they do not do POST requests. This is just the current state of affairs and is not dictated anywhere, I believe. Some search engines are experimenting with crawling forms, but these are still GET requests.
molf
2009-06-19 19:08:50
A:
Polite crawlers only use GET requests. Just like polite crawlers obey the bots.txt.
However, even google is playing fast and loose with the definition of polite, their crawler has done serious bandwidth damage to some sites including stackoverflow.
marr75
2009-06-19 20:37:53