Do web crawlers use cookies, or discard them?
A:
Crawlers do not react to commands to issue cookies and store them until the next visit to a page.
Crawlers do not execute JavaScript.
Crawlers just read default pages (what gets served with no cookies and JavaScript turned off) and rank them based on what they see.
User
2009-07-03 08:31:59
Sorry -1 - there are webcrawlers that use cookies - here's the documentation to how to turn it on for IBM's Omnifind - http://publib.boulder.ibm.com/infocenter/discover/v8r5m0/index.jsp?topic=/com.ibm.discovery.es.ad.doc/administering/iiysatcookie.htm
DanSingerman
2009-07-03 09:16:40
Then that's new for me. Thanks.
User
2009-07-03 09:24:44
And as of 2010, some crawlers (e.g. GoogleBot, but also some spambots) have JS, even Flash execution capabilities.
Piskvor
2010-10-13 10:38:30
+4
A:
A crawler is just a piece of software that downloads files over http, and tries to find further URLs to download more files.
It is up to the creator of the crawler to determine whether they wish to use or ignore javascript and cookies.
Most crawlers, I believe, currently will ignore javascript and cookies, but there is no reason why that has to be the case (and you certainly should not rely on it, say for security reasons)
DanSingerman
2009-07-03 08:42:31