Looking through my search logs from time to time, I notice that by far the biggest user of my search engine is the google-bot. What gives? Is it looking for content that might not be directly accessible through navigation? If so, how does it know which words and phrases to look for (they're surprisingly relevant). Does it check the most popular keywords on the site? I know I seem to be answering my own question here, but this is really only working it out from first principles. I'd like to hear from someone who knows what they're talking about (i.e. not me).
+7
A:
If your search form's method is get instead of post, each search has its own url, and people might be posting those urls elsewhere. Or if you have a (possibly inadvertently) publicly accessible webstats page that listed those urls, that's another common way for search engines to stumble upon your internal search urls. A third way I've seen is sites that list recent searches on their pages, but this is more intentional. "MySQL Performance Blog" does this to an annoying extent, so any search of their site from google yields hundreds of pages of similar searches, even if none of them found what they were looking for.
Edit: Looks like it does on occasion, but only GET forms: http://googlewebmastercentral.blogspot.com/2008/04/crawling-through-html-forms.html
David
2009-08-04 06:00:09
Actually, that's a good point. The search form's method is GET so that staff and users can bookmark their searches. I've double-checked my stats and they are pw protected. The thing is, googlebot is making its own unique searches; they don't appear anywhere else in the logs. Curious, huh?
Iain Fraser
2009-08-04 06:05:03
Just curious, what words does it enter?
Daniel S
2009-11-13 01:23:24