Selenium (RC) is being used to test an ASP.NET 1.1 site.
When we make a request via Selenium RC (which in turn automates the request via a configured browser - in this case Firefox) the http verb is "HEAD". We have several form action methods that have separate GET and POST methods decorated with AcceptVerbs(HttpVerbs.Get) or HttpVerbs.Post respectively. These methods are returning a 404 and logging a "a public action method could not be found" error message.
Questions:
When writing separate Get/Post action methods what is the best practice for handling the Head verb? Should we always decorate with an AcceptVerbs(HttpVerbs.Get | HttpVerbs.Head)?
Why is the HEAD verb being generated when Selenium RC is automating the browser in lieu of an If-Modified-Since header?
We've also seen log entries from (non-mainstream) crawlers that are using the HEAD verb. We created robots.txt entries to stop these crawlers from indexing the site, but now we're wondering what the best practice from an SEO perspective is as well. Is it important to respond to HEAD for crawlers? Are there mainstream crawlers that use it? Does it impact SEO rank?