According to RFC2616 if I return 401 in response to a request to my (Ruby) server, I "MUST include a WWW-Authenticate header field." Is this really true? Not setting the header seems to have no negative impact. I'm using Merb as a web framework and it doesn't force me to set the header.
Am I missing something or is this a rule more honoured in the breach?
The issue is whether you expect users to be able to navigate from the 401 failure to a successful authentication going forward. If you fail to provide a WWW-Authenticate header, then you are changing the meaning of the 401 from 'You must supply credentials' to 'we don't like your kind around here'. This might be fine for your purposes, but the inherent impoliteness in the concept of rejecting credentials without offering a way to fix the problem is the root behind the 'MUST'.
You send 401 if you want the client to authenticate, in which case you need to tell it how.
So what do you want the client to do? If it's just a "you can't do this" message, consider 403.