I've set my ErrorDocument 401 to point to my website's account creation page, but not all browsers seem to honor this redirect (Safari).
Also, other browsers (Firefox, Chrome) never quit asking for the password and show the ErrorDocument. This causes a good number of users to give up trying after many password attempts without seeing the account creation page.
Is there any way to make the redirect more reliable, without trashing basic authentication altogether?
I suspect that your firefox and safari users are not entering the domain before the username i.e. MYDOMAIN\USERNAME. There are some settings in firefox that will allow pass-through authentication; I don't know about safari.
The simple answer to your question is no, you can't make this more reliable without implementing custom authentication.
The only way that Firefox and Chrome will display page that you specified in the ErrorDocument 401 directive is if you click cancel button. Also, there is no redirect sent with the 401 HTTP code; rather, it is a content of the document specified with ErrorDocument 401 directive. You can do redirect using HTML meta tag:
<Location "/protected">
AuthUserFile /path/to/users
AuthName "This is protected area"
AuthGroupFile /dev/null
AuthType Basic
Require valid-user
#ErrorDocument 401 /register.html
ErrorDocument 401 "<html><meta http-equiv=\"refresh\" content=\"0;url=/register.html\"></html>"
</Location>
Possible solutions to your problem are to create custom basic HTTP authentication module or to use language like php that supports basic HTTP authentication hooks