I was wondering if protecting a web directory with an .htaccess file (with the .htpasswd file outside the public apache webroot) is a secure way to protect a directory (and its contents).
Could someone explain the Do's and Don'ts of this kind of protection?
.htaccess is pretty standard way how to make per-directory configuration changes for resources served by Apache HTTPD in cases where you don't have access to main configuration file/not having root access.
If you have access to main configuration, it's much easier to have all configuration (including authentication) placed in one central location (even if it's split into multiple files), where it's not so easy to overlook it. From my experience I can tell, it's just a matter of time before you forget about your .htaccess files.
Official documentation is mentioning on several occasions that use of .htaccess files should be avoided when possible.
If use of .htaccess is your only choice, make sure you follow general security precautions as with main HTTPD config, i.e. prevent unauthorized users from reading them, files are readable by server, make sure you got directory listing disabled, always make sure password are stored in encrypted/hashed format, etc.
For more info, please, check Apache htaccess tutorial
A .htaccess file is quite handy, but... this will not protect You from someone exploiting Your code and reading any file he wants to. If Your site (even one small script) will be exploited, nor .htaccess nor other solutions will protect You, as the hacker will gain rights of the user executing the script (usually www-data).
This is especially painfull in CGI, but other scripts are being hacked as well.
As far as I know, htaccess is easy to hack if it's intercepted (f.e. You are logging in from a internet cafe with a network sniffer running). As far as I know, Digest authentication helps to get over that problem.
The authentication will be passed unprotected over the net if you use the standard http protocol. This is not regarded as safe, as someone could sniff the password.
If you restrict acces to https it is quite safe. This would mean to install and enable the apache module for ssl encrypted http traffic (port 433, https:// in the adress line of the browser) and disable the standard http traffic for this directory on port 80. Username and Password will be ssl encrypted. Be sure to select a good password (long and complex enough, not possible to guess or brute force).
Apache configuration can be tricky, so take a lot of care to keep it simple and test against possible mistakes.
It can be a good idea to move the access restriction configuration from the .htaccess file to the main apache configuration file if you have knowledge and control of it. Could also be easier for you to keep it in the .htacces file. And "easy" can be safer. Do it the way it feels simple and safe and easy to maintain and remember for you.
This is a simple setup to enhance security and protect against accidents:
If you have php and email configured on the machine where the protected directory is you can write a simple alarm script. Just a php file "alarm.php" with a single line with the php mail function that sends you an e-mail, telling you that htaccess protection does not work.
If your domain and directory path is "http://mybox.example.com/secretdir/alarm.php" you can enter this in a browser on a different machine and you should get that mail as long as htaccess is "opened". If it is protected you can enter the username and password and you will also get the mail.
To make an automated alarm out of this you could use a different unix box that tries to get this url every 15 minutes or so. The line for the crontab:
*/15 * * * * user1 wget http://mybox.example.com/secretdir/alarm.php
user1 is a user on this machine who is allowed to run wget, and wget must be installed.
You can disable the htaccess protection as a test and should get the mail every 15 minutes.
From my experience it is a common security flaw that a directory that you think is protected looses its protection when you change something and you are not aware, this way you get the email that warns you.