How to stop unintentional human errors?

Question

We maintain a huge set of files in our web servers. Yesterday we were surprised to see a very important core file of the system being reverted by an older version (probably by some human beings since we do not have any automated scripts). Users partially got affected for half a day for this.

Shall I run a sort of automated script to scan the last uploaded date (possible?) of each file and send us an automated alert if there is a mismatch of the core files? Or are there any s/w available to take fingerprints of each file and send alerts? Please suggest what else can be done to stop such sort of human errors.

Answer 1

Humans will always make mistakes unfortunately.

Sounds like you need to restrict access to the web server and ensure changes to go through a defined process. If "anyone" (using the term loosely) can go on to a production server and just change files, then you're increasing your exposure to this kind of problem.

In fact, this kind of issue can highlight a problem further up the production line. For example, usually if people need to make direct changes to files on a production server it's to patch an urgent bug. Setting up a continuous integration environment, with continual automated testing of code changes and running a beta version alongside production can greatly reduce these types of bugs making out live, therefore reducing (ideally completely) the need to ever need to patch live files.

Answer 2

Treat it as if it were an intrusion detection problem -take a look at tripwire for example.