Should I log my website's 404 errors?

Question

I have an ASP.NET website, but this question isn't really about technology, it is rather about practice. Should we log our 404 errors?

My reasoning:

1. This is a potential vulnerable point because a simple unfriendly user may fill up your hard drive in no time just by requesting wrong URLs!
2. Some browsers often request resources up front - like for example favicon.ico, even if its not there. This is really annoying.

But really I would like to know about a broken link if there exists one in my websites. Should I depend on the URL referrer? The problem with the URL referrer is that I cannot distinguish my internal redirect which may be broken with an unfriendly one from outside.

What does the practice suggest?

Answer 1

I would recommend logging unique URLs (one row per url, with a count field specifying how many times it was requested), and deleting the least-frequently requested ones if the log gets too large.

Answer 2

Yes Log all HTTP Access/Errors.

Then when you encounter 404's that are errors in your design, fix them!

As for the favicon 404 (IE), be sure to put a tiny (cachable) image there for IE to download vs. causing a 404 entry.

As for a naughty user that decides to hammer your site with invalid URL's... use the log as a tool to find them, and ban them. ;-)

Answer 3

This is a potential vulnerable point because a simple unfriendly user may fill up your hard drive in no time just by requesting wrong URLs!

Make sure your hard disk has a reasonable amount of free space, and rotate your logs.

Some browsers often request resources up front - like for example favicon.ico, even if its not there. This is really annoying.

There aren't many such resources (favicon.ico and robots.txt being the primary ones). Just provide them.

Error logs are useful, they clue you in to broken links which you might be able to do something about.