Best place for log files in an in-house IT environment

Question

All of my users are a short walk down the hall, and all of my programs run on workstations on the same LAN. Some years ago, I had the staff write the log files for all of their programs to a shared folder hierarchy, naming each log file after the machine name in a sub-directory named after the app.

But this arrangement wasn't that great, since if the file server went down then none of the programs anywhere could write logs. Yet keeping logs local to each workstation would make it a pain in the ass to read them whenever we had to debug a problem.

We tried making a DNS alias for the logging fileserver so we could point it to a backup machine when necessary, but DNS aliases don't work with Windows file shares.

Putting the path to the shared log folder in each program isn't great--even if it's field configurable--because we have dozens of programs on dozens of machines.

We've also looked into using Microsoft's distributed file system, but the price is ridiculous.

I'd like a way to gather the logging for many programs on the local network into one place so I can tail and analyze them without paying a visit to the remote machine. We use .Net for all our programs.

Edit: I'd like to avoid setting up a file-share on each user's workstation, or solutions that trawl for logs each night, since I want to be able to read fresh logs on demand, or moments after a problem is reported.

Answer 1

In an IT environment which I worked in a few years back, we had each machine write it's log files locally and wipe them every five days. The server would log in every night to grab the latest logs from each machine. If the server went down, it would just grab two days worth of logs from everyone. If a client went down, it's log could be grabbed the next day as well.

Answer 2

Can you have the log files writtin to peoples local machine and then have a script that pulls them to a common file server as a nightly batch job?

Answer 3

What about having your application's logger as a subscriber service (using remoting). Then implement LogServers which subscribe to the application. When the app has something to log, it sends it to the subscribers. If there is a problem, such as the subscriber not responding/dead and you have no working subscribers, write locally but sound an alarm. E.g. a multi-machine version of the TraceListener and Debug classes.

You'd almost certainly want the application to send log messages from a different thread to the rest of your application.

Answer 4

You can use MSMQ. Write your logs to a MSMQ queue, and then have a service that picks these logs up and puts them in a database or out to a file in a central location if you want. It wouldn't be instantaneous, but you could tell it to run whenever you want to get new log entries. Plus it would be reliable since it uses MSMQ.

Answer 5

We wrote a single application that hosts other applications (plug-in architecture). When an exception occurs, the necessary information is written to an XML file on the user's machine (which is named <USERNAME>.xml). Either upon our request or when the application exits, it sends the XML to a web service where it writes the files to a location where we can go and check them.

If the web service is down, it's no big deal since the XML file is still on the user's machine and will get uploaded the next time they run the application.

Answer 6

DFS is included with Windows Server, and you can regedit to enable NetBIOS with a CNAME.

I've also tried Splunk for pulling and analyzing log files, but it's a bit of a CPU hog and I couldn't spare the horsepower. Otherwise, though, it was a pretty nice product.

Answer 7

Our shop has had good experience with storing logs in a database table. You can schedule cleanup as a database job whenever you feel like it, to prevent it from getting too large.

Answer 8

write to a database also (like CALM) so you only have to troll the log files when the database was down

Answer 9

We use the approach of logging to a known location locally which is shared. Then it's easy to have a second process pull these logs for later processing (dump to database, collect and archive etc). If the collection process dies, then nothing is really is affected.

Make sure you don't have a central point of failure. I've seen examples of this that used a central logging database that all apps depended on and when it went down everything else did too. Not clever.

Answer 10

I agree with craigb's comment. Creating additional single points of failure should be avoided in a LAN environment.

My preference is to have the clients log locally, and then copy these log files to a central location. In this case if your log server falls over then there won't be any serious outage. Additionally, sometimes writing a log file to a network can be performance bottleneck.

Another issue with the central log server is if one application misbehaves, it can take down all the apps. I've seen countless bugs, which when encountered cause the program to enter an infinite loop and spew logs messages until the device fills up.

If you really, want the central logging, for compliance reasons or something else, I would either write to a network drive, or write to the socket in the fashion of syslogd. Since syslogd is quite standard, you shouldn't have any trouble setting one up on a server, and there's plenty of example code and libraries to write to it. One can even configure a hardware load balance to sit in front of two or more syslogd servers, providing higher availability of the service.