Best way to read in a rolling log file in .NET?

Question

Hi folks,

i need to read in a rolling log file in .NET 3.5sp1. I'm wondering what's the best way to do this? The file can get really big. An idea i had was to

• Open File
• Read file to last read line (in the first case, it was 0)
• Read all remaining lines
• Close stream and remember the last line number read.
• Wait a bit.
• Rinse/repeat.

i'm not sure if this the best way though and efficient for the memory, considering the file can be pretty huge. And i can't have the log file being locked either :(

thoughts?

Answer 1

Here's an example of using a FileSystem Watcher to open the file and read the new data as it is written:

http://www.codeproject.com/KB/cs/wintail.aspx

The example looks good, but do not for the love of Pete and all things fuzzy use lock(this). It is bad. Use a new object instead that you can share amongst all of the objects in the class.

If you open the file ReadOnly you shouldn't have a sharing violation depending on how your file has been opened by the program appending to the log file.

Answer 2

Get the file length and read the last piece of the file. It won't be hard to find the first line break and show everything from there on. FileStream has a Length property, and the read method allows you to read from one place to another.

Steven is right. If the other program doesn't require exclusive access, you can open read-only and be fine.

Answer 3

Can you just append to the file, rather than keeping track of where you are?

    Object mLogLock = new Object(); //to make logging thread safe
    string mLogFile = ""; //set your log location
    string mLogDirectory = "";
    public void HandleMessage(string inMessage)
    {

        lock (mLogLock)
        {
            if (!System.IO.Directory.Exists(mLogDirectory ))
            {
                System.IO.Directory.CreateDirectory(mLogDirectory );
            }

            String theMessage = DateTime.Now.ToString("s");
            if (inMessage != null)
                theMessage += " : " + inMessage;

            StreamWriter sw = new StreamWriter(mLogFile, true);

            sw.WriteLine(theMessage );

            sw.Dispose();
            sw.Close();
        }
    }

Otherwise, if you have a writer writing at one point and something else consuming, and you have no locking, then your program will corrupt your logs. That's just not even a question. You can try to lock your file by having the file opened for exclusive read/write by whatever wants to read or write to it, but then the other code will throw exceptions when it can't read it.

If the log file is in the same program, you can just keep all of the information in one large stream, like a MemoryStream, and use the StreamWriter as above instead (or as well as) to the MemoryStream. Then, your readers and writers can lock on the MemoryStream, rather than on the file itself. Would that help?