Keeping the UI responsive while parsing a very large logfile

Question

I'm writing an app that parses a very large logfile, so that the user can see the contents in a treeview format. I've used a BackGroundWorker to read the file, and as it parses each message, I use a BeginInvoke to get the GUI thread to add a node to my treeview. Unfortunately, there's two issues:

• The treeview is unresponsive to clicks or scrolls while the file is being parsed. I would like users to be able to examine (ie expand) nodes while the file is parsing, so that they don't have to wait for the whole file to finish parsing.
• The treeview flickers each time a new node is added.

Here's the code inside the form:

private void btnChangeDir_Click(object sender, EventArgs e)
{
    OpenFileDialog browser = new OpenFileDialog();

    if (browser.ShowDialog() == DialogResult.OK)
    {
        tbSearchDir.Text = browser.FileName;
        BackgroundWorker bgw = new BackgroundWorker();
        bgw.DoWork += (ob, evArgs) => ParseFile(tbSearchDir.Text);
        bgw.RunWorkerAsync();
    }
}

private void ParseFile(string inputfile)
{
    FileStream logFileStream = new FileStream(inputfile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
    StreamReader LogsFile = new StreamReader(logFileStream);

    while (!LogsFile.EndOfStream)
    {
        string Msgtxt = LogsFile.ReadLine();
        Message msg = new Message(Msgtxt.Substring(26)); //Reads the text into a class with appropriate members
        AddTreeViewNode(msg);
    }
}

private void AddTreeViewNode(Message msg)
{
    TreeNode newNode = new TreeNode(msg.SeqNum);

    BeginInvoke(new Action(() =>
                               {
                                   treeView1.BeginUpdate();
                                   treeView1.Nodes.Add(newNode);
                                   treeView1.EndUpdate();
                                   Refresh();
                               }
                    )); 
}

What needs to be changed?

EDIT

New code, to replace the last function above:
        List<TreeNode> nodeQueue = new List<TreeNode>(1000);

        private void AddTreeViewNode(Message msg)
        {
            TreeNode newNode = new TreeNode(msg.SeqNum);

            nodeQueue.Add(newNode);

            if (nodeQueue.Count == 1000)
            {
                var buffer = nodeQueue.ToArray();
                nodeQueue.Clear();
                BeginInvoke(new Action(() =>
                                           {
                                               treeView1.BeginUpdate();
                                               treeView1.Nodes.AddRange(buffer);
                                               treeView1.EndUpdate();
                                               Refresh();
                                               Application.DoEvents();
                                           }
                                ));
            }
        }

Not sure why I left the Refresh and the DoEvents in there. A bit of testing other comments...

Answer 1

did you tryed Application.Doevents() method?

Answer 2

Invoke/BeginInvoke uses PostMessage internally to marshal a request from an arbitrary thread to the UI thread. BeginInvoke will be flooding your windows message queue with messages which need to be processed by the UI thread, that accompanied by the fact that you are disabling the tree redraw with every node you add is probably impacting your ability to interact with the tree while it is loading.

One option would be to batch a number of updates together and then send them update the tree in batches. So parse the file and update the tree with every 100 or some number of nodes rather than 1 at a time.

Update: After your edit to add nodes in batches I would suggest the following.

1- Rather use Invoke than BeginInvoke, otherwise the queue fills up while the tree is being updated and then once the tree is updated the next thousand nodes are ready to be inserted which puts you right back where you where.

2- Sleep a few 100 milliseconds after inserting each batch so that there is a period that UI can respond. You can play with this, this will balance performance vs. user experience. Longer sleep will feel more responsive for the user, but will ultimately take longer to get all the data loaded.

3- Note that your current batching solution will miss the last few nodes if the total count is not a multiple of 1000

    private void AddTreeViewNode(Message msg) 
    { 
        TreeNode newNode = new TreeNode(msg.SeqNum); 

        nodeQueue.Add(newNode); 

        if (nodeQueue.Count == 1000) 
        { 
            var buffer = nodeQueue.ToArray(); 
            nodeQueue.Clear(); 
            Invoke(new Action(() => 
                { 
                    treeView1.BeginUpdate(); 
                    treeView1.Nodes.AddRange(buffer); 
                    treeView1.EndUpdate(); 
                }));
            System.Threading.Thread.Sleep(500); 
        } 
    }

Answer 3

I didn't run your code with profiler but if you consider slow I/O as botleneck you may try MemoryMappedFile from .NET 4.0.

So instead of seeking and reading line by line from disk you will have an access to the same data in memory instead of disk IO.

Answer 4

First thing you'll want to do is enable double-buffering on the TreeView so it will stop flickering. That's supported since Vista but unfortunately not by Windows Forms. Add a new class to your project and paste the code shown below. Compile. Drop the control from the top of the toolbox onto your form.

using System;
using System.Windows.Forms;
using System.Runtime.InteropServices;

class BufferedTreeView : TreeView {
    protected override void OnHandleCreated(EventArgs e) {
        base.OnHandleCreated(e);
        IntPtr style = (IntPtr)TVS_EX_DOUBLEBUFFER;
        SendMessage(this.Handle, TVM_SETEXTENDEDSTYLE, (IntPtr)style, (IntPtr)style);
    }
    // P/Invoke:
    private const int TVS_EX_DOUBLEBUFFER = 0x004;
    private const int TVM_SETEXTENDEDSTYLE = 0x1100 + 44;
    [DllImport("user32.dll")]
    private static extern IntPtr SendMessage(IntPtr hWnd, int msg, IntPtr wp, IntPtr lp);
}

Keeping your UI responsive requires a redesign of your ParseFile() method. As written, it calls BeginInvoke() much too frequently. That floods the UI thread with requests it cannot keep up with. It doesn't get around its normal duties anymore, like painting and responding to mouse clicks.

It is wasted effort, the human eye cannot perceive updates that happen at a rate faster than 25 times per second. Store the data in a collection, BeginInvoke and pass that collection at a much slower rate.