realtime logging

Question

I have an application which has a loop, part of a "Scheduler", which runs at all time and is the heart of the application. Pretty much like a game loop, just that my application is a WPF application and it's not a game. Naturally the application does logging at many points, but the Scheduler does some sensitive monitoring, and sometimes it's impossible just from the logs to tell what may have gotten wrong (and by wrong I don't mean exceptions) or the current status.

Because Scheduler's inner loop runs at short intervals, you can't do file I/O-based logging (or using the Event Viewer) in there. First, you need to watch it in real-time, and secondly the log file would grow in size very fast. So I was thinking of ways to show this data to the user in the realtime, some things I considered:

• Display the data in realtime in the UI
• Use AllocConsole/WriteConsole to display this information in a console
• Use a different console application which would display this information, communicate between the Scheduler and the console app using pipes or other IPC techniques
• Use Windows' Performance Monitor and somehow feed it with this information
• ETW

Displaying in the UI would have its issues. First it doesn't integrate with the UI I had in mind for my application, and I don't want to complicate the UI just for this. This diagnostics would only happen rarely. Secondly, there is going to be some non-trivial data protection, as the Scheduler has it's own thread.

A separate console window would work probably, but I'm still worried if it's not too much threshold. Allocating my own console, as this is a windows app, would probably be better than a different console application (3), as I don't need to worry about IPC communication, and non-blocking communication. However a user could close the console I allocated, and it would be problematic in that case. With a separate process you don't have to worry about it.

Assuming there is an API for Performance Monitor, it wouldn't be integrated too well with my app or apparent to the users. Using ETW also doesn't solve anything, just a random idea, I still need to display this information somehow.

What others think, would there be other ways I missed?

Answer 1

Back to basics - seperate concerns.

My usual solution would be use the Microsoft Enterprise Libraries to handle the actual logging; I'd use a database as the repository, you can then query it at will, from any application (your existing one or something completely stand-alone).

The thing I like about the MS Ent Libs is you can configure them to log to a wide variety of repository types. You could extend them with if need be; I'm not sure if you want to work asynchronously for performance / execution constraints.

I prefer logging to a DB as it gives a good level of control: it's easy to query and reasonably easy to mamage the data. Having sia dthat the Ent Libs do allow rolling file based logging - that would help you manage files sizes - but using a Db would be quicker than reading a file.

I guess it comes down to what you mean by "realtime" as to whether logging to a db would be fast enough. - real-time to a computer is very different from real-time to a person.

You could log to memory, and then asynchronously iterate through those logs entries and commit them to log term storage (DB). For reporting you could use the in-memory copy to show the 'current' state, and refer to the DB for longer periods of time / stuff in the more distant past.

Answer 2

If you'd like to use a third party solution, look at logFaces. It gives real-time monitoring provided that your application can use one of the apache log4xxx logging frameworks. It can work over UDP, so the remote calls are pretty cheap. Database is an option in case you will want to keep the history, can be a standalone viewer or a log server. Has a cool log viewer too.

Answer 3

Respectfully - both Adrian K and Dima's answers are not correct. The right answer is to use Event Tracing For Windows (ETW). This is what we use for all logging in Windows. Its extremely robust and very well performing. For example W7 logs an ETW event on many OS events - all the time - including processor context switch. Ever use the performance monitor in W7? It is consuming ETW events from the kernel.

I recommend you do all your logging with ETW. Why? Several reasons:

1. Its ubiquitous
2. You can enable disable logging in your running process. No process restarts required. (yes, other loggers do this, but some do not).
3. Its designed for including in shipping code.
4. Logging an event is guaranteed to be non-blocking: it will not cause a 'wait'.
5. We provide lots of tools for ETW trace processing. most notably the XPERF tools (link, link, link)

A big benefit of instrumenting your performance paths with ETW events is that your events can be seen integral with the kernel events using the XPERF tools.

Its also pretty easy to write a 'watch' application that watches ETW events from your components. I have one for one of our components that simply displays the events to the console.

I highly recommended to not try and write your own high performance logging system. This is challenging to do well, but in terms of performance and reliability. The Windows ETW system is super-robust and very well performing.