What is the best way to store usage reports over time?

Question

Hello,

I currently have a few server reports that return usage statistics whenever run. The data is collected from several different sources (mostly log files), so they're not in a database to begin with.

The returned data are simple lists, for example, detailing how much disk space a user is using (user => space) average percent memory they've used for the month (user => memory), avg CPU time, etc.

Some of the information is a running total (like disk usage) and others are averages of snapshots taken throughout the month.

Running these reports and looking at the results works perfect, but I'd like to start storing these results to look at long-term trends.

What would be the best way to do this?

Thanks

Answer 1

RRDtool seems the obvious solution for this.

Or for that matter, one of the out-of-the box monitoring tools, some of which happen to use rrdtool for storing their data. E.g. Munin.

Answer 2

CACTI is very useful and highly configurable. Utilizes RRD Tool.

RRD Tool is great, b/c it stores data in a circular format and summarizes it. When RRD creates a data file, it creates it with every data point that it will ever store, so it never gets larger. You don't have to worry about log files getting too large. The key is to configure it to summarize in time periods, e.g., daily, monthly, yearly. The downside is that next year, you might not be able to know the CPU usage for the five minute period from January 1 of this year. But who really needs that?