How do applications collect statistics?

Question

I need to collect statistics from my server application written in python. I am looking for some general guidance on how to setup models and exactly how to store the statistics information. I was thinking of storing and organizing all this information in a database, but my implementation is turning out to be too specific.

I need to collect stats like active users, requests processed and things like that over time.

Are there any guides or techniques out there to create some more generic statistics storage systems?

Answer 1

One approach is to do this by stages: Store activity logs, including requests and users, as text files. Later, mine the logs into data points (python should be able to do this easily). You may want to use the logging library for python for the logging stage. In general, start with high time-resolution logging, which you can later aggregate into hourly, daily, weekly summaries etc.

Answer 2

Like most software solutions there is no single solution that I can recommend that will solve your problem. But I have created a few similar programs and here's some things that I found that worked well.

• Create an asynchronous logging service so the logging doesn't adversely affect your code's performance. (You need to be mindful of where you are storing your data, where it is processed, etc. because you can still significantly degrade performance if you're not careful.) I have found that creating a web service is often convenient.
• Try and save as much information about the request as possible. In the future this will make it easier to add new queries and reports.
  • Normalize your data
  • Always include the time the action was performed. If you can capture run time that it typically useful too.