storing + evaluating performance data

Question

since we suffer from creeping degradation in our web application we decided to monitor our application performance and measure individual actions.

for example we will measure the duration of each request, the duration of individual actions like editing a customer or creating an appointment, searching for a contract. in most cases the database is the bottleneck for these actions.

i expect that the culminated data will be quite large, since we will gather 1-5 individual actions per request.

of course it would be nonsense to insert each an every element to the database, since this would slow down every request even more.

what is a good strategy for storing and evaluating those per-request data. i thought about having a global Queue object which is appended and a seperate thread that empties the queue and handles the persistent storage/file. but where to store such data? are there any prebuilt tools for such a visualisation?

we use java, spring, mixed hibernate+jdbc+pl/sql, oracle.

the question should be language-agnostic, though.

edit: the measurement will be taken in production over a large period of time.

Answer 1

It seems like your archive strategy will be at least partially dependent on the scope of your tests:

• How long do you intend to collect performance data?
• What are you trying to demonstrate? Performance improvements over time? Improvements associated with specific changes? (Like perf issues for a specific set of releases)

As for visualization tools, I've found excel to be pretty useful for small to moderate amounts of data.