Persistence based on priority??

Question

I have a monitoring application which via gui displays that status of a number of application servers. It works by sending a status request to the app server. The app server in turns queries the status of all it components, builds up a status response message and sends that message to the monitoring app. The monitoring app writes information in the status message to the database, which in turn is displayed on the GUI. The entire monitoring app is done in JavaSE.

My problem is this...the database struggles to keep up due to the sheer amount of data it needs to persist.

My solution that i thought up is to assign priorities to components of the status message and based on the priority i will persist it to the DB.

Is there possibly a better solution to this problem?

Answer 1

I'm wondering that if your database can't keep up with storing the data that it might also be a heavy load on the application servers to keep responding to the status requests.

I don't know which information is in the status messages and what the requirements are. Some of the options are:

• The status messages might be (almost) identical to the previous status message. In that case you can only store new messages or only the differences.
• Keep more (recent) messages in memory and only persist every minute/10 seconds etc.
• Make persisting in the database more efficient. Maybe you can reduce indexes or constraints. You can also try batching a large amount of status reports (and maybe keep these in memory for the UI until batch is persisted).
• Reduce the number of status report messages. Maybe some components can be messaged less frequently.