Best method for saving values of a queue in case of program failure

Question

Say I have a multithreaded application made up of two separate threads and a Queue. Thread 1 finishes it's computation and puts the result in the Queue. Meanwhile thread 2 is constantly looping and checking if there is any data in the Queue for it to process.

How can I save the values in the queue to disk temporarily in case the for some reason the program faults or the computer is restarted? Would it be reasonable to save the value in a SQLite DB after the first thread is finished and then delete it after thread 2 is done?

If SQLite is in fact the best solution is it quicker to insert a flag on a row saying that it's been processed and have another thread come through every half hour and delete those rows that are flagged or just immediately delete that row?

Answer 1

I would not store the queue in memory at all - just have it stored on SQL Lite. If thread 1 is only adding to the queue and thread 2 is the only one reading, then just have thread 2 delete the row immediately after it has been processed.

Of course for performance reasons, the longer the thread 2 wait interval is before checking the queue, the better.

Answer 2

Don't roll your own here. Asynchronous processing routines, restartability, program faults, etc. are best managed with message queues.

You could use the built-in one to Windows, go open source with ActiveMQ from the Apache Software Foundation, or leverage web services in the cloud with Amazon's Simple Queue Service.

There are a lot of different ways you can go here. Investigate these options before repeating the functionality yourself. Reliability in these areas is more challenging than you might expect.

Answer 3

One solution would be to use a queue server, such as Zero-MQ, RabbitMQ or (commercial) Tibco EMS or Websphere MQ.