Persistance Queue Implementation

Question

I was reading an article on Batch Processing in java over at JDJ http://java.sys-con.com/node/415321 . The article mentioned using a persistence queue as a Batch Updater instead of immediately sending an individual insert or update to the database. The author doesn't give a concrete example of this concept so I googled Persistence Queue but that didn't come up with much. Does anyone know of a good example of this?

Answer 1

http://en.wikipedia.org/wiki/Java_Message_Service scroll down to Provider implementations

Answer 2

This pattern is known by a few names (Ex. Backing Store, Write-Behind, etc...). This is often seen in Data Grid/Cache technologies as well as others. Generally, domain objects are stored in some kind of FIFO or priority queue and then a dispatcher dequeus them on an alternate thread or process and passes them to your real DAL. The queues can be in-memory or actually placed on a network queue such as ActiveMQ or MSMQ. Also, I've seen scenario's where there are seperate queues for each type of DB operation.

In many cases this is implemented as a facade in front of your DAL representing the same interface. In this way, other applications think they are communicating with the 'real' DAL and are abstracted from other related concerns. This decoupling allows you to change it if you need to. When object data is submitted to the facade the data is enqueued and then control is returned to the calling application.

Once queued you can just do batch updates/inserts or continue to process items one at a time. Keep in mind that the notion of a transaction takes on a very different meaning so you will need think that through.

Although there are technologies that do this... This is more of a pattern then a single technology. I don't have an example handy but you can look at data grid documentation from products like Oracle Coherence. Look for backing stores.

Answer 3

Take a look at this previous Stackoverflow question:

http://stackoverflow.com/questions/2332537/producer-consumer-threads-using-a-queue

The first answer talks of implementing a Producer-Consumer (which is what you want) using the ExecutorService. This is one way to do what you want, but it uses an in memory queue - you could quite easily change this to a JMS Queue though.