views:

41

answers:

1

Concerning ActiveMQ: I have a scenario where I have one producer which sends small (around 10KB) files to the consumers. Although the files are small, the consumers need around 10 seconds to analyze them and return the result to the producer. I've researched a lot, but I still cannot find answers to the following questions:

  1. How do I make the broker store the files (completely) in a queue?
  2. Should I use ObjectMessage (because the files are small) or blob messages?
  3. Because the consumers are slow processing, should I lower their prefetchLimit or use a round-robin dispatch policy? Which one is better?
  4. And finally, in the ActiveMQ FAQ, I read this - "If a consumer receives a message and does not acknowledge it before closing then the message will be redelivered to another consumer.". So my question here is, does ActiveMQ guarantee that only 1 consumer will process the message (and therefore there will be only 1 answer to the producer), or not? When does the consumer acknowledge a message (in the default, automatic acknowledge settings) - when receiving the message and storing it in a session, or when the onMessage handler finishes? And also, because the consumers are so slow in processing, should I change some "timeout limit" so the broker knows how much to wait before giving the work to another consumer (this is kind of related to my previous questions)?
A: 

Not sure about others, but here are some thoughts.

First: I am not sure what your exact concern is. ActiveMQ does store messages in a data store; all data need NOT reside in memory in any single place (either broker or client). So you should actually be good in that regard; earlier versions did require that all ids needed to fit in memory (not sure if that was resolved), but even that memory usage would be low enough unless you had tens of millions of in-queue messages.

As to ObjectMessage vs blob; raw byte array (blob) should be most compact representation, but since all of these get serialized for storage, it only affects memory usage on client. Pre-fetch mostly helps with access latency; but given that they are slow to process, you probably don't need any prefetching; so yes, either set it to 1 or 2 or disable altogether.

As to guarantees: best that distributed message queues can guarantee is either at-least-once (with possible duplicates), or at-most-once (no duplicates, can lose messages). It is usually better to take at-least-once, and make clients to de-duping using client-provided ids. How acknowledgement is sent is defiend by JMS specification so you can read more about JMS; this is not ActiveMQ specific. And yes, you should set timeout high enough that worker typically can finish up work, including all network latencies. This can slow down re-transmit of dropped messages (if worked dies), but it is probably not a problem for you.

StaxMan
Thank you very much for the quick reply! Actually, some of the things I read about JMS (example - http://en.wikipedia.org/wiki/Java_Message_Service), are not mentioned or are contradictory to what I've read about ActiveMQ. For example, in that article, it says that messages are stored in the queue (so the producer doesn't have to be active at the time the consumer consumes the message), which I wasn't able to find on the activeMQ web site. Also, it says that in the queue model (the one I talk about) - only one consumer gets the message (which as you said, isn't guaranteed by ActiveMQ).
Petar
Yeah docs for ActiveMQ may not be all that good. But it certainly does store messages, as that is the whole point of Message Queues (asynchronous operation). There are modes that do not have full persistence, and just keep them in broker's memory queue, but that is something you have to configure.
StaxMan