Implementing a file based queue

Question

Hi folks I have an in memory bounded queue in which multiple threads queue objects. Normally the queue should be emptied by a single reader thread that processes the items in the queue.

However, there is a possibility that the queue is filled up. In such a case I would like to persist any additional items on the disk that would be processed by another background reader thread that scans a directory for such files and processes the entries within the files. I am familiar with Active MQ but prefer a more light weight solution. It is ok if the "FIFO" is not strictly followed (since the persisted entries may be processed out of order).

Are there any open source solutions out there? I did not find any but thought I would ping this list for suggestions before I embark on the implementation myself.

Thank you!

Answer 1

EHCache can overflow to disk. It's also highly concurrent, though you dont really need that

Answer 2

You could use something like SQLLite to store the objects in.

Answer 3

Why is the queue bounded? Why not use a dynamically expandable data structure? That seems much simpler than involving the disk.

Edit: It's hard to answer your question with out more context.

Can you clarify what you mean by "run out of memory"? How big is the queue? How much memory do you have?

Are you on an embedded system with very little memory? Or do you have 2 GB or more of stuff in the queue?

If either is true, you really aught to use a "swappable" data structure like a BTree. Implementing one your self for one queue seems like overkill. I would just use an embedded database like SQL lite.

If neither of those us true, then just use a vector or a linked list.

Edit 2: You probably don't need a BTree or a database. You could just use a linked list of pages. But again, I have to ask: is this necessary?

Or, if you are willing to process things non serially, why not have multiple reader threads all the time?

Ultimately though I don't think your proposal is the way to go.