Which Java blocking queue is best for multiple producer and single or multiple consumers scenarios?

Question

Which Java blocking queue is best for multiple producer and single or multiple consumers scenarios ?

I am testing with LinkedBlockingQueue but i am getting OutOfMemoryError exception.

I am trying to achieve following things.

1. producer creates a object & put in a queue.
2. consumer grabs the data from queue & insert into DB. There would be 400 producers and i can adjust consumers as i wish.

let me know any idea. Thanks,

Update 10/07,

Thanks for your input gents,

Producer : It should listen to Server Socket. It reads the data from socket & construct the object (Domain objects) & put in a queue. Consumer : Take the object from queue & insert into DB (Hiberante & connection pooling supported) This my real environment.Process should be able to process at least 200 records/sec.I am testing the scalability of process and how to improve that. I hope, This will give better idea.

Helpful links : link vmoptions link Monitoring and Managing Java SE 6 Platform Applications link BlockingQueue

Thanks.

Answer 1

Is each producer a separate thread ? Don't forget that each thread will allocate (by default) 512k of memory for its stack (in your case requiring 200Mb of VM memory just for the threads). You can reduce this via -Xss.

Alternatively, just how big are the objects you're queuing ? I don't think you have a queue problem so much as some sort of scaling issue - e.g. producers producing faster than consumers can consume.

Answer 2

It seems that LinkedBlockingQueue is the best choice for your problem. I would suggest you to run your program with the switch -XX:+HeapDumpOnOutOfMemoryError and analyze the dump file to see what has caused the problem. Then you'll be able to solve it much easier.

Answer 3

The problem is not which Queue implementation you use but rather how to approach the problem of throttling your producers if your consumers cannot keep up. One possible solution is to create a LinkedBlockingQueue with a fixed capacity, and have your producers call offer(E e), which will return false if the queue is full.

Another possible solution is to tailor the number of producers and consumers accordiingly.