Multi-Threaded Application - Help with some pseudo code!!

I am working on a multi-threaded application and need help with some pseudo-code. To make it simpler for implementation I will try to explain that in simple terms / test case.

Here is the scenario -

I have an array list of strings (say 100 strings)

I have a Reader Class that reads the strings and passes them to a Writer Class that prints the strings to the console. Right now this runs in a Single Thread Model.

I wanted to make this multi-threaded but with the following features -

Ability to set MAX_READERS

Ability to set MAX_WRITERS

Ability to set BATCH_SIZE

So basically the code should instantiate those many Readers and Writers and do the work in parallel.

Any pseudo code will really be helpful to keep me going!

Aside from using the producer-consumer pattern that has been suggested, I would recommend that you use the CopyOnWriteArrayList so you can have lock-free read/write/iteration of your list. Since you're only working with a couple of hundred strings you will probably not have any performance issues with the CopyOnWriteArrayList.

If you're concerned about performance then I actually think it might be better if you use the BlockingQueue or a ConcurrentHashMap. They will allow you to maximize throughput with your multithreaded application.

The recommended option:
A BlockingQueue works very well with multiple producers and consumers, but of course it implies an order of data processing (FIFO). If you're OK with FIFO ordering, then you will probably find that the BlockingQueue is a faster and more robust option.

I think that the Wikipedia article has sufficient pseudo code for you to use, but you can also check out some of the following SO questions:
http://stackoverflow.com/search?q=java+producer+consumer

Java Producer-Consumer Designs:
http://stackoverflow.com/questions/2332537/producer-consumer-threads-using-a-queue
http://stackoverflow.com/questions/669294/design-of-a-producer-consumer-app

@Lirik: Personally I find this very confusing. First, you normally don't want to use a CopyOnWriteArrayList in a typical producer/consumer model (if you produce a lot, it's not a good choice). Also I think it is unclear why a ConcurrentHashMap could be useful in this case. I mean, yes it performs good for a map, but where do you need a map?

Enno Shioji 2010-04-14 22:34:47

@Zwei, the only reason I suggested the CopyOnWriteArrayList is because the OP is using an array and the OP stated that there will be around 100 strings in the array. Obviously BlockingQueue would be the better option for the Producer/Consumer design, so perhaps I should highlight that portion of my answer.There are many situations where a map is useful, here is one example: your producer is frequently updating a fixed number data fields and you only care for the latest updates (i.e. high frequency market data)... a queue will probably be a bad choice in that case.

Lirik 2010-04-14 23:58:12

@ZWei - I stated in my post that I am using that particular example to make things understand better. The data that needs to be copied need not be strings...in general its actually database...so basically Readers would read sets of data from database and writers will persist them in a different database..

HonorGod 2010-04-15 11:55:37

@Lirik, @HonorGod, I see. I still think though that List is generally a poor choice here.. I see that it is because the OP suggested, but I'd go as far as to say that it would be wrong to use a CopyOnWriteArrayList here. Especially because CopyOnWriteArrayList should only be used if you write rarely to the List. Since multiple writers are fetching data from the source, you normally want to de-queue things (for which Queue fits very well). This would be extremely cumbersome with List, and very inefficient with a CopyOnWriteArrayList.. Every time you remove/add, the entire List must be copied..

Enno Shioji 2010-04-15 15:28:33

ansaurus

tags:

views:

answers:

Multi-Threaded Application - Help with some pseudo code!!

related questions