views:

67

answers:

2

What should be the ideal number of parallel java threads for copying a large set of files from a quad core linux box to an external shared folder? I can see that with a single thread it is taking a hell lot of time to move the files one by one. Multiple threads is improving the copy performance, but I don't know what should be the exact number of threads.

I am using Java executor service to create the thread pool.

+4  A: 

Just try different variants and measure performance. There are also such bottlenecks as HDD and network speed, so I guess there is no definite answer.

Dmitry
+3  A: 

Absolutely you should work this out by testing. As Dmitry points out, there are many factors involved. In fact, CPU is almost certainly not the bottleneck with an IO- and network-based operation. You'll probably find things level off and start to get worse before you have too many threads, but if you want to minimise testing, use a graph of a few results to interpolate a good guess for the ideal value.

To get a better understanding of what's taking all the time, use metrics tools to measure utilisation of your resources - in this case that should include network interface traffic and disk queue length.

Steven Mackenzie
I'd *guess* (even though guessing is not a good idea here) that the resulting "ideal number" will be mostly independent from the number of CPUs/Cores available and depend a lot more on I/O- and Network-performance.
Joachim Sauer
I'd expect so too.
Steven Mackenzie