Should I use multiple threads in this situation? [Ruby]

Question

I'm opening multiple files and processing them, one line at a time. The files contain tokens separating the data, such that sometimes the processing of one file may have to wait for others to catch up to that same token.

I was doing this initially with only one thread and an array indicating with true/false if the file should be read in the current iteration or if it should wait for some of the others to catch up.

Would using threads make this simpler? More efficient? Does Ruby have a mechanism for this?

Answer 1

Firstly, Threads never make anything simpler. Threading is only applicable for helping to speed up applications. Threading introduces a host of new complications, it may seem handy to be able to describe multiple threads of execution but it always makes life harder.

Secondly, premature optimization is the root of all evil. Do not attempt to speed up the file processing unless you know that it is a bottleneck. Do the simplest thing that could possibly work (but no simpler).

Thirdly, threading might help if the process of reading the files was independent so that thread can process a file without worrying about what the other threads are doing. It sounds like this is not true in your case. Since the different threads would have to communicate with each other you are unlikely to see a speed benefit in applying threads.

Fourthly, I don't know Ruby and therefore can't comment on what mechanisms it has.

Answer 2

I'm not sure if using threads in ruby is beneficial. Recently I've written and tested an application which was supposed to do parallel computations, but I didn't get what I expected even on quad core processor, it performed computations sequentially, one thread after another. Read this article, it has discussion about threads scheduling, it may turn out that things haven't changed at least for original ruby.