ruby working on array elements in groups of four

Question

I have a ruby script array when each element needs processing :

threads = []
elemets.each do  |element|
    threads.push(Thread.new{process(element)}}
end
threads.each { |aThread|  aThread.join }

how ever due to resource limitations, the script works in an optimal way if no more the four elements are processed at a time.

no I know I can dump the each loop and use a variable to count 4 elements and then wait but is there a cooler ruby way to do it ?

Answer 1

Not sure if the following variant counts as just using a "variable to count 4 elements", or could be considered cool, but it gives you an array in slices of size no greater than 4 elements:

x = (1..10).to_a
0.step(x.size - 1, 4) do |i|
    # Choose one
    p x.slice(i, 4)
    p x[i, 4]
end

Answer 2

Yes, but you need to do some method overriding. Usual approach is to override '/' for Array like so:

class Array
  def / len
    a = []
    each_with_index do |x,i|
      a << [] if i % len == 0
      a.last << x
    end
    a
  end
end 

And with that defined you can now easily do:

foo = [1,2,3,4,5,6]
foo / 2
# Result is [[1,2], [3,4], [5,6]]

Answer 3

You can enumerate in groups of 4 for an array:

>> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].each_slice(4) {|a| p a}
[1, 2, 3, 4]
[5, 6, 7, 8]
[9, 10, 11, 12]

So you can try something like

elements.each_slice(4) do | batch |
    batch.each do | element |
        threads.push(Thread.new{process(element)}}

    end
    (do stuff to check to see if the threads are done, otherwise wait )
end

Its may not be what you need, though - I have been up since 3 AM and I only had a couple of hours sleep. :/

Answer 4

If I read you right, you want to have no more than 4 threads processing at a time.

Sounds to me like you should launch only 4 threads, and have them all read from a shared Queue (part of the standard thread lib) to process the elements.

You can have the threads end when the queue is empty.

Slicing the array into 4 equal arrays, and having each thread process 1/4 of the elements assumes that each element processes in the same time. If some take longer than others, some of your threads will finish early.

Using a queue, no thread stops until the shared queue is empty, so it is I think a more efficient solution.

Here is a working program based on your code to demonstrate:

require 'thread'

elements = [1,2,3,4,5,6,7,8,9,10]

def process(element)
    puts "working on #{element}"
    sleep rand * 10
end

queue = Queue.new
elements.each{|e| queue << e }

threads = []
4.times do
    threads << Thread.new do
      while (e = queue.pop(true) rescue nil)
        process(e)
      end
    end
end

threads.each {|t| t.join }