Process n items at a time (using threads)

Question

I'm doing what a lot of people probably need to do, processing tasks that have a variable execution time. I have the following proof of concept code:

threads = []

(1...10000).each do |n|
  threads << Thread.new do
    run_for = rand(10)
    puts "Starting thread #{n}(#{run_for})"
    time=Time.new
    while 1 do
      if Time.new - time >= run_for then
          break
      else
          sleep 1
      end
    end
    puts "Ending thread #{n}(#{run_for})"
  end
  finished_threads = []
  while threads.size >= 10 do
    threads.each do |t|
      finished_threads << t unless t.alive?
    end
    finished_threads.each do |t|
      threads.delete(t)
    end
  end
end

It doesn't start a new thread until one of the previous threads has dropped off. Does anyone know a better, more elegant way of doing this?

Answer 1

I'd suggest creating a work pool. See http://snippets.dzone.com/posts/show/3276. Then submit all of your variable length work to the pool, and call join to wait for all the threads to complete.

Answer 2

EDITED: Try using the work_queue gem.