Asynchronous HTTP request

Question

require 'net/http'

urls = [
  {'link' => 'http://www.google.com/'},
  {'link' => 'http://www.yandex.ru/'},
  {'link' => 'http://www.baidu.com/'}
]

urls.each do |u|
  u['content'] = Net::HTTP.get( URI.parse(u['link']) )
end

print urls

This code works in synchronous style. First request, second, third. I would like to send all requests asynchronously and print urls after all of them is done.

What the best way to do it? Is Fiber suited for that?

Answer 1

This can be done with the C library cURL. A ruby binding for that library exists, but it doesn't seem to support this functionality out of the box. However, it looks like there is a patch adding/fixing it (example code is available on the page). I know this doesn't sound great, but it might be worth a try if there aren't any better suggestions.

Answer 2

It depends what you want to do after the function afterwards. You can do it with simple threads:

see: http://snipplr.com/view/3966/simple-example-of-threading-in-ruby/

Answer 3

You could have a different thread execute each one of the Net::HTTP.get. And just wait for all the threads to finish.

BTW printing urls will print both the link and the content.

Answer 4

Here's an example using threads.

require 'net/http'

urls = [
  {'link' => 'http://www.google.com/'},
  {'link' => 'http://www.yandex.ru/'},
  {'link' => 'http://www.baidu.com/'}
]

urls.each do |u|
  Thread.new do
    u['content'] = Net::HTTP.get( URI.parse(u['link']) )
    puts "Successfully requested #{u['link']}"

    if urls.all? {|u| u.has_key?("content") }
      puts "Fetched all urls!"
      exit
    end
  end
end

sleep