How to fetch multiple feeds concurrently

Question

Hello,

I am new to ruby on rails and I have just started watching rails casts tutorials.

To parse feeds, I have started using feed zirra.

To fetch multiple feeds at once, feedzirra has this feature

feed_urls = ["http://feeds.feedburner.com/PaulDixExplainsNothing",
"http://feeds.feedburner.com/trottercashion"]
feeds = Feedzirra::Feed.fetch_and_parse(feed_urls)

If I have 100 feeds, this procedure takes some time to index the 100th feed so,

How to parse fetch all these lets say 100 feeds concurrently?

Looking forward for your help and support

Answer 1

You can try to combine Feedzirra for feed parsing and Typhoeus for concurency, something like:

#!/usr/bin/env ruby
require 'rubygems'
require 'typhoeus'
require 'feedzirra'

feed_urls = ["http://feeds.feedburner.com/PaulDixExplainsNothing", "http://feeds.feedburner.com/trottercashion"]

hydra = Typhoeus::Hydra.new
feeds = {}

feed_urls.each do |feed|
        r = Typhoeus::Request.new(feed)
        r.on_complete do |response|
                feeds[r.url] = Feedzirra::Feed.parse(response.body)
        end
        hydra.queue r
end
hydra.run
puts feeds.inspect

The idea is to use hydra to be concurent. Up to you then to do something else than inspect or populating the feeds variable.