Concurrent web requests with Ruby (Sinatra?)?

Question

I have a Sinatra app that basically takes some input values and then finds data matching those values from external services like Flickr, Twitter, etc.

For example:

input:"Chattanooga Choo Choo" Would go out and find images at Flickr on the Chattanooga Choo Choo and tweets from Twitter, etc.

Right now I have something like:

@images = Flickr::...find...images..

@tweets = Twitter::...find...tweets...

@results << @images

@results << @tweets

So my question is, is there an efficient way in Ruby to run those requests concurrently? Instead of waiting for the images to finish before the tweets finish.

Thanks.

Answer 1

Yes why not threads?

As i understood. As soon as the user submit a form, you want to process all request in parallel right? You can have one multithread controller (Ruby threads support works really well.) where you receive one request, then you execute in parallel the external queries services and then you answer back in one response or in the client side you send one ajax post for each service and process it (maybe each external service has your own controller/actions?)

Answer 2

Threads would work, but it's a crude tool. You could try something like this:

flickr_thread = Thread.start do
  @flickr_result = ... # make the Flickr request
end

twitter_thread = Thread.start do
  @twitter_result = ... # make the Twitter request
end

# this makes the main thread wait for the other two threads
# before continuing with its execution
flickr_thread.join
twitter_thread.join

# now both @flickr_result and @twitter_result have
# their values (unless an error occurred)

You'd have to tinker a bit with the code though, and add proper error detection. I can't remember right now if instance variables work when declared inside the thread block, local variables wouldn't unless they were explicitly declared outside.

I wouldn't call this an elegant solution, but I think it works, and it's not too complex. In this case there is luckily no need for locking or synchronizations apart from the joins, so the code reads quite well.

Perhaps a tool like EventMachine (in particular the em-http-request subproject) might help you, if you do a lot of things like this. It could probably make it easier to code at a higher level. Threads are hard to get right.

Answer 3

You might consider making a client side change to use asynchronous Ajax requests to get each type (image, twitter) independently. The problem with server threads (one of them anyway) is that if one service hangs, the entire request hangs waiting for that thread to finish. With Ajax, you can load an images section, a twitter section, etc, and if one hangs the others will still show their results; eventually you can timeout the requests and show a fail whale or something in that section only.

Answer 4

Consider using YQL for this. It supports subqueries, so that you can pull everything you need with a single (client-side, even) call that just spits out JSON of what you need to render. There are tons of tutorials out there already: http://j.mp/aprsHT

Answer 5

http://github.com/pauldix/typhoeus

parallel/concurrent http requests