My web site need to read a slow web site, how to improve the performance

Question

I'm writing a web site with rails, which can let visitors inputing some domains and check if they had been regiestered.

When user clicked "Submit" button, my web site will try to post some data to another web site, and read the result back. But that website is slow for me, each request need 2 or 3 seconds. So I'm worried about the performance.

For example, if my web server allows 100 processes at most, that there are only 30 or 40 users can visit my website at the same time. This is not acceptable, is there any way to improve the performance?

PS: At first, I want to use ajax reading that web site, but because of the "cross-domain" problem, it doesn't work. So I have to use this "ajax proxy" solution.

Answer 1

Rather than posting some data to the websites, you could use an HTTP HEAD request, which (I believe) should return only the header information for that URL.

I found this code by googling around a bit:

require "net/http"
req = Net::HTTP.new('google.com', 80)
p req.request_head('/')

This will probably be faster than a POST request, and you won't have to wait to receive the entire contents of that resource. You should be able to determine whether the site is in use based on the response code.

Answer 2

It's a bit more work, but you can use something like DelayedJob to process the requests to the other site in the background.

DelayedJob creates separate worker processes that look at a jobs table for stuff to do. When the user clicks submit, such a job is created, and starts running in one of those workers. This off-loads your Rails workers, and keeps your website snappy.

However, you will have to create some sort of polling mechanism in the browser while the job is running. Perhaps using a refresh or some simple AJAX. That way, the visitor could see a message such as “One moment, please...”, and after a while, the actual results.

Answer 3

Try using typhoeus rather than AJAX to get the body. You can POST the domain names for that site to check using typhoeus and can parse the response fetched. Its extremely fast compared to other solutions. A snippet that i ripped from the wiki page from the github repo http://github.com/pauldix/typhoeus shows that you can run requests in parallel (Which is probably what you want considering that it takes 1 to 2 seconds for an ajax request!!) :

hydra = Typhoeus::Hydra.new

first_request = Typhoeus::Request.new("http://localhost:3000/posts/1.json")
first_request.on_complete do |response|
  post = JSON.parse(response.body)
  third_request = Typhoeus::Request.new(post.links.first) # get the first url in the post
  third_request.on_complete do |response|
    # do something with that
  end
  hydra.queue third_request
  return post
end
second_request = Typhoeus::Request.new("http://localhost:3000/users/1.json")
second_request.on_complete do |response|
  JSON.parse(response.body)
end
hydra.queue first_request
hydra.queue second_request
hydra.run # this is a blocking call that returns once all requests are complete

first_request.handled_response # the value returned from the on_complete block
second_request.handled_response # the value returned from the on_complete block (parsed JSON)

Also Typhoeus + delayed_job = AWESOME!