High-performance Screen Scrape with Ruby on Rails!

Question

I need to get many data and performance is an essential requirement. Do you have any suggestions?

Thanks in advance!

Answer 1

I've been using the mechanize gem (http://mechanize.rubyforge.org/mechanize/) with good results. Performance is always a problem because HTTP response times can vary, but there is nothing inherently slow about mechanize.

I was looking to gain optimal performance by adding concurrency where the order of retrieval was not important. It can take quite a bit of tweaking to get the receive buffers isolated so the buffer from one does not get corrupted by a response from another thread.

Good luck with this.