Prevent images from downloading with ScrAPI

Question

I need to scrape some websites, and would like to avoid downloading images from the pages I am scraping - I only need the text. I am hoping this will speed up the process. Any ideas on how to manage this?

Thanks, Jon

Answer 1

While scraping you do not download images but the reference IMG tag along with the entire body. You can always remove the IMG tag on the server side before storing into your database/rendering to the view. I would suggest you use nokogiri to parse the content received and remove all occurrences of the IMG tag.

This however does not speed up the process. Its just plain old html that is scraped. If you want fast fetching and parsing go for Feedzirra if you are dealing with feeds or Typhoeus for fetching just the html content.