How can I retrieve the main image of a blog post/news article?

Question

Hello,

I have a made a news aggregator Newzupp which I want to modify. Right now I am simply displaying the titles of the news stories and I am linking them to their urls.

I am planning to make it more graphical, by using images + titles instead of plain titles. I want to know how can I get the main image of each article (somewhat similar to google news).

One way that I can think of is I can strip all the images and display the image which points the the same article. But I do not think that will be efficient. Is there any other way of doing this?

---

I have found a solution to it.

1. Fetch the contents of the url [html/xml]
2. Scrape the content using hpricot
3. Find all elements with tag "img"
4. Do some research to find which of them is the main display image. [Like 6th image in case of Wired.com's rss feed]

I still think this is highly inefficient. I would like to know how services like Google News scrape the sites/blogs and display relevant images.

Answer 1

Perhaps you could filter/ sort by image size or position in the DOM hierarchy (i.e. nearest the top of the body/ immediately after an h1 tag).

Answer 2

Why not just convert all the scraped images(using hpricot/nokogiri) to square thumbnail images(using rmagick or the likes of it or just resizing them on the server side) and group those images in one DIV just below the topic body. You can then use a lightbox w/ slideshow to show the actual images only when the user clicks on them. That way it looks more graphical and still not spoil the look of your site. Finding the most relevant image is tricky.

Answer 3

What about a blacklist of advert hosts, from whom you would ignore images?

Since, generally speaking, adverts are hosted elsewhere while story-related images are hosted within the same domain, perhaps you could filter the page for those images that have same base url as the site itself.