Which web crawler to use to save news articles from a website into .txt files?

Question

Hi, i am currently in dire need of news articles to test a LSI implementation (it's in a foreign language, so there isnt the usual packs of files ready to use).

So i need a crawler that given a starting url, let's say http://news.bbc.co.uk/ follows all the contained links and saves their content into .txt files, if we could specify the format to be UTF8 i would be in heaven.

I have 0 expertise in this area, so i beg you for some sugestions in which crawler to use for this task.

Answer 1

What you are looking for is a "Scraper", and you will have to write one. Further more you may be in violation of the BBC's Terms of Use like anyone cares.

Answer 2

you can grab the site with wget. Then run it all through some HTML renderer (Lynx text browser does the job adequately with --dump-html option) to convert HTML to TXT. You will need to write the script to call Lynx on each downloaded file yourself, but that should be easy enough.