How to parse/leach/extract a ASP.net website Content using Ruby or PHP or Java?

Question

Hi ,

I am doing a hobby project to scrape the content of an ASP.net website using Ruby or PHP or Java . For example if the website url " www.myaspnet.com/home.aspx" . i would like to extract the unicode text content from home.aspx and paste it to a notepad . Is there any libraries available in any of the above mentioned languages ? . Can anybody help me regarding this by providing suitable resources .

Thanks!

Answer 1

If you are using PHP you should be able to do without too much difficulties, you can use curl for getting the content and simplexml & dom to navigate into html.

You might also want to take a look at xpath for extracting content easily.

Answer 2

Sounds like you just want to scrape content.

You don't "paste it in notepad", you just write it to a simple text file (which you could then open in notepad, or any other program that can read a text file).

Generally:

$content = file_get_contents('http://example.com/url.asp');
//do some stuff to extract what you want and format it.  Probably using simplexml, or regular expressions to do the extraction
$fp = fopen('some_file.txt');
fwrite($fp,$stuff);
fclose($fp);

And you're done.

Answer 3

If you need some kind of screen scraping and you favor Ruby,then you can check scRubyt or watch this screen cast by Ryan Bates.

Answer 4

With Java you could write a simple utility using HttpUnit to extract the text from a page, then follow each of the links presented on the page - visiting those recursively and performing the same task.

It should be a fairly trivial exercise, external links need to be treated with some caution though, depending on how far you want to go with indexing.

Answer 5

This screencast basically shows how to scrap a website step by step in Ruby. It uses it the output a rails application, it should be easy to focus on just the Ruby library ( ScrAPI ).