Get source of a website which requires post login

Question

Hello, I wan't to write a script to get the source of a website which requires a post login. I need a shell script to do this. I want to parse some information. Any idea which language is the best choice for handling the http request and maybe cookies? Thank you.

Answer 1

Starting with shell programming, curl can do a POST to get you the source of a web page. -- See the -d option.

You can then use shell scripting with grep, sed or other options to parse the results.

More automated/sophisticated/complicated is to write a web scraper program using Perl, Ruby, pick-your-language

Answer 2

Take a look at cURL. Never used it from the command line myself, but it's able to send POST variables.

Answer 3

Thank you very much.

This is nice too for ruby: http://scrubyt.org/