Possible to Authenticate with an website with POST / Download CAPTCHA

Question

I've often wanted to create applications that provide a simpler front-end to other websites that require users to login before the pages I want to use can be accessed. I was wondering, if

(1) any website with a POST to an http page can be authenticated by POSTing

postField1name=pf1Value&postField2name=pf2Value

to the website, if that's true how can you inspect the HTML to POST correctly?

(2) I wanted to know if you could parse HTML, say for a sign up form, and display all the fields in an application UI, including downloading a Captcha, and displaying it to the user, and allowing them to type the value in, to send back to the website, and process the response.

Also if anyone knows how I might accomplish (2) using Apache HTTP Client in java, I'd greatly appreciate it!

http://hc.apache.org/httpcomponents-client/httpclient/index.html

Answer 1

(1) An easy way to find out what's actually being POST'd is to look at the actual HTTP requests. You can do that with a tool like LiveHTTPHeaders. Then have your script simulate that.

(2) Yes. You can use cURL, which is excellent for things like this.

Answer 2

(1) Try FireBug. There's actually a lot of options for authentication.

(2) Try JTidy