Screen Scraping in PHP with login

Question

Looking around for a solution to this, I have found different methods. Some use regex, some use DOM scripting or something.

I want to go to a site, log in, fill out a form and then check if the form sent. The logging in part is the part I can't find anything on.

Anyone know of an easy way to do this?

Answer 1

You might be better off with some sort of scriptable browser if you need to do a lot of GUI stuff. If you need to use PHP, check out curl: http://us2.php.net/curl

Answer 2

what I usually do is fire up charles go through the login process in a browser and record the raw requests. Copy+paste the requests and throw them through fopen or curl (with some small adjustments according to the responses).

Answer 3

You may want to take a look at Perl's LWP library (I know it isn't PHP, but it's very useful for screen scraping, web unit testing, and such):

• Perl LWP at CPAN
• LWP::Simple tutorial

Answer 4

I'd agree with Les. Curl + Charles (or Fiddler, Firefox's Tamper Data extension, wireshark, etc.) is the way I've always done this. The one trick I've found is that some sites require a three step process:

1. Hit the login page with a GET request first to get any session ids, cookies, and/or required fields (e.g. .net sites have __VIEWSTATE and __EVENTVALIDATION).
2. Once you have these values, then you post to the login page
3. Finally, request whatever resource you're after.

Don't plan on curl's cookie jar and cookie file being much help. You'll probably be best off parsing out the session id and cookies from the headers using a simple regex.

Hope this helps!