Programmatically log on to forum and then screenscrape

Question

I'd like to login to the Forums part of community-server (e.g. http://tinyurl.com/cs-login) and then download a specific page and perform a regex (to see if there are any posts waiting for moderation). If there is, I'd like to send an email.

I'd like to do this from a Linux server.

Currently I know how to download a page (using e.g. wget) but have a problem logging in. Any bright idea how that works?

Answer 1

You might have better luck with Selenium or see this question for more suggestions:

http://stackoverflow.com/questions/300788/script-for-college-class-registration

Answer 2

Looking at the source of the login page it appears to be an asp.net app so you'd need to probably do a couple things to achieve this -

Manage the form hidden __viewstate field and post that back when you submit the login details.

Once you get past that I'm guessing you can reference the specific page in question just using an absolute URL but you'd need to handle the ASP.NET Forms authentication cookie and send that as part of the GET request.

Cheers
Kev

Answer 3

Personally, I'd write it in Perl, using WWW::Mechanize, and do something like:

my $login_url = 'login url here';
my $username = 'username';
my $password = 'password';
my $mech = new WWW::Mechanize;
$mech->get($login_url)
    or die "Failed to fetch login page";
$mech->set_visible($username, $password)
    or die "Failed to find fields to complete";
$mech->submit
    or die "Failed to submit form";

if ($mech->content() =~ /posts awaiting moderation/i) {
    # Do something here
}

I've no idea whether the above will work, as I don't have login details to a Community Server (whatever that is) to test it against, but it should give you something you could work from easily enough, and shows the power of WWW::Mechanize.

Answer 4

You can do it all with wget. You need to submit form using POST and need to store cookies. Relevant stuff from the wget man page:

--post-data=string
--post-file=file

Use POST as the method for all HTTP requests and send the specified data in the request body.
"--post-data" sends string as data, whereas "--post-file" sends the contents of file.  Other than
that, they work in exactly the same way.

This example shows how to log to a server using POST and then proceed to download the desired pages,
presumably only accessible to authorized users:

       # Log in to the server.  This can be done only once.
       wget --save-cookies cookies.txt \
            --post-data 'user=foo&password=bar' \
            http://server.com/auth.php

       # Now grab the page or pages we care about.
       wget --load-cookies cookies.txt \
            -p http://server.com/interesting/article.php