Scraping and using data using PHP from a website that must be logged on to (Reddit)?

Question

I would like to create a webpage that, given two reddit usernames and their passwords, subscribes user2 to all of the subreddits that user1 is subscribed to. So I need to:

1. Get the subreddits that user1 is subscribed to.
2. Subscribe user2 to those reddits

I have experience using PHP, but I have no experience with scraping (especially when the user must be logged in) and also submitting the type of information that would be necessary to "subscribe" a user to a subreddit. Does anyone have any ideas on how this can be done?

Regards,

Tim

Answer 1

Assuming this isn't against reddits' terms of service, using cURL to login, one could probably easily regex the necessary information. From there it's a matter of checking how reddit subscribes favorites and either navigating to the proper urls or posting form data.

I'd call it a medium-level task, as long as it's not against the reddit terms of service.

Answer 2

The open source product TestPlan is very good at such things. Using a simple language you can login to the site with one user, grab the names of the subreddits, then login as th other user to subscribe to the groups.

For example, if you just wanted the titles of the top entries you could use this code:

GotoURL http://www.reddit.com/top/

set %Topics% as response //p[@class='title']
foreach %Topic% in %Topics%
    set %Title% as selectIn %Topic% string(.)
    Notice %Title%
end

Which produces output like this:

00000000-00 GOTOURL http://www.reddit.com/top/
00000001-00 NOTICE LEGAL DVD vs. PIRATED COPY (i.imgur.com)
00000002-00 NOTICE Don't just shorten your URL, make it suspicious and frightening. - ShadyURL (shadyurl.com)
00000003-00 NOTICE HOLY CRAP! IS THAT A ROOM FOR RENT ON MY CRAIGSLIST??!?!? (houston.craigslist.org)
00000004-00 NOTICE Years from now when our children ask us, "What did we do after 9/11?" we shall explain it to them using this... (4gifs.com)
00000005-00 NOTICE TSA forces disabled boy to remove leg braces and walk through the metal detector. "I told him, 'This is overkill. He's 4 years old. I don't think he's a terrorist.' " (philly.com)
00000006-00 NOTICE This picture scares the shit out of me. (imgur.com)
00000007-00 NOTICE Civilization V Announced, in Development at Firaxis Games (hellforge.gameriot.com)
00000008-00 NOTICE I don't know, the price seems a little steep... [pic] (i.imgur.com)
00000009-00 NOTICE Reddit, last week we saw the depth of the ocean scaled relative to human size. I made a figure of the depth of the ocean accurately scaled to the width. It's really very shallow from this perspective. (i.imgur.com)