Grabbing text from a webpage

Question

I would like to write a program that will find bus stop times and update my personal webpage accordingly.

If I were to do this manually I would

1. Visit www.calgarytransit.com
2. Enter a stop number. ie) 9510
3. Click the button "next bus"

The results may look like the following:

10:16p Route 154 10:46p Route 154 11:32p Route 154

Once I've grabbed the time and routes then I will update my webpage accordingly.

I have no idea where to start. I know diddly squat about web programming but can write some C and Python. What are some topics/libraries I could look into?

Answer 1

Since you write in C, you may want to check out cURL; in particular, take a look at libcurl. It's great.

Answer 2

Beautiful Soup is a Python library designed for parsing web pages. Between it and urllib2 (urllib.request in Python 3) you should be able to figure out what you need. :)

Answer 3

What you're asking about is called "web scraping." I'm sure if you google around you'll find some stuff, but the core notion is that you want to open a connection to the website, slurp in the HTML, parse it and identify the chunks you want.

The Python Wiki has a good lot of stuff on this.

Answer 4

That site doesnt offer an API for you to be able to get the appropriate data that you need. In that case you'll need to parse the actual HTML page returned by, for example, a CURL request .

Answer 5

As long as the layout of the web page your trying to 'scrape' doesnt regularly change, you should be able to parse the html with any modern day programming language.

Answer 6

This is called Web scraping, and it even has its own Wikipedia article where you can find more information.

Also, you might find more details in this SO discussion.

Answer 7

You can use Perl to help you complete your task.

use strict;
use LWP;

my $browser = LWP::UserAgent->new;

my $responce = $browser->get("http://google.com");
print $responce->content;

Your responce object can tell you if it suceeded as well as returning the content of the page.You can also use this same library to post to a page.

Here is some documentation. http://search.cpan.org/~gaas/libwww-perl-5.822/lib/LWP/UserAgent.pm

Answer 8

You can use the mechanize library that is available for Python http://wwwsearch.sourceforge.net/mechanize/