data-harvest

Automatically pressing a "submit" button using python

Hi, The bus company I use runs an awful website (Hebrew,English) which making a simple "From A to B timetable today" query a nightmare. I suspect they are trying to encourage the usage of the costly SMS query system. I'm trying to harvest the entire timetable from the site, by submitting the query for every possible point to every poss...

Crawling news articles

Does anyone know if there are standards / api to crawl news articles from most of the biggest news sources. I'm using rss to index them but I would like to classify them with more data than just their titles. ...