How can I make a web crawler gather data?

Question

I know its a big question but I'm a complete beginner. I've limited experience in HTML, PHP etc and want to knock something together but don't even know where to start.

Although I can't necessarily program every language, with a little guidance I do a mean cut and paste and can learn anything. I'm a school teacher so I have a long summer holiday to fill.

I was wondering if I could get something that would search dates and prices of events and/or travel itineries eg train times etc to make a list of educational trips and stuff like that.

Any help would be greatly appreciated.

Answer 1

1. Choose a framework/library/language
2. You will need something to make HTTP GET requests to the pages in question.
3. The GET will return you html data. Use whatever language/framework you chose to parse out the data you are interested in
4. store your crawled data (in a database, or xml file, or text file etc)

If you post specific requests for detail we can help, but that is the general idea

Answer 2

One way to get going is to use a tool that doesn't require much coding. SiteScraper learns what you want itself once you give it a pattern to match.