web crawler needed

views:

answers:

web crawler needed

does anybody know where i can get a free web crawler that actually works with minimal coding by me. ive googled it and can only find really old ones that dont work or openwebspider which doesnt seem to work. ideally id like to store just the web addresses and which links that page contains

any suggestions? thanks

+1 A:

I would recommend writing one from scratch. You can use the time you spend searching for the crawler to write one.

codaddict 2010-04-12 11:38:52

+1. I don't think there's such thing as minimal coding in this (any?) area as every web crawler is subtly different.

Ninefingers 2010-04-12 11:40:35

i dont think i have the coding ability, i only have a bit of experience in c++ and i need to have it start crawling in three days max so i dont have the time.

2010-04-12 12:36:10

now writing my own crawler, hope this works

2010-04-12 13:56:15

@nighcoder1: Good choice. Its bound to work cuz if you have have any trouble with any part of it the community will the happy to help you. All the best :)

codaddict 2010-04-12 13:57:43