hi I wanted to know which is the best webservice provider to choose if I want to write a multi threaded crawler. Appengine,ec2....
Appengine java apparently has some restrictions .. also a free would be great as i am just testing the app.
views:
18answers:
1
+1
A:
The thing to be aware of is, a crawler is not a Web application. Well, other than using the Web for input. You may eventually build a Web application that uses your crawler's output, but a crawler and a Web service provider are a poor match. A "real" Web application is one that people from all over the world access with their browsers, and which returns Web pages to those browsers.
Assuming that the world needs yet another crawler, why don't you run it on a PC at your home and hook that into the 'net with a standard consumer cable connection? You'll have great bandwidth, you're probably cheaper off to buy your own CPU power rather than to rent it from elsewhere, and you have full control of your system.
Carl Smotricz
2009-12-16 20:57:47
Also, that's as close to free as you'll probably get for quite a while.
anonymous coward
2009-12-16 22:18:11
Thanks carl,Now what I am really trying to do is generate useful data through my crawlers and serve it to users. Now over time my comp may not be able to handle the load and a PC is not equivalent to a server as it wud require my comp to be on all the time.what do u suggest?
eklavya
2009-12-17 11:44:36
The main problem is that you want something for nothing. While still in the "testing" phase, I suggest starting with a PC on a cable and see what kind of response you get. Chances are 10,000 to 1 that your service will not be the next Facebook, Twitter or Google. If business picks up, you will be able to afford a "real" paid hosting service. If not, you won't have wasted any money.
Carl Smotricz
2009-12-17 14:00:44