im building a large-scale web crawler, how many instances is optimal when crawling a web when running it on dedicated web server located in internet server farms.
+2
A:
spare_memory_on_machine / memory_footprint_of_crawler_process * 0.95
chaos
2009-06-21 10:06:13