I have a spider on our website that requests about 42000 pages to click, it takes between 2-4 days to run but recently it has been reaching about 20000 and then every single page request fails with a timeout until eventually the whole service crashes out. Does anyone have any idea why this might be happening and how i can relieve some pressure? These 42000 requests are spread between 5 different threads all running the same process to make it run slightly faster. (C#)