Can 1330316 AJAX requests crash my server?

Question

I'm building a small PHP/Javascript app which will do some processing for all cities in all US states. This rounds up to a total of (52 x 25583) = 1330316 or less items that will need to be processed.

The processing of each item will take about 2-3 seconds, so its possible that the user could have to stare at this page for 1-2 hours (or at least keep it minimized while he did other stuff).

In order to give the user the maximum feedback, I was thinking of controlling the processing of the page via javascript, basically something like this:

var current = 1;
var max = userItems.length; // 1330316 or less

process();

function process()
{
  if (current >= max)
  {
     alert('done');
     return;
  }

  $.post("http://example.com/process", {id: current}, function()
     {
        $("#current").html(current);
        current ++;
        process();
     }
  );
}

In the html i will have the following status message which will be updated whenever the process() function is called:

<div id="progress">
   Please wait while items are processed.
   <span id="current">0</span> / <span id="max">1330316</span> items have been processed.
</div>

Hopefully you can all see how I want this to work.

My only concern is that, if those 1330316 requests are made simultaneously to the server, is there a possibility that this crashes/brings down the server? If so, if I put in an extra wait of 2 seconds per request using sleep(3); in the server-side PHP code, will that make things better?

Or is there a different mechanism for showing the user the rapid feedback such as polling which doesn't require me to mess with apache or the server?

Answer 1

If you can place a cronjob in the server, I believe it'd work much better. What about using a cronjob to do the actual processing and use Javascript to update periodically the status (say, every 10 seconds)?

Then, the first step would be to trigger some flag that the cronjob PHP will check. If it's active, then the task must be performed (you could use some temporary file to tell the script which records must be processsed).

The cronjob would do the task and then, when its iteration is complete, turn off the flag.

This way, the user can even close your application and check it back later, and the server will handle all the processing, uninterrupted by client activity.

Answer 2

Putting a sleep inside your server-side php script can only make it worse. It leads to more processes sticking around, which turns out to increase parallel working/sleeping processes count, which adds up to increased memory usage.

Don't fear that so much processes can be done in parallel. Usually an apache server is configured to process no more than 150 requests in parallel. A well configured server does not process more requests in parallel than resources are available (good administrators do some calculations beforehand). The other requests have to wait - and given your count of requests it's probable that they are going to timeout before being processed.

Your concerns should however be about client-side resources but it looks like your script only starts a new request when the previous returned. BTW: Well behaving HTTP clients (which your browser should be) start no more than 6 requests in parallel to the same IP.

Update: Besides the above you should seriously consider redesigning your approach to mass-processing (similar to as @Joel suggested) - but this should go to another question.