What's the best way to handle long running process in an ASP.Net application?

Question

In my web application there is a process that queries data from all over the web, filters it, and saves it to the database. As you can imagine this process takes some time. My current solution is to increase the page timeout and give an AJAX progress bar to the user while it loads. This is a problem for two reasons - 1) it still takes to long and the user must wait 2) it sometimes still times out.

I've dabbled in threading the process and have read I should async post it to a web service ("Fire and forget").

Some references I've read:
- MSDN
- Fire and Forget

So my question is - what is the best method?

UPDATE: After the user inputs their data I would like to redirect them to the results page that incrementally updates as the process is running in the background.

Answer 1

I ran into this exact problem at my last job. The best way I found was to fire off an asychronous process, and notify the user when it's done (email or something else). Making them wait that long is going to be problematic because of timeouts and wasted productivity for them. Having them wait for a progress bar can give them false sense of security that they can cancel the process when they close the browser which may not be the case depending on how you set up the system.

Answer 2

1. How are you querying the remote data?
2. How often does it change?
3. Are the results something that could be cached for a period of time?
4. How long a period of time are we actually talking about here?

The 'best method' is likely to depend in some way on the answers to these questions...

Answer 3

We had a similar issue and solved it by starting the work via an asychronous web service call (which meant that the user did not have to wait for the work to finish). The web service then started a SQL Job which performed the work and periodically updated a table with the status of the work. We provided a UI which allowed the user to query the table.

Answer 4

You can create another thread and store a reference to the thread in the session or application state, depending on wether the thread can run only once per website, or once per user session.
You can then redirect the user to a page where he can monitor the threads progress. You can set the page to refresh automatically, or display a refresh button to the user.
Upon completion of the thread, you can send an email to the user.

Answer 5

My solution to this, has been an out of band service that does these and caches them in db.

When the person asks for something the first time, they get a bit of a wait, and then it shows up but if they refresh, its immediate, and then, because its int he db, its now part of the hourly update for the next 24 hours from the last request.

Answer 6

To avoid excessive architecture astronomy, I often use a hidden iframe to call the long running process and stream back progress information. Coupled with something like jsProgressBarHandler, you can pretty easily create great out-of-band progress indication for longer tasks where a generic progress animation doesn't cut it.

In your specific situation, you may want to use one LongRunningProcess.aspx call per task, to avoid those page timeouts.

For example, call LongRunningProcess.aspx?taskID=1 to kick it off and then at the end of that task, emit a document.location = "LongRunningProcess.aspx?taskID=2". Ad nauseum.

Answer 7

Add the job, with its relevant parameters, to a job queue table. Then, write a windows service that will pick up these jobs and process them, save the results to an appropriate location, and email the requester with a link to the results. It is also a nice touch to give some sort of a UI so the user can check the status of their job(s).

This way is much better than launching a seperate thread or increasing the timeout, especially if your application is larger and needs to scale, as you can simply add multiple servers to process jobs if necessary.