long running http process - how to put in separate process?

Question

I know that similar questions have been asked all over the place, but I'm having trouble finding one that relates directly to what I'm after.

I have a website where a user uploads a data file, then that file is transformed and imported into SQL. The file could be up to 50mb in size, and some times this process can take 30 minutes or sometimes even longer.

I realise I need to palm off the actual work to another process, and poll that process on the web page. I'm wondering what the best approach would be though? Being a web developer by trade, I'm finding all this new Windows Service stuff a bit confusing, and I just wanted somewhere to start.

So:

• Can I do / should I being doing this with a windows service? if so, how?
• Should I use WCF? If this runs under IIS, will I have problems with aspnet_wp.exe recycling and timing out my process?

clarifications

The data is imported into sql, there's no file distribution taking place.

If there is a failure, it absolutely MUST be reported to the user. The web page will poll every, lets say, 5 seconds, from the time the async task begins, to get the 'status' of the import. Once it's finished another response will tell the page to stop polling for status updates.

queries on final decision

ok, so as I thought, it seems that a windows service is the best idea. So as to HOW to get it to work, it seems the 'put the file there and wait for the service to pick it up' idea is the generally accepted way, is there a way I can start a process run by the service, without it having to constantly be checking a database table / folder? As I said earlier, I don't have any experience with Windows Services - I wondered if I put a public method in the service, can I call it somehow?

Answer 1

well ...

var thread = new Thread(() => {
    // your action
});
thread.Start();

but you will have problems with that:

• what if the import to sql fails? should there be any response to the client
• if it fails, how do you ensure the file on a later request
• what if the applications shuts down ... this newly created and started thread will be killed either
• ...

it's not always a good idea to store everything in sql (especially files...). if you want to make the file available to several servers why not distribute them via ftp ...?

you believe that your whole concept is a bit fucked (sry assuming this), and it might be helpful if you elaborate and give us more information about your intentions!

edit:

Can I do / should I being doing this with a windows service? if so, how?

you can :) i suggest you create a simple console-program and convert this with srvany and sc. you can get a rough overview howto here (note: insert blanks after =... that's a silly pitfall)

the term should is relative, due you did not answer the most important question

what if a record is persisted to the database, telling a consumer that file test.img should be persisted, but your service hasn't captured it or did not transform it yet?

so ... next on

Should I use WCF? If this runs under IIS, will I have problems with aspnet_wp.exe recycling and timing out my process?

why the hell you do you want to use WCF? do you even know what this techology is for?... you probably could create a WCF-service which recieves some binary-data and then stores this to a database. this request could be async. yes. but what for?

once again: please give us more insight to your workflow: what you exactly try to achieve? which "environmental-conditions" to you have (eg. app A pools db and expects file-records which are referenced in table x to be persisted) ...

edit:
so you want to import a .csv-file. well that changes everything :)

but i won't advise you to use a wcf-service (there could be a usage: eg. a wcf-service which has a method to insert a single row, then your iteration through the file would be implemented in another app... not that good though).

i would suggest following:
at first do everything in your webapp (as you've already done), but rather use some sort of bulk-insert and do your transformation/logic on the database.

if you then have some sort of bottle-neck, i would suggest you something like a minor job-service, eg:

webapp will upload the file and insert a row to a job-table. the job-service is continiously polling the table/or gets informed via wcf by the webapp (hey, hey, finally some sort of usage for WCF in your scenario... :) ) and then does the import-job, writing a finish-note to a table/or set the state of the job to finished ...
but this is a bit overkill :)

Answer 2

Please see if my below comments helps you to resolve your issue:

•Can I do / should I being doing this with a windows service? if so, how?

Yes you can do this with a windows service. And I think that is the way you should be doing it. You can implement your own service to process your request or you can use the open source code Job Proccessor

Basically the idea is..

• You submit a request for processing the csv file in database table with some status as not started.
• Then your windows service picks up the request from database table which are not started and update them as in progress status.
• Once the processing is complete succesfully /unsuccesfuly your service updated the database table with status as Completed / Failed.
• And your asp.net page can poll to database table for the current status every 5 sec or so.

•Should I use WCF? If this runs under IIS, will I have problems with aspnet_wp.exe recycling and timing out my process?

you should not be using WCF for this purpose.