Download a large file with a c# webservice

Question

I am trying to create a webservice that would allow its consumers to download files (can be very huge files). At the server, I have many files that need to be sent back to the consumer, so I am compressing all those file into one big zip file and streaming them back to the user. Right now, my webservice will start compressing the files, when the request comes in, forms the zip files and streams it back.Sometimes compression can take a lot of time, and the request may time out. What can I do to avoid such situations? My solution right now is to, seperate the data into smaller zip files, send a response to consumer saying there would be these many smaller files, and let consumers send request for individual smaller files. So, if i have 1GB zip file, i will break it into 10 smaller zip files, and ask the consumer to request for smaller files in 10 requests. Is this the correct approach? What problems can I be facing? Has anyone dealth with such issues before? I would be glad if you can share your experiences. Also, is it possible to start streaming the zip files without forming them fully?

Answer 1

Treat the request and the delivery as asynchronous operations.

The client can make a request for the file using one method. Another method can let the client know the status of the file packaging (whether they are ready for download yet). A third method can actually download the files.

Answer 2

It may be worth looking at a restful approach. Rather than a soap web service. As OrbMan, suggested an asynch approach may be best.

With REST you could expose a resource as: http://yourlocation/generatefile Which (when called with a post) returns a http response with a response code of 301 'accepted' and a location header value of location=http://yourlocation/generatefile/id00124 which suggests the location of the data.

You can then poll the http://yourlocation/generatefile/id00124 resource (maybe just header request) to get the status i.e. processing / complete.

When processing is complete. Do a get on the http://yourlocation/generatefile/id00124 to download your file. The response http message should identify you file and the format i.e. encryption and compression types so any consumer knows how to read it.

This is a nice solution to problems which are long running and returns data in formats other than soap anbd general xml.

I hope this helps

Answer 3

I would poll from the calling client as part of the method which gets the file. The client code might flow something like this:

byte[] GetFile()
{
   response = request.Post(http://yourlocation/generatefile);
   string dataResource = response.Headers["Location"];
   bool resourceReady = false;
   while(!reasourceReady)
   {
     resH = request.Header(dataResource);
     if(resH.Headers[Status] == "complete")
         break;
     else
         Thread.Sleep(OneSecond); ?? or whetever
   }

   fileRes = request.Get(dataResource);

return fileRes.ToByteArray();
}

This is only psuedo, but I hope it makes sense...