How to pass large quantity of data to web service

Question

I'm building a client-server (c#) application that uses a web services to synchronize the data.

Basically I am passing the XML of a DataSet back and forth. But depending on various parameters the dataset can actually be quite large. I'm compressing the XML using Gzip and want to pass that to the web server and the get the resulting compressed XML back.

What is the best way to pass potentially large chunks of data back and forth?

Clarification: I guess I'm asking what format is the best to pass the data. JSON, SOAP, ordinary POST (I'm not very familiar with Web Services so I'm sure there's more that I'm not thinking of).

Answer 1

If you use WCF for your web services you can use MTOM to send data over as a binary attachment. See:

http://stackoverflow.com/questions/1020956/what-is-the-best-way-to-send-large-data-from-wcf-service-to-client/1020989#1020989

Answer 2

What you probably want to investigate is using MTOM. This allows you to send binary SOAP in a chunked manner. This is cool because chunking the data, coupled with an AJAX UI, will allow you to present a progress bar to the user that is uploading it.

Here is an example of using MTOM to send/receive data in a chunked manner. I hope this helps.

Answer 3

Best probably depends on a lot of factors. If by best you mean most performant, here are some points to consider:

• XML is not the best way at all
• Binary serialization is far more efficient.
• You may not have this option, however, if you need to concern yourself with interoperability. In that case, you may want to consider using a flat file or delimited format.
• If neither of those are doable, then you might consider just sending what has changed instead of all of the data.
• If that isn't an option, then compressing the XML is an... ok solution.

I'd use XML as a last resort, since there's a lot of unnecessary metadata that is included in XML that will easily increase the size of your payload by more than 5X.

Answer 4

Check this question&answer

Compressing web service request

Answer 5

You can try compressing the file and sending it as binary via webservice. also, don't forget to modify your upload limits.

how to send binary data via webservice: http://support.microsoft.com/kb/318425

Answer 6

If all you want is to apply gzip compression to your xml data you might consider enabling transparent gzip compression at the http transport level (Accept-encoding: gzip, Content-Encoding: gzip) and let the iis handle it.
see http://weblogs.asp.net/owscott/archive/2004/01/12/IIS-Compression-in-IIS6.0.aspx#58745

Answer 7

For web services, you have multiple concepts and possibilities that you can consider.

Do you need multiple clients managing the same data set? Concurrently?

Do you have a disconnected set where it is managed on both ends or just one manages the other views/stores/reports?

Are you simply using the client to use as an interface to the data set, and if so, would it make sense to use a different UI (web) to manage different parts of the dataset independently as a web UI based application?

Are you making incremental changes in a transactional type interaction? (One end sends the big dataset to client and transactional changes are returned based on that?)

Difficult to recommend specifically without some thoughts about if/how you might dissect the dataset for more optimal management or knowing more of the specifics here.

If you HAVE to send the large data entity, then it just becomes a matter of which method you wish to use for your pipeline, its simply and upload/download and you manage the endpoints appropriately.

For some random thoughts:

If you want to manipulate the data in the DOM, consider JSON, which is quite compatible with JavaScript.

If you desire the XML dataset manipulation, you could place zip or some other compression if you need to pass the entire dataset, just wire it up on each end.