Output large dataset from web service

Question

Hi, I wonder if anyone experienced with returning large dataset from webservice. The dataset is around 10,000 x 60 floats.

I will be using http wcf for my webservice. Any ideas to approach it are welcome :)

Thanks.

Answer 1

There's no technical reason you can't do it.

You just have to consider the amount of data that is being transfered and realize that it may take your client a while to download and deserialize the results.

If you're really worried about the amount of data going over the wire, you could use a library like Google's protocol buffers to do binary serialization (rather than the XML or JSON that you get out of the box with WCF). You can find the .NET port of Protocol Buffers at:

protobuf-net - Project Hosting on Google Code

Answer 2

This is not big data set. You can use web service to return such dataset without any implementation problems. You just need to set maxReceivedMessageSize and maxArrayLength on the client.

The real set of questions you should ask is:

• How many concurrent clients can use this service?
• What is expected response time?
• How often does a client call this service?
• What bandwidth is available on production server?
• What bandwidth is available on clients?

Answers to these questions show you if 2.3MB is a big data set. If you are affraid of performance and response time you should definitely plan load tests.