Large data! stream tcp/ip or a web service?

Question

Hi,

What's the best way to retrieve huge data (only once when you create a local database) like 200 MB (may be in chunks) streaming tcp/ip or web service or directly from the DB server or something else?

Factors: Speed and may be security. Languages: C# client anything on the server preferred open source.

If we take direct DB access how can I provide security?

Thanks Vishal

Answer 1

if you control the client side and it's not over the internet, direct access might be good

if not, web services for one major reason, security

web service is tcp/ip

Answer 2

Streaming TCP/IP is probably your best bet, because you will not have the overhead of the HTTP protocol which is really meant for textual data. But if you go with something webservice related, make sure that whatever solution you choose supports sending "chunked" data. Because with out it, your server is going to have to buffer the entire request before sending.

To answer you second question, you really don't want anybody besides you in your data. So I wouldn't even consider direct DB access.

Answer 3

if your collection is under a terabyte I recommend a binary table in a database. If it's larger, consider a key-value storage system.

Answer 4

Use WCF- you can easily change the underlying protocol if it doesn't meet your requirements. If you going to use WCF MTOM, streaming, chunking and compression are some options depending on which binding you go for. This blog post might be helpful.