Best ways for transfering data over the Internet (for .NET developers)

Question

I need to read some data from a server, transfer it over the internet (no VPN), and write it to the disk of another server. I could have thought many choices, but in the end I implemented:

1. The sender as windows service, that sends new data every X minutes
2. The receiver as a WCF self hosted service, with WSHttpBinding.
3. The data being transfered as a string, which is the object serialized to XML

But after I done that, I realized that there are many ways to do that:

1. Other protocols such as FTP.
2. Other bindings listed in http://msdn.microsoft.com/en-us/library/ms731092.aspx
3. Other parameter types, such as the original object type.

And I wonder if any of these options would result in a faster, more reliable or more secure transfer. What would you recommend?

Answer 1

SFTP (SSH File Transfer Protocol) gets my vote if it's a possibility.

Pretty much tunnels FTP through an SSH connection.

...you could also try creating web services that conform to the WS-Security policy for message based security (depending on the model you go with, your message is encrypted using anything from username/password pair to an X.509 Cert).

That just seems like a lot of work to build something custom when an off the shelf package/protocol would work and is perfectly legit for your purposes.

Answer 2

In terms of security, you could leverage System.Security.Cryptography.CryptoStream to provide a layer of encryption when you're passing data over the wire.

Answer 3

The sender is pretty wide open -- pretty much any choice would be good there.

For a receiver, one option to consider is an HttpHandler -- just an ashx file in an ASP.NET project. You can use whatever custom http headers you want to provide options/information to the receiver, and then the data is just the payload of the HTTP request. You don't need to increase the size of your data by text encoding it, security is built in with HTTPS. It's pretty drop-dead simple.

The only downside would be if you wanted to implement some sort of "broken connection, pick up where I left off functionality" which is still possible but a little difficult -- that might be easier with a dedicated custom service.

Answer 4

I use Google Protocol Buffers. Simple enough to write a socket layer for it.

It's very compact, multi-language, fast, and best of all, adding data fields is trivial, without breaking compatibility.

Check out protobuf-net written by Marc Gravell, who is a regular contributor on Stack overflow.

Answer 5

If you are working with Windows servers you can also use MSMQ and get message delivery guarantees, transactions, reliability and a whole bunch of other features. It exposes HTTP endpoints and is pretty easy to access and use from .NET

Answer 6

Based on the fact that you only want to transfer 1MB per minute, with no security required, and since you said both sides are running Windows, I'd simply use WCF and netTcpBinding. That would send the data as binary over TCP/IP.

No SOAP, no XML, not HTTP overhead.