Large file transfers are indeed a problem in wcf and the streaming option does not solve anything (you even need more memory on the server).
If you don't want to use sockets can solve the problem with implementing your own "protocol" for splitting up the file in blocks and transfer only separate blocks. I used reliableSessions and TransportWithMessageCredential.
The Server (or client) interface looks something like this:
[ServiceContract(CallbackContract = typeof(IClient), SessionMode = SessionMode.Required)]
public interface IServer
{
[OperationContract]
FilePart GetFileChunk(string identifier, int number, int blockSize);
}
As DataContract you can use something like this:
[DataContract]
public class FilePart
{
[DataMember] public int Part;
[DataMember] public byte[] Data;
[DataMember] public int BlockSize;
}
To find the "right" block size you have to play around a little, i recommend something about 64-512 kb. When they are too small you have a lot request, when they are to large it gets slow and you have more load on the server side.
It's also important that the maxReceivedMessageSize, maxBufferSize, and the timeouts are high enough (in the binding configuration) and the reader quotas. For testing i recommend to use the maximum for all the fields, when it works use values that better fits.
If you work with duplex bindings you can pass objects with by ref. With this way you can pass callback objects, so you are better able to view the progress of the transfer and so on...
[OperationContract IsOneWay=true]
FilePart GetFileChunk(string identifier, int number, int blockSize, ref TransferState callback);
I think this are all tricks and hints I can give. I hope it helps.